[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:19:39.352095 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:36359
I20260709 12:19:39.355281 16792 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:19:39.357868 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:39.381134 16798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:39.381140 16799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:39.387183 16792 server_base.cc:1061] running on GCE node
W20260709 12:19:39.387092 16801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:40.621610 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:40.622087 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:40.622380 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599580622328 us; error 0 us; skew 500 ppm
I20260709 12:19:40.623425 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:40.631879 16792 webserver.cc:533] Webserver started at http://127.16.102.62:40557/ using document root <none> and password file <none>
I20260709 12:19:40.633033 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:40.633324 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:40.633997 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:40.641335 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "ef63d7d07c374f7da87449fef4fc6585"
format_stamp: "Formatted at 2026-07-09 12:19:40 on dist-test-slave-7xt3"
I20260709 12:19:40.652251 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:19:40.659307 16807 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:40.661026 16792 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:19:40.661563 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "ef63d7d07c374f7da87449fef4fc6585"
format_stamp: "Formatted at 2026-07-09 12:19:40 on dist-test-slave-7xt3"
I20260709 12:19:40.662117 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:40.733807 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:40.735867 16792 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:19:40.736464 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:40.826336 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:36359
I20260709 12:19:40.826465 16858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:36359 every 8 connection(s)
I20260709 12:19:40.836568 16859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:40.864409 16859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585: Bootstrap starting.
I20260709 12:19:40.871807 16859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:40.874327 16859 log.cc:826] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585: Log is configured to *not* fsync() on all Append() calls
I20260709 12:19:40.880649 16859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585: No bootstrap required, opened a new log
I20260709 12:19:40.905435 16859 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d7d07c374f7da87449fef4fc6585" member_type: VOTER }
I20260709 12:19:40.906313 16859 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:40.906649 16859 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef63d7d07c374f7da87449fef4fc6585, State: Initialized, Role: FOLLOWER
I20260709 12:19:40.907912 16859 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d7d07c374f7da87449fef4fc6585" member_type: VOTER }
I20260709 12:19:40.908596 16859 raft_consensus.cc:399] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:19:40.908982 16859 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:19:40.909442 16859 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:40.914677 16859 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d7d07c374f7da87449fef4fc6585" member_type: VOTER }
I20260709 12:19:40.915798 16859 leader_election.cc:304] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef63d7d07c374f7da87449fef4fc6585; no voters: 
I20260709 12:19:40.918339 16859 leader_election.cc:290] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:19:40.918844 16862 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:40.922647 16862 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 1 LEADER]: Becoming Leader. State: Replica: ef63d7d07c374f7da87449fef4fc6585, State: Running, Role: LEADER
I20260709 12:19:40.923993 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d7d07c374f7da87449fef4fc6585" member_type: VOTER }
I20260709 12:19:40.925505 16859 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:19:40.938447 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef63d7d07c374f7da87449fef4fc6585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d7d07c374f7da87449fef4fc6585" member_type: VOTER } }
I20260709 12:19:40.938416 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef63d7d07c374f7da87449fef4fc6585. Latest consensus state: current_term: 1 leader_uuid: "ef63d7d07c374f7da87449fef4fc6585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d7d07c374f7da87449fef4fc6585" member_type: VOTER } }
I20260709 12:19:40.939951 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:40.940111 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:40.945250 16871 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:19:40.960734 16871 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:19:40.963644 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:19:40.985385 16871 catalog_manager.cc:1383] Generated new cluster ID: 2efd9fbb07c34421848c52042a8cd4b5
I20260709 12:19:40.985766 16871 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:19:41.007697 16871 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:19:41.009680 16871 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:19:41.028831 16871 catalog_manager.cc:6070] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585: Generated new TSK 0
I20260709 12:19:41.030431 16871 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:19:41.096210 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:41.107395 16881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:41.108862 16882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:41.114074 16884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:41.114128 16792 server_base.cc:1061] running on GCE node
I20260709 12:19:41.115484 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:41.115836 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:41.116070 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599581116048 us; error 0 us; skew 500 ppm
I20260709 12:19:41.117014 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:41.122694 16792 webserver.cc:533] Webserver started at http://127.16.102.1:38847/ using document root <none> and password file <none>
I20260709 12:19:41.123653 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:41.123987 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:41.124394 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:41.126863 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "0e10c2ade849401e8ba5d8141abea6ec"
format_stamp: "Formatted at 2026-07-09 12:19:41 on dist-test-slave-7xt3"
I20260709 12:19:41.135967 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.000s	sys 0.007s
I20260709 12:19:41.142813 16889 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.144160 16792 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.003s
I20260709 12:19:41.144608 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "0e10c2ade849401e8ba5d8141abea6ec"
format_stamp: "Formatted at 2026-07-09 12:19:41 on dist-test-slave-7xt3"
I20260709 12:19:41.145064 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:41.169626 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:41.171828 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:41.174855 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:41.181519 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:41.181975 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.001s
I20260709 12:19:41.182322 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:41.182583 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.309393 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:35527
I20260709 12:19:41.309507 16951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:35527 every 8 connection(s)
I20260709 12:19:41.318125 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:41.329907 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:41.336326 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:41.350533 16792 server_base.cc:1061] running on GCE node
W20260709 12:19:41.351871 16959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:41.353302 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:41.353586 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:41.353868 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599581353793 us; error 0 us; skew 500 ppm
I20260709 12:19:41.354537 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:41.358124 16792 webserver.cc:533] Webserver started at http://127.16.102.2:46369/ using document root <none> and password file <none>
I20260709 12:19:41.358762 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:41.359027 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:41.359369 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:41.360932 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "012c695806b6424dad30e97211e4fcc6"
format_stamp: "Formatted at 2026-07-09 12:19:41 on dist-test-slave-7xt3"
I20260709 12:19:41.365669 16952 heartbeater.cc:344] Connected to a master server at 127.16.102.62:36359
I20260709 12:19:41.366308 16952 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:41.368009 16952 heartbeater.cc:507] Master 127.16.102.62:36359 requested a full tablet report, sending...
I20260709 12:19:41.368791 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.010s	sys 0.000s
I20260709 12:19:41.372720 16824 ts_manager.cc:194] Registered new tserver with Master: 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527)
I20260709 12:19:41.374881 16964 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.376055 16792 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:19:41.376438 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "012c695806b6424dad30e97211e4fcc6"
format_stamp: "Formatted at 2026-07-09 12:19:41 on dist-test-slave-7xt3"
I20260709 12:19:41.376683 16824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32970
I20260709 12:19:41.376797 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:41.429301 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:41.431093 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:41.434401 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:41.438458 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:41.438864 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.439215 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:41.439419 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.558082 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:38333
I20260709 12:19:41.558291 17026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:38333 every 8 connection(s)
I20260709 12:19:41.566332 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:41.581012 17030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:41.587618 17027 heartbeater.cc:344] Connected to a master server at 127.16.102.62:36359
I20260709 12:19:41.588160 17027 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:41.589366 17027 heartbeater.cc:507] Master 127.16.102.62:36359 requested a full tablet report, sending...
W20260709 12:19:41.589571 17031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:41.592641 16824 ts_manager.cc:194] Registered new tserver with Master: 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:41.594837 16824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32978
I20260709 12:19:41.600947 16792 server_base.cc:1061] running on GCE node
W20260709 12:19:41.602247 17033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:41.603609 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:41.603935 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:41.604149 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599581604131 us; error 0 us; skew 500 ppm
I20260709 12:19:41.604802 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:41.608217 16792 webserver.cc:533] Webserver started at http://127.16.102.3:38721/ using document root <none> and password file <none>
I20260709 12:19:41.608871 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:41.609127 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:41.609457 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:41.610788 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "849bbec7c1154b2d9b70c1709ebfa534"
format_stamp: "Formatted at 2026-07-09 12:19:41 on dist-test-slave-7xt3"
I20260709 12:19:41.616595 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20260709 12:19:41.621021 17038 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.622108 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:19:41.622501 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "849bbec7c1154b2d9b70c1709ebfa534"
format_stamp: "Formatted at 2026-07-09 12:19:41 on dist-test-slave-7xt3"
I20260709 12:19:41.622826 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:41.646833 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:41.648670 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:41.650549 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:41.654479 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:41.654796 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.655103 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:41.655313 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:41.781044 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:37359
I20260709 12:19:41.781231 17100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:37359 every 8 connection(s)
I20260709 12:19:41.803301 17101 heartbeater.cc:344] Connected to a master server at 127.16.102.62:36359
I20260709 12:19:41.803800 17101 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:41.804881 17101 heartbeater.cc:507] Master 127.16.102.62:36359 requested a full tablet report, sending...
I20260709 12:19:41.807624 16824 ts_manager.cc:194] Registered new tserver with Master: 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:41.808485 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017551516s
I20260709 12:19:41.809748 16824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32990
I20260709 12:19:41.883857 16824 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32996:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:19:41.889922 16824 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260709 12:19:41.950114 16872 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:42.011394 16992 tablet_service.cc:1511] Processing CreateTablet for tablet 52b372c5d74648458c4589b25e07b070 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:42.010028 16991 tablet_service.cc:1511] Processing CreateTablet for tablet 7f135dc093fd4d3d93612ace8d903bc2 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:42.012413 16990 tablet_service.cc:1511] Processing CreateTablet for tablet ecc62f04a8d84076896751bcf7de0727 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:42.015990 16991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f135dc093fd4d3d93612ace8d903bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.016867 16990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc62f04a8d84076896751bcf7de0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.020006 16989 tablet_service.cc:1511] Processing CreateTablet for tablet 468a3e5634ee46fbbd8f8b4df97ce770 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:42.021808 16989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 468a3e5634ee46fbbd8f8b4df97ce770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.025588 16988 tablet_service.cc:1511] Processing CreateTablet for tablet 94077101e4f74b77bcd819cd3f835b32 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:42.027360 16988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94077101e4f74b77bcd819cd3f835b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.032387 16992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b372c5d74648458c4589b25e07b070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.033258 16987 tablet_service.cc:1511] Processing CreateTablet for tablet 24b40b9dd1114b498e4dcc91c97be5b2 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:42.036362 16987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b40b9dd1114b498e4dcc91c97be5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.040787 16986 tablet_service.cc:1511] Processing CreateTablet for tablet 46b0e485bf754c3a92586c68f25e4c1a (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:42.042675 16986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b0e485bf754c3a92586c68f25e4c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.045101 16985 tablet_service.cc:1511] Processing CreateTablet for tablet 748d2378aa78450ba4cef89d9c643cea (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:42.046895 16985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748d2378aa78450ba4cef89d9c643cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.060806 16984 tablet_service.cc:1511] Processing CreateTablet for tablet 41e4984c9b7f4173b47582e4e794e607 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:42.062637 16984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e4984c9b7f4173b47582e4e794e607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.073649 16917 tablet_service.cc:1511] Processing CreateTablet for tablet 52b372c5d74648458c4589b25e07b070 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:42.075508 16915 tablet_service.cc:1511] Processing CreateTablet for tablet ecc62f04a8d84076896751bcf7de0727 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:42.077092 16916 tablet_service.cc:1511] Processing CreateTablet for tablet 7f135dc093fd4d3d93612ace8d903bc2 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:42.075567 16917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b372c5d74648458c4589b25e07b070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.082674 16911 tablet_service.cc:1511] Processing CreateTablet for tablet 46b0e485bf754c3a92586c68f25e4c1a (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:42.082653 16910 tablet_service.cc:1511] Processing CreateTablet for tablet 748d2378aa78450ba4cef89d9c643cea (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:42.087268 16915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc62f04a8d84076896751bcf7de0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.083022 16913 tablet_service.cc:1511] Processing CreateTablet for tablet 94077101e4f74b77bcd819cd3f835b32 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:42.097791 16914 tablet_service.cc:1511] Processing CreateTablet for tablet 468a3e5634ee46fbbd8f8b4df97ce770 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:42.099505 16909 tablet_service.cc:1511] Processing CreateTablet for tablet 41e4984c9b7f4173b47582e4e794e607 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:42.087397 16912 tablet_service.cc:1511] Processing CreateTablet for tablet 24b40b9dd1114b498e4dcc91c97be5b2 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:42.103933 16916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f135dc093fd4d3d93612ace8d903bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.108757 16909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e4984c9b7f4173b47582e4e794e607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.117144 16914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 468a3e5634ee46fbbd8f8b4df97ce770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.118940 16912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b40b9dd1114b498e4dcc91c97be5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.124204 16910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748d2378aa78450ba4cef89d9c643cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.140817 17065 tablet_service.cc:1511] Processing CreateTablet for tablet 7f135dc093fd4d3d93612ace8d903bc2 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:42.146349 16911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b0e485bf754c3a92586c68f25e4c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.149616 17063 tablet_service.cc:1511] Processing CreateTablet for tablet 468a3e5634ee46fbbd8f8b4df97ce770 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:42.154382 17066 tablet_service.cc:1511] Processing CreateTablet for tablet 52b372c5d74648458c4589b25e07b070 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:42.156687 17064 tablet_service.cc:1511] Processing CreateTablet for tablet ecc62f04a8d84076896751bcf7de0727 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:42.154917 17065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f135dc093fd4d3d93612ace8d903bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.170740 17063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 468a3e5634ee46fbbd8f8b4df97ce770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.173477 16913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94077101e4f74b77bcd819cd3f835b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.176043 17061 tablet_service.cc:1511] Processing CreateTablet for tablet 24b40b9dd1114b498e4dcc91c97be5b2 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:42.171401 17062 tablet_service.cc:1511] Processing CreateTablet for tablet 94077101e4f74b77bcd819cd3f835b32 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:42.182786 17059 tablet_service.cc:1511] Processing CreateTablet for tablet 748d2378aa78450ba4cef89d9c643cea (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:42.182626 17060 tablet_service.cc:1511] Processing CreateTablet for tablet 46b0e485bf754c3a92586c68f25e4c1a (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:42.189483 17066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b372c5d74648458c4589b25e07b070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.195420 17058 tablet_service.cc:1511] Processing CreateTablet for tablet 41e4984c9b7f4173b47582e4e794e607 (DEFAULT_TABLE table=test-workload [id=eeefc25fad2c41daa43a5d06fc0813a5]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:42.202024 17062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94077101e4f74b77bcd819cd3f835b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.206296 17059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748d2378aa78450ba4cef89d9c643cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.209964 17058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e4984c9b7f4173b47582e4e794e607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.212781 17060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b0e485bf754c3a92586c68f25e4c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.214392 17119 tablet_bootstrap.cc:492] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.235253 17061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b40b9dd1114b498e4dcc91c97be5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.268319 17064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc62f04a8d84076896751bcf7de0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:42.308008 17119 tablet_bootstrap.cc:654] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.325632 17121 tablet_bootstrap.cc:492] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.351926 17120 tablet_bootstrap.cc:492] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:42.363912 17119 tablet_bootstrap.cc:492] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.364545 17119 ts_tablet_manager.cc:1403] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.193s	user 0.022s	sys 0.019s
I20260709 12:19:42.370430 17121 tablet_bootstrap.cc:654] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.370548 17119 raft_consensus.cc:359] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.372990 17119 raft_consensus.cc:385] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.374083 17119 raft_consensus.cc:740] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.377629 17119 consensus_queue.cc:260] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.382477 16952 heartbeater.cc:499] Master 127.16.102.62:36359 was elected leader, sending a full tablet report...
I20260709 12:19:42.382774 17119 ts_tablet_manager.cc:1434] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.002s
I20260709 12:19:42.385371 17027 heartbeater.cc:499] Master 127.16.102.62:36359 was elected leader, sending a full tablet report...
I20260709 12:19:42.386369 17120 tablet_bootstrap.cc:654] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.388980 17119 tablet_bootstrap.cc:492] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.403234 17121 tablet_bootstrap.cc:492] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.404050 17121 ts_tablet_manager.cc:1403] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.080s	user 0.017s	sys 0.007s
I20260709 12:19:42.412091 17121 raft_consensus.cc:359] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.413031 17121 raft_consensus.cc:385] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.413372 17121 raft_consensus.cc:740] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.414281 17121 consensus_queue.cc:260] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.415427 17119 tablet_bootstrap.cc:654] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.417444 17120 tablet_bootstrap.cc:492] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:42.418169 17120 ts_tablet_manager.cc:1403] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.099s	user 0.018s	sys 0.000s
I20260709 12:19:42.423081 17120 raft_consensus.cc:359] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.424098 17120 raft_consensus.cc:385] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.424494 17120 raft_consensus.cc:740] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.425833 17120 consensus_queue.cc:260] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.432644 17101 heartbeater.cc:499] Master 127.16.102.62:36359 was elected leader, sending a full tablet report...
I20260709 12:19:42.434659 17119 tablet_bootstrap.cc:492] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.435307 17119 ts_tablet_manager.cc:1403] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.047s	user 0.012s	sys 0.005s
I20260709 12:19:42.439889 17121 ts_tablet_manager.cc:1434] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.034s	user 0.013s	sys 0.008s
I20260709 12:19:42.441113 17121 tablet_bootstrap.cc:492] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.439141 17119 raft_consensus.cc:359] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.448232 17119 raft_consensus.cc:385] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.454571 17123 raft_consensus.cc:493] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.455284 17120 ts_tablet_manager.cc:1434] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.037s	user 0.021s	sys 0.005s
I20260709 12:19:42.455235 17123 raft_consensus.cc:515] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.457141 17120 tablet_bootstrap.cc:492] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:42.448673 17119 raft_consensus.cc:740] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.461123 17123 leader_election.cc:290] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.463085 17119 consensus_queue.cc:260] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.471078 17121 tablet_bootstrap.cc:654] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.482023 17119 ts_tablet_manager.cc:1434] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.003s
I20260709 12:19:42.484876 17119 tablet_bootstrap.cc:492] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.488617 17120 tablet_bootstrap.cc:654] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.497391 17119 tablet_bootstrap.cc:654] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.507552 17121 tablet_bootstrap.cc:492] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.507315 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.508212 17121 ts_tablet_manager.cc:1403] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.067s	user 0.009s	sys 0.009s
W20260709 12:19:42.510157 16965 leader_election.cc:343] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.515419 17128 raft_consensus.cc:493] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.515252 17121 raft_consensus.cc:359] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.516083 17128 raft_consensus.cc:515] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.516861 17121 raft_consensus.cc:385] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.517288 17121 raft_consensus.cc:740] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.519580 17128 leader_election.cc:290] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.523000 17123 raft_consensus.cc:493] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.525949 17119 tablet_bootstrap.cc:492] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.526628 17119 ts_tablet_manager.cc:1403] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.042s	user 0.014s	sys 0.007s
I20260709 12:19:42.523612 17123 raft_consensus.cc:515] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.518256 17121 consensus_queue.cc:260] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.533455 17123 leader_election.cc:290] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.548951 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.535801 17121 ts_tablet_manager.cc:1434] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260709 12:19:42.536082 17119 raft_consensus.cc:359] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
W20260709 12:19:42.551194 16965 leader_election.cc:343] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.551280 17119 raft_consensus.cc:385] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.551764 17119 raft_consensus.cc:740] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.551964 17121 tablet_bootstrap.cc:492] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.552618 17119 consensus_queue.cc:260] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
W20260709 12:19:42.543735 17102 tablet.cc:2406] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:19:42.556958 17119 ts_tablet_manager.cc:1434] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260709 12:19:42.558522 17119 tablet_bootstrap.cc:492] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.569226 17119 tablet_bootstrap.cc:654] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.569962 17121 tablet_bootstrap.cc:654] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.573652 17120 tablet_bootstrap.cc:492] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:42.574501 17120 ts_tablet_manager.cc:1403] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.118s	user 0.021s	sys 0.021s
I20260709 12:19:42.578613 17120 raft_consensus.cc:359] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.579182 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.579715 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.581925 17120 raft_consensus.cc:385] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.582836 17120 raft_consensus.cc:740] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.581485 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.585301 17120 consensus_queue.cc:260] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.591615 17120 ts_tablet_manager.cc:1434] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.002s
I20260709 12:19:42.592965 17120 tablet_bootstrap.cc:492] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
W20260709 12:19:42.593928 16967 leader_election.cc:343] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.594553 16967 leader_election.cc:304] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6; no voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:42.598038 17123 raft_consensus.cc:2749] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:19:42.596196 16967 leader_election.cc:343] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:19:42.597651 16892 leader_election.cc:343] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.599725 16967 leader_election.cc:304] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6; no voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:42.600605 17123 raft_consensus.cc:2749] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:42.600946 17120 tablet_bootstrap.cc:654] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.604453 17119 tablet_bootstrap.cc:492] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.605047 17119 ts_tablet_manager.cc:1403] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.047s	user 0.012s	sys 0.003s
I20260709 12:19:42.606678 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:42.607681 17002 raft_consensus.cc:2468] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 0.
I20260709 12:19:42.609715 16890 leader_election.cc:304] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:42.611732 17128 raft_consensus.cc:2804] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:42.612298 17128 raft_consensus.cc:493] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.612752 17128 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.614403 17119 raft_consensus.cc:359] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.615408 17119 raft_consensus.cc:385] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.615742 17119 raft_consensus.cc:740] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.616562 17119 consensus_queue.cc:260] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.617686 17121 tablet_bootstrap.cc:492] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.618340 17121 ts_tablet_manager.cc:1403] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.013s
I20260709 12:19:42.621585 17128 raft_consensus.cc:515] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.621778 17121 raft_consensus.cc:359] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.622710 17121 raft_consensus.cc:385] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.623111 17121 raft_consensus.cc:740] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.624176 17121 consensus_queue.cc:260] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.625478 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:42.631443 17002 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.632123 17121 ts_tablet_manager.cc:1434] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 12:19:42.633380 17121 tablet_bootstrap.cc:492] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.625861 17119 ts_tablet_manager.cc:1434] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.001s
I20260709 12:19:42.640416 17002 raft_consensus.cc:2468] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 1.
I20260709 12:19:42.641314 17119 tablet_bootstrap.cc:492] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.641809 16890 leader_election.cc:304] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:42.628026 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
W20260709 12:19:42.650118 16892 leader_election.cc:343] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.626573 17128 leader_election.cc:290] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.651319 17121 tablet_bootstrap.cc:654] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.651489 17128 raft_consensus.cc:493] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.653141 17128 raft_consensus.cc:515] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.661098 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
W20260709 12:19:42.663465 16892 leader_election.cc:343] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.665030 17120 tablet_bootstrap.cc:492] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:42.666046 17119 tablet_bootstrap.cc:654] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.666181 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:42.666805 17120 ts_tablet_manager.cc:1403] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.074s	user 0.014s	sys 0.005s
W20260709 12:19:42.667901 16890 leader_election.cc:343] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.668419 16890 leader_election.cc:304] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec; no voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:42.669770 17142 raft_consensus.cc:2749] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:42.670835 17128 leader_election.cc:290] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.661368 17140 raft_consensus.cc:2804] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:42.671959 17140 raft_consensus.cc:697] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:42.673193 17121 tablet_bootstrap.cc:492] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.672958 17140 consensus_queue.cc:237] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.673852 17121 ts_tablet_manager.cc:1403] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.041s	user 0.012s	sys 0.005s
I20260709 12:19:42.679533 17120 raft_consensus.cc:359] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.680810 17120 raft_consensus.cc:385] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.681314 17120 raft_consensus.cc:740] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.682542 17120 consensus_queue.cc:260] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.686144 17120 ts_tablet_manager.cc:1434] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 12:19:42.678071 17121 raft_consensus.cc:359] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.686817 17121 raft_consensus.cc:385] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.687264 17121 raft_consensus.cc:740] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.688278 17121 consensus_queue.cc:260] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.690706 17119 tablet_bootstrap.cc:492] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.691563 17119 ts_tablet_manager.cc:1403] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.051s	user 0.020s	sys 0.011s
I20260709 12:19:42.693245 17121 ts_tablet_manager.cc:1434] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:19:42.694448 17121 tablet_bootstrap.cc:492] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.694741 17120 tablet_bootstrap.cc:492] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:42.703457 17120 tablet_bootstrap.cc:654] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.704329 16821 catalog_manager.cc:5697] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 1 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:42.707011 17121 tablet_bootstrap.cc:654] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.707188 17119 raft_consensus.cc:359] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.709232 17119 raft_consensus.cc:385] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.710269 17119 raft_consensus.cc:740] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.712215 17123 raft_consensus.cc:493] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.713932 17119 consensus_queue.cc:260] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.714193 17123 raft_consensus.cc:515] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.721256 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.722693 16927 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:42.723181 17123 leader_election.cc:290] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.725162 16965 leader_election.cc:304] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:42.726511 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.727587 17074 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:42.730116 17123 raft_consensus.cc:2804] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:42.730588 17123 raft_consensus.cc:493] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.730985 17123 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.747213 17121 tablet_bootstrap.cc:492] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.747876 17121 ts_tablet_manager.cc:1403] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.054s	user 0.015s	sys 0.016s
I20260709 12:19:42.749008 17119 ts_tablet_manager.cc:1434] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.007s
I20260709 12:19:42.750321 17119 tablet_bootstrap.cc:492] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.750670 17120 tablet_bootstrap.cc:492] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:42.751242 17120 ts_tablet_manager.cc:1403] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.057s	user 0.020s	sys 0.002s
I20260709 12:19:42.754285 17121 raft_consensus.cc:359] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.756021 17121 raft_consensus.cc:385] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.756434 17121 raft_consensus.cc:740] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.756911 17120 raft_consensus.cc:359] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.757380 17121 consensus_queue.cc:260] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.757864 17120 raft_consensus.cc:385] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.759513 17120 raft_consensus.cc:740] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.762151 17119 tablet_bootstrap.cc:654] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.762234 17121 ts_tablet_manager.cc:1434] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.000s
I20260709 12:19:42.760679 17120 consensus_queue.cc:260] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.763535 17121 tablet_bootstrap.cc:492] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.766335 17120 ts_tablet_manager.cc:1434] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.009s
I20260709 12:19:42.759035 17123 raft_consensus.cc:515] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.767670 17120 tablet_bootstrap.cc:492] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:42.770610 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:42.771003 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:42.771999 17121 tablet_bootstrap.cc:654] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.772207 16927 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.772312 17074 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.781085 17074 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 1.
I20260709 12:19:42.782675 16967 leader_election.cc:304] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:42.783875 17148 raft_consensus.cc:2804] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:42.785466 17121 tablet_bootstrap.cc:492] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.789620 17121 ts_tablet_manager.cc:1403] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.007s
I20260709 12:19:42.793064 17121 raft_consensus.cc:359] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.785470 16927 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 1.
I20260709 12:19:42.794934 17126 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.800577 17121 raft_consensus.cc:385] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.800951 17121 raft_consensus.cc:740] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.800839 17126 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.801941 17121 consensus_queue.cc:260] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.804325 17126 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:42.796471 17120 tablet_bootstrap.cc:654] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.811983 17121 ts_tablet_manager.cc:1434] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.002s
I20260709 12:19:42.813194 17121 tablet_bootstrap.cc:492] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.800246 17148 raft_consensus.cc:697] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:42.797797 17123 leader_election.cc:290] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.818825 17140 raft_consensus.cc:493] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.819584 17148 consensus_queue.cc:237] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.814777 17156 raft_consensus.cc:493] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.819979 17140 raft_consensus.cc:515] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.821686 17156 raft_consensus.cc:515] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.799806 17154 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.830233 17140 leader_election.cc:290] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.831661 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
W20260709 12:19:42.835481 16890 leader_election.cc:343] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.833386 17156 leader_election.cc:290] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:42.828331 17154 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.838416 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.839517 17074 raft_consensus.cc:2468] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 0.
I20260709 12:19:42.841727 17154 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.843787 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.845520 17121 tablet_bootstrap.cc:654] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.853925 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.854848 17074 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:42.845583 16892 leader_election.cc:304] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:42.857311 17140 raft_consensus.cc:2804] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:42.857872 17140 raft_consensus.cc:493] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.858300 17140 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.849774 17119 tablet_bootstrap.cc:492] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.850214 17120 tablet_bootstrap.cc:492] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:42.860595 17120 ts_tablet_manager.cc:1403] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.093s	user 0.004s	sys 0.034s
W20260709 12:19:42.863147 16965 leader_election.cc:343] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.866209 17120 raft_consensus.cc:359] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.867141 17120 raft_consensus.cc:385] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.867483 17120 raft_consensus.cc:740] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.860633 17119 ts_tablet_manager.cc:1403] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.111s	user 0.017s	sys 0.004s
I20260709 12:19:42.868381 16967 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:42.869527 17154 raft_consensus.cc:2804] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:42.870463 17154 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.870997 17154 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.872517 17119 raft_consensus.cc:359] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.872747 17120 consensus_queue.cc:260] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.873499 17119 raft_consensus.cc:385] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.874140 17119 raft_consensus.cc:740] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.874998 17119 consensus_queue.cc:260] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.882879 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.884918 17119 ts_tablet_manager.cc:1434] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20260709 12:19:42.886327 17119 tablet_bootstrap.cc:492] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:42.889060 17154 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.883139 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.890904 17120 ts_tablet_manager.cc:1434] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260709 12:19:42.892210 17120 tablet_bootstrap.cc:492] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:42.893574 17123 raft_consensus.cc:493] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:42.893803 17039 leader_election.cc:343] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.894197 17123 raft_consensus.cc:515] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
W20260709 12:19:42.895222 17039 leader_election.cc:343] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.896749 17123 leader_election.cc:290] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.898890 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:42.899154 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:42.906246 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.907351 16927 raft_consensus.cc:2468] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:42.904160 17002 raft_consensus.cc:2393] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 849bbec7c1154b2d9b70c1709ebfa534 in current term 1: Already voted for candidate 012c695806b6424dad30e97211e4fcc6 in this term.
I20260709 12:19:42.904368 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:42.916373 16965 leader_election.cc:304] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
W20260709 12:19:42.917495 16967 leader_election.cc:343] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.923095 17154 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.923923 17154 raft_consensus.cc:2804] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:42.924333 17154 raft_consensus.cc:493] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:42.924701 17154 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.925120 16824 catalog_manager.cc:5697] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:42.927942 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
W20260709 12:19:42.930198 16965 leader_election.cc:343] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.930609 17120 tablet_bootstrap.cc:654] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.933125 17121 tablet_bootstrap.cc:492] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:42.933694 17121 ts_tablet_manager.cc:1403] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.121s	user 0.014s	sys 0.018s
W20260709 12:19:42.932372 17039 leader_election.cc:343] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.936599 17140 raft_consensus.cc:515] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.940898 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6"
W20260709 12:19:42.942903 16890 leader_election.cc:343] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:42.935545 17039 leader_election.cc:304] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:42.944172 17156 raft_consensus.cc:2749] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:42.937901 17119 tablet_bootstrap.cc:654] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:42.946228 17121 raft_consensus.cc:359] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.947186 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:42.947350 17121 raft_consensus.cc:385] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.947955 17074 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.948009 17121 raft_consensus.cc:740] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.949492 17121 consensus_queue.cc:260] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.944173 17154 raft_consensus.cc:515] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
W20260709 12:19:42.954960 16872 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:42.957837 17156 raft_consensus.cc:493] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.958729 17156 raft_consensus.cc:515] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.960474 17039 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:42.962335 17156 leader_election.cc:290] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:42.962643 17074 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 1.
I20260709 12:19:42.965519 17154 leader_election.cc:290] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.966504 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:42.967228 16927 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.967778 17121 ts_tablet_manager.cc:1434] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.003s
I20260709 12:19:42.967908 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:42.968828 17156 raft_consensus.cc:2749] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:42.969018 17075 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:42.970096 17140 leader_election.cc:290] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:42.974814 17121 tablet_bootstrap.cc:492] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:42.980376 17120 tablet_bootstrap.cc:492] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:42.982180 17120 ts_tablet_manager.cc:1403] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.090s	user 0.016s	sys 0.024s
I20260709 12:19:42.985069 17075 raft_consensus.cc:2468] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 1.
I20260709 12:19:42.986552 16892 leader_election.cc:304] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:42.986141 17120 raft_consensus.cc:359] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.987491 17120 raft_consensus.cc:385] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:42.987833 17119 tablet_bootstrap.cc:492] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:42.988127 17120 raft_consensus.cc:740] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:42.989383 17119 ts_tablet_manager.cc:1403] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.103s	user 0.003s	sys 0.013s
I20260709 12:19:42.989969 17120 consensus_queue.cc:260] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.991402 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:42.992830 17128 raft_consensus.cc:493] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.999066 17128 raft_consensus.cc:515] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:42.993070 17140 raft_consensus.cc:2804] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:43.058905 16927 raft_consensus.cc:2468] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 1.
I20260709 12:19:43.059926 17120 ts_tablet_manager.cc:1434] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.077s	user 0.006s	sys 0.031s
I20260709 12:19:43.060725 16965 leader_election.cc:304] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:43.061224 17120 tablet_bootstrap.cc:492] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:43.061357 17121 tablet_bootstrap.cc:654] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.062278 17154 raft_consensus.cc:2804] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:43.064590 17154 raft_consensus.cc:697] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:43.065603 17154 consensus_queue.cc:237] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.066146 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:43.067303 16967 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.068315 17148 raft_consensus.cc:2804] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:43.068873 17148 raft_consensus.cc:697] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:43.069172 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:43.069734 17148 consensus_queue.cc:237] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.070045 17156 raft_consensus.cc:493] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.080472 17002 raft_consensus.cc:2393] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 849bbec7c1154b2d9b70c1709ebfa534 in current term 1: Already voted for candidate 0e10c2ade849401e8ba5d8141abea6ec in this term.
I20260709 12:19:43.103936 17039 leader_election.cc:304] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.105489 17126 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.072069 17156 raft_consensus.cc:515] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.073643 17140 raft_consensus.cc:697] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
W20260709 12:19:43.111537 16967 leader_election.cc:343] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.068325 17119 raft_consensus.cc:359] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.131753 17156 leader_election.cc:290] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:43.124854 17128 leader_election.cc:290] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.129077 17119 raft_consensus.cc:385] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.137567 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:43.138713 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:43.138957 17140 consensus_queue.cc:237] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.139586 17002 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 0.
I20260709 12:19:43.139910 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
W20260709 12:19:43.141167 17039 leader_election.cc:343] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.142369 17039 leader_election.cc:304] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.141193 17075 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 0.
I20260709 12:19:43.144285 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:43.144958 16892 leader_election.cc:304] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:43.145145 17002 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 0.
I20260709 12:19:43.144690 16821 catalog_manager.cc:5697] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:43.147686 17156 raft_consensus.cc:2804] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:43.148247 17156 raft_consensus.cc:493] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.148702 17156 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.148473 17128 raft_consensus.cc:2804] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:43.157161 17128 raft_consensus.cc:493] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.157711 17128 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.160066 17120 tablet_bootstrap.cc:654] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.150352 17121 tablet_bootstrap.cc:492] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:43.135284 17119 raft_consensus.cc:740] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.166534 17121 ts_tablet_manager.cc:1403] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.192s	user 0.018s	sys 0.080s
I20260709 12:19:43.166217 17119 consensus_queue.cc:260] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.156265 17126 raft_consensus.cc:2749] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:43.174965 17119 ts_tablet_manager.cc:1434] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.185s	user 0.010s	sys 0.001s
I20260709 12:19:43.175254 17121 raft_consensus.cc:359] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.176203 17121 raft_consensus.cc:385] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.176620 17121 raft_consensus.cc:740] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.177721 17121 consensus_queue.cc:260] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.180812 17121 ts_tablet_manager.cc:1434] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.002s
I20260709 12:19:43.171111 17128 raft_consensus.cc:515] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.186443 16821 catalog_manager.cc:5697] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 1 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:43.189729 17156 raft_consensus.cc:515] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.190922 17119 tablet_bootstrap.cc:492] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:43.202327 17128 leader_election.cc:290] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.204625 17121 tablet_bootstrap.cc:492] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534: Bootstrap starting.
I20260709 12:19:43.205271 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:43.206524 17075 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.208652 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:43.209378 17002 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.217034 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:43.217033 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:43.218731 17001 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.216204 17156 leader_election.cc:290] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:43.224030 17119 tablet_bootstrap.cc:654] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.228813 17002 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 1.
W20260709 12:19:43.229907 17039 leader_election.cc:343] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.230865 17039 leader_election.cc:304] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.232394 17001 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 1.
I20260709 12:19:43.232872 17156 raft_consensus.cc:2804] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:43.233458 17156 raft_consensus.cc:697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:43.233888 16824 catalog_manager.cc:5697] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:43.235306 16890 leader_election.cc:304] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:43.234395 17156 consensus_queue.cc:237] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.236451 17128 raft_consensus.cc:2804] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:43.237952 17128 raft_consensus.cc:697] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:43.238770 17075 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 1.
I20260709 12:19:43.239108 17128 consensus_queue.cc:237] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.240516 17121 tablet_bootstrap.cc:654] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.248805 17120 tablet_bootstrap.cc:492] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:43.249507 17120 ts_tablet_manager.cc:1403] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.189s	user 0.008s	sys 0.012s
I20260709 12:19:43.253764 16821 catalog_manager.cc:5697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:43.261197 17119 tablet_bootstrap.cc:492] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:43.262107 17119 ts_tablet_manager.cc:1403] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.080s	user 0.018s	sys 0.001s
I20260709 12:19:43.263440 17120 raft_consensus.cc:359] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.264552 17120 raft_consensus.cc:385] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.265622 17120 raft_consensus.cc:740] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.271415 17120 consensus_queue.cc:260] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.275797 17120 ts_tablet_manager.cc:1434] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.003s
I20260709 12:19:43.278642 17119 raft_consensus.cc:359] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.279520 17119 raft_consensus.cc:385] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.279968 17119 raft_consensus.cc:740] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.280814 17119 consensus_queue.cc:260] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.281481 17121 tablet_bootstrap.cc:492] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534: No bootstrap required, opened a new log
I20260709 12:19:43.282428 17121 ts_tablet_manager.cc:1403] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent bootstrapping tablet: real 0.078s	user 0.016s	sys 0.016s
I20260709 12:19:43.283874 17128 consensus_queue.cc:1048] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.286247 17121 raft_consensus.cc:359] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.287062 17121 raft_consensus.cc:385] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.287122 17119 ts_tablet_manager.cc:1434] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.001s
I20260709 12:19:43.287418 17121 raft_consensus.cc:740] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.287097 17142 consensus_queue.cc:1048] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.288443 17119 tablet_bootstrap.cc:492] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: Bootstrap starting.
I20260709 12:19:43.288309 17121 consensus_queue.cc:260] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.291443 17121 ts_tablet_manager.cc:1434] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:19:43.275141 16823 catalog_manager.cc:5697] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 1 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:43.294266 17120 tablet_bootstrap.cc:492] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:43.310528 17148 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.311185 17148 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.315842 17156 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.316514 17156 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.326200 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
W20260709 12:19:43.328011 17039 leader_election.cc:343] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.333321 17156 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:43.333738 17120 tablet_bootstrap.cc:654] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.333997 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:43.335615 17119 tablet_bootstrap.cc:654] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.336838 17148 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.337160 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
W20260709 12:19:43.337581 16965 leader_election.cc:343] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.338331 17002 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 0.
I20260709 12:19:43.338652 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:43.340003 17074 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:43.342061 16967 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.348181 17148 raft_consensus.cc:2804] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:43.348677 17148 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.349114 17148 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.357705 17039 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.359898 17156 raft_consensus.cc:2804] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:43.360625 17156 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.361202 17156 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.364711 17128 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.365391 17128 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.368435 17128 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.375238 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:43.376703 17074 raft_consensus.cc:2393] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e10c2ade849401e8ba5d8141abea6ec in current term 1: Already voted for candidate 012c695806b6424dad30e97211e4fcc6 in this term.
I20260709 12:19:43.379331 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:43.380617 17120 tablet_bootstrap.cc:492] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:43.381287 17120 ts_tablet_manager.cc:1403] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.088s	user 0.012s	sys 0.006s
I20260709 12:19:43.369520 17156 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.402201 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:43.402954 17148 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.403889 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:43.404697 17002 raft_consensus.cc:2393] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 849bbec7c1154b2d9b70c1709ebfa534 in current term 1: Already voted for candidate 012c695806b6424dad30e97211e4fcc6 in this term.
W20260709 12:19:43.405763 17039 leader_election.cc:343] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.407330 17156 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:43.407577 16892 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec; no voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:43.407861 17039 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.413329 17126 raft_consensus.cc:2749] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:43.413650 17140 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.415261 17119 tablet_bootstrap.cc:492] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: No bootstrap required, opened a new log
I20260709 12:19:43.415130 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:43.415850 17119 ts_tablet_manager.cc:1403] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: Time spent bootstrapping tablet: real 0.128s	user 0.013s	sys 0.011s
I20260709 12:19:43.417167 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:43.418072 17074 raft_consensus.cc:2393] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 012c695806b6424dad30e97211e4fcc6 in current term 1: Already voted for candidate 849bbec7c1154b2d9b70c1709ebfa534 in this term.
I20260709 12:19:43.419590 17119 raft_consensus.cc:359] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.420310 17119 raft_consensus.cc:385] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.420621 17119 raft_consensus.cc:740] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.421633 17120 raft_consensus.cc:359] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.422472 17120 raft_consensus.cc:385] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.422870 17120 raft_consensus.cc:740] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.422505 17148 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.421344 17119 consensus_queue.cc:260] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
W20260709 12:19:43.432770 16965 leader_election.cc:343] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:43.433399 16965 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6; no voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:43.423810 17120 consensus_queue.cc:260] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.436034 17120 ts_tablet_manager.cc:1434] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.054s	user 0.005s	sys 0.004s
I20260709 12:19:43.437404 17120 tablet_bootstrap.cc:492] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec: Bootstrap starting.
I20260709 12:19:43.441059 17156 raft_consensus.cc:493] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.441727 17156 raft_consensus.cc:515] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.444352 17156 leader_election.cc:290] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527)
I20260709 12:19:43.444690 17120 tablet_bootstrap.cc:654] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:43.446767 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:43.447789 16927 raft_consensus.cc:2393] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 849bbec7c1154b2d9b70c1709ebfa534 in current term 1: Already voted for candidate 012c695806b6424dad30e97211e4fcc6 in this term.
I20260709 12:19:43.453614 17140 raft_consensus.cc:2749] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:43.454607 17120 tablet_bootstrap.cc:492] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec: No bootstrap required, opened a new log
I20260709 12:19:43.455245 17120 ts_tablet_manager.cc:1403] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.003s
I20260709 12:19:43.457810 17154 raft_consensus.cc:2749] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:43.458765 17120 raft_consensus.cc:359] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.459383 17119 ts_tablet_manager.cc:1434] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 12:19:43.459704 17120 raft_consensus.cc:385] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:43.460101 17120 raft_consensus.cc:740] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Initialized, Role: FOLLOWER
I20260709 12:19:43.460951 17120 consensus_queue.cc:260] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.463450 17120 ts_tablet_manager.cc:1434] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:19:43.465346 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:43.467198 17039 leader_election.cc:304] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 849bbec7c1154b2d9b70c1709ebfa534; no voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:43.467592 17154 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:43.468999 17156 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.470758 17154 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:43.477799 17156 raft_consensus.cc:2749] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:43.560778 17154 raft_consensus.cc:493] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.561408 17154 raft_consensus.cc:515] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.563514 17154 leader_election.cc:290] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.564611 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:43.564903 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:43.565290 16927 raft_consensus.cc:2468] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:43.565752 17075 raft_consensus.cc:2468] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 0.
I20260709 12:19:43.566656 16965 leader_election.cc:304] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:43.567515 17154 raft_consensus.cc:2804] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:43.567929 17154 raft_consensus.cc:493] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.568235 17154 raft_consensus.cc:3060] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.569146 17140 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.569664 17140 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.573086 17140 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.573302 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:43.574244 17001 raft_consensus.cc:2393] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e10c2ade849401e8ba5d8141abea6ec in current term 1: Already voted for candidate 012c695806b6424dad30e97211e4fcc6 in this term.
I20260709 12:19:43.574084 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534" is_pre_election: true
I20260709 12:19:43.574978 17075 raft_consensus.cc:2393] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e10c2ade849401e8ba5d8141abea6ec in current term 1: Already voted for candidate 849bbec7c1154b2d9b70c1709ebfa534 in this term.
I20260709 12:19:43.576195 17154 raft_consensus.cc:515] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.576195 16892 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec; no voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:43.577383 17140 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.578652 17154 leader_election.cc:290] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:43.579746 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:43.580454 16927 raft_consensus.cc:3060] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.585928 17140 raft_consensus.cc:2749] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:43.587600 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:43.588196 16927 raft_consensus.cc:2468] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 1.
I20260709 12:19:43.588376 17075 raft_consensus.cc:3060] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.589679 16965 leader_election.cc:304] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:43.590665 17154 raft_consensus.cc:2804] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:43.591207 17154 raft_consensus.cc:697] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:43.592809 17154 consensus_queue.cc:237] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:43.594957 17075 raft_consensus.cc:2468] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 1.
I20260709 12:19:43.602980 16824 catalog_manager.cc:5697] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:43.623468 17001 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.633630 17140 consensus_queue.cc:1048] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.642261 17154 consensus_queue.cc:1048] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.653483 17140 consensus_queue.cc:1048] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.662719 17123 consensus_queue.cc:1048] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.677597 16927 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:43.680999 17128 consensus_queue.cc:1048] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.709136 17142 consensus_queue.cc:1048] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.713781 17126 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.723412 17154 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.733752 17156 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:43.756026 17123 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:19:43.961624 16872 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:44.049172 17148 consensus_queue.cc:1048] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:44.067724 17123 consensus_queue.cc:1048] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:19:44.966895 16872 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:44.979341 17194 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.980003 17194 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:44.981935 17194 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:44.983160 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec" is_pre_election: true
I20260709 12:19:44.983363 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6" is_pre_election: true
I20260709 12:19:44.983911 16927 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 1.
I20260709 12:19:44.983986 17001 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 1.
I20260709 12:19:44.985239 17039 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:44.986021 17194 raft_consensus.cc:2804] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:19:44.986375 17194 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:44.986701 17194 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:44.992305 17194 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:44.994318 17194 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:44.995393 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:44.995733 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:44.996031 16927 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:44.996428 17001 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.002990 16927 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:45.002990 17001 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:45.004482 17039 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:45.005407 17194 raft_consensus.cc:2804] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:45.006593 17194 raft_consensus.cc:697] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:45.007462 17194 consensus_queue.cc:237] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.016078 16824 catalog_manager.cc:5697] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 0 to 2, leader changed from <none> to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.497437 17001 raft_consensus.cc:1275] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:19:45.498977 17194 consensus_queue.cc:1048] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:45.510784 16927 raft_consensus.cc:1275] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:19:45.513190 17205 consensus_queue.cc:1048] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:45.576850 17001 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52b372c5d74648458c4589b25e07b070"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:57106
I20260709 12:19:45.577694 17001 raft_consensus.cc:606] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Received request to transfer leadership to TS 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:45.583870 17001 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:57106
I20260709 12:19:45.584524 17001 raft_consensus.cc:606] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Received request to transfer leadership to TS 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:45.603430 17075 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
mode: GRACEFUL
new_leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:47488
I20260709 12:19:45.604041 17075 raft_consensus.cc:606] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 LEADER]: Received request to transfer leadership to TS 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:19:45.613656 17074 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:47488
I20260709 12:19:45.614372 17074 raft_consensus.cc:606] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:45.617643 17123 raft_consensus.cc:993] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:19:45.618208 17154 raft_consensus.cc:1081] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Signalling peer 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:19:45.620271 16927 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:41192
I20260709 12:19:45.620975 16927 raft_consensus.cc:493] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:45.621412 16927 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.624444 17123 raft_consensus.cc:993] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:19:45.624979 17154 raft_consensus.cc:1081] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Signalling peer 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:19:45.626804 16926 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "52b372c5d74648458c4589b25e07b070"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:41192
I20260709 12:19:45.627445 16926 raft_consensus.cc:493] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:45.627806 16926 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.629093 16927 raft_consensus.cc:515] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.632361 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:45.633059 17001 raft_consensus.cc:3055] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:45.633342 17001 raft_consensus.cc:740] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:45.634224 17001 consensus_queue.cc:260] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.635308 16926 raft_consensus.cc:515] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.635696 17001 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.636049 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:45.636631 17074 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.638317 16925 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:46188
I20260709 12:19:45.638973 16925 raft_consensus.cc:606] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:45.640242 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:45.640823 17002 raft_consensus.cc:3055] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:45.641183 17002 raft_consensus.cc:740] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:45.642062 17002 consensus_queue.cc:260] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.643273 17002 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.645650 17001 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 2.
I20260709 12:19:45.645901 16925 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:46188
I20260709 12:19:45.646636 16925 raft_consensus.cc:606] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:45.648226 16890 leader_election.cc:304] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:45.648955 17074 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 2.
I20260709 12:19:45.649171 17142 raft_consensus.cc:2804] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:45.650602 16927 leader_election.cc:290] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 2 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:45.654976 17142 raft_consensus.cc:697] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:45.655833 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:45.656545 17074 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.656672 17002 raft_consensus.cc:2468] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 2.
I20260709 12:19:45.656728 17001 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:57106
I20260709 12:19:45.657840 17001 raft_consensus.cc:606] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Received request to transfer leadership to TS 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:45.657008 17142 consensus_queue.cc:237] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.657961 16890 leader_election.cc:304] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:45.659126 17213 raft_consensus.cc:2804] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:45.660910 17213 raft_consensus.cc:697] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:45.662068 17213 consensus_queue.cc:237] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.665935 16926 leader_election.cc:290] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 2 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:45.665804 17074 raft_consensus.cc:2468] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 2.
I20260709 12:19:45.668499 16822 catalog_manager.cc:5697] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 1 to 2, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 2 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:45.687551 16822 catalog_manager.cc:5697] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 1 to 2, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 2 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:45.946316 17205 raft_consensus.cc:993] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: : Instructing follower 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:19:45.946864 17194 raft_consensus.cc:1081] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 LEADER]: Signalling peer 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:19:45.948796 16926 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:41208
I20260709 12:19:45.949433 16926 raft_consensus.cc:493] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:45.949803 16926 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.955658 16926 raft_consensus.cc:515] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.957697 16926 leader_election.cc:290] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 2 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:45.958866 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:45.958881 17001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:45.959606 17074 raft_consensus.cc:3055] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:45.959620 17001 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.959997 17074 raft_consensus.cc:740] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:45.960740 17074 consensus_queue.cc:260] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.961793 17074 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:45.965471 17001 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 2.
I20260709 12:19:45.966810 16890 leader_election.cc:304] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:45.967754 17074 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 2.
I20260709 12:19:45.967778 17213 raft_consensus.cc:2804] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:45.968470 17213 raft_consensus.cc:697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:45.969426 17213 consensus_queue.cc:237] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:45.979131 16822 catalog_manager.cc:5697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 1 to 2, leader changed from 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3) to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 2 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.009845 17142 raft_consensus.cc:993] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:46.010368 17213 raft_consensus.cc:1081] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:46.012228 17001 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49296
I20260709 12:19:46.012887 17001 raft_consensus.cc:493] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:46.013233 17001 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.019101 17001 raft_consensus.cc:515] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.021075 17001 leader_election.cc:290] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 2 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:46.022292 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:46.022291 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:46.023082 17074 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.023170 16926 raft_consensus.cc:3055] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:46.023531 16926 raft_consensus.cc:740] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:46.024307 16926 consensus_queue.cc:260] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.025483 16926 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.029428 17074 raft_consensus.cc:2468] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 2.
I20260709 12:19:46.030967 16967 leader_election.cc:304] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:46.031391 16926 raft_consensus.cc:2468] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 2.
I20260709 12:19:46.032008 17154 raft_consensus.cc:2804] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:46.032524 17154 raft_consensus.cc:697] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:46.033345 17154 consensus_queue.cc:237] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.042999 16822 catalog_manager.cc:5697] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 1 to 2, leader changed from 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.110651 17205 raft_consensus.cc:993] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:46.111222 17222 raft_consensus.cc:1081] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:46.113370 17001 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49298
I20260709 12:19:46.114122 17001 raft_consensus.cc:493] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:46.114550 17001 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:46.120616 17001 raft_consensus.cc:515] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.122584 17001 leader_election.cc:290] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:46.123778 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:46.123804 17074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24b40b9dd1114b498e4dcc91c97be5b2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:46.124675 17074 raft_consensus.cc:3055] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:46.124675 16926 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:46.125170 17074 raft_consensus.cc:740] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:46.126055 17074 consensus_queue.cc:260] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.127265 17074 raft_consensus.cc:3060] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:46.131594 16926 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:46.133222 17074 raft_consensus.cc:2468] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:46.133267 16965 leader_election.cc:304] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:46.134307 17154 raft_consensus.cc:2804] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:46.134841 17154 raft_consensus.cc:697] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:46.135761 17154 consensus_queue.cc:237] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.140815 17001 raft_consensus.cc:1275] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.142436 17213 consensus_queue.cc:1048] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:46.144479 16822 catalog_manager.cc:5697] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 2 to 3, leader changed from 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.168380 17213 raft_consensus.cc:993] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:46.170894 17213 raft_consensus.cc:1081] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:46.171527 17154 raft_consensus.cc:993] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:19:46.172036 17154 raft_consensus.cc:1081] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Signalling peer 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:19:46.174232 17001 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49296
I20260709 12:19:46.174777 17074 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:47794
I20260709 12:19:46.175140 17001 raft_consensus.cc:493] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:46.175436 17074 raft_consensus.cc:493] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:46.175637 17001 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.175913 17074 raft_consensus.cc:3060] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.178157 17075 raft_consensus.cc:1275] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.180544 17142 consensus_queue.cc:1048] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:46.191267 17002 raft_consensus.cc:1275] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.192711 17074 raft_consensus.cc:515] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.194391 17142 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:46.197470 17001 raft_consensus.cc:515] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.208619 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:46.209411 16926 raft_consensus.cc:3055] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:46.209836 16926 raft_consensus.cc:740] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:46.211077 17001 leader_election.cc:290] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 2 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:46.210724 16926 consensus_queue.cc:260] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.212443 17075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:46.218461 17075 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.218868 17000 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:46.219576 17000 raft_consensus.cc:3055] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:46.219919 17000 raft_consensus.cc:740] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:46.220866 17000 consensus_queue.cc:260] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.222229 17000 raft_consensus.cc:3060] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.216377 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4984c9b7f4173b47582e4e794e607" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:46.225054 16927 raft_consensus.cc:3060] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.226115 17076 raft_consensus.cc:1275] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.228340 17213 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.232928 17000 raft_consensus.cc:2468] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:46.234404 17039 leader_election.cc:304] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:46.235493 17222 raft_consensus.cc:2804] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:46.236106 17222 raft_consensus.cc:697] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:46.237130 17222 consensus_queue.cc:237] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.239874 16926 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.217100 17074 leader_election.cc:290] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:46.252108 16927 raft_consensus.cc:2468] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:46.256081 17075 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 2.
I20260709 12:19:46.257355 17073 raft_consensus.cc:1240] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Rejecting Update request from peer 0e10c2ade849401e8ba5d8141abea6ec for earlier term 1. Current term is 2. Ops: []
I20260709 12:19:46.258003 16967 leader_election.cc:304] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:46.259236 17154 raft_consensus.cc:2804] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:46.260130 17154 raft_consensus.cc:697] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:46.260538 16926 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 2.
I20260709 12:19:46.261336 17154 consensus_queue.cc:237] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:46.272523 16824 catalog_manager.cc:5697] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 1 to 2, leader changed from 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.282114 16822 catalog_manager.cc:5697] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 1 to 2, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:46.422645 17076 raft_consensus.cc:1275] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.424283 17227 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.435423 17000 raft_consensus.cc:1275] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.437763 17142 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:46.479354 17076 raft_consensus.cc:1275] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.480953 17154 consensus_queue.cc:1048] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.491489 16926 raft_consensus.cc:1275] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.494179 17123 consensus_queue.cc:1048] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:46.514811 16926 raft_consensus.cc:1275] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:19:46.516237 17123 consensus_queue.cc:1048] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.525144 17076 raft_consensus.cc:1275] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:19:46.527019 17154 consensus_queue.cc:1048] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:46.664986 16926 raft_consensus.cc:1275] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.666745 17222 consensus_queue.cc:1048] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.678241 17000 raft_consensus.cc:1275] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.680677 17241 consensus_queue.cc:1048] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.772356 17073 raft_consensus.cc:1275] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.774086 17154 consensus_queue.cc:1048] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:46.786134 16926 raft_consensus.cc:1275] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:46.788043 17154 consensus_queue.cc:1048] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:48.695271 16792 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:19:48.696408 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:48.726212 17000 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:57112
I20260709 12:19:48.726868 17000 raft_consensus.cc:606] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Received request to transfer leadership to TS 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:48.743680 16926 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
mode: GRACEFUL
new_leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:46202
I20260709 12:19:48.744906 16926 raft_consensus.cc:606] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Received request to transfer leadership to TS 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:19:48.746876 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:19:48.747541 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:48.748289 17154 raft_consensus.cc:993] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:19:48.748723 17154 raft_consensus.cc:1081] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Signalling peer 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:19:48.750571 17073 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:47794
I20260709 12:19:48.751228 17073 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:48.751569 17073 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:48.757798 17073 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:48.759943 17073 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:48.761066 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:48.761320 17000 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:48.761704 16926 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:48.762004 17000 raft_consensus.cc:3055] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:48.762416 17000 raft_consensus.cc:740] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:48.763172 17000 consensus_queue.cc:260] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:48.764241 17000 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:48.768532 16926 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:48.770160 17039 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:48.770829 17000 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:48.771256 17241 raft_consensus.cc:2804] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:48.772802 17241 raft_consensus.cc:697] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:48.773867 17241 consensus_queue.cc:237] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:48.782760 16824 catalog_manager.cc:5697] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 1 to 2, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:49.181272 17000 raft_consensus.cc:1275] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:49.182977 17241 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:49.195230 16926 raft_consensus.cc:1275] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:49.198043 17241 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:49.218281 17263 raft_consensus.cc:993] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: : Instructing follower 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:19:49.218853 17224 raft_consensus.cc:1081] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Signalling peer 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:19:49.220674 17073 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:47806
I20260709 12:19:49.221302 17073 raft_consensus.cc:493] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:49.221698 17073 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:49.229058 17073 raft_consensus.cc:515] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:49.230954 17073 leader_election.cc:290] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:19:49.232136 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:49.232352 17000 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:19:49.232884 16926 raft_consensus.cc:3055] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:49.232952 17000 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:49.233234 16926 raft_consensus.cc:740] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:49.234097 16926 consensus_queue.cc:260] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:49.235138 16926 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:49.239267 17000 raft_consensus.cc:2468] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:49.240805 17039 leader_election.cc:304] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:49.240970 16926 raft_consensus.cc:2468] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 2.
I20260709 12:19:49.241954 17241 raft_consensus.cc:2804] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:49.242859 17241 raft_consensus.cc:697] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:49.243695 17241 consensus_queue.cc:237] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:49.252231 16822 catalog_manager.cc:5697] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 1 to 2, leader changed from 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1) to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:49.749257 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:49.755214 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:49.755657 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:49.793018 16926 raft_consensus.cc:1275] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:49.794849 17241 consensus_queue.cc:1048] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:49.809216 17000 raft_consensus.cc:1275] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:49.811319 17259 consensus_queue.cc:1048] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:50.757045 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:50.762318 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:50.762818 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:51.764024 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:51.769214 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:51.769797 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:52.771135 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:52.777070 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:52.777612 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:53.778813 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:53.784085 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:53.784667 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:54.786119 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:54.791898 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:54.792510 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:55.793802 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:55.799983 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:55.800464 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:56.801442 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:56.808071 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:56.808631 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:57.809973 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:57.816982 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:57.817746 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:58.973467 16792 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:19:58.979853 16926 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52b372c5d74648458c4589b25e07b070"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:46188
I20260709 12:19:58.980592 16926 raft_consensus.cc:606] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:58.986851 16926 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:46188
I20260709 12:19:58.987468 16926 raft_consensus.cc:606] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:58.993667 17076 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:47488
I20260709 12:19:58.994388 17076 raft_consensus.cc:606] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:59.001201 16926 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:46188
I20260709 12:19:59.001899 16926 raft_consensus.cc:606] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:59.005951 17280 raft_consensus.cc:993] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.006628 17283 raft_consensus.cc:1081] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.007874 17076 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
mode: GRACEFUL
new_leader_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:47488
I20260709 12:19:59.008476 17002 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "52b372c5d74648458c4589b25e07b070"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49296
I20260709 12:19:59.008616 17076 raft_consensus.cc:606] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Received request to transfer leadership to TS 012c695806b6424dad30e97211e4fcc6
I20260709 12:19:59.009155 17002 raft_consensus.cc:493] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:59.009639 17002 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.017709 17002 raft_consensus.cc:515] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.021220 17002 leader_election.cc:290] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:59.022210 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:59.022533 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52b372c5d74648458c4589b25e07b070" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:59.023036 16926 raft_consensus.cc:3055] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:59.023061 17076 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.023406 16926 raft_consensus.cc:740] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:59.024307 16926 consensus_queue.cc:260] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.025338 16926 raft_consensus.cc:3060] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.030604 17076 raft_consensus.cc:2468] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.032366 16967 leader_election.cc:304] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:59.033108 16926 raft_consensus.cc:2468] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.033406 17282 raft_consensus.cc:2804] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:59.034049 17282 raft_consensus.cc:697] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:59.035113 17282 consensus_queue.cc:237] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.047240 16822 catalog_manager.cc:5697] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 2 to 3, leader changed from 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:59.151322 17281 raft_consensus.cc:993] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.152981 17289 raft_consensus.cc:1081] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.154954 17000 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49298
I20260709 12:19:59.155547 17000 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:59.155928 17290 raft_consensus.cc:993] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.156045 17000 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.156522 17289 raft_consensus.cc:1081] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.158419 17002 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49298
I20260709 12:19:59.159039 17002 raft_consensus.cc:493] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:59.159392 17002 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.162352 17000 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.164312 17000 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:59.165261 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:59.165489 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:59.165956 16926 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.166144 17076 raft_consensus.cc:3055] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:59.165927 17002 raft_consensus.cc:515] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.166558 17076 raft_consensus.cc:740] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:59.167441 17076 consensus_queue.cc:260] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.169199 17002 leader_election.cc:290] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:59.169350 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:59.169589 17073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc62f04a8d84076896751bcf7de0727" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:59.170083 16927 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.170181 17073 raft_consensus.cc:3055] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:59.168678 17076 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.170529 17073 raft_consensus.cc:740] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:19:59.171646 17073 consensus_queue.cc:260] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.175092 16926 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.177719 16965 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:59.177942 17073 raft_consensus.cc:3060] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.178840 17282 raft_consensus.cc:2804] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:59.179364 17076 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.179384 17282 raft_consensus.cc:697] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:59.181265 17282 consensus_queue.cc:237] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.182870 16927 raft_consensus.cc:2468] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.184463 16965 leader_election.cc:304] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:59.185590 17272 raft_consensus.cc:2804] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:59.187355 17272 raft_consensus.cc:697] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:59.187312 17073 raft_consensus.cc:2468] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.188643 17272 consensus_queue.cc:237] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.194231 16822 catalog_manager.cc:5697] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 2 to 3, leader changed from 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:59.202451 17280 raft_consensus.cc:993] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.203048 17283 raft_consensus.cc:1081] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
W20260709 12:19:59.203043 17097 debug-util.cc:398] Leaking SignalData structure 0x7b080016d060 after lost signal to thread 16795
W20260709 12:19:59.204444 17097 debug-util.cc:398] Leaking SignalData structure 0x7b080016d740 after lost signal to thread 16858
W20260709 12:19:59.205524 17097 debug-util.cc:398] Leaking SignalData structure 0x7b08001ea5a0 after lost signal to thread 16951
I20260709 12:19:59.205950 17002 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49296
W20260709 12:19:59.206534 17097 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7760 after lost signal to thread 17026
I20260709 12:19:59.206620 17002 raft_consensus.cc:493] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:59.207150 17002 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:19:59.207680 17097 debug-util.cc:398] Leaking SignalData structure 0x7b08001bcb80 after lost signal to thread 17100
I20260709 12:19:59.212513 16822 catalog_manager.cc:5697] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 2 to 3, leader changed from 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:59.213539 17002 raft_consensus.cc:515] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.217240 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:59.217999 16927 raft_consensus.cc:3055] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:59.218375 16927 raft_consensus.cc:740] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:59.219357 16927 consensus_queue.cc:260] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.220645 16927 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.225395 17073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:59.226152 17073 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.229166 16927 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.230513 16965 leader_election.cc:304] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 0e10c2ade849401e8ba5d8141abea6ec; no voters: 
I20260709 12:19:59.231457 17282 raft_consensus.cc:2804] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:59.233517 17282 raft_consensus.cc:697] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:59.234495 17073 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.234727 17282 consensus_queue.cc:237] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.235396 17002 leader_election.cc:290] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:59.247258 16822 catalog_manager.cc:5697] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 2 to 3, leader changed from 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:59.265200 17280 raft_consensus.cc:993] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: : Instructing follower 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.265794 17283 raft_consensus.cc:1081] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Signalling peer 012c695806b6424dad30e97211e4fcc6 to start an election
I20260709 12:19:59.267809 17002 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
 from {username='slave'} at 127.0.0.1:49296
I20260709 12:19:59.268603 17002 raft_consensus.cc:493] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:59.269061 17002 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.279196 17002 raft_consensus.cc:515] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.281980 17002 leader_election.cc:290] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:19:59.283090 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:19:59.283095 16927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "012c695806b6424dad30e97211e4fcc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:19:59.283896 17076 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.283907 16927 raft_consensus.cc:3055] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:59.285640 16927 raft_consensus.cc:740] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:19:59.286878 16927 consensus_queue.cc:260] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.288345 16927 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:59.292317 17076 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.293689 16967 leader_election.cc:304] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012c695806b6424dad30e97211e4fcc6, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:19:59.294674 17282 raft_consensus.cc:2804] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:59.295341 17282 raft_consensus.cc:697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Leader. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:19:59.296501 17282 consensus_queue.cc:237] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:19:59.297508 16927 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012c695806b6424dad30e97211e4fcc6 in term 3.
I20260709 12:19:59.309312 16822 catalog_manager.cc:5697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 reported cstate change: term changed from 2 to 3, leader changed from 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1) to 012c695806b6424dad30e97211e4fcc6 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "012c695806b6424dad30e97211e4fcc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:59.513489 16927 raft_consensus.cc:1275] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.516052 17282 consensus_queue.cc:1048] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:59.548332 17076 raft_consensus.cc:1275] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.551656 17282 consensus_queue.cc:1048] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:59.593096 16927 raft_consensus.cc:1275] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.594617 17282 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:59.611400 17076 raft_consensus.cc:1275] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.613183 17282 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:59.640278 17076 raft_consensus.cc:1275] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.641225 16927 raft_consensus.cc:1275] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.641980 17282 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:59.643183 17294 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:59.780777 17076 raft_consensus.cc:1275] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.782372 17282 consensus_queue.cc:1048] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:59.794157 16927 raft_consensus.cc:1275] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.796062 17294 consensus_queue.cc:1048] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:59.893612 16927 raft_consensus.cc:1275] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.894996 17294 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:59.909459 17076 raft_consensus.cc:1275] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 012c695806b6424dad30e97211e4fcc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:59.911736 17294 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:02.070858 16792 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:20:02.071856 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:02.080698 17002 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:57112
I20260709 12:20:02.081633 17002 raft_consensus.cc:606] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Received request to transfer leadership to TS 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:20:02.084821 17002 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:57112
I20260709 12:20:02.085362 17002 raft_consensus.cc:606] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Received request to transfer leadership to TS 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:20:02.088083 17002 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:57112
I20260709 12:20:02.088613 17002 raft_consensus.cc:606] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Received request to transfer leadership to TS 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:20:02.091207 17002 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:57112
I20260709 12:20:02.091748 17002 raft_consensus.cc:606] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Received request to transfer leadership to TS 0e10c2ade849401e8ba5d8141abea6ec
I20260709 12:20:02.094436 17002 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32"
dest_uuid: "012c695806b6424dad30e97211e4fcc6"
mode: GRACEFUL
new_leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:57112
I20260709 12:20:02.094940 17002 raft_consensus.cc:606] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Received request to transfer leadership to TS 849bbec7c1154b2d9b70c1709ebfa534
I20260709 12:20:02.096101 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:02.096576 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:02.128043 17282 raft_consensus.cc:993] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:20:02.128754 17294 raft_consensus.cc:1081] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Signalling peer 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:20:02.130913 16926 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:02.131544 16926 raft_consensus.cc:493] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:02.132028 16926 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.139006 16926 raft_consensus.cc:515] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.141498 16926 leader_election.cc:290] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 4 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:20:02.142984 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:20:02.142935 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f135dc093fd4d3d93612ace8d903bc2" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:20:02.143764 17076 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.143888 17002 raft_consensus.cc:3055] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:02.144284 17002 raft_consensus.cc:740] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:20:02.145042 17002 consensus_queue.cc:260] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.146117 17002 raft_consensus.cc:3060] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.149820 17076 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 4.
I20260709 12:20:02.151476 16892 leader_election.cc:304] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:20:02.152225 17002 raft_consensus.cc:2468] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 4.
I20260709 12:20:02.153502 17315 raft_consensus.cc:2804] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:02.154806 17315 raft_consensus.cc:697] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:20:02.156163 17315 consensus_queue.cc:237] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.165025 16822 catalog_manager.cc:5697] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 3 to 4, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 4 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:02.231043 17282 raft_consensus.cc:993] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:20:02.231581 17294 raft_consensus.cc:1081] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Signalling peer 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:20:02.233417 17076 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:47794
I20260709 12:20:02.234114 17076 raft_consensus.cc:493] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:02.234398 17076 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.240543 17076 raft_consensus.cc:515] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.242929 17076 leader_election.cc:290] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 4 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:20:02.244153 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:20:02.244372 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "468a3e5634ee46fbbd8f8b4df97ce770" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:20:02.244871 17002 raft_consensus.cc:3055] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:02.244871 16926 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.245368 17002 raft_consensus.cc:740] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:20:02.246205 17002 consensus_queue.cc:260] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.247478 17002 raft_consensus.cc:3060] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.253583 16926 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 4.
I20260709 12:20:02.255628 17002 raft_consensus.cc:2468] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 4.
I20260709 12:20:02.255628 17039 leader_election.cc:304] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:20:02.256940 17313 raft_consensus.cc:2804] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:02.257550 17313 raft_consensus.cc:697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:20:02.258522 17313 consensus_queue.cc:237] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.267593 16822 catalog_manager.cc:5697] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 3 to 4, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 4 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.379503 17282 raft_consensus.cc:993] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:20:02.380021 17294 raft_consensus.cc:1081] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Signalling peer 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:20:02.381983 16926 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:02.382593 16926 raft_consensus.cc:493] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:02.382926 16926 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:02.388981 16926 raft_consensus.cc:515] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.391016 16926 leader_election.cc:290] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 3 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:20:02.392100 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:20:02.392100 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46b0e485bf754c3a92586c68f25e4c1a" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:20:02.392843 17076 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:02.392970 17002 raft_consensus.cc:3055] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:02.393352 17002 raft_consensus.cc:740] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:20:02.394115 17002 consensus_queue.cc:260] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.395087 17002 raft_consensus.cc:3060] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:02.398675 17076 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 3.
I20260709 12:20:02.399889 16892 leader_election.cc:304] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:20:02.400676 17002 raft_consensus.cc:2468] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 3.
I20260709 12:20:02.400698 17315 raft_consensus.cc:2804] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:02.401468 17315 raft_consensus.cc:697] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 3 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:20:02.402588 17315 consensus_queue.cc:237] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.412458 16822 catalog_manager.cc:5697] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 2 to 3, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 3 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:02.458643 17282 raft_consensus.cc:993] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:20:02.459201 17282 raft_consensus.cc:1081] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Signalling peer 849bbec7c1154b2d9b70c1709ebfa534 to start an election
I20260709 12:20:02.460773 17076 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32"
dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
 from {username='slave'} at 127.0.0.1:47794
I20260709 12:20:02.461333 17076 raft_consensus.cc:493] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:02.461627 17076 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.467370 17076 raft_consensus.cc:515] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.469656 17076 leader_election.cc:290] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 4 election: Requested vote from peers 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527), 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333)
I20260709 12:20:02.470770 16926 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
I20260709 12:20:02.470831 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94077101e4f74b77bcd819cd3f835b32" candidate_uuid: "849bbec7c1154b2d9b70c1709ebfa534" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:20:02.471542 17002 raft_consensus.cc:3055] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:02.471513 16926 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.472013 17002 raft_consensus.cc:740] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:20:02.472882 17002 consensus_queue.cc:260] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.474359 17002 raft_consensus.cc:3060] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:02.480125 16926 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 4.
I20260709 12:20:02.481613 17039 leader_election.cc:304] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:20:02.482008 17002 raft_consensus.cc:2468] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849bbec7c1154b2d9b70c1709ebfa534 in term 4.
I20260709 12:20:02.482550 17313 raft_consensus.cc:2804] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:02.483119 17313 raft_consensus.cc:697] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 LEADER]: Becoming Leader. State: Replica: 849bbec7c1154b2d9b70c1709ebfa534, State: Running, Role: LEADER
I20260709 12:20:02.484163 17313 consensus_queue.cc:237] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.494277 16822 catalog_manager.cc:5697] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 reported cstate change: term changed from 3 to 4, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3). New cstate: current_term: 4 leader_uuid: "849bbec7c1154b2d9b70c1709ebfa534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:02.576860 17294 raft_consensus.cc:993] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: : Instructing follower 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:20:02.577445 17282 raft_consensus.cc:1081] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Signalling peer 0e10c2ade849401e8ba5d8141abea6ec to start an election
I20260709 12:20:02.579305 16926 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea"
dest_uuid: "0e10c2ade849401e8ba5d8141abea6ec"
 from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:02.579845 16926 raft_consensus.cc:493] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:02.580158 16926 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:02.585750 16926 raft_consensus.cc:515] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.587630 16926 leader_election.cc:290] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 3 election: Requested vote from peers 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333), 849bbec7c1154b2d9b70c1709ebfa534 (127.16.102.3:37359)
I20260709 12:20:02.588802 17076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "849bbec7c1154b2d9b70c1709ebfa534"
I20260709 12:20:02.588802 17002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "748d2378aa78450ba4cef89d9c643cea" candidate_uuid: "0e10c2ade849401e8ba5d8141abea6ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "012c695806b6424dad30e97211e4fcc6"
I20260709 12:20:02.589522 17076 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:02.589558 17002 raft_consensus.cc:3055] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:02.590029 17002 raft_consensus.cc:740] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 012c695806b6424dad30e97211e4fcc6, State: Running, Role: LEADER
I20260709 12:20:02.590814 17002 consensus_queue.cc:260] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.591820 17002 raft_consensus.cc:3060] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:02.595871 17076 raft_consensus.cc:2468] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 3.
I20260709 12:20:02.597172 16892 leader_election.cc:304] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e10c2ade849401e8ba5d8141abea6ec, 849bbec7c1154b2d9b70c1709ebfa534; no voters: 
I20260709 12:20:02.598032 17315 raft_consensus.cc:2804] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:02.598068 17002 raft_consensus.cc:2468] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e10c2ade849401e8ba5d8141abea6ec in term 3.
I20260709 12:20:02.598567 17315 raft_consensus.cc:697] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 3 LEADER]: Becoming Leader. State: Replica: 0e10c2ade849401e8ba5d8141abea6ec, State: Running, Role: LEADER
I20260709 12:20:02.599529 17315 consensus_queue.cc:237] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } }
I20260709 12:20:02.607333 16822 catalog_manager.cc:5697] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec reported cstate change: term changed from 2 to 3, leader changed from 012c695806b6424dad30e97211e4fcc6 (127.16.102.2) to 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1). New cstate: current_term: 3 leader_uuid: "0e10c2ade849401e8ba5d8141abea6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:02.700708 17002 raft_consensus.cc:1275] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:02.702495 17315 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:02.711028 17076 raft_consensus.cc:1275] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:02.717243 17316 consensus_queue.cc:1048] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:02.817595 17002 raft_consensus.cc:1275] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:02.819016 17313 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:02.829936 16926 raft_consensus.cc:1275] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:02.832780 17332 consensus_queue.cc:1048] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:02.898312 17076 raft_consensus.cc:1275] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:02.899768 17316 consensus_queue.cc:1048] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:02.911288 17002 raft_consensus.cc:1275] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:02.913064 17315 consensus_queue.cc:1048] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:03.037484 16926 raft_consensus.cc:1275] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:03.039110 17314 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e10c2ade849401e8ba5d8141abea6ec" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:03.049348 17002 raft_consensus.cc:1275] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Refusing update from remote peer 849bbec7c1154b2d9b70c1709ebfa534: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:03.052045 17314 consensus_queue.cc:1048] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:03.057451 17000 raft_consensus.cc:1275] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:03.059155 17315 consensus_queue.cc:1048] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "012c695806b6424dad30e97211e4fcc6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:03.072348 17076 raft_consensus.cc:1275] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Refusing update from remote peer 0e10c2ade849401e8ba5d8141abea6ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:03.074608 17335 consensus_queue.cc:1048] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bbec7c1154b2d9b70c1709ebfa534" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:03.097890 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:03.103350 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:03.103919 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:04.105180 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:04.110916 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:04.111480 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:05.112847 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:05.118947 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:05.119544 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:06.121016 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:06.128108 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:06.128667 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:07.130038 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:07.135541 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:07.136101 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:08.137419 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:08.142841 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:08.143425 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:09.144562 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:09.150467 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:09.151012 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:10.152803 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:10.160674 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:10.161291 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:11.162773 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:11.168562 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:11.169154 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:12.203157 16792 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:20:12.203883 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:20:12.241284 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:12.242435 16792 tablet_replica.cc:333] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.243340 16792 raft_consensus.cc:2243] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.244124 16792 raft_consensus.cc:2272] T 468a3e5634ee46fbbd8f8b4df97ce770 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.247195 16792 tablet_replica.cc:333] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.248018 16792 raft_consensus.cc:2243] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.248600 16792 raft_consensus.cc:2272] T 94077101e4f74b77bcd819cd3f835b32 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.251257 16792 tablet_replica.cc:333] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.251816 16792 raft_consensus.cc:2243] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.252858 16792 raft_consensus.cc:2272] T 46b0e485bf754c3a92586c68f25e4c1a P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.255026 16792 tablet_replica.cc:333] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.255656 16792 raft_consensus.cc:2243] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.256659 16792 raft_consensus.cc:2272] T 748d2378aa78450ba4cef89d9c643cea P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.258881 16792 tablet_replica.cc:333] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.259465 16792 raft_consensus.cc:2243] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.259974 16792 raft_consensus.cc:2272] T 24b40b9dd1114b498e4dcc91c97be5b2 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.261979 16792 tablet_replica.cc:333] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.262638 16792 raft_consensus.cc:2243] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.263564 16792 raft_consensus.cc:2272] T 7f135dc093fd4d3d93612ace8d903bc2 P 0e10c2ade849401e8ba5d8141abea6ec [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.265563 16792 tablet_replica.cc:333] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.266188 16792 raft_consensus.cc:2243] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.266661 16792 raft_consensus.cc:2272] T 41e4984c9b7f4173b47582e4e794e607 P 0e10c2ade849401e8ba5d8141abea6ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.268428 16792 tablet_replica.cc:333] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.269021 16792 raft_consensus.cc:2243] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.269490 16792 raft_consensus.cc:2272] T 52b372c5d74648458c4589b25e07b070 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.271330 16792 tablet_replica.cc:333] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec: stopping tablet replica
I20260709 12:20:12.271840 16792 raft_consensus.cc:2243] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.272320 16792 raft_consensus.cc:2272] T ecc62f04a8d84076896751bcf7de0727 P 0e10c2ade849401e8ba5d8141abea6ec [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:12.443853 17039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35527: connect: Connection refused (error 111)
W20260709 12:20:12.449141 17039 consensus_peers.cc:597] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 -> Peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Couldn't send request to peer 0e10c2ade849401e8ba5d8141abea6ec. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:12.450124 17039 consensus_peers.cc:597] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 -> Peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Couldn't send request to peer 0e10c2ade849401e8ba5d8141abea6ec. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:12.451547 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
W20260709 12:20:12.453411 16965 consensus_peers.cc:597] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 -> Peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Couldn't send request to peer 0e10c2ade849401e8ba5d8141abea6ec. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:12.454106 16965 consensus_peers.cc:597] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 -> Peer 0e10c2ade849401e8ba5d8141abea6ec (127.16.102.1:35527): Couldn't send request to peer 0e10c2ade849401e8ba5d8141abea6ec. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:12.477236 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
W20260709 12:20:12.484949 17039 consensus_peers.cc:597] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 -> Peer 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333): Couldn't send request to peer 012c695806b6424dad30e97211e4fcc6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:12.511376 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:12.512071 16792 tablet_replica.cc:333] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.512810 16792 raft_consensus.cc:2243] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.513422 16792 raft_consensus.cc:2272] T 41e4984c9b7f4173b47582e4e794e607 P 012c695806b6424dad30e97211e4fcc6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.515836 16792 tablet_replica.cc:333] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.516515 16792 raft_consensus.cc:2243] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.517464 16792 raft_consensus.cc:2272] T 52b372c5d74648458c4589b25e07b070 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.519974 16792 tablet_replica.cc:333] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.520645 16792 raft_consensus.cc:2243] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.521113 16792 raft_consensus.cc:2272] T 94077101e4f74b77bcd819cd3f835b32 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.523027 16792 tablet_replica.cc:333] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.523525 16792 raft_consensus.cc:2243] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.524333 16792 raft_consensus.cc:2272] T 24b40b9dd1114b498e4dcc91c97be5b2 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.526454 16792 tablet_replica.cc:333] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.526975 16792 raft_consensus.cc:2243] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.527405 16792 raft_consensus.cc:2272] T 7f135dc093fd4d3d93612ace8d903bc2 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.529230 16792 tablet_replica.cc:333] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.529723 16792 raft_consensus.cc:2243] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.530630 16792 raft_consensus.cc:2272] T ecc62f04a8d84076896751bcf7de0727 P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.532608 16792 tablet_replica.cc:333] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.533130 16792 raft_consensus.cc:2243] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.533622 16792 raft_consensus.cc:2272] T 468a3e5634ee46fbbd8f8b4df97ce770 P 012c695806b6424dad30e97211e4fcc6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.535543 16792 tablet_replica.cc:333] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.536026 16792 raft_consensus.cc:2243] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.536449 16792 raft_consensus.cc:2272] T 748d2378aa78450ba4cef89d9c643cea P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.538309 16792 tablet_replica.cc:333] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6: stopping tablet replica
I20260709 12:20:12.538775 16792 raft_consensus.cc:2243] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.539194 16792 raft_consensus.cc:2272] T 46b0e485bf754c3a92586c68f25e4c1a P 012c695806b6424dad30e97211e4fcc6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.571975 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
I20260709 12:20:12.593559 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
I20260709 12:20:12.627092 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:12.627871 16792 tablet_replica.cc:333] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.628615 16792 raft_consensus.cc:2243] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.629231 16792 raft_consensus.cc:2272] T ecc62f04a8d84076896751bcf7de0727 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.631567 16792 tablet_replica.cc:333] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.632164 16792 raft_consensus.cc:2243] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.632678 16792 raft_consensus.cc:2272] T 52b372c5d74648458c4589b25e07b070 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.634749 16792 tablet_replica.cc:333] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.635273 16792 raft_consensus.cc:2243] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.636165 16792 raft_consensus.cc:2272] T 41e4984c9b7f4173b47582e4e794e607 P 849bbec7c1154b2d9b70c1709ebfa534 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.638258 16792 tablet_replica.cc:333] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.638785 16792 raft_consensus.cc:2243] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.639209 16792 raft_consensus.cc:2272] T 24b40b9dd1114b498e4dcc91c97be5b2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.641165 16792 tablet_replica.cc:333] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.641741 16792 raft_consensus.cc:2243] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.642403 16792 raft_consensus.cc:2272] T 7f135dc093fd4d3d93612ace8d903bc2 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.645048 16792 tablet_replica.cc:333] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.645797 16792 raft_consensus.cc:2243] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.647090 16792 raft_consensus.cc:2272] T 468a3e5634ee46fbbd8f8b4df97ce770 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:12.648236 17039 consensus_peers.cc:597] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 -> Peer 012c695806b6424dad30e97211e4fcc6 (127.16.102.2:38333): Couldn't send request to peer 012c695806b6424dad30e97211e4fcc6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:38333: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:12.649523 16792 tablet_replica.cc:333] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.650163 16792 raft_consensus.cc:2243] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.651015 16792 raft_consensus.cc:2272] T 46b0e485bf754c3a92586c68f25e4c1a P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.653055 16792 tablet_replica.cc:333] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.653625 16792 raft_consensus.cc:2243] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:12.654152 16792 raft_consensus.cc:2272] T 748d2378aa78450ba4cef89d9c643cea P 849bbec7c1154b2d9b70c1709ebfa534 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.656096 16792 tablet_replica.cc:333] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534: stopping tablet replica
I20260709 12:20:12.656569 16792 raft_consensus.cc:2243] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.657518 16792 raft_consensus.cc:2272] T 94077101e4f74b77bcd819cd3f835b32 P 849bbec7c1154b2d9b70c1709ebfa534 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.707338 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:20:12.729591 16792 master.cc:562] Master@127.16.102.62:36359 shutting down...
I20260709 12:20:12.884799 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:12.885674 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.886147 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P ef63d7d07c374f7da87449fef4fc6585: stopping tablet replica
I20260709 12:20:12.908020 16792 master.cc:584] Master@127.16.102.62:36359 shutdown complete.
012c695806b6424dad30e97211e4fcc6  4
0e10c2ade849401e8ba5d8141abea6ec  4
849bbec7c1154b2d9b70c1709ebfa534  1
012c695806b6424dad30e97211e4fcc6  3
0e10c2ade849401e8ba5d8141abea6ec  3
849bbec7c1154b2d9b70c1709ebfa534  3
012c695806b6424dad30e97211e4fcc6  8
0e10c2ade849401e8ba5d8141abea6ec  0
849bbec7c1154b2d9b70c1709ebfa534  1
012c695806b6424dad30e97211e4fcc6  3
0e10c2ade849401e8ba5d8141abea6ec  3
849bbec7c1154b2d9b70c1709ebfa534  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (33612 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:20:12.957562 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:35967
I20260709 12:20:12.959000 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:12.965296 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:12.966117 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:12.967626 16792 server_base.cc:1061] running on GCE node
W20260709 12:20:12.967479 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:12.969190 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:12.969409 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:12.969545 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599612969526 us; error 0 us; skew 500 ppm
I20260709 12:20:12.970229 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:12.973481 16792 webserver.cc:533] Webserver started at http://127.16.102.62:44057/ using document root <none> and password file <none>
I20260709 12:20:12.974313 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:12.974551 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:12.974814 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:12.976207 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "229dc4b0a15d4803972a0f8e85d7e1f9"
format_stamp: "Formatted at 2026-07-09 12:20:12 on dist-test-slave-7xt3"
I20260709 12:20:12.981884 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:20:12.986244 17366 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:12.987214 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:20:12.987532 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "229dc4b0a15d4803972a0f8e85d7e1f9"
format_stamp: "Formatted at 2026-07-09 12:20:12 on dist-test-slave-7xt3"
I20260709 12:20:12.987977 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:13.002157 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:13.003612 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:13.052724 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:35967
I20260709 12:20:13.052846 17417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:35967 every 8 connection(s)
I20260709 12:20:13.057976 17418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.069808 17418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9: Bootstrap starting.
I20260709 12:20:13.075492 17418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:13.080798 17418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9: No bootstrap required, opened a new log
I20260709 12:20:13.083667 17418 raft_consensus.cc:359] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" member_type: VOTER }
I20260709 12:20:13.084194 17418 raft_consensus.cc:385] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:13.084489 17418 raft_consensus.cc:740] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229dc4b0a15d4803972a0f8e85d7e1f9, State: Initialized, Role: FOLLOWER
I20260709 12:20:13.085209 17418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" member_type: VOTER }
I20260709 12:20:13.085754 17418 raft_consensus.cc:399] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:13.086033 17418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:13.086376 17418 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:13.092458 17418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" member_type: VOTER }
I20260709 12:20:13.093492 17418 leader_election.cc:304] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229dc4b0a15d4803972a0f8e85d7e1f9; no voters: 
I20260709 12:20:13.095257 17418 leader_election.cc:290] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:13.095762 17421 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:13.097388 17421 raft_consensus.cc:697] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 1 LEADER]: Becoming Leader. State: Replica: 229dc4b0a15d4803972a0f8e85d7e1f9, State: Running, Role: LEADER
I20260709 12:20:13.098297 17421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" member_type: VOTER }
I20260709 12:20:13.099153 17418 sys_catalog.cc:565] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:13.104919 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229dc4b0a15d4803972a0f8e85d7e1f9. Latest consensus state: current_term: 1 leader_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" member_type: VOTER } }
I20260709 12:20:13.105345 17422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229dc4b0a15d4803972a0f8e85d7e1f9" member_type: VOTER } }
I20260709 12:20:13.105914 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:13.105943 17422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:13.111781 17430 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:13.121408 17430 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:13.122742 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:13.133700 17430 catalog_manager.cc:1383] Generated new cluster ID: 4e18b77db1e7476189ab8dda000a9a77
I20260709 12:20:13.134126 17430 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:13.154268 17430 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:13.156149 17430 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:13.179900 17430 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9: Generated new TSK 0
I20260709 12:20:13.180704 17430 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:13.190178 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:13.198139 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.198711 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.200201 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:13.201697 16792 server_base.cc:1061] running on GCE node
I20260709 12:20:13.202919 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:13.203147 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:13.203269 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599613203256 us; error 0 us; skew 500 ppm
I20260709 12:20:13.203811 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:13.207125 16792 webserver.cc:533] Webserver started at http://127.16.102.1:42421/ using document root <none> and password file <none>
I20260709 12:20:13.207719 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:13.207904 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:13.208137 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:13.209472 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-7xt3"
I20260709 12:20:13.215085 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20260709 12:20:13.219115 17447 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.220191 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:20:13.220598 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-7xt3"
I20260709 12:20:13.220983 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:13.237135 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:13.238945 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:13.240780 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:13.244724 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:13.245028 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.245376 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:13.245602 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.297397 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:43577
I20260709 12:20:13.297503 17509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:43577 every 8 connection(s)
I20260709 12:20:13.303563 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:13.312362 17514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.314051 17515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.317297 17517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:13.317448 16792 server_base.cc:1061] running on GCE node
I20260709 12:20:13.318971 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:13.319289 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:13.319522 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599613319498 us; error 0 us; skew 500 ppm
I20260709 12:20:13.320165 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:13.323890 17510 heartbeater.cc:344] Connected to a master server at 127.16.102.62:35967
I20260709 12:20:13.324362 17510 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:13.325222 17510 heartbeater.cc:507] Master 127.16.102.62:35967 requested a full tablet report, sending...
I20260709 12:20:13.325394 16792 webserver.cc:533] Webserver started at http://127.16.102.2:45183/ using document root <none> and password file <none>
I20260709 12:20:13.326251 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:13.326582 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:13.327279 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:13.328359 17383 ts_manager.cc:194] Registered new tserver with Master: 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577)
I20260709 12:20:13.329051 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "33dcc8fb352e4344bee7055ce74134b4"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-7xt3"
I20260709 12:20:13.330850 17383 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59700
I20260709 12:20:13.335013 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.003s
I20260709 12:20:13.339459 17522 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.340723 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:20:13.341187 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "33dcc8fb352e4344bee7055ce74134b4"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-7xt3"
I20260709 12:20:13.341538 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:13.359933 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:13.361536 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:13.363485 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:13.366276 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:13.366570 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.366856 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:13.367049 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.417807 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:36453
I20260709 12:20:13.418094 17584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:36453 every 8 connection(s)
I20260709 12:20:13.423358 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:13.433419 17588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.433667 17589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.437914 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:13.438292 16792 server_base.cc:1061] running on GCE node
I20260709 12:20:13.439231 17585 heartbeater.cc:344] Connected to a master server at 127.16.102.62:35967
I20260709 12:20:13.439615 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:20:13.439682 17585 heartbeater.cc:461] Registering TS with master...
W20260709 12:20:13.439977 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:13.440227 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599613440208 us; error 0 us; skew 500 ppm
I20260709 12:20:13.440888 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:13.441020 17585 heartbeater.cc:507] Master 127.16.102.62:35967 requested a full tablet report, sending...
I20260709 12:20:13.443900 17383 ts_manager.cc:194] Registered new tserver with Master: 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:13.444619 16792 webserver.cc:533] Webserver started at http://127.16.102.3:39649/ using document root <none> and password file <none>
I20260709 12:20:13.445281 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:13.445554 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:13.445941 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:13.445887 17383 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59702
I20260709 12:20:13.447820 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-7xt3"
I20260709 12:20:13.453197 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:20:13.457479 17596 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.458618 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:20:13.459052 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-7xt3"
I20260709 12:20:13.459436 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:13.471895 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:13.473476 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:13.475360 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:13.478066 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:13.478338 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.478619 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:13.478811 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.527904 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:43895
I20260709 12:20:13.527999 17659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:43895 every 8 connection(s)
I20260709 12:20:13.545373 17660 heartbeater.cc:344] Connected to a master server at 127.16.102.62:35967
I20260709 12:20:13.545804 17660 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:13.546859 17660 heartbeater.cc:507] Master 127.16.102.62:35967 requested a full tablet report, sending...
I20260709 12:20:13.549213 17383 ts_manager.cc:194] Registered new tserver with Master: b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:13.549749 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018083329s
I20260709 12:20:13.550889 17383 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53398
I20260709 12:20:13.582144 17383 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53402:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:20:13.586020 17383 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:20:13.684207 17474 tablet_service.cc:1511] Processing CreateTablet for tablet bbd3acbbf5dc43338e386061158ebe56 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:20:13.684526 17475 tablet_service.cc:1511] Processing CreateTablet for tablet cc6f3f060b6e401cbcadf965f547df10 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:20:13.686178 17475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6f3f060b6e401cbcadf965f547df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.687050 17472 tablet_service.cc:1511] Processing CreateTablet for tablet a993238f88674f6898cddd4975026ca8 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:20:13.688683 17624 tablet_service.cc:1511] Processing CreateTablet for tablet bbd3acbbf5dc43338e386061158ebe56 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:20:13.689338 17622 tablet_service.cc:1511] Processing CreateTablet for tablet a993238f88674f6898cddd4975026ca8 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:20:13.689525 17473 tablet_service.cc:1511] Processing CreateTablet for tablet 5eec864561e648359d9f9b828abe58cb (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:20:13.689338 17625 tablet_service.cc:1511] Processing CreateTablet for tablet cc6f3f060b6e401cbcadf965f547df10 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:20:13.690419 17624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd3acbbf5dc43338e386061158ebe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.687314 17474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd3acbbf5dc43338e386061158ebe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.692116 17623 tablet_service.cc:1511] Processing CreateTablet for tablet 5eec864561e648359d9f9b828abe58cb (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:20:13.692149 17625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6f3f060b6e401cbcadf965f547df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.695086 17472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a993238f88674f6898cddd4975026ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.698604 17471 tablet_service.cc:1511] Processing CreateTablet for tablet 075c63c2c56446948478c94e194a13cf (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:20:13.700205 17471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075c63c2c56446948478c94e194a13cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.702275 17623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eec864561e648359d9f9b828abe58cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.702078 17621 tablet_service.cc:1511] Processing CreateTablet for tablet 075c63c2c56446948478c94e194a13cf (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:20:13.704890 17549 tablet_service.cc:1511] Processing CreateTablet for tablet bbd3acbbf5dc43338e386061158ebe56 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:20:13.705295 17550 tablet_service.cc:1511] Processing CreateTablet for tablet cc6f3f060b6e401cbcadf965f547df10 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:20:13.706619 17549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd3acbbf5dc43338e386061158ebe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.708002 17547 tablet_service.cc:1511] Processing CreateTablet for tablet a993238f88674f6898cddd4975026ca8 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:20:13.709712 17547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a993238f88674f6898cddd4975026ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.710520 17621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075c63c2c56446948478c94e194a13cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.713038 17470 tablet_service.cc:1511] Processing CreateTablet for tablet 8ec98de19e3848ba8368ee133aa41b57 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:20:13.713047 17548 tablet_service.cc:1511] Processing CreateTablet for tablet 5eec864561e648359d9f9b828abe58cb (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:20:13.715497 17548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eec864561e648359d9f9b828abe58cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.715327 17546 tablet_service.cc:1511] Processing CreateTablet for tablet 075c63c2c56446948478c94e194a13cf (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:20:13.716837 17546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075c63c2c56446948478c94e194a13cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.718642 17470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec98de19e3848ba8368ee133aa41b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.719630 17545 tablet_service.cc:1511] Processing CreateTablet for tablet 8ec98de19e3848ba8368ee133aa41b57 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:20:13.722302 17473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eec864561e648359d9f9b828abe58cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.723013 17620 tablet_service.cc:1511] Processing CreateTablet for tablet 8ec98de19e3848ba8368ee133aa41b57 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:20:13.724532 17620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec98de19e3848ba8368ee133aa41b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.727837 17469 tablet_service.cc:1511] Processing CreateTablet for tablet eef99b25d1b748188b63a97e4a2b734f (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:20:13.729498 17469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef99b25d1b748188b63a97e4a2b734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.730248 17545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec98de19e3848ba8368ee133aa41b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.733389 17550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6f3f060b6e401cbcadf965f547df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.739151 17619 tablet_service.cc:1511] Processing CreateTablet for tablet eef99b25d1b748188b63a97e4a2b734f (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:20:13.741067 17619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef99b25d1b748188b63a97e4a2b734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.742189 17622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a993238f88674f6898cddd4975026ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.743950 17544 tablet_service.cc:1511] Processing CreateTablet for tablet eef99b25d1b748188b63a97e4a2b734f (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:20:13.745618 17544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef99b25d1b748188b63a97e4a2b734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.747900 17468 tablet_service.cc:1511] Processing CreateTablet for tablet 5344382fd65b4249a153f3ee791b0c75 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:20:13.749640 17468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5344382fd65b4249a153f3ee791b0c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.751991 17543 tablet_service.cc:1511] Processing CreateTablet for tablet 5344382fd65b4249a153f3ee791b0c75 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:20:13.756728 17618 tablet_service.cc:1511] Processing CreateTablet for tablet 5344382fd65b4249a153f3ee791b0c75 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:20:13.761214 17618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5344382fd65b4249a153f3ee791b0c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.753682 17543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5344382fd65b4249a153f3ee791b0c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.764990 17542 tablet_service.cc:1511] Processing CreateTablet for tablet d9004ba90bee446faa1d33c66886f446 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:20:13.767270 17542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9004ba90bee446faa1d33c66886f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.767206 17625 tablet_service.cc:1511] Processing CreateTablet for tablet d9004ba90bee446faa1d33c66886f446 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:20:13.768836 17625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9004ba90bee446faa1d33c66886f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.776559 17541 tablet_service.cc:1511] Processing CreateTablet for tablet 37aa0221b88d4bb3a96df77e6c0659ec (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:20:13.778369 17541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37aa0221b88d4bb3a96df77e6c0659ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.783713 17475 tablet_service.cc:1511] Processing CreateTablet for tablet d9004ba90bee446faa1d33c66886f446 (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:20:13.784111 17467 tablet_service.cc:1511] Processing CreateTablet for tablet 37aa0221b88d4bb3a96df77e6c0659ec (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:20:13.785380 17475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9004ba90bee446faa1d33c66886f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.787066 17467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37aa0221b88d4bb3a96df77e6c0659ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.806161 17617 tablet_service.cc:1511] Processing CreateTablet for tablet 37aa0221b88d4bb3a96df77e6c0659ec (DEFAULT_TABLE table=test-workload [id=79d4efff1ad94ac9ab74a701bee61b29]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:20:13.808283 17617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37aa0221b88d4bb3a96df77e6c0659ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.875360 17676 tablet_bootstrap.cc:492] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:13.883142 17676 tablet_bootstrap.cc:654] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:13.934701 17675 tablet_bootstrap.cc:492] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:13.945046 17677 tablet_bootstrap.cc:492] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:13.964561 17676 tablet_bootstrap.cc:492] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:13.965240 17676 ts_tablet_manager.cc:1403] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.090s	user 0.022s	sys 0.030s
I20260709 12:20:13.965379 17675 tablet_bootstrap.cc:654] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:13.966164 17677 tablet_bootstrap.cc:654] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:13.969520 17676 raft_consensus.cc:359] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:13.971160 17676 raft_consensus.cc:385] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:13.971652 17676 raft_consensus.cc:740] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:13.973217 17676 consensus_queue.cc:260] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:13.994864 17676 ts_tablet_manager.cc:1434] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.029s	user 0.013s	sys 0.001s
I20260709 12:20:13.995065 17660 heartbeater.cc:499] Master 127.16.102.62:35967 was elected leader, sending a full tablet report...
I20260709 12:20:13.996145 17675 tablet_bootstrap.cc:492] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:13.996913 17675 ts_tablet_manager.cc:1403] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.063s	user 0.014s	sys 0.008s
I20260709 12:20:13.996147 17677 tablet_bootstrap.cc:492] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:13.999993 17675 raft_consensus.cc:359] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.000955 17677 ts_tablet_manager.cc:1403] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.056s	user 0.010s	sys 0.014s
I20260709 12:20:14.000905 17675 raft_consensus.cc:385] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.001435 17675 raft_consensus.cc:740] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.002326 17675 consensus_queue.cc:260] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:13.996230 17676 tablet_bootstrap.cc:492] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.004892 17677 raft_consensus.cc:359] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.005952 17677 raft_consensus.cc:385] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.006319 17677 raft_consensus.cc:740] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.007277 17677 consensus_queue.cc:260] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.016515 17676 tablet_bootstrap.cc:654] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.018994 17675 ts_tablet_manager.cc:1434] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.022s	user 0.020s	sys 0.000s
I20260709 12:20:14.019007 17510 heartbeater.cc:499] Master 127.16.102.62:35967 was elected leader, sending a full tablet report...
I20260709 12:20:14.020568 17675 tablet_bootstrap.cc:492] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.028496 17675 tablet_bootstrap.cc:654] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:14.032927 17661 tablet.cc:2406] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:14.035748 17677 ts_tablet_manager.cc:1434] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.034s	user 0.012s	sys 0.020s
I20260709 12:20:14.037604 17585 heartbeater.cc:499] Master 127.16.102.62:35967 was elected leader, sending a full tablet report...
I20260709 12:20:14.038285 17677 tablet_bootstrap.cc:492] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.047657 17676 tablet_bootstrap.cc:492] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.048408 17676 ts_tablet_manager.cc:1403] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.053s	user 0.013s	sys 0.010s
I20260709 12:20:14.049396 17677 tablet_bootstrap.cc:654] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.052238 17676 raft_consensus.cc:359] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.053215 17676 raft_consensus.cc:385] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.055235 17676 raft_consensus.cc:740] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.056694 17676 consensus_queue.cc:260] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.068583 17675 tablet_bootstrap.cc:492] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.069252 17675 ts_tablet_manager.cc:1403] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.049s	user 0.016s	sys 0.028s
I20260709 12:20:14.069660 17676 ts_tablet_manager.cc:1434] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.021s	user 0.011s	sys 0.000s
I20260709 12:20:14.070915 17676 tablet_bootstrap.cc:492] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.073176 17675 raft_consensus.cc:359] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.074108 17675 raft_consensus.cc:385] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.074482 17675 raft_consensus.cc:740] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.075368 17675 consensus_queue.cc:260] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.078248 17675 ts_tablet_manager.cc:1434] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.009s	user 0.011s	sys 0.000s
I20260709 12:20:14.079438 17675 tablet_bootstrap.cc:492] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.084444 17677 tablet_bootstrap.cc:492] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.085331 17677 ts_tablet_manager.cc:1403] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.028s
I20260709 12:20:14.087679 17675 tablet_bootstrap.cc:654] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.089584 17677 raft_consensus.cc:359] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.090682 17677 raft_consensus.cc:385] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.091113 17677 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.092483 17677 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.093773 17676 tablet_bootstrap.cc:654] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.096709 17677 ts_tablet_manager.cc:1434] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:20:14.098635 17677 tablet_bootstrap.cc:492] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.105561 17675 tablet_bootstrap.cc:492] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.106405 17675 ts_tablet_manager.cc:1403] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.027s	user 0.017s	sys 0.008s
I20260709 12:20:14.107998 17677 tablet_bootstrap.cc:654] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.110268 17675 raft_consensus.cc:359] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.111280 17675 raft_consensus.cc:385] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.111752 17675 raft_consensus.cc:740] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.112614 17676 tablet_bootstrap.cc:492] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.113255 17676 ts_tablet_manager.cc:1403] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.043s	user 0.009s	sys 0.011s
I20260709 12:20:14.113292 17675 consensus_queue.cc:260] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
W20260709 12:20:14.115064 17431 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:14.116752 17676 raft_consensus.cc:359] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.117583 17676 raft_consensus.cc:385] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.117605 17675 ts_tablet_manager.cc:1434] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:20:14.118076 17676 raft_consensus.cc:740] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.119052 17676 consensus_queue.cc:260] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.120021 17675 tablet_bootstrap.cc:492] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.127404 17677 tablet_bootstrap.cc:492] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.127240 17682 raft_consensus.cc:493] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.128062 17682 raft_consensus.cc:515] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.129942 17676 ts_tablet_manager.cc:1434] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.001s
I20260709 12:20:14.130980 17675 tablet_bootstrap.cc:654] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.131305 17676 tablet_bootstrap.cc:492] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.141016 17676 tablet_bootstrap.cc:654] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.144094 17682 leader_election.cc:290] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.146497 17677 ts_tablet_manager.cc:1403] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.048s	user 0.022s	sys 0.006s
I20260709 12:20:14.148759 17676 tablet_bootstrap.cc:492] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.149430 17676 ts_tablet_manager.cc:1403] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.002s
I20260709 12:20:14.149925 17677 raft_consensus.cc:359] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.154035 17677 raft_consensus.cc:385] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.154814 17677 raft_consensus.cc:740] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.152915 17676 raft_consensus.cc:359] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.156479 17676 raft_consensus.cc:385] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.155946 17677 consensus_queue.cc:260] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.156891 17676 raft_consensus.cc:740] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.158689 17677 ts_tablet_manager.cc:1434] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:20:14.159981 17677 tablet_bootstrap.cc:492] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.158380 17676 consensus_queue.cc:260] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.167920 17677 tablet_bootstrap.cc:654] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.171797 17676 ts_tablet_manager.cc:1434] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.001s
I20260709 12:20:14.190356 17676 tablet_bootstrap.cc:492] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.190913 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.192196 17675 tablet_bootstrap.cc:492] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.192785 17675 ts_tablet_manager.cc:1403] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.073s	user 0.031s	sys 0.031s
W20260709 12:20:14.192945 17451 leader_election.cc:343] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.194425 17677 tablet_bootstrap.cc:492] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.195113 17677 ts_tablet_manager.cc:1403] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.035s	user 0.029s	sys 0.002s
I20260709 12:20:14.196159 17675 raft_consensus.cc:359] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.196985 17675 raft_consensus.cc:385] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.197305 17675 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.197779 17676 tablet_bootstrap.cc:654] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.198325 17675 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.201186 17675 ts_tablet_manager.cc:1434] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:20:14.202337 17675 tablet_bootstrap.cc:492] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.202472 17635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
W20260709 12:20:14.204545 17448 leader_election.cc:343] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.205046 17448 leader_election.cc:304] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.205947 17682 raft_consensus.cc:2749] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.198941 17677 raft_consensus.cc:359] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.206679 17677 raft_consensus.cc:385] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.207101 17677 raft_consensus.cc:740] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.208276 17677 consensus_queue.cc:260] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.210875 17677 ts_tablet_manager.cc:1434] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.004s
I20260709 12:20:14.212173 17677 tablet_bootstrap.cc:492] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.214093 17675 tablet_bootstrap.cc:654] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.220535 17677 tablet_bootstrap.cc:654] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.225554 17676 tablet_bootstrap.cc:492] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.226310 17676 ts_tablet_manager.cc:1403] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.036s	user 0.007s	sys 0.027s
I20260709 12:20:14.229694 17676 raft_consensus.cc:359] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.230672 17676 raft_consensus.cc:385] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.231068 17676 raft_consensus.cc:740] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.232086 17676 consensus_queue.cc:260] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.233233 17682 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.234055 17682 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.238153 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.239209 17635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.239791 17681 raft_consensus.cc:493] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.240401 17681 raft_consensus.cc:515] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
W20260709 12:20:14.241519 17448 leader_election.cc:343] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.242810 17676 ts_tablet_manager.cc:1434] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.002s
I20260709 12:20:14.242691 17682 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
W20260709 12:20:14.246958 17451 leader_election.cc:343] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.247532 17451 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.249486 17682 raft_consensus.cc:2749] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.252846 17676 tablet_bootstrap.cc:492] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.255254 17681 leader_election.cc:290] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.258502 17677 tablet_bootstrap.cc:492] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.259311 17677 ts_tablet_manager.cc:1403] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.047s	user 0.024s	sys 0.016s
I20260709 12:20:14.259228 17683 raft_consensus.cc:493] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.246884 17675 tablet_bootstrap.cc:492] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.259944 17683 raft_consensus.cc:515] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.261047 17675 ts_tablet_manager.cc:1403] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.059s	user 0.017s	sys 0.012s
I20260709 12:20:14.263885 17676 tablet_bootstrap.cc:654] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.266095 17675 raft_consensus.cc:359] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.267169 17675 raft_consensus.cc:385] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.267645 17675 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.267729 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37aa0221b88d4bb3a96df77e6c0659ec" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.269131 17675 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
W20260709 12:20:14.270313 17599 leader_election.cc:343] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.272912 17675 ts_tablet_manager.cc:1434] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20260709 12:20:14.274367 17675 tablet_bootstrap.cc:492] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.275219 17683 leader_election.cc:290] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.308046 17675 tablet_bootstrap.cc:654] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.308784 17677 raft_consensus.cc:359] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.309778 17677 raft_consensus.cc:385] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.310173 17677 raft_consensus.cc:740] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.310699 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.311594 17485 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:14.311165 17677 consensus_queue.cc:260] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.314646 17525 leader_election.cc:304] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:14.316349 17683 raft_consensus.cc:2804] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.316977 17683 raft_consensus.cc:493] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.317474 17683 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.318962 17677 ts_tablet_manager.cc:1434] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.005s
I20260709 12:20:14.320343 17677 tablet_bootstrap.cc:492] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.326499 17676 tablet_bootstrap.cc:492] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.327184 17676 ts_tablet_manager.cc:1403] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.075s	user 0.030s	sys 0.013s
I20260709 12:20:14.330925 17676 raft_consensus.cc:359] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.331818 17676 raft_consensus.cc:385] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.332150 17676 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.332445 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37aa0221b88d4bb3a96df77e6c0659ec" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.333039 17676 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.333478 17560 raft_consensus.cc:2468] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 0.
I20260709 12:20:14.335965 17600 leader_election.cc:304] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:14.336023 17676 ts_tablet_manager.cc:1434] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:14.340390 17681 raft_consensus.cc:2804] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.340875 17681 raft_consensus.cc:493] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.341214 17681 raft_consensus.cc:3060] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.342094 17676 tablet_bootstrap.cc:492] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.349293 17677 tablet_bootstrap.cc:654] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.354826 17683 raft_consensus.cc:515] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.359974 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:14.361066 17485 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.371037 17681 raft_consensus.cc:515] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.373970 17683 leader_election.cc:290] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.374527 17635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.374611 17681 leader_election.cc:290] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.375481 17635 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:14.378326 17635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:14.378774 17485 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
I20260709 12:20:14.379244 17635 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.380615 17681 raft_consensus.cc:493] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.379758 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37aa0221b88d4bb3a96df77e6c0659ec" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.380023 17484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37aa0221b88d4bb3a96df77e6c0659ec" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:14.381597 17560 raft_consensus.cc:3060] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.379221 17675 tablet_bootstrap.cc:492] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.382535 17675 ts_tablet_manager.cc:1403] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.108s	user 0.032s	sys 0.041s
W20260709 12:20:14.382892 17599 leader_election.cc:343] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.381446 17681 raft_consensus.cc:515] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.386858 17484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.386658 17675 raft_consensus.cc:359] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.387555 17675 raft_consensus.cc:385] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.387897 17675 raft_consensus.cc:740] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.380613 17676 tablet_bootstrap.cc:654] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:14.388939 17599 leader_election.cc:343] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.388825 17675 consensus_queue.cc:260] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.392596 17675 ts_tablet_manager.cc:1434] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 12:20:14.393998 17675 tablet_bootstrap.cc:492] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.389922 17525 leader_election.cc:304] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:14.396349 17683 raft_consensus.cc:2804] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.399927 17682 raft_consensus.cc:493] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.400426 17709 raft_consensus.cc:493] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.400614 17682 raft_consensus.cc:515] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.400914 17709 raft_consensus.cc:515] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.403158 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.405308 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.405283 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.406441 17559 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 0.
I20260709 12:20:14.406869 17634 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 0.
I20260709 12:20:14.407568 17709 leader_election.cc:290] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.408563 17448 leader_election.cc:304] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:14.403445 17682 leader_election.cc:290] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.409477 17682 raft_consensus.cc:493] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.410207 17682 raft_consensus.cc:515] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.412437 17710 raft_consensus.cc:2804] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:20:14.412832 17600 leader_election.cc:343] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.412984 17710 raft_consensus.cc:493] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.411046 17675 tablet_bootstrap.cc:654] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.413362 17600 leader_election.cc:304] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2a71e6ebd8442caba3a20dda6936d8e; no voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:14.415086 17682 leader_election.cc:290] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.415412 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.421164 17712 raft_consensus.cc:2749] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.415904 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6f3f060b6e401cbcadf965f547df10" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
W20260709 12:20:14.424784 17451 leader_election.cc:343] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:14.426685 17523 leader_election.cc:343] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.416620 17683 raft_consensus.cc:697] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:14.413379 17710 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.428855 17683 consensus_queue.cc:237] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.425711 17560 raft_consensus.cc:2468] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:14.447958 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6f3f060b6e401cbcadf965f547df10" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.451308 17600 leader_election.cc:304] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:14.452817 17712 raft_consensus.cc:2804] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.433410 17681 leader_election.cc:290] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.458777 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.456920 17635 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
I20260709 12:20:14.467710 17485 raft_consensus.cc:2468] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:14.454280 17634 raft_consensus.cc:2468] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 0.
I20260709 12:20:14.473361 17525 leader_election.cc:304] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.475402 17709 raft_consensus.cc:2804] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.475834 17709 raft_consensus.cc:493] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.476171 17709 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.476979 17710 raft_consensus.cc:515] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.473400 17711 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.478145 17711 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.482275 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
W20260709 12:20:14.484131 17451 leader_election.cc:343] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.485114 17681 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.485489 17709 raft_consensus.cc:515] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.491071 17709 leader_election.cc:290] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.491562 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:14.492413 17485 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.485896 17681 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.493217 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:14.494647 17448 leader_election.cc:304] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:14.500686 17682 raft_consensus.cc:2804] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.501379 17682 raft_consensus.cc:493] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.501948 17682 raft_consensus.cc:3060] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:14.505987 17523 leader_election.cc:343] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.508072 17676 tablet_bootstrap.cc:492] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.508836 17676 ts_tablet_manager.cc:1403] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.167s	user 0.029s	sys 0.053s
I20260709 12:20:14.516209 17677 tablet_bootstrap.cc:492] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.517261 17677 ts_tablet_manager.cc:1403] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.197s	user 0.017s	sys 0.023s
I20260709 12:20:14.519063 17711 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.522045 17484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.523218 17484 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 0.
I20260709 12:20:14.526325 17599 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:14.526463 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.529507 17634 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 0.
I20260709 12:20:14.529703 17712 raft_consensus.cc:697] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:14.513124 17676 raft_consensus.cc:359] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.538237 17714 raft_consensus.cc:2804] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.539222 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.530603 17675 tablet_bootstrap.cc:492] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.539083 17710 leader_election.cc:290] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.557633 17677 raft_consensus.cc:359] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.558696 17677 raft_consensus.cc:385] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.540951 17712 consensus_queue.cc:237] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.559099 17677 raft_consensus.cc:740] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.560129 17675 ts_tablet_manager.cc:1403] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.166s	user 0.018s	sys 0.012s
I20260709 12:20:14.538707 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.560206 17677 consensus_queue.cc:260] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.561305 17560 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.563726 17677 ts_tablet_manager.cc:1434] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.000s
I20260709 12:20:14.565196 17677 tablet_bootstrap.cc:492] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.553164 17714 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:14.556515 17600 leader_election.cc:343] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.532823 17681 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.552830 17676 raft_consensus.cc:385] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.531718 17448 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:14.570204 17714 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.572413 17710 raft_consensus.cc:2804] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.572889 17710 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.573315 17710 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.576333 17675 raft_consensus.cc:359] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.577214 17675 raft_consensus.cc:385] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.577389 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:14.577555 17675 raft_consensus.cc:740] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.578045 17634 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.570745 17383 catalog_manager.cc:5697] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:14.579756 17560 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 1.
I20260709 12:20:14.581241 17451 leader_election.cc:304] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:14.582345 17711 raft_consensus.cc:2804] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.571447 17676 raft_consensus.cc:740] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.584124 17683 raft_consensus.cc:493] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.583755 17676 consensus_queue.cc:260] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.584671 17683 raft_consensus.cc:515] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.586398 17676 ts_tablet_manager.cc:1434] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.077s	user 0.021s	sys 0.008s
I20260709 12:20:14.570621 17682 raft_consensus.cc:515] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.591120 17484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.597748 17676 tablet_bootstrap.cc:492] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
W20260709 12:20:14.598121 17525 leader_election.cc:343] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.592437 17683 leader_election.cc:290] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.592803 17714 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.587056 17485 raft_consensus.cc:2468] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
I20260709 12:20:14.594185 17635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.596133 17634 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 1.
I20260709 12:20:14.602064 17633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6f3f060b6e401cbcadf965f547df10" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:14.602638 17682 leader_election.cc:290] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Requested vote from peers b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.605165 17633 raft_consensus.cc:3060] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.605933 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6f3f060b6e401cbcadf965f547df10" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.606853 17559 raft_consensus.cc:3060] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.605742 17710 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.603749 17711 raft_consensus.cc:697] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:14.578600 17675 consensus_queue.cc:260] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
W20260709 12:20:14.611354 17523 leader_election.cc:343] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.613677 17523 leader_election.cc:304] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4; no voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.614594 17677 tablet_bootstrap.cc:654] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.614706 17683 raft_consensus.cc:2749] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.615145 17525 leader_election.cc:304] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.616138 17683 raft_consensus.cc:2804] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.616669 17683 raft_consensus.cc:697] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:14.616645 17711 consensus_queue.cc:237] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.617956 17683 consensus_queue.cc:237] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.627547 17714 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.630059 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.633220 17676 tablet_bootstrap.cc:654] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.634367 17382 catalog_manager.cc:5697] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 1 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:14.635667 17675 ts_tablet_manager.cc:1434] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.075s	user 0.009s	sys 0.000s
I20260709 12:20:14.636940 17675 tablet_bootstrap.cc:492] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.638566 17710 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
W20260709 12:20:14.638875 17600 leader_election.cc:343] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.643035 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:14.648886 17485 raft_consensus.cc:2393] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b2a71e6ebd8442caba3a20dda6936d8e in current term 1: Already voted for candidate 7677bb01c3ee4e0ca1378309a562a7cc in this term.
I20260709 12:20:14.642719 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.651508 17599 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2a71e6ebd8442caba3a20dda6936d8e; no voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:14.652841 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
W20260709 12:20:14.653426 17451 leader_election.cc:343] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:14.653759 17634 raft_consensus.cc:2393] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7677bb01c3ee4e0ca1378309a562a7cc in current term 1: Already voted for candidate b2a71e6ebd8442caba3a20dda6936d8e in this term.
I20260709 12:20:14.654306 17714 raft_consensus.cc:2749] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.655277 17448 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.656050 17711 raft_consensus.cc:2749] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.658282 17559 raft_consensus.cc:2468] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 1.
I20260709 12:20:14.658389 17633 raft_consensus.cc:2468] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 1.
I20260709 12:20:14.659853 17451 leader_election.cc:304] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:14.661254 17711 raft_consensus.cc:2804] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.663180 17711 raft_consensus.cc:697] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:14.663897 17677 tablet_bootstrap.cc:492] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.664695 17677 ts_tablet_manager.cc:1403] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.100s	user 0.008s	sys 0.006s
I20260709 12:20:14.664229 17711 consensus_queue.cc:237] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } }
I20260709 12:20:14.669556 17677 raft_consensus.cc:359] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.670955 17677 raft_consensus.cc:385] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.671408 17677 raft_consensus.cc:740] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.672052 17675 tablet_bootstrap.cc:654] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.673193 17381 catalog_manager.cc:5697] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 1 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:14.674868 17677 consensus_queue.cc:260] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.690953 17676 tablet_bootstrap.cc:492] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.691491 17676 ts_tablet_manager.cc:1403] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.094s	user 0.015s	sys 0.011s
I20260709 12:20:14.692667 17677 ts_tablet_manager.cc:1434] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.027s	user 0.011s	sys 0.002s
I20260709 12:20:14.696448 17677 tablet_bootstrap.cc:492] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.697163 17676 raft_consensus.cc:359] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.697988 17676 raft_consensus.cc:385] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.698259 17676 raft_consensus.cc:740] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.699110 17676 consensus_queue.cc:260] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.702410 17676 ts_tablet_manager.cc:1434] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20260709 12:20:14.703642 17676 tablet_bootstrap.cc:492] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.710029 17676 tablet_bootstrap.cc:654] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.711506 17675 tablet_bootstrap.cc:492] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.712224 17675 ts_tablet_manager.cc:1403] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.076s	user 0.019s	sys 0.005s
I20260709 12:20:14.713606 17382 catalog_manager.cc:5697] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:14.715161 17675 raft_consensus.cc:359] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.715904 17675 raft_consensus.cc:385] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.716192 17675 raft_consensus.cc:740] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.717172 17675 consensus_queue.cc:260] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.718418 17677 tablet_bootstrap.cc:654] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.719641 17675 ts_tablet_manager.cc:1434] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:14.720683 17675 tablet_bootstrap.cc:492] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.727640 17675 tablet_bootstrap.cc:654] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.730824 17677 tablet_bootstrap.cc:492] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.731508 17677 ts_tablet_manager.cc:1403] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.035s	user 0.012s	sys 0.005s
I20260709 12:20:14.734714 17677 raft_consensus.cc:359] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.735574 17677 raft_consensus.cc:385] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.735910 17677 raft_consensus.cc:740] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.736771 17675 tablet_bootstrap.cc:492] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.736667 17677 consensus_queue.cc:260] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.737311 17675 ts_tablet_manager.cc:1403] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.003s
I20260709 12:20:14.739121 17677 ts_tablet_manager.cc:1434] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:14.738722 17381 catalog_manager.cc:5697] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 1 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:14.740746 17675 raft_consensus.cc:359] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.741508 17675 raft_consensus.cc:385] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.741789 17675 raft_consensus.cc:740] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.742794 17675 consensus_queue.cc:260] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.743323 17676 tablet_bootstrap.cc:492] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.745611 17675 ts_tablet_manager.cc:1434] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:14.746404 17677 tablet_bootstrap.cc:492] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.746817 17675 tablet_bootstrap.cc:492] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc: Bootstrap starting.
I20260709 12:20:14.753695 17677 tablet_bootstrap.cc:654] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.753695 17675 tablet_bootstrap.cc:654] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.754647 17676 ts_tablet_manager.cc:1403] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.051s	user 0.009s	sys 0.014s
I20260709 12:20:14.759451 17676 raft_consensus.cc:359] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.760303 17676 raft_consensus.cc:385] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.760612 17676 raft_consensus.cc:740] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.761585 17676 consensus_queue.cc:260] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.763896 17676 ts_tablet_manager.cc:1434] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:20:14.765100 17676 tablet_bootstrap.cc:492] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: Bootstrap starting.
I20260709 12:20:14.767283 17677 tablet_bootstrap.cc:492] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.767295 17675 tablet_bootstrap.cc:492] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc: No bootstrap required, opened a new log
I20260709 12:20:14.767910 17675 ts_tablet_manager.cc:1403] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.009s
I20260709 12:20:14.767910 17677 ts_tablet_manager.cc:1403] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.002s
I20260709 12:20:14.771159 17677 raft_consensus.cc:359] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.771271 17675 raft_consensus.cc:359] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.772017 17676 tablet_bootstrap.cc:654] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.772089 17675 raft_consensus.cc:385] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.772014 17677 raft_consensus.cc:385] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.772611 17675 raft_consensus.cc:740] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.772673 17677 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.773649 17677 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.773649 17675 consensus_queue.cc:260] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.776455 17677 ts_tablet_manager.cc:1434] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 12:20:14.776499 17675 ts_tablet_manager.cc:1434] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:20:14.777858 17677 tablet_bootstrap.cc:492] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: Bootstrap starting.
I20260709 12:20:14.780566 17676 tablet_bootstrap.cc:492] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: No bootstrap required, opened a new log
I20260709 12:20:14.783303 17676 ts_tablet_manager.cc:1403] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.005s
I20260709 12:20:14.785279 17677 tablet_bootstrap.cc:654] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.786614 17676 raft_consensus.cc:359] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.787468 17676 raft_consensus.cc:385] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.787775 17676 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.788570 17676 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.791461 17676 ts_tablet_manager.cc:1434] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:14.804144 17677 tablet_bootstrap.cc:492] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: No bootstrap required, opened a new log
I20260709 12:20:14.804711 17677 ts_tablet_manager.cc:1403] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: Time spent bootstrapping tablet: real 0.027s	user 0.016s	sys 0.008s
I20260709 12:20:14.806726 17714 raft_consensus.cc:493] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.806912 17677 raft_consensus.cc:359] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.807233 17714 raft_consensus.cc:515] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.807577 17677 raft_consensus.cc:385] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.807839 17677 raft_consensus.cc:740] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.808461 17677 consensus_queue.cc:260] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.811146 17677 ts_tablet_manager.cc:1434] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:20:14.813158 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.813984 17559 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 0.
I20260709 12:20:14.809582 17714 leader_election.cc:290] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.814610 17714 raft_consensus.cc:493] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.815145 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.815081 17714 raft_consensus.cc:515] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.815446 17600 leader_election.cc:304] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:14.815864 17485 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 0.
I20260709 12:20:14.816814 17681 raft_consensus.cc:2804] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.817206 17681 raft_consensus.cc:493] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.817564 17681 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.818711 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.819438 17559 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 0.
I20260709 12:20:14.820254 17714 leader_election.cc:290] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.820843 17600 leader_election.cc:304] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:14.821954 17714 raft_consensus.cc:2804] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:14.822332 17714 raft_consensus.cc:493] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:14.822818 17714 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.824810 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.825625 17485 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 0.
I20260709 12:20:14.828817 17681 raft_consensus.cc:515] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.832397 17484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:14.833215 17484 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.833326 17681 leader_election.cc:290] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.833449 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.833739 17683 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:14.834136 17559 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.832361 17714 raft_consensus.cc:515] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.838950 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:14.839638 17560 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.841590 17484 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:14.843626 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:14.844233 17599 leader_election.cc:304] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:14.844532 17483 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.845273 17681 raft_consensus.cc:2804] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.846123 17681 raft_consensus.cc:697] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:14.847154 17681 consensus_queue.cc:237] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.849141 17560 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:14.851141 17600 leader_election.cc:304] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:14.849666 17714 leader_election.cc:290] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.852258 17712 raft_consensus.cc:2804] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.852831 17712 raft_consensus.cc:697] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:14.854112 17712 consensus_queue.cc:237] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.855005 17559 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:14.858923 17483 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:14.875805 17707 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:14.881116 17381 catalog_manager.cc:5697] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 1 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:14.887709 17709 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.888325 17709 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.890548 17709 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:14.891474 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.892287 17483 raft_consensus.cc:2393] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33dcc8fb352e4344bee7055ce74134b4 in current term 1: Already voted for candidate 7677bb01c3ee4e0ca1378309a562a7cc in this term.
I20260709 12:20:14.893808 17633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:14.894798 17633 raft_consensus.cc:2393] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33dcc8fb352e4344bee7055ce74134b4 in current term 1: Already voted for candidate b2a71e6ebd8442caba3a20dda6936d8e in this term.
I20260709 12:20:14.896028 17523 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4; no voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:14.896858 17709 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.899766 17381 catalog_manager.cc:5697] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 1 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:14.903504 17709 raft_consensus.cc:2749] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.935580 17681 raft_consensus.cc:493] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:14.936244 17681 raft_consensus.cc:515] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:14.938434 17681 leader_election.cc:290] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:14.939566 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:14.939749 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:14.940395 17483 raft_consensus.cc:2393] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2a71e6ebd8442caba3a20dda6936d8e in current term 1: Already voted for candidate 33dcc8fb352e4344bee7055ce74134b4 in this term.
I20260709 12:20:14.941785 17599 leader_election.cc:304] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2a71e6ebd8442caba3a20dda6936d8e; no voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:14.942699 17681 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.949205 17681 raft_consensus.cc:2749] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:14.980127 17681 consensus_queue.cc:1048] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:14.990547 17483 raft_consensus.cc:3060] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.000270 17714 consensus_queue.cc:1048] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:15.028043 17709 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.028697 17709 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:15.030993 17709 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:15.032230 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:15.032084 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:15.032557 17711 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:15.032969 17634 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:15.032970 17483 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:15.034401 17523 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:15.035396 17709 raft_consensus.cc:2804] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.035825 17709 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.036170 17709 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.043741 17709 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:15.047154 17682 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.047771 17709 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:15.048285 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:15.049062 17634 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.054977 17707 raft_consensus.cc:493] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.055670 17707 raft_consensus.cc:515] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:15.058211 17707 leader_election.cc:290] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:15.059146 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:15.059969 17485 raft_consensus.cc:2468] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:15.060469 17633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" is_pre_election: true
I20260709 12:20:15.062050 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:15.062726 17483 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.061242 17633 raft_consensus.cc:2468] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 0.
I20260709 12:20:15.066496 17634 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
I20260709 12:20:15.065907 17523 leader_election.cc:304] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:15.071911 17483 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
I20260709 12:20:15.072762 17709 consensus_queue.cc:1048] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:15.073158 17523 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:15.074065 17683 raft_consensus.cc:2804] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.074160 17707 raft_consensus.cc:2804] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:15.074497 17683 raft_consensus.cc:493] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.074765 17707 raft_consensus.cc:697] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:15.074936 17683 raft_consensus.cc:3060] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.075758 17707 consensus_queue.cc:237] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:15.083160 17709 consensus_queue.cc:1048] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.085531 17683 raft_consensus.cc:515] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:15.090390 17633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:15.091473 17633 raft_consensus.cc:3060] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.092453 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:15.093189 17483 raft_consensus.cc:3060] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.097689 17381 catalog_manager.cc:5697] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:15.102020 17683 leader_election.cc:290] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:15.105273 17633 raft_consensus.cc:2468] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
I20260709 12:20:15.107237 17523 leader_election.cc:304] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:15.108448 17683 raft_consensus.cc:2804] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:15.109025 17683 raft_consensus.cc:697] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:15.110000 17683 consensus_queue.cc:237] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:15.110570 17483 raft_consensus.cc:2468] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 1.
W20260709 12:20:15.122202 17431 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:15.132570 17381 catalog_manager.cc:5697] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:15.155820 17682 consensus_queue.cc:1048] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:15.169606 17682 consensus_queue.cc:1048] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.338668 17714 consensus_queue.cc:1048] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:15.353710 17712 consensus_queue.cc:1048] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.372097 17712 consensus_queue.cc:1048] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.385725 17681 consensus_queue.cc:1048] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.583413 17750 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:15.594151 17709 consensus_queue.cc:1048] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.596941 17709 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:15.616209 17709 consensus_queue.cc:1048] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:16.131013 17431 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:16.197000 17681 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.197562 17681 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.199491 17681 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:16.200726 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4" is_pre_election: true
I20260709 12:20:16.200726 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" is_pre_election: true
I20260709 12:20:16.201625 17559 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:16.201697 17485 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 1.
I20260709 12:20:16.203141 17599 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.203984 17681 raft_consensus.cc:2804] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:16.204352 17681 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.204674 17681 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.210520 17681 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.212545 17681 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 2 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:16.213581 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:16.213732 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:16.214471 17559 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.214471 17485 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.222862 17559 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 2.
I20260709 12:20:16.222885 17485 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 2.
I20260709 12:20:16.224726 17600 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.225595 17681 raft_consensus.cc:2804] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.226181 17681 raft_consensus.cc:697] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:16.227124 17681 consensus_queue.cc:237] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.235596 17380 catalog_manager.cc:5697] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 0 to 2, leader changed from <none> to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 2 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:16.362326 17559 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:16.363061 17559 raft_consensus.cc:606] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:16.369354 17560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:16.370115 17560 raft_consensus.cc:606] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:16.375696 17560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a993238f88674f6898cddd4975026ca8"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:16.376292 17560 raft_consensus.cc:606] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:16.382553 17560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:16.383255 17560 raft_consensus.cc:606] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:16.399956 17485 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:40976
I20260709 12:20:16.400650 17485 raft_consensus.cc:606] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Received request to transfer leadership to TS 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:16.418362 17633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
mode: GRACEFUL
new_leader_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:42682
I20260709 12:20:16.419144 17633 raft_consensus.cc:606] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Received request to transfer leadership to TS 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:16.426528 17633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
mode: GRACEFUL
new_leader_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:42682
I20260709 12:20:16.427162 17633 raft_consensus.cc:606] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Received request to transfer leadership to TS 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:16.435570 17633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
mode: GRACEFUL
new_leader_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:42682
I20260709 12:20:16.436270 17633 raft_consensus.cc:606] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 LEADER]: Received request to transfer leadership to TS 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:16.475240 17750 raft_consensus.cc:993] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.475766 17709 raft_consensus.cc:1081] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.477510 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:16.478214 17485 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.478513 17485 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.486050 17485 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.487980 17485 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.489238 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:16.489283 17633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.490109 17560 raft_consensus.cc:3055] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.490188 17633 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.490480 17560 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.491432 17560 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.492607 17560 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.496795 17633 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.498570 17448 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.499500 17710 raft_consensus.cc:2804] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.498980 17560 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.501550 17710 raft_consensus.cc:697] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:16.502746 17710 consensus_queue.cc:237] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.512554 17380 catalog_manager.cc:5697] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 1 to 2, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 2 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.521227 17750 raft_consensus.cc:993] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.522075 17709 raft_consensus.cc:993] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.522186 17707 raft_consensus.cc:1081] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.522701 17709 raft_consensus.cc:1081] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.525372 17483 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a993238f88674f6898cddd4975026ca8"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:16.525862 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:16.526270 17483 raft_consensus.cc:493] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.526579 17485 raft_consensus.cc:493] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.526906 17483 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.527168 17485 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.535737 17485 raft_consensus.cc:515] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.535737 17483 raft_consensus.cc:515] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.538487 17485 leader_election.cc:290] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.538487 17483 leader_election.cc:290] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.539530 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:16.540144 17633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.540259 17560 raft_consensus.cc:3055] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.540335 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:16.540560 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.540833 17633 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.540858 17560 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.541245 17634 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.542232 17560 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.541065 17559 raft_consensus.cc:3055] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.543501 17559 raft_consensus.cc:740] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.543998 17560 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.544486 17559 consensus_queue.cc:260] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.546007 17559 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.550288 17633 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.550534 17634 raft_consensus.cc:2468] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.552534 17448 leader_election.cc:304] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.553040 17560 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.553669 17710 raft_consensus.cc:2804] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.553884 17448 leader_election.cc:304] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.555058 17770 raft_consensus.cc:2804] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.555104 17559 raft_consensus.cc:2468] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.555840 17770 raft_consensus.cc:697] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:16.555325 17710 raft_consensus.cc:697] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:16.557353 17710 consensus_queue.cc:237] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.557055 17770 consensus_queue.cc:237] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.567196 17380 catalog_manager.cc:5697] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 1 to 2, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 2 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.573683 17750 raft_consensus.cc:993] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.574316 17771 raft_consensus.cc:1081] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:16.576020 17483 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:16.576653 17483 raft_consensus.cc:493] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.577086 17483 raft_consensus.cc:3060] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.583287 17483 raft_consensus.cc:515] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.582975 17380 catalog_manager.cc:5697] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 1 to 2, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 2 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.585371 17483 leader_election.cc:290] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.586331 17559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:16.586680 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eec864561e648359d9f9b828abe58cb" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.586962 17559 raft_consensus.cc:3055] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.587307 17634 raft_consensus.cc:3060] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.587355 17559 raft_consensus.cc:740] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.588441 17559 consensus_queue.cc:260] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.593549 17559 raft_consensus.cc:3060] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.596853 17634 raft_consensus.cc:2468] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.598416 17448 leader_election.cc:304] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.599424 17770 raft_consensus.cc:2804] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.599927 17770 raft_consensus.cc:697] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:16.600890 17770 consensus_queue.cc:237] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.601742 17559 raft_consensus.cc:2468] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 2.
I20260709 12:20:16.608698 17380 catalog_manager.cc:5697] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 1 to 2, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 2 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.611124 17559 raft_consensus.cc:1275] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:16.613050 17681 consensus_queue.cc:1048] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:16.621721 17712 raft_consensus.cc:993] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: : Instructing follower 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.622648 17681 raft_consensus.cc:1081] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 LEADER]: Signalling peer 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.625891 17560 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:33406
I20260709 12:20:16.629685 17483 raft_consensus.cc:1275] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:16.631860 17560 raft_consensus.cc:493] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.632007 17712 consensus_queue.cc:1048] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:16.632519 17560 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:16.641908 17560 raft_consensus.cc:515] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.648559 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:16.649448 17483 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:16.649484 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9004ba90bee446faa1d33c66886f446" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.650343 17634 raft_consensus.cc:3055] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:16.650758 17634 raft_consensus.cc:740] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:16.651723 17634 consensus_queue.cc:260] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.653013 17634 raft_consensus.cc:3060] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:16.658028 17560 leader_election.cc:290] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 3 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.658423 17483 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 3.
I20260709 12:20:16.659700 17525 leader_election.cc:304] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:16.660548 17771 raft_consensus.cc:2804] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:16.661160 17771 raft_consensus.cc:697] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.660995 17634 raft_consensus.cc:2468] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 3.
I20260709 12:20:16.662257 17771 consensus_queue.cc:237] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.670204 17383 catalog_manager.cc:5697] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 2 to 3, leader changed from b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3) to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.709208 17681 raft_consensus.cc:993] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: : Instructing follower 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.709798 17712 raft_consensus.cc:1081] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Signalling peer 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.711777 17560 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:33406
I20260709 12:20:16.712508 17560 raft_consensus.cc:493] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.712942 17560 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.719308 17560 raft_consensus.cc:515] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.721267 17560 leader_election.cc:290] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 2 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.722462 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.722388 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:16.723279 17634 raft_consensus.cc:3055] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.723279 17483 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.723771 17634 raft_consensus.cc:740] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:16.724715 17634 consensus_queue.cc:260] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.726135 17634 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.732345 17483 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 2.
I20260709 12:20:16.733975 17525 leader_election.cc:304] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:16.734233 17634 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 2.
I20260709 12:20:16.734865 17771 raft_consensus.cc:2804] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.735419 17771 raft_consensus.cc:697] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.736240 17771 consensus_queue.cc:237] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.744801 17383 catalog_manager.cc:5697] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 1 to 2, leader changed from b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3) to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.795817 17681 raft_consensus.cc:993] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: : Instructing follower 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.796366 17712 raft_consensus.cc:1081] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Signalling peer 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.798316 17560 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:33406
I20260709 12:20:16.798947 17560 raft_consensus.cc:493] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.799311 17560 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.805090 17560 raft_consensus.cc:515] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.806993 17560 leader_election.cc:290] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 2 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.808069 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:16.808322 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.808816 17483 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.808969 17634 raft_consensus.cc:3055] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.809341 17634 raft_consensus.cc:740] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:16.810164 17634 consensus_queue.cc:260] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.811298 17634 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.815093 17483 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 2.
I20260709 12:20:16.816419 17525 leader_election.cc:304] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:16.817245 17771 raft_consensus.cc:2804] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.817200 17634 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 2.
I20260709 12:20:16.817937 17771 raft_consensus.cc:697] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.818881 17771 consensus_queue.cc:237] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.827333 17383 catalog_manager.cc:5697] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 1 to 2, leader changed from b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3) to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.952551 17560 raft_consensus.cc:1275] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:16.954288 17770 consensus_queue.cc:1048] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:16.967720 17634 raft_consensus.cc:1275] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:16.970067 17710 consensus_queue.cc:1048] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:16.971484 17772 raft_consensus.cc:993] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.971976 17772 raft_consensus.cc:1081] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Signalling peer 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:16.973858 17559 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:33404
I20260709 12:20:16.974437 17559 raft_consensus.cc:493] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:16.974820 17559 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.983673 17559 raft_consensus.cc:515] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.986424 17559 leader_election.cc:290] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 2 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:16.986711 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:16.987426 17483 raft_consensus.cc:3055] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:16.987555 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:16.987829 17483 raft_consensus.cc:740] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:16.988255 17634 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.988739 17483 consensus_queue.cc:260] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:16.990007 17483 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:16.995168 17634 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 2.
I20260709 12:20:16.996383 17483 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 2.
I20260709 12:20:16.996460 17523 leader_election.cc:304] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:16.997445 17771 raft_consensus.cc:2804] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:16.997974 17771 raft_consensus.cc:697] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:16.998836 17771 consensus_queue.cc:237] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:17.007148 17383 catalog_manager.cc:5697] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 1 to 2, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.041046 17634 raft_consensus.cc:1275] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.042647 17772 consensus_queue.cc:1048] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:17.048199 17483 raft_consensus.cc:1275] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:20:17.049475 17633 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.050423 17771 consensus_queue.cc:1048] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.051296 17710 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:17.055588 17559 raft_consensus.cc:1275] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.058065 17710 consensus_queue.cc:1048] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.072919 17560 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.075891 17634 raft_consensus.cc:1275] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:20:17.078423 17560 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.080413 17771 consensus_queue.cc:1048] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.090955 17770 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.096918 17770 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:20:17.116103 17634 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.118358 17787 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.155740 17634 raft_consensus.cc:1275] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.157308 17790 consensus_queue.cc:1048] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:17.168025 17483 raft_consensus.cc:1275] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.169898 17750 consensus_queue.cc:1048] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.298846 17634 raft_consensus.cc:1275] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.300539 17750 consensus_queue.cc:1048] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:17.310731 17483 raft_consensus.cc:1275] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.312837 17790 consensus_queue.cc:1048] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:17.411748 17634 raft_consensus.cc:1275] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.413404 17790 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:17.424448 17483 raft_consensus.cc:1275] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:17.426715 17750 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:19.484797 16792 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:20:19.485900 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:19.543002 17483 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:40978
I20260709 12:20:19.545467 17483 raft_consensus.cc:606] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Received request to transfer leadership to TS b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:19.551663 17483 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:40978
I20260709 12:20:19.552242 17483 raft_consensus.cc:606] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Received request to transfer leadership to TS b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:19.553544 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:20:19.554705 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:19.796247 17805 raft_consensus.cc:993] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:19.796948 17770 raft_consensus.cc:1081] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Signalling peer b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:19.798858 17634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:42670
I20260709 12:20:19.799470 17634 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:19.799845 17634 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:19.807132 17634 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:19.809242 17634 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 3 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:19.810861 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:19.810415 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:19.811539 17560 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:19.811540 17483 raft_consensus.cc:3055] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:19.812077 17483 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:19.812891 17483 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:19.814347 17483 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:19.820183 17560 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 3.
I20260709 12:20:19.821982 17600 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:19.822839 17483 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 3.
I20260709 12:20:19.823081 17712 raft_consensus.cc:2804] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:19.824329 17712 raft_consensus.cc:697] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:19.825580 17712 consensus_queue.cc:237] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:19.839133 17383 catalog_manager.cc:5697] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 2 to 3, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 3 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.855675 17805 raft_consensus.cc:993] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:19.856284 17770 raft_consensus.cc:1081] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Signalling peer b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:19.858239 17634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:42670
I20260709 12:20:19.858943 17634 raft_consensus.cc:493] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:19.859234 17634 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:19.865037 17634 raft_consensus.cc:515] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:19.866918 17634 leader_election.cc:290] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 3 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:19.868136 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:19.868196 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:19.868871 17560 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:19.868840 17483 raft_consensus.cc:3055] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:19.869413 17483 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:19.870203 17483 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:19.871301 17483 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:19.875200 17560 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 3.
I20260709 12:20:19.876765 17600 leader_election.cc:304] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:19.877169 17483 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 3.
I20260709 12:20:19.877799 17712 raft_consensus.cc:2804] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:19.878369 17712 raft_consensus.cc:697] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:19.879226 17712 consensus_queue.cc:237] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:19.888352 17383 catalog_manager.cc:5697] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 2 to 3, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 3 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:20.301119 17483 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:20.302511 17712 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:20.314064 17560 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:20.316434 17815 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:20.437165 17560 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:20.438710 17712 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:20.449419 17483 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:20.451743 17712 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:20.556202 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:20.562382 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:20.562914 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:21.564054 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:21.569862 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:21.570461 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:22.617480 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:22.805190 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:22.806028 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:20:22.929838 17450 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.007s	sys 0.008s
W20260709 12:20:22.930398 17450 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.007s	sys 0.010s
W20260709 12:20:23.254520 17414 debug-util.cc:398] Leaking SignalData structure 0x7b08000522a0 after lost signal to thread 16795
W20260709 12:20:23.255587 17414 debug-util.cc:398] Leaking SignalData structure 0x7b0800052980 after lost signal to thread 17417
W20260709 12:20:23.256661 17414 debug-util.cc:398] Leaking SignalData structure 0x7b08000f3ce0 after lost signal to thread 17509
W20260709 12:20:23.257578 17414 debug-util.cc:398] Leaking SignalData structure 0x7b08000f34a0 after lost signal to thread 17584
W20260709 12:20:23.258741 17414 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3720 after lost signal to thread 17659
I20260709 12:20:23.807540 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:23.815347 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:23.816037 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:24.818535 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:24.828995 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:24.829783 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:25.831073 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:25.838718 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:25.839304 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:26.840401 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:26.846195 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:26.846782 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:27.847961 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:27.853996 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:27.854573 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:28.855777 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:28.862213 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:28.863343 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:29.904878 16792 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:20:29.914193 17634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:42682
I20260709 12:20:29.915040 17634 raft_consensus.cc:606] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:29.928730 17634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:42682
I20260709 12:20:29.929622 17634 raft_consensus.cc:606] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:29.935948 17558 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:29.936633 17558 raft_consensus.cc:606] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:29.943218 17558 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:29.943877 17558 raft_consensus.cc:606] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:29.949995 17558 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
mode: GRACEFUL
new_leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:33408
I20260709 12:20:29.950680 17558 raft_consensus.cc:606] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Received request to transfer leadership to TS 7677bb01c3ee4e0ca1378309a562a7cc
I20260709 12:20:30.153772 17837 raft_consensus.cc:993] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.154479 17832 raft_consensus.cc:1081] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.156440 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40958
I20260709 12:20:30.156981 17485 raft_consensus.cc:493] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:30.157289 17485 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:30.163975 17485 raft_consensus.cc:515] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.166544 17485 leader_election.cc:290] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 4 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:30.167678 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:30.167676 17558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:30.168642 17634 raft_consensus.cc:3055] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:30.168658 17558 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:30.169001 17634 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:30.170042 17634 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.171135 17634 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:30.175360 17558 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 4.
I20260709 12:20:30.176873 17451 leader_election.cc:304] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:30.176944 17634 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 4.
I20260709 12:20:30.177939 17836 raft_consensus.cc:2804] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:30.178557 17836 raft_consensus.cc:697] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:30.179692 17836 consensus_queue.cc:237] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.195655 17381 catalog_manager.cc:5697] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 3 to 4, leader changed from b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 4 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.199620 17837 raft_consensus.cc:993] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.200300 17832 raft_consensus.cc:1081] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.202736 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40958
I20260709 12:20:30.203305 17485 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:30.203635 17485 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:30.211979 17485 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.214417 17485 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 4 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:30.215689 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:30.215998 17558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:30.216447 17634 raft_consensus.cc:3055] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:30.216658 17558 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:30.216892 17634 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:30.217942 17634 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.219007 17634 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:30.223105 17558 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 4.
I20260709 12:20:30.224435 17451 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, 7677bb01c3ee4e0ca1378309a562a7cc; no voters: 
I20260709 12:20:30.225306 17836 raft_consensus.cc:2804] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:30.225513 17634 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 4.
I20260709 12:20:30.225878 17836 raft_consensus.cc:697] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:30.226943 17836 consensus_queue.cc:237] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.235786 17381 catalog_manager.cc:5697] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 3 to 4, leader changed from b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 4 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.303784 17835 raft_consensus.cc:993] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.304385 17825 raft_consensus.cc:1081] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.306008 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:30.306480 17485 raft_consensus.cc:493] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:30.306808 17485 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.312572 17485 raft_consensus.cc:515] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.314324 17485 leader_election.cc:290] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 3 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:30.315461 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:30.315478 17558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:30.316035 17634 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.316448 17558 raft_consensus.cc:3055] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:30.316799 17558 raft_consensus.cc:740] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:30.317669 17558 consensus_queue.cc:260] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.318910 17558 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.322325 17634 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 3.
I20260709 12:20:30.324327 17448 leader_election.cc:304] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:30.325430 17836 raft_consensus.cc:2804] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:30.326049 17836 raft_consensus.cc:697] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:30.327059 17558 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 3.
I20260709 12:20:30.327121 17836 consensus_queue.cc:237] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.336347 17381 catalog_manager.cc:5697] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 2 to 3, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 3 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.366479 17835 raft_consensus.cc:993] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.367067 17825 raft_consensus.cc:1081] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.368742 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:30.369323 17485 raft_consensus.cc:493] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:30.369647 17485 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.375326 17485 raft_consensus.cc:515] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.377058 17485 leader_election.cc:290] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 3 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:30.378436 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:30.378264 17558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5344382fd65b4249a153f3ee791b0c75" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:30.379047 17634 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.379086 17558 raft_consensus.cc:3055] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:30.379585 17558 raft_consensus.cc:740] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:30.380581 17558 consensus_queue.cc:260] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.382124 17558 raft_consensus.cc:3060] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.387429 17634 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 3.
I20260709 12:20:30.388896 17448 leader_election.cc:304] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:30.389911 17836 raft_consensus.cc:2804] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:30.390101 17558 raft_consensus.cc:2468] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 3.
I20260709 12:20:30.390501 17836 raft_consensus.cc:697] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:30.391582 17836 consensus_queue.cc:237] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.400334 17381 catalog_manager.cc:5697] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 2 to 3, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 3 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.514227 17835 raft_consensus.cc:993] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: : Instructing follower 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.514765 17825 raft_consensus.cc:1081] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Signalling peer 7677bb01c3ee4e0ca1378309a562a7cc to start an election
I20260709 12:20:30.516376 17485 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
 from {username='slave'} at 127.0.0.1:40970
I20260709 12:20:30.516850 17485 raft_consensus.cc:493] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:30.517100 17485 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.522517 17485 raft_consensus.cc:515] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.524147 17485 leader_election.cc:290] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 3 election: Requested vote from peers 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:30.525243 17558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:30.525243 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef99b25d1b748188b63a97e4a2b734f" candidate_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:30.526165 17634 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.526165 17558 raft_consensus.cc:3055] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:30.526808 17558 raft_consensus.cc:740] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:30.527797 17558 consensus_queue.cc:260] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.529263 17558 raft_consensus.cc:3060] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:30.534685 17634 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 3.
I20260709 12:20:30.536113 17448 leader_election.cc:304] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7677bb01c3ee4e0ca1378309a562a7cc, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:30.537070 17836 raft_consensus.cc:2804] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:30.537441 17558 raft_consensus.cc:2468] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7677bb01c3ee4e0ca1378309a562a7cc in term 3.
I20260709 12:20:30.537561 17836 raft_consensus.cc:697] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Becoming Leader. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:30.538807 17836 consensus_queue.cc:237] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:30.547180 17381 catalog_manager.cc:5697] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc reported cstate change: term changed from 2 to 3, leader changed from 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2) to 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1). New cstate: current_term: 3 leader_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.668411 17558 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:30.669745 17836 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:30.683569 17634 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:30.685976 17830 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:30.689687 17634 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:30.691978 17836 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:30.704586 17558 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:30.706367 17849 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:30.838253 17558 raft_consensus.cc:1275] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:30.839886 17830 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:30.851145 17634 raft_consensus.cc:1275] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:30.853430 17830 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:30.860935 17633 raft_consensus.cc:1275] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:30.864025 17830 consensus_queue.cc:1048] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:30.877183 17558 raft_consensus.cc:1275] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:30.878535 17830 consensus_queue.cc:1048] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:31.139200 17558 raft_consensus.cc:1275] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:31.140714 17830 consensus_queue.cc:1048] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:31.151099 17634 raft_consensus.cc:1275] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Refusing update from remote peer 7677bb01c3ee4e0ca1378309a562a7cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:31.152949 17849 consensus_queue.cc:1048] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.999184 16792 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:20:33.000075 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:33.008599 17485 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:40978
I20260709 12:20:33.009218 17485 raft_consensus.cc:606] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Received request to transfer leadership to TS b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:33.012246 17485 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:40978
I20260709 12:20:33.012844 17485 raft_consensus.cc:606] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Received request to transfer leadership to TS 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:33.015437 17485 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a993238f88674f6898cddd4975026ca8"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:40978
I20260709 12:20:33.015977 17485 raft_consensus.cc:606] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Received request to transfer leadership to TS 33dcc8fb352e4344bee7055ce74134b4
I20260709 12:20:33.018455 17485 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
mode: GRACEFUL
new_leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:40978
I20260709 12:20:33.019006 17485 raft_consensus.cc:606] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Received request to transfer leadership to TS b2a71e6ebd8442caba3a20dda6936d8e
I20260709 12:20:33.020092 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:33.020694 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:33.172578 17861 raft_consensus.cc:993] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:33.173130 17849 raft_consensus.cc:1081] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Signalling peer b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:33.174760 17634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "075c63c2c56446948478c94e194a13cf"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:42670
I20260709 12:20:33.175307 17634 raft_consensus.cc:493] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:33.175618 17634 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:33.182876 17634 raft_consensus.cc:515] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.185446 17634 leader_election.cc:290] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 5 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:33.186414 17485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:33.186476 17861 raft_consensus.cc:993] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:33.187214 17485 raft_consensus.cc:3055] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:20:33.187574 17485 raft_consensus.cc:740] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:33.187467 17849 raft_consensus.cc:1081] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Signalling peer 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:33.186931 17560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075c63c2c56446948478c94e194a13cf" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:33.188309 17560 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:33.188560 17485 consensus_queue.cc:260] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.189595 17558 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a993238f88674f6898cddd4975026ca8"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:33404
I20260709 12:20:33.190114 17485 raft_consensus.cc:3060] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:33.190177 17558 raft_consensus.cc:493] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:33.190799 17558 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:33.196175 17558 raft_consensus.cc:515] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.196668 17560 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 5.
I20260709 12:20:33.197949 17485 raft_consensus.cc:2468] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 5.
I20260709 12:20:33.198357 17600 leader_election.cc:304] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:33.198899 17558 leader_election.cc:290] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 3 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:33.199286 17832 raft_consensus.cc:2804] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:20:33.200403 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:33.200510 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a993238f88674f6898cddd4975026ca8" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:33.201041 17832 raft_consensus.cc:697] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 5 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:33.201143 17483 raft_consensus.cc:3055] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:33.201205 17634 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:33.201660 17483 raft_consensus.cc:740] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:33.202147 17832 consensus_queue.cc:237] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.202530 17483 consensus_queue.cc:260] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.203616 17483 raft_consensus.cc:3060] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:33.211261 17634 raft_consensus.cc:2468] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 3.
I20260709 12:20:33.211797 17483 raft_consensus.cc:2468] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 3.
I20260709 12:20:33.212791 17523 leader_election.cc:304] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:33.212198 17381 catalog_manager.cc:5697] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 4 to 5, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 5 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:33.214003 17835 raft_consensus.cc:2804] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:33.215260 17835 raft_consensus.cc:697] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:33.216209 17835 consensus_queue.cc:237] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.226147 17383 catalog_manager.cc:5697] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 2 to 3, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:33.312672 17861 raft_consensus.cc:993] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:33.313230 17849 raft_consensus.cc:1081] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Signalling peer b2a71e6ebd8442caba3a20dda6936d8e to start an election
I20260709 12:20:33.314824 17634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56"
dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
 from {username='slave'} at 127.0.0.1:42670
I20260709 12:20:33.315282 17634 raft_consensus.cc:493] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:33.315541 17634 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:33.320569 17634 raft_consensus.cc:515] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.322258 17634 leader_election.cc:290] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 5 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453)
I20260709 12:20:33.323098 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:33.323280 17558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbd3acbbf5dc43338e386061158ebe56" candidate_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
I20260709 12:20:33.323756 17483 raft_consensus.cc:3055] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:20:33.323844 17558 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:33.324066 17483 raft_consensus.cc:740] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:33.324787 17483 consensus_queue.cc:260] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.325723 17483 raft_consensus.cc:3060] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:33.329334 17558 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 5.
I20260709 12:20:33.330674 17600 leader_election.cc:304] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:33.331287 17483 raft_consensus.cc:2468] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a71e6ebd8442caba3a20dda6936d8e in term 5.
I20260709 12:20:33.331411 17832 raft_consensus.cc:2804] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:20:33.331992 17832 raft_consensus.cc:697] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 5 LEADER]: Becoming Leader. State: Replica: b2a71e6ebd8442caba3a20dda6936d8e, State: Running, Role: LEADER
I20260709 12:20:33.332896 17832 consensus_queue.cc:237] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.338061 17849 raft_consensus.cc:993] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: : Instructing follower 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:33.338636 17849 raft_consensus.cc:1081] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Signalling peer 33dcc8fb352e4344bee7055ce74134b4 to start an election
I20260709 12:20:33.340492 17558 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57"
dest_uuid: "33dcc8fb352e4344bee7055ce74134b4"
 from {username='slave'} at 127.0.0.1:33404
I20260709 12:20:33.341149 17558 raft_consensus.cc:493] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:33.341602 17558 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:33.343438 17383 catalog_manager.cc:5697] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e reported cstate change: term changed from 4 to 5, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3). New cstate: current_term: 5 leader_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:33.348510 17558 raft_consensus.cc:515] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.351747 17558 leader_election.cc:290] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 4 election: Requested vote from peers 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577), b2a71e6ebd8442caba3a20dda6936d8e (127.16.102.3:43895)
I20260709 12:20:33.352161 17483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7677bb01c3ee4e0ca1378309a562a7cc"
I20260709 12:20:33.353158 17483 raft_consensus.cc:3055] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:33.353471 17634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ec98de19e3848ba8368ee133aa41b57" candidate_uuid: "33dcc8fb352e4344bee7055ce74134b4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b2a71e6ebd8442caba3a20dda6936d8e"
I20260709 12:20:33.353557 17483 raft_consensus.cc:740] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7677bb01c3ee4e0ca1378309a562a7cc, State: Running, Role: LEADER
I20260709 12:20:33.354188 17634 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:33.354646 17483 consensus_queue.cc:260] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.355940 17483 raft_consensus.cc:3060] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:33.362215 17634 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 4.
I20260709 12:20:33.363791 17483 raft_consensus.cc:2468] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dcc8fb352e4344bee7055ce74134b4 in term 4.
I20260709 12:20:33.363724 17523 leader_election.cc:304] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dcc8fb352e4344bee7055ce74134b4, b2a71e6ebd8442caba3a20dda6936d8e; no voters: 
I20260709 12:20:33.364880 17835 raft_consensus.cc:2804] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:33.365397 17835 raft_consensus.cc:697] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 LEADER]: Becoming Leader. State: Replica: 33dcc8fb352e4344bee7055ce74134b4, State: Running, Role: LEADER
I20260709 12:20:33.366387 17835 consensus_queue.cc:237] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } }
I20260709 12:20:33.374599 17383 catalog_manager.cc:5697] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 reported cstate change: term changed from 3 to 4, leader changed from 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1) to 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2). New cstate: current_term: 4 leader_uuid: "33dcc8fb352e4344bee7055ce74134b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:33.603451 17483 raft_consensus.cc:1275] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:33.605146 17835 consensus_queue.cc:1048] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:33.620077 17634 raft_consensus.cc:1275] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:33.621665 17835 consensus_queue.cc:1048] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:33.628839 17483 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:33.630649 17832 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:33.644783 17558 raft_consensus.cc:1275] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:33.646546 17832 consensus_queue.cc:1048] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:33.767024 17483 raft_consensus.cc:1275] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:33.768533 17835 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:33.779320 17634 raft_consensus.cc:1275] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Refusing update from remote peer 33dcc8fb352e4344bee7055ce74134b4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:33.781170 17882 consensus_queue.cc:1048] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a71e6ebd8442caba3a20dda6936d8e" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:33.866848 17483 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:33.868247 17832 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7677bb01c3ee4e0ca1378309a562a7cc" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:33.877548 17558 raft_consensus.cc:1275] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Refusing update from remote peer b2a71e6ebd8442caba3a20dda6936d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:33.879246 17832 consensus_queue.cc:1048] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dcc8fb352e4344bee7055ce74134b4" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:34.021991 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:34.027661 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:34.028141 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:35.029241 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:35.035092 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:35.035748 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:36.036965 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:36.043339 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:36.043879 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:37.044914 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:37.050547 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:37.051148 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:38.052317 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:38.059106 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:38.059620 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:39.060602 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:39.066445 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:39.067068 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:40.068466 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:40.080549 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:40.081282 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:41.082768 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:41.088286 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:41.088800 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:42.090133 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:42.096025 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:42.096697 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:43.138031 16792 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:20:43.138685 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:20:43.169332 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:43.170367 16792 tablet_replica.cc:333] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.171284 16792 raft_consensus.cc:2243] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.172528 16792 raft_consensus.cc:2272] T eef99b25d1b748188b63a97e4a2b734f P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.175251 16792 tablet_replica.cc:333] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.176020 16792 raft_consensus.cc:2243] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.177493 16792 raft_consensus.cc:2272] T 5eec864561e648359d9f9b828abe58cb P 7677bb01c3ee4e0ca1378309a562a7cc [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.180553 16792 tablet_replica.cc:333] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.181274 16792 raft_consensus.cc:2243] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.182360 16792 raft_consensus.cc:2272] T 5344382fd65b4249a153f3ee791b0c75 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.184558 16792 tablet_replica.cc:333] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.185201 16792 raft_consensus.cc:2243] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.185719 16792 raft_consensus.cc:2272] T 075c63c2c56446948478c94e194a13cf P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.187887 16792 tablet_replica.cc:333] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.188517 16792 raft_consensus.cc:2243] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.189474 16792 raft_consensus.cc:2272] T cc6f3f060b6e401cbcadf965f547df10 P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.192062 16792 tablet_replica.cc:333] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.192634 16792 raft_consensus.cc:2243] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.193120 16792 raft_consensus.cc:2272] T bbd3acbbf5dc43338e386061158ebe56 P 7677bb01c3ee4e0ca1378309a562a7cc [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.195122 16792 tablet_replica.cc:333] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.195657 16792 raft_consensus.cc:2243] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.196115 16792 raft_consensus.cc:2272] T 8ec98de19e3848ba8368ee133aa41b57 P 7677bb01c3ee4e0ca1378309a562a7cc [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.198220 16792 tablet_replica.cc:333] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.198817 16792 raft_consensus.cc:2243] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.199337 16792 raft_consensus.cc:2272] T d9004ba90bee446faa1d33c66886f446 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.201906 16792 tablet_replica.cc:333] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.202495 16792 raft_consensus.cc:2243] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.203025 16792 raft_consensus.cc:2272] T 37aa0221b88d4bb3a96df77e6c0659ec P 7677bb01c3ee4e0ca1378309a562a7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.205117 16792 tablet_replica.cc:333] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc: stopping tablet replica
I20260709 12:20:43.205660 16792 raft_consensus.cc:2243] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.206297 16792 raft_consensus.cc:2272] T a993238f88674f6898cddd4975026ca8 P 7677bb01c3ee4e0ca1378309a562a7cc [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:43.216480 17525 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20260709 12:20:43.220350 17525 consensus_peers.cc:597] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 -> Peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Couldn't send request to peer 7677bb01c3ee4e0ca1378309a562a7cc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:43.225667 17599 connection.cc:573] client connection to 127.16.102.1:43577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260709 12:20:43.232343 17599 consensus_peers.cc:597] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e -> Peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Couldn't send request to peer 7677bb01c3ee4e0ca1378309a562a7cc. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:43577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:43.234993 17525 consensus_peers.cc:597] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 -> Peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Couldn't send request to peer 7677bb01c3ee4e0ca1378309a562a7cc. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:43577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:43.243419 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
W20260709 12:20:43.246822 17599 consensus_peers.cc:597] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e -> Peer 7677bb01c3ee4e0ca1378309a562a7cc (127.16.102.1:43577): Couldn't send request to peer 7677bb01c3ee4e0ca1378309a562a7cc. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:43577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:43.271236 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
I20260709 12:20:43.308601 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:43.309590 16792 tablet_replica.cc:333] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.310915 16792 raft_consensus.cc:2243] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.311870 16792 raft_consensus.cc:2272] T bbd3acbbf5dc43338e386061158ebe56 P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.315220 16792 tablet_replica.cc:333] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.315949 16792 raft_consensus.cc:2243] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.316497 16792 raft_consensus.cc:2272] T cc6f3f060b6e401cbcadf965f547df10 P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.318914 16792 tablet_replica.cc:333] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.319581 16792 raft_consensus.cc:2243] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.320137 16792 raft_consensus.cc:2272] T 37aa0221b88d4bb3a96df77e6c0659ec P 33dcc8fb352e4344bee7055ce74134b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.322470 16792 tablet_replica.cc:333] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.323154 16792 raft_consensus.cc:2243] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.324162 16792 raft_consensus.cc:2272] T a993238f88674f6898cddd4975026ca8 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.326627 16792 tablet_replica.cc:333] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.327255 16792 raft_consensus.cc:2243] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.327765 16792 raft_consensus.cc:2272] T 5eec864561e648359d9f9b828abe58cb P 33dcc8fb352e4344bee7055ce74134b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.330123 16792 tablet_replica.cc:333] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.330716 16792 raft_consensus.cc:2243] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.331231 16792 raft_consensus.cc:2272] T eef99b25d1b748188b63a97e4a2b734f P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.333847 16792 tablet_replica.cc:333] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.334511 16792 raft_consensus.cc:2243] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.335606 16792 raft_consensus.cc:2272] T d9004ba90bee446faa1d33c66886f446 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:43.337143 17600 consensus_peers.cc:597] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e -> Peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Couldn't send request to peer 33dcc8fb352e4344bee7055ce74134b4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:43.338389 16792 tablet_replica.cc:333] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.339112 16792 raft_consensus.cc:2243] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.340162 16792 raft_consensus.cc:2272] T 8ec98de19e3848ba8368ee133aa41b57 P 33dcc8fb352e4344bee7055ce74134b4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.342489 16792 tablet_replica.cc:333] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.343253 16792 raft_consensus.cc:2243] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.343892 16792 raft_consensus.cc:2272] T 5344382fd65b4249a153f3ee791b0c75 P 33dcc8fb352e4344bee7055ce74134b4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.347087 16792 tablet_replica.cc:333] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4: stopping tablet replica
I20260709 12:20:43.347873 16792 raft_consensus.cc:2243] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.348564 16792 raft_consensus.cc:2272] T 075c63c2c56446948478c94e194a13cf P 33dcc8fb352e4344bee7055ce74134b4 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:43.378435 17600 consensus_peers.cc:597] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e -> Peer 33dcc8fb352e4344bee7055ce74134b4 (127.16.102.2:36453): Couldn't send request to peer 33dcc8fb352e4344bee7055ce74134b4. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:36453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:43.382920 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
I20260709 12:20:43.408680 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
I20260709 12:20:43.445945 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:43.446817 16792 tablet_replica.cc:333] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.447574 16792 raft_consensus.cc:2243] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.448275 16792 raft_consensus.cc:2272] T 5344382fd65b4249a153f3ee791b0c75 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.450805 16792 tablet_replica.cc:333] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.451531 16792 raft_consensus.cc:2243] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.452608 16792 raft_consensus.cc:2272] T 075c63c2c56446948478c94e194a13cf P b2a71e6ebd8442caba3a20dda6936d8e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.454974 16792 tablet_replica.cc:333] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.455636 16792 raft_consensus.cc:2243] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.456205 16792 raft_consensus.cc:2272] T 8ec98de19e3848ba8368ee133aa41b57 P b2a71e6ebd8442caba3a20dda6936d8e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.458442 16792 tablet_replica.cc:333] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.459064 16792 raft_consensus.cc:2243] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.459604 16792 raft_consensus.cc:2272] T d9004ba90bee446faa1d33c66886f446 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.461741 16792 tablet_replica.cc:333] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.462400 16792 raft_consensus.cc:2243] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.462945 16792 raft_consensus.cc:2272] T a993238f88674f6898cddd4975026ca8 P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.464947 16792 tablet_replica.cc:333] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.465582 16792 raft_consensus.cc:2243] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.466543 16792 raft_consensus.cc:2272] T 37aa0221b88d4bb3a96df77e6c0659ec P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.468433 16792 tablet_replica.cc:333] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.469003 16792 raft_consensus.cc:2243] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.470131 16792 raft_consensus.cc:2272] T bbd3acbbf5dc43338e386061158ebe56 P b2a71e6ebd8442caba3a20dda6936d8e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.472093 16792 tablet_replica.cc:333] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.472626 16792 raft_consensus.cc:2243] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.473062 16792 raft_consensus.cc:2272] T cc6f3f060b6e401cbcadf965f547df10 P b2a71e6ebd8442caba3a20dda6936d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.475679 16792 tablet_replica.cc:333] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.476292 16792 raft_consensus.cc:2243] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.476823 16792 raft_consensus.cc:2272] T eef99b25d1b748188b63a97e4a2b734f P b2a71e6ebd8442caba3a20dda6936d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.479055 16792 tablet_replica.cc:333] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e: stopping tablet replica
I20260709 12:20:43.479576 16792 raft_consensus.cc:2243] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.480068 16792 raft_consensus.cc:2272] T 5eec864561e648359d9f9b828abe58cb P b2a71e6ebd8442caba3a20dda6936d8e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.512104 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:20:43.538983 16792 master.cc:562] Master@127.16.102.62:35967 shutting down...
I20260709 12:20:43.570684 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:43.571411 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.571724 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P 229dc4b0a15d4803972a0f8e85d7e1f9: stopping tablet replica
I20260709 12:20:43.613247 16792 master.cc:584] Master@127.16.102.62:35967 shutdown complete.
33dcc8fb352e4344bee7055ce74134b4  4
7677bb01c3ee4e0ca1378309a562a7cc  5
b2a71e6ebd8442caba3a20dda6936d8e  1
33dcc8fb352e4344bee7055ce74134b4  4
7677bb01c3ee4e0ca1378309a562a7cc  3
b2a71e6ebd8442caba3a20dda6936d8e  3
33dcc8fb352e4344bee7055ce74134b4  1
7677bb01c3ee4e0ca1378309a562a7cc  8
b2a71e6ebd8442caba3a20dda6936d8e  1
33dcc8fb352e4344bee7055ce74134b4  3
7677bb01c3ee4e0ca1378309a562a7cc  4
b2a71e6ebd8442caba3a20dda6936d8e  3
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30707 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:20:43.664856 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:42293
I20260709 12:20:43.666277 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:43.672811 17897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:43.672981 17898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:43.674394 17900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:43.675670 16792 server_base.cc:1061] running on GCE node
I20260709 12:20:43.676889 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:43.677095 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:43.677315 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599643677270 us; error 0 us; skew 500 ppm
I20260709 12:20:43.678020 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:43.680816 16792 webserver.cc:533] Webserver started at http://127.16.102.62:41941/ using document root <none> and password file <none>
I20260709 12:20:43.681370 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:43.681592 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:43.681967 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:43.683338 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "7b17aa4109194c3c9f5ca41c3594e03c"
format_stamp: "Formatted at 2026-07-09 12:20:43 on dist-test-slave-7xt3"
I20260709 12:20:43.688620 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 12:20:43.693096 17905 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:43.694201 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260709 12:20:43.694630 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "7b17aa4109194c3c9f5ca41c3594e03c"
format_stamp: "Formatted at 2026-07-09 12:20:43 on dist-test-slave-7xt3"
I20260709 12:20:43.695014 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:43.718878 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:43.720434 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:43.767141 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:42293
I20260709 12:20:43.767230 17956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:42293 every 8 connection(s)
I20260709 12:20:43.772362 17957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:43.784722 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Bootstrap starting.
I20260709 12:20:43.790967 17957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:43.797358 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: No bootstrap required, opened a new log
I20260709 12:20:43.800369 17957 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" member_type: VOTER }
I20260709 12:20:43.800966 17957 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:43.801224 17957 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b17aa4109194c3c9f5ca41c3594e03c, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.801980 17957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" member_type: VOTER }
I20260709 12:20:43.802573 17957 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:43.802811 17957 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:43.803079 17957 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:43.808588 17957 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" member_type: VOTER }
I20260709 12:20:43.809263 17957 leader_election.cc:304] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b17aa4109194c3c9f5ca41c3594e03c; no voters: 
I20260709 12:20:43.810815 17957 leader_election.cc:290] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:43.811332 17960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:43.812954 17960 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 1 LEADER]: Becoming Leader. State: Replica: 7b17aa4109194c3c9f5ca41c3594e03c, State: Running, Role: LEADER
I20260709 12:20:43.813751 17960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" member_type: VOTER }
I20260709 12:20:43.814683 17957 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:43.819294 17961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" member_type: VOTER } }
I20260709 12:20:43.819514 17962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b17aa4109194c3c9f5ca41c3594e03c. Latest consensus state: current_term: 1 leader_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b17aa4109194c3c9f5ca41c3594e03c" member_type: VOTER } }
I20260709 12:20:43.820197 17961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:43.820358 17962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:43.824805 17966 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:43.830776 17966 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:43.837523 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:43.845721 17966 catalog_manager.cc:1383] Generated new cluster ID: 0966dae56a604962a6e8057efd271e30
I20260709 12:20:43.846228 17966 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:43.864094 17966 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:43.865649 17966 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:43.883651 17966 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Generated new TSK 0
I20260709 12:20:43.884430 17966 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:43.904971 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:43.912793 17978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:43.913748 17979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:43.917078 16792 server_base.cc:1061] running on GCE node
W20260709 12:20:43.917253 17981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:43.918560 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:43.918833 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:43.918982 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599643918967 us; error 0 us; skew 500 ppm
I20260709 12:20:43.919579 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:43.922649 16792 webserver.cc:533] Webserver started at http://127.16.102.1:37729/ using document root <none> and password file <none>
I20260709 12:20:43.923226 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:43.923418 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:43.923663 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:43.924961 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
format_stamp: "Formatted at 2026-07-09 12:20:43 on dist-test-slave-7xt3"
I20260709 12:20:43.930884 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:20:43.934875 17986 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:43.935853 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:20:43.936177 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
format_stamp: "Formatted at 2026-07-09 12:20:43 on dist-test-slave-7xt3"
I20260709 12:20:43.936494 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:43.980226 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:43.981776 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:43.983634 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:43.986688 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:43.986945 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:43.987321 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:43.987495 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.037808 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:45965
I20260709 12:20:44.037982 18048 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:45965 every 8 connection(s)
I20260709 12:20:44.043900 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:44.053843 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:44.054945 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:44.057675 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:44.058772 16792 server_base.cc:1061] running on GCE node
I20260709 12:20:44.060196 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:44.060415 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:44.060549 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599644060536 us; error 0 us; skew 500 ppm
I20260709 12:20:44.061079 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:44.064483 16792 webserver.cc:533] Webserver started at http://127.16.102.2:34057/ using document root <none> and password file <none>
I20260709 12:20:44.065109 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:44.065393 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:44.065737 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:44.066074 18049 heartbeater.cc:344] Connected to a master server at 127.16.102.62:42293
I20260709 12:20:44.066596 18049 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:44.067580 18049 heartbeater.cc:507] Master 127.16.102.62:42293 requested a full tablet report, sending...
I20260709 12:20:44.067580 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "768723648a9444fda53659e6a8cc3091"
format_stamp: "Formatted at 2026-07-09 12:20:44 on dist-test-slave-7xt3"
I20260709 12:20:44.070542 17922 ts_manager.cc:194] Registered new tserver with Master: cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:20:44.073302 17922 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41312
I20260709 12:20:44.074656 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:44.079432 18061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.080466 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:20:44.080785 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "768723648a9444fda53659e6a8cc3091"
format_stamp: "Formatted at 2026-07-09 12:20:44 on dist-test-slave-7xt3"
I20260709 12:20:44.081076 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:44.092638 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:44.094215 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:44.095973 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:44.098946 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:44.099172 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.099503 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:44.099648 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.149739 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:40047
I20260709 12:20:44.149932 18123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:40047 every 8 connection(s)
I20260709 12:20:44.155514 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:44.165215 18127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:44.167491 18128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:44.169720 18130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:44.171241 16792 server_base.cc:1061] running on GCE node
I20260709 12:20:44.172500 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:44.172780 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:44.172907 18124 heartbeater.cc:344] Connected to a master server at 127.16.102.62:42293
I20260709 12:20:44.173049 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599644173024 us; error 0 us; skew 500 ppm
I20260709 12:20:44.173345 18124 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:44.173914 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:44.174403 18124 heartbeater.cc:507] Master 127.16.102.62:42293 requested a full tablet report, sending...
I20260709 12:20:44.176939 16792 webserver.cc:533] Webserver started at http://127.16.102.3:43907/ using document root <none> and password file <none>
I20260709 12:20:44.177212 17922 ts_manager.cc:194] Registered new tserver with Master: 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:44.177747 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:44.178066 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:44.178427 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:44.179352 17922 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41324
I20260709 12:20:44.180269 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "ae9641d5c3184572a102d51a75f670ee"
format_stamp: "Formatted at 2026-07-09 12:20:44 on dist-test-slave-7xt3"
I20260709 12:20:44.186656 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20260709 12:20:44.190610 18135 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.191542 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:20:44.191967 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "ae9641d5c3184572a102d51a75f670ee"
format_stamp: "Formatted at 2026-07-09 12:20:44 on dist-test-slave-7xt3"
I20260709 12:20:44.192330 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:44.214926 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:44.216408 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:44.218233 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:44.220944 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:44.221216 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.221487 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:44.221688 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:44.268404 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:39905
I20260709 12:20:44.268502 18197 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:39905 every 8 connection(s)
I20260709 12:20:44.284700 18198 heartbeater.cc:344] Connected to a master server at 127.16.102.62:42293
I20260709 12:20:44.285154 18198 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:44.286031 18198 heartbeater.cc:507] Master 127.16.102.62:42293 requested a full tablet report, sending...
I20260709 12:20:44.288336 17922 ts_manager.cc:194] Registered new tserver with Master: ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:44.289187 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01724032s
I20260709 12:20:44.290298 17922 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41336
I20260709 12:20:44.322384 17922 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41338:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:20:44.329794 17922 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:20:44.562501 18162 tablet_service.cc:1511] Processing CreateTablet for tablet 18093241e4df42869e3d96cdf66a53e6 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:44.562767 18163 tablet_service.cc:1511] Processing CreateTablet for tablet a83278d09c914df888a57be57eb08f8e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:44.564733 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18093241e4df42869e3d96cdf66a53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.565757 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83278d09c914df888a57be57eb08f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.566336 18161 tablet_service.cc:1511] Processing CreateTablet for tablet 5240f7371cc74cb48dffa1dadcbeec49 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:44.567968 18161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5240f7371cc74cb48dffa1dadcbeec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.571378 18089 tablet_service.cc:1511] Processing CreateTablet for tablet a83278d09c914df888a57be57eb08f8e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:44.574044 18086 tablet_service.cc:1511] Processing CreateTablet for tablet b7fd19d9bf374e6da56b7ead11c666e9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:44.575999 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fd19d9bf374e6da56b7ead11c666e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.580363 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 5240f7371cc74cb48dffa1dadcbeec49 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:44.583784 18085 tablet_service.cc:1511] Processing CreateTablet for tablet 804b0137cb0b4850b912b6519bae7417 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:44.585469 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83278d09c914df888a57be57eb08f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.583703 18088 tablet_service.cc:1511] Processing CreateTablet for tablet 18093241e4df42869e3d96cdf66a53e6 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:44.590384 18160 tablet_service.cc:1511] Processing CreateTablet for tablet b7fd19d9bf374e6da56b7ead11c666e9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:44.592417 18159 tablet_service.cc:1511] Processing CreateTablet for tablet 804b0137cb0b4850b912b6519bae7417 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:44.593361 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18093241e4df42869e3d96cdf66a53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.592458 18160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fd19d9bf374e6da56b7ead11c666e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.595706 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804b0137cb0b4850b912b6519bae7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.595709 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804b0137cb0b4850b912b6519bae7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.600589 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5240f7371cc74cb48dffa1dadcbeec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.613304 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 4cc94b1c76be48d6813e7d1c89a4d66c (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:44.615072 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc94b1c76be48d6813e7d1c89a4d66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.619484 18083 tablet_service.cc:1511] Processing CreateTablet for tablet a3c91f3d9da3442f89360b0c316ee411 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:44.621320 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c91f3d9da3442f89360b0c316ee411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.624531 18161 tablet_service.cc:1511] Processing CreateTablet for tablet 4cc94b1c76be48d6813e7d1c89a4d66c (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:44.624743 18158 tablet_service.cc:1511] Processing CreateTablet for tablet a3c91f3d9da3442f89360b0c316ee411 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:44.625646 18082 tablet_service.cc:1511] Processing CreateTablet for tablet aef65d70106f4f458174493864b4ba3d (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:44.626343 18161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc94b1c76be48d6813e7d1c89a4d66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.627173 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef65d70106f4f458174493864b4ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.627430 18158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c91f3d9da3442f89360b0c316ee411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.631839 18010 tablet_service.cc:1511] Processing CreateTablet for tablet 804b0137cb0b4850b912b6519bae7417 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:44.632903 18014 tablet_service.cc:1511] Processing CreateTablet for tablet a83278d09c914df888a57be57eb08f8e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:44.635210 18157 tablet_service.cc:1511] Processing CreateTablet for tablet aef65d70106f4f458174493864b4ba3d (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:44.635462 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 18093241e4df42869e3d96cdf66a53e6 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:44.636852 18157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef65d70106f4f458174493864b4ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.637897 18012 tablet_service.cc:1511] Processing CreateTablet for tablet 5240f7371cc74cb48dffa1dadcbeec49 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:44.639146 18009 tablet_service.cc:1511] Processing CreateTablet for tablet 4cc94b1c76be48d6813e7d1c89a4d66c (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:44.640113 18081 tablet_service.cc:1511] Processing CreateTablet for tablet 3412b19684e44e3d89ca8150a50a9d14 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:44.640494 18011 tablet_service.cc:1511] Processing CreateTablet for tablet b7fd19d9bf374e6da56b7ead11c666e9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:44.641721 18081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3412b19684e44e3d89ca8150a50a9d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.641763 18008 tablet_service.cc:1511] Processing CreateTablet for tablet a3c91f3d9da3442f89360b0c316ee411 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:44.644629 18006 tablet_service.cc:1511] Processing CreateTablet for tablet 3412b19684e44e3d89ca8150a50a9d14 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:44.645220 18007 tablet_service.cc:1511] Processing CreateTablet for tablet aef65d70106f4f458174493864b4ba3d (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:44.647753 18156 tablet_service.cc:1511] Processing CreateTablet for tablet 3412b19684e44e3d89ca8150a50a9d14 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:44.648324 18005 tablet_service.cc:1511] Processing CreateTablet for tablet d440327a63754b889f7edd91e1373942 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:44.648905 18080 tablet_service.cc:1511] Processing CreateTablet for tablet d440327a63754b889f7edd91e1373942 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:44.653509 18156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3412b19684e44e3d89ca8150a50a9d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.653995 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d440327a63754b889f7edd91e1373942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.633679 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804b0137cb0b4850b912b6519bae7417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.658573 18011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fd19d9bf374e6da56b7ead11c666e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.662535 18012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5240f7371cc74cb48dffa1dadcbeec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.664516 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18093241e4df42869e3d96cdf66a53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.666344 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc94b1c76be48d6813e7d1c89a4d66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.668150 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83278d09c914df888a57be57eb08f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.670511 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3412b19684e44e3d89ca8150a50a9d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.675073 18005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d440327a63754b889f7edd91e1373942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.677302 18007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef65d70106f4f458174493864b4ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.688432 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c91f3d9da3442f89360b0c316ee411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.720383 18155 tablet_service.cc:1511] Processing CreateTablet for tablet d440327a63754b889f7edd91e1373942 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:44.722070 18155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d440327a63754b889f7edd91e1373942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.749970 18162 tablet_service.cc:1511] Processing CreateTablet for tablet 7c3aa170318f4e1f95796426dc71d60b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:44.751639 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3aa170318f4e1f95796426dc71d60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.761260 18213 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:44.797057 18088 tablet_service.cc:1511] Processing CreateTablet for tablet 7c3aa170318f4e1f95796426dc71d60b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:44.798787 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3aa170318f4e1f95796426dc71d60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.807829 18010 tablet_service.cc:1511] Processing CreateTablet for tablet 7c3aa170318f4e1f95796426dc71d60b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:44.809480 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3aa170318f4e1f95796426dc71d60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.824702 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 4843c80e9d9348a397ce35e196deb9bd (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:44.825117 18163 tablet_service.cc:1511] Processing CreateTablet for tablet 4843c80e9d9348a397ce35e196deb9bd (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:44.826723 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4843c80e9d9348a397ce35e196deb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:44.830171 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:44.826726 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4843c80e9d9348a397ce35e196deb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.859503 18014 tablet_service.cc:1511] Processing CreateTablet for tablet 4843c80e9d9348a397ce35e196deb9bd (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:44.860694 18082 tablet_service.cc:1511] Processing CreateTablet for tablet f9731c63bc7f406db2c77f5ef0415a78 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:44.861649 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4843c80e9d9348a397ce35e196deb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.862197 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9731c63bc7f406db2c77f5ef0415a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.880625 18008 tablet_service.cc:1511] Processing CreateTablet for tablet f9731c63bc7f406db2c77f5ef0415a78 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:44.882275 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9731c63bc7f406db2c77f5ef0415a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.895313 18213 tablet_bootstrap.cc:654] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:44.905393 18214 tablet_bootstrap.cc:492] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:44.923555 18215 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:44.924149 18155 tablet_service.cc:1511] Processing CreateTablet for tablet f9731c63bc7f406db2c77f5ef0415a78 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:44.925583 18155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9731c63bc7f406db2c77f5ef0415a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.921115 18214 tablet_bootstrap.cc:654] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:44.934288 18013 tablet_service.cc:1511] Processing CreateTablet for tablet c53769058ca3443f805983b61a69328e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:44.936625 18082 tablet_service.cc:1511] Processing CreateTablet for tablet c53769058ca3443f805983b61a69328e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:44.941956 18162 tablet_service.cc:1511] Processing CreateTablet for tablet c53769058ca3443f805983b61a69328e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:44.949352 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53769058ca3443f805983b61a69328e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.952780 18215 tablet_bootstrap.cc:654] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:44.954802 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53769058ca3443f805983b61a69328e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.956857 18213 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:44.957428 18213 ts_tablet_manager.cc:1403] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.197s	user 0.007s	sys 0.006s
I20260709 12:20:44.957752 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53769058ca3443f805983b61a69328e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.958878 18080 tablet_service.cc:1511] Processing CreateTablet for tablet 578328e4e1fd478cbc58dd1048417480 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:44.960433 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578328e4e1fd478cbc58dd1048417480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.963276 18156 tablet_service.cc:1511] Processing CreateTablet for tablet 578328e4e1fd478cbc58dd1048417480 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:44.965095 18156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578328e4e1fd478cbc58dd1048417480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.961246 18213 raft_consensus.cc:359] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:44.970588 18213 raft_consensus.cc:385] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:44.971148 18213 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:44.973970 18008 tablet_service.cc:1511] Processing CreateTablet for tablet 7e793f93c4f740b78f31d2f0dd643095 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:44.975816 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e793f93c4f740b78f31d2f0dd643095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.974078 18213 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:44.972707 18012 tablet_service.cc:1511] Processing CreateTablet for tablet 578328e4e1fd478cbc58dd1048417480 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:44.978057 18012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578328e4e1fd478cbc58dd1048417480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.983057 18214 tablet_bootstrap.cc:492] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:44.983851 18214 ts_tablet_manager.cc:1403] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.079s	user 0.018s	sys 0.000s
I20260709 12:20:44.987658 18214 raft_consensus.cc:359] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:44.989101 18163 tablet_service.cc:1511] Processing CreateTablet for tablet 7e793f93c4f740b78f31d2f0dd643095 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:44.991354 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e793f93c4f740b78f31d2f0dd643095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:44.998190 18006 tablet_service.cc:1511] Processing CreateTablet for tablet c289d2c3e9fc40fbb7102b5788f74aab (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:44.989727 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 7e793f93c4f740b78f31d2f0dd643095 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:45.000042 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e793f93c4f740b78f31d2f0dd643095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.001019 18088 tablet_service.cc:1511] Processing CreateTablet for tablet c289d2c3e9fc40fbb7102b5788f74aab (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:45.002622 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c289d2c3e9fc40fbb7102b5788f74aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.002552 18158 tablet_service.cc:1511] Processing CreateTablet for tablet c289d2c3e9fc40fbb7102b5788f74aab (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:45.004196 18158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c289d2c3e9fc40fbb7102b5788f74aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.005986 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c289d2c3e9fc40fbb7102b5788f74aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.007056 18214 raft_consensus.cc:385] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.007524 18214 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.008428 18010 tablet_service.cc:1511] Processing CreateTablet for tablet 666440ca642c4715bd2f211bb5f14d29 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:45.008496 18214 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.010241 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 666440ca642c4715bd2f211bb5f14d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.012444 18083 tablet_service.cc:1511] Processing CreateTablet for tablet 666440ca642c4715bd2f211bb5f14d29 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:45.014488 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 666440ca642c4715bd2f211bb5f14d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.041926 18162 tablet_service.cc:1511] Processing CreateTablet for tablet 666440ca642c4715bd2f211bb5f14d29 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:45.043694 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 666440ca642c4715bd2f211bb5f14d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.075183 18198 heartbeater.cc:499] Master 127.16.102.62:42293 was elected leader, sending a full tablet report...
I20260709 12:20:45.076141 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 14d2c4b70e164856a205483966ca4963 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:45.076999 18049 heartbeater.cc:499] Master 127.16.102.62:42293 was elected leader, sending a full tablet report...
I20260709 12:20:45.077760 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d2c4b70e164856a205483966ca4963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.080673 18161 tablet_service.cc:1511] Processing CreateTablet for tablet 14d2c4b70e164856a205483966ca4963 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:45.082474 18080 tablet_service.cc:1511] Processing CreateTablet for tablet 14d2c4b70e164856a205483966ca4963 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:45.083709 18014 tablet_service.cc:1511] Processing CreateTablet for tablet 1236f612fda5419089c0c9aa244bb5bc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:45.085248 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1236f612fda5419089c0c9aa244bb5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.084056 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d2c4b70e164856a205483966ca4963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.082365 18161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d2c4b70e164856a205483966ca4963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.089280 18213 ts_tablet_manager.cc:1434] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.132s	user 0.010s	sys 0.000s
I20260709 12:20:45.090577 18213 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:45.093225 18214 ts_tablet_manager.cc:1434] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.104s	user 0.007s	sys 0.026s
I20260709 12:20:45.094264 18124 heartbeater.cc:499] Master 127.16.102.62:42293 was elected leader, sending a full tablet report...
I20260709 12:20:45.098846 18214 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:45.094295 18220 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.106621 18220 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.108201 18215 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:45.108940 18215 ts_tablet_manager.cc:1403] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.186s	user 0.046s	sys 0.027s
I20260709 12:20:45.118821 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 1236f612fda5419089c0c9aa244bb5bc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:45.120350 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1236f612fda5419089c0c9aa244bb5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.124009 18155 tablet_service.cc:1511] Processing CreateTablet for tablet 1236f612fda5419089c0c9aa244bb5bc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:45.125672 18155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1236f612fda5419089c0c9aa244bb5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.135293 18089 tablet_service.cc:1511] Processing CreateTablet for tablet fd8fe387b2164c30aa94afed5a7c218d (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:45.136721 18213 tablet_bootstrap.cc:654] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.136940 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8fe387b2164c30aa94afed5a7c218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.139675 18008 tablet_service.cc:1511] Processing CreateTablet for tablet fd8fe387b2164c30aa94afed5a7c218d (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:45.140614 18220 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:45.141254 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8fe387b2164c30aa94afed5a7c218d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.146509 18214 tablet_bootstrap.cc:654] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.152587 18162 tablet_service.cc:1511] Processing CreateTablet for tablet fd8fe387b2164c30aa94afed5a7c218d (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:45.154271 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8fe387b2164c30aa94afed5a7c218d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:45.155237 18125 tablet.cc:2406] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:45.166712 18009 tablet_service.cc:1511] Processing CreateTablet for tablet a29e00b535474906948e66659110db3b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:45.168442 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e00b535474906948e66659110db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.171078 18085 tablet_service.cc:1511] Processing CreateTablet for tablet a29e00b535474906948e66659110db3b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:45.172673 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e00b535474906948e66659110db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.173840 18159 tablet_service.cc:1511] Processing CreateTablet for tablet a29e00b535474906948e66659110db3b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:45.175401 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e00b535474906948e66659110db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.122031 18215 raft_consensus.cc:359] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.188727 18215 raft_consensus.cc:385] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.191663 18215 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.204455 18215 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.216104 18214 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:45.218798 18214 ts_tablet_manager.cc:1403] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.120s	user 0.018s	sys 0.004s
I20260709 12:20:45.228717 18158 tablet_service.cc:1511] Processing CreateTablet for tablet d525bcd825eb49d9be935bc89de569c7 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:45.230463 18158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d525bcd825eb49d9be935bc89de569c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.232260 18213 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:45.233026 18213 ts_tablet_manager.cc:1403] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.143s	user 0.015s	sys 0.004s
I20260709 12:20:45.238111 18010 tablet_service.cc:1511] Processing CreateTablet for tablet 9ba0c897fcb346488dfe124a973cb2d1 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:45.239498 18083 tablet_service.cc:1511] Processing CreateTablet for tablet 9ba0c897fcb346488dfe124a973cb2d1 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:45.246984 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ba0c897fcb346488dfe124a973cb2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.246845 18087 tablet_service.cc:1511] Processing CreateTablet for tablet d525bcd825eb49d9be935bc89de569c7 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:45.250972 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ba0c897fcb346488dfe124a973cb2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.251924 18213 raft_consensus.cc:359] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.252879 18213 raft_consensus.cc:385] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.253279 18213 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.203372 18006 tablet_service.cc:1511] Processing CreateTablet for tablet d525bcd825eb49d9be935bc89de569c7 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:45.268970 18215 ts_tablet_manager.cc:1434] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.160s	user 0.011s	sys 0.001s
I20260709 12:20:45.270373 18215 tablet_bootstrap.cc:492] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:45.270692 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d525bcd825eb49d9be935bc89de569c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.278900 18215 tablet_bootstrap.cc:654] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.295928 18214 raft_consensus.cc:359] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.286362 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d525bcd825eb49d9be935bc89de569c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.299156 18214 raft_consensus.cc:385] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.334681 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:45.336203 18227 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.336815 18227 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.303208 18214 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.337659 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:45.337839 18064 leader_election.cc:343] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:45.340966 18064 leader_election.cc:343] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.341528 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:45.307332 18213 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.342744 18220 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:45.345204 18227 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:45.349325 18219 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.341146 18214 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.350051 18219 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.377761 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 95f78d7b0f7f422eacbfa07bbc248bfa (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:45.379577 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f78d7b0f7f422eacbfa07bbc248bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.386679 18213 ts_tablet_manager.cc:1434] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.140s	user 0.007s	sys 0.004s
I20260709 12:20:45.381502 18219 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:45.421800 18083 tablet_service.cc:1511] Processing CreateTablet for tablet 95f78d7b0f7f422eacbfa07bbc248bfa (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:45.430830 18215 tablet_bootstrap.cc:492] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:45.430830 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f78d7b0f7f422eacbfa07bbc248bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.431372 18214 ts_tablet_manager.cc:1434] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.212s	user 0.012s	sys 0.000s
I20260709 12:20:45.431639 18215 ts_tablet_manager.cc:1403] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.162s	user 0.031s	sys 0.027s
I20260709 12:20:45.432598 18214 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:45.435436 18215 raft_consensus.cc:359] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.436295 18215 raft_consensus.cc:385] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.436606 18215 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.437394 18215 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.439922 18215 ts_tablet_manager.cc:1434] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:45.427302 18159 tablet_service.cc:1511] Processing CreateTablet for tablet 9ba0c897fcb346488dfe124a973cb2d1 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:45.441074 18215 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:45.441710 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ba0c897fcb346488dfe124a973cb2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.449786 18215 tablet_bootstrap.cc:654] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.455535 18085 tablet_service.cc:1511] Processing CreateTablet for tablet a6ac73d7138d4200bbfcbf2f84c2c775 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:45.456207 18009 tablet_service.cc:1511] Processing CreateTablet for tablet a6ac73d7138d4200bbfcbf2f84c2c775 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:45.457145 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ac73d7138d4200bbfcbf2f84c2c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.426900 18213 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:45.462772 18232 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.463445 18232 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.466257 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ac73d7138d4200bbfcbf2f84c2c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.472882 18232 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:45.484762 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:45.485284 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:45.486232 18098 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:45.490483 18158 tablet_service.cc:1511] Processing CreateTablet for tablet 95f78d7b0f7f422eacbfa07bbc248bfa (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:45.492241 18158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f78d7b0f7f422eacbfa07bbc248bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.492408 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:45.494470 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:45.497188 18014 tablet_service.cc:1511] Processing CreateTablet for tablet 47bac54b53844de9814420f86103a6f8 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:45.498509 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:45.498817 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47bac54b53844de9814420f86103a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:45.488025 18136 leader_election.cc:343] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.500909 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:45.501976 18232 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:45.502446 18232 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:45.502929 18232 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:45.502381 18138 leader_election.cc:343] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:45.514744 17987 leader_election.cc:343] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.517293 18138 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:45.523396 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:45.524045 18089 tablet_service.cc:1511] Processing CreateTablet for tablet 47bac54b53844de9814420f86103a6f8 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:45.525527 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47bac54b53844de9814420f86103a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:45.526352 17989 leader_election.cc:343] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.526898 17989 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:45.529412 18227 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:45.530575 18008 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe4c9950f4d4f4588d08c7cd436b1dc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:45.531669 18214 tablet_bootstrap.cc:654] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.532011 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe4c9950f4d4f4588d08c7cd436b1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.534794 18232 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.536394 18215 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:45.536931 18215 ts_tablet_manager.cc:1403] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.096s	user 0.009s	sys 0.009s
I20260709 12:20:45.537184 18232 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:45.540453 18215 raft_consensus.cc:359] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:45.541216 18215 raft_consensus.cc:385] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.541509 18215 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.542351 18215 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:45.544804 18215 ts_tablet_manager.cc:1434] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
W20260709 12:20:45.548663 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46485.4us
W20260709 12:20:45.550016 18138 leader_election.cc:343] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.553691 18215 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:45.554047 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:45.554718 18098 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:45.558112 18232 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:45.563167 18162 tablet_service.cc:1511] Processing CreateTablet for tablet a6ac73d7138d4200bbfcbf2f84c2c775 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:45.565030 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ac73d7138d4200bbfcbf2f84c2c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.567638 18215 tablet_bootstrap.cc:654] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.570907 18213 tablet_bootstrap.cc:654] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.572296 18156 tablet_service.cc:1511] Processing CreateTablet for tablet 47bac54b53844de9814420f86103a6f8 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:45.574561 18156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47bac54b53844de9814420f86103a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.578567 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:45.580806 18138 leader_election.cc:343] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.592116 18155 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe4c9950f4d4f4588d08c7cd436b1dc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:45.594656 18155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe4c9950f4d4f4588d08c7cd436b1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.596447 18083 tablet_service.cc:1511] Processing CreateTablet for tablet aeba5cb4432947efbaa2fa537357e992 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:45.598208 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeba5cb4432947efbaa2fa537357e992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.602442 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe4c9950f4d4f4588d08c7cd436b1dc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:45.604094 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe4c9950f4d4f4588d08c7cd436b1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.622964 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 5c743e221c2c477e9b998557d9b9749a (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:45.624068 18161 tablet_service.cc:1511] Processing CreateTablet for tablet aeba5cb4432947efbaa2fa537357e992 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:45.624670 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c743e221c2c477e9b998557d9b9749a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.625578 18161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeba5cb4432947efbaa2fa537357e992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.628080 18098 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:45.629719 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:45.630276 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 5c743e221c2c477e9b998557d9b9749a (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:45.631278 18232 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:45.631858 18232 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:45.632578 18163 tablet_service.cc:1511] Processing CreateTablet for tablet 5c743e221c2c477e9b998557d9b9749a (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:45.632802 18232 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.634092 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c743e221c2c477e9b998557d9b9749a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.631865 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c743e221c2c477e9b998557d9b9749a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.638504 18080 tablet_service.cc:1511] Processing CreateTablet for tablet af9dff14df364074b9d5e7e769f198b5 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:45.640082 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9dff14df364074b9d5e7e769f198b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.647257 18014 tablet_service.cc:1511] Processing CreateTablet for tablet af9dff14df364074b9d5e7e769f198b5 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:45.648895 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9dff14df364074b9d5e7e769f198b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.654873 18009 tablet_service.cc:1511] Processing CreateTablet for tablet aeba5cb4432947efbaa2fa537357e992 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:45.656590 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeba5cb4432947efbaa2fa537357e992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.663331 18159 tablet_service.cc:1511] Processing CreateTablet for tablet af9dff14df364074b9d5e7e769f198b5 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:45.664891 18008 tablet_service.cc:1511] Processing CreateTablet for tablet a07f730a06794e2cb530edca57871763 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:45.667369 18160 tablet_service.cc:1511] Processing CreateTablet for tablet a07f730a06794e2cb530edca57871763 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:45.670310 18085 tablet_service.cc:1511] Processing CreateTablet for tablet a07f730a06794e2cb530edca57871763 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:45.667469 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07f730a06794e2cb530edca57871763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.675474 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07f730a06794e2cb530edca57871763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.682418 18214 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:45.683173 18214 ts_tablet_manager.cc:1403] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.251s	user 0.014s	sys 0.025s
I20260709 12:20:45.686491 18227 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.687175 18010 tablet_service.cc:1511] Processing CreateTablet for tablet f7a707b110a44c8c97606fdb8d7dece0 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:45.687100 18227 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:45.688761 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a707b110a44c8c97606fdb8d7dece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.694655 18215 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:45.695335 18215 ts_tablet_manager.cc:1403] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.142s	user 0.031s	sys 0.011s
I20260709 12:20:45.695230 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:45.695508 18227 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:45.696955 17987 leader_election.cc:343] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.699589 18214 raft_consensus.cc:359] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.700743 18214 raft_consensus.cc:385] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.701174 18214 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.703527 18215 raft_consensus.cc:359] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.704695 18215 raft_consensus.cc:385] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.705025 18215 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.715550 18160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07f730a06794e2cb530edca57871763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.716434 18088 tablet_service.cc:1511] Processing CreateTablet for tablet f7a707b110a44c8c97606fdb8d7dece0 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:45.716575 18157 tablet_service.cc:1511] Processing CreateTablet for tablet f7a707b110a44c8c97606fdb8d7dece0 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:45.718039 18157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a707b110a44c8c97606fdb8d7dece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.718039 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a707b110a44c8c97606fdb8d7dece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.736178 18006 tablet_service.cc:1511] Processing CreateTablet for tablet 555e520f5efe4bafb5947749ee16e734 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:45.737871 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 555e520f5efe4bafb5947749ee16e734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.739200 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9dff14df364074b9d5e7e769f198b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.739553 18213 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:45.740128 18213 ts_tablet_manager.cc:1403] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.321s	user 0.014s	sys 0.012s
I20260709 12:20:45.727578 18214 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.747828 18214 ts_tablet_manager.cc:1434] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.064s	user 0.009s	sys 0.000s
I20260709 12:20:45.748418 17921 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:45.751871 18214 tablet_bootstrap.cc:492] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:45.725910 18215 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.766682 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:45.770190 17989 leader_election.cc:343] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.770779 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:45.772626 18227 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:45.769528 18089 tablet_service.cc:1511] Processing CreateTablet for tablet 555e520f5efe4bafb5947749ee16e734 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:45.779824 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 555e520f5efe4bafb5947749ee16e734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.781169 18162 tablet_service.cc:1511] Processing CreateTablet for tablet 555e520f5efe4bafb5947749ee16e734 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:45.798301 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 555e520f5efe4bafb5947749ee16e734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.799551 18215 ts_tablet_manager.cc:1434] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.104s	user 0.009s	sys 0.006s
I20260709 12:20:45.800936 18215 tablet_bootstrap.cc:492] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:45.811755 18213 raft_consensus.cc:359] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:45.812739 18213 raft_consensus.cc:385] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.813088 18213 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.814225 18214 tablet_bootstrap.cc:654] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.817655 18213 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:45.823792 18213 ts_tablet_manager.cc:1434] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.083s	user 0.016s	sys 0.000s
I20260709 12:20:45.825199 18213 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:45.827241 18013 tablet_service.cc:1511] Processing CreateTablet for tablet b5eedbd3c223402caf9a71dac2fcaa98 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:45.828666 18215 tablet_bootstrap.cc:654] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.836733 18159 tablet_service.cc:1511] Processing CreateTablet for tablet b5eedbd3c223402caf9a71dac2fcaa98 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:45.838357 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5eedbd3c223402caf9a71dac2fcaa98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.847124 18014 tablet_service.cc:1511] Processing CreateTablet for tablet 17018f4f3f0d4fd5a11cac014a5abd36 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:45.831964 18227 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:45.850529 18227 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.853060 18227 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:45.854592 18214 tablet_bootstrap.cc:492] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:45.855171 18214 ts_tablet_manager.cc:1403] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.104s	user 0.007s	sys 0.005s
I20260709 12:20:45.855237 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:45.857285 17987 leader_election.cc:343] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.833003 18082 tablet_service.cc:1511] Processing CreateTablet for tablet b5eedbd3c223402caf9a71dac2fcaa98 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:45.828909 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5eedbd3c223402caf9a71dac2fcaa98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.859867 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:45.862244 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17018f4f3f0d4fd5a11cac014a5abd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.862931 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5eedbd3c223402caf9a71dac2fcaa98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.862741 18214 raft_consensus.cc:359] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:45.862174 17989 leader_election.cc:343] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:45.863565 18214 raft_consensus.cc:385] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.863950 18214 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.863732 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:45.870887 18227 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:45.874684 18214 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.878407 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 17018f4f3f0d4fd5a11cac014a5abd36 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:45.879592 18213 tablet_bootstrap.cc:654] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.880379 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17018f4f3f0d4fd5a11cac014a5abd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.877727 18214 ts_tablet_manager.cc:1434] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.004s
I20260709 12:20:45.890560 18214 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:45.891662 18215 tablet_bootstrap.cc:492] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:45.892438 18215 ts_tablet_manager.cc:1403] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.092s	user 0.016s	sys 0.003s
I20260709 12:20:45.897356 18215 raft_consensus.cc:359] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.898753 18215 raft_consensus.cc:385] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.899309 18215 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.900971 18159 tablet_service.cc:1511] Processing CreateTablet for tablet 17018f4f3f0d4fd5a11cac014a5abd36 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:45.900683 18215 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.902930 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17018f4f3f0d4fd5a11cac014a5abd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.907517 18214 tablet_bootstrap.cc:654] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.909128 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 2217b6329873485ead60419dc5acbff0 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:45.909798 18215 ts_tablet_manager.cc:1434] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.001s
I20260709 12:20:45.910743 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2217b6329873485ead60419dc5acbff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.910548 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 2217b6329873485ead60419dc5acbff0 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:45.911767 18215 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:45.912040 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2217b6329873485ead60419dc5acbff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.924888 18215 tablet_bootstrap.cc:654] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.929328 18213 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:45.930022 18213 ts_tablet_manager.cc:1403] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.105s	user 0.016s	sys 0.007s
I20260709 12:20:45.934333 18214 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:45.938643 18214 ts_tablet_manager.cc:1403] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.048s	user 0.017s	sys 0.002s
I20260709 12:20:45.933513 18213 raft_consensus.cc:359] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.942634 18213 raft_consensus.cc:385] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.943048 18213 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.944500 18215 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:45.945438 18215 ts_tablet_manager.cc:1403] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.034s	user 0.009s	sys 0.010s
I20260709 12:20:45.951171 18213 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.951525 18215 raft_consensus.cc:359] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.952394 18215 raft_consensus.cc:385] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.952739 18215 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.954094 18213 ts_tablet_manager.cc:1434] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.005s
I20260709 12:20:45.953734 18215 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:45.954519 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:45.955406 18213 tablet_bootstrap.cc:492] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:45.960114 18214 raft_consensus.cc:359] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.961066 18214 raft_consensus.cc:385] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:45.961434 18214 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:45.962250 18214 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:45.964838 18214 ts_tablet_manager.cc:1434] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.001s
I20260709 12:20:45.965472 18213 tablet_bootstrap.cc:654] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.967875 18214 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:45.956459 18215 ts_tablet_manager.cc:1434] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:45.970125 18215 tablet_bootstrap.cc:492] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:45.980361 18215 tablet_bootstrap.cc:654] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.981130 18214 tablet_bootstrap.cc:654] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:45.984787 18155 tablet_service.cc:1511] Processing CreateTablet for tablet 2217b6329873485ead60419dc5acbff0 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:45.986733 18155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2217b6329873485ead60419dc5acbff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.999881 18232 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.000310 18085 tablet_service.cc:1511] Processing CreateTablet for tablet 9cf56c3bba4a4641a72b648d02c669f6 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:46.000773 18232 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:46.018754 18232 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:46.019958 18227 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.020460 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.020593 18227 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.019960 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:46.022693 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:46.023106 18227 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.024981 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:46.025488 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 9cf56c3bba4a4641a72b648d02c669f6 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:46.027079 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf56c3bba4a4641a72b648d02c669f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.028337 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.029896 18006 tablet_service.cc:1511] Processing CreateTablet for tablet e84e056b63c34d8da15d2fde3877279b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:46.030741 18160 tablet_service.cc:1511] Processing CreateTablet for tablet 9cf56c3bba4a4641a72b648d02c669f6 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:46.031374 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84e056b63c34d8da15d2fde3877279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.032130 18160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf56c3bba4a4641a72b648d02c669f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.034113 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf56c3bba4a4641a72b648d02c669f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:46.034916 18136 leader_election.cc:343] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.025857 18173 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:46.036377 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.039666 18232 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:46.047614 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.051178 18213 tablet_bootstrap.cc:492] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:46.051986 18213 ts_tablet_manager.cc:1403] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.097s	user 0.011s	sys 0.019s
I20260709 12:20:46.052353 18227 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:46.050267 18232 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:46.052877 18227 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:46.053169 18232 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.053294 18227 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.054889 18088 tablet_service.cc:1511] Processing CreateTablet for tablet 5c6e41c6a46d4fc99220183ed20b6921 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:46.056499 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6e41c6a46d4fc99220183ed20b6921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.056044 18213 raft_consensus.cc:359] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.057008 18213 raft_consensus.cc:385] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.057389 18213 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.058362 18213 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.061390 18213 ts_tablet_manager.cc:1434] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:20:46.061662 18162 tablet_service.cc:1511] Processing CreateTablet for tablet e84e056b63c34d8da15d2fde3877279b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:46.063019 18213 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:46.063383 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84e056b63c34d8da15d2fde3877279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.064863 18215 tablet_bootstrap.cc:492] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:46.065500 18215 ts_tablet_manager.cc:1403] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.096s	user 0.011s	sys 0.017s
I20260709 12:20:46.052287 18087 tablet_service.cc:1511] Processing CreateTablet for tablet e84e056b63c34d8da15d2fde3877279b (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:46.066862 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84e056b63c34d8da15d2fde3877279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.069047 18215 raft_consensus.cc:359] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:46.071012 17987 leader_election.cc:343] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.070631 18227 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.071695 18213 tablet_bootstrap.cc:654] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:46.070844 18215 raft_consensus.cc:385] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.075546 18215 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.076093 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:46.076478 18227 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.076972 18173 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.077520 18215 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.080036 18215 ts_tablet_manager.cc:1434] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.004s
I20260709 12:20:46.081239 18215 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:46.082840 18220 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.083673 18220 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.088321 18220 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.093796 18227 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.094481 18227 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.095049 18009 tablet_service.cc:1511] Processing CreateTablet for tablet 5c6e41c6a46d4fc99220183ed20b6921 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:46.097270 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6e41c6a46d4fc99220183ed20b6921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.098935 18227 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.116117 18080 tablet_service.cc:1511] Processing CreateTablet for tablet 85764dcebf974f7cbec322be36e3dd5f (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:46.117951 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85764dcebf974f7cbec322be36e3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.090404 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:46.099229 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.101696 18214 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:46.120968 18214 ts_tablet_manager.cc:1403] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.153s	user 0.033s	sys 0.023s
W20260709 12:20:46.121577 17987 leader_election.cc:343] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.121356 18163 tablet_service.cc:1511] Processing CreateTablet for tablet acbc4a634f954522904f2b865531f456 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:46.123104 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acbc4a634f954522904f2b865531f456. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:46.123059 17987 leader_election.cc:343] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.124447 18214 raft_consensus.cc:359] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.125286 18214 raft_consensus.cc:385] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.125592 18214 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.126354 18023 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.126869 18214 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.130363 18161 tablet_service.cc:1511] Processing CreateTablet for tablet 85764dcebf974f7cbec322be36e3dd5f (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:46.132123 18161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85764dcebf974f7cbec322be36e3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.133870 18214 ts_tablet_manager.cc:1434] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:20:46.133746 18008 tablet_service.cc:1511] Processing CreateTablet for tablet f0dc8b13a3294835a091ca099ee49b29 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:46.102568 18014 tablet_service.cc:1511] Processing CreateTablet for tablet acbc4a634f954522904f2b865531f456 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:46.135576 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0dc8b13a3294835a091ca099ee49b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.104099 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:46.137698 18064 leader_election.cc:343] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.138942 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acbc4a634f954522904f2b865531f456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.104677 18157 tablet_service.cc:1511] Processing CreateTablet for tablet 5c6e41c6a46d4fc99220183ed20b6921 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:46.105317 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:46.141453 18083 tablet_service.cc:1511] Processing CreateTablet for tablet f0dc8b13a3294835a091ca099ee49b29 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
W20260709 12:20:46.142236 18064 leader_election.cc:343] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.142174 18219 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:46.143128 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0dc8b13a3294835a091ca099ee49b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.144146 18173 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:46.145506 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:46.146234 18157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6e41c6a46d4fc99220183ed20b6921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.146993 18227 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:46.142904 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:46.148227 18220 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:46.112623 18010 tablet_service.cc:1511] Processing CreateTablet for tablet 85764dcebf974f7cbec322be36e3dd5f (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:46.159730 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85764dcebf974f7cbec322be36e3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.098855 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:46.164134 17989 leader_election.cc:343] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.164489 18214 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:46.164650 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:46.102082 18232 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:46.177153 18215 tablet_bootstrap.cc:654] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:46.107867 18089 tablet_service.cc:1511] Processing CreateTablet for tablet acbc4a634f954522904f2b865531f456 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:46.185863 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acbc4a634f954522904f2b865531f456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.186327 18213 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:46.186980 18213 ts_tablet_manager.cc:1403] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.124s	user 0.020s	sys 0.039s
I20260709 12:20:46.188860 18219 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.189436 18219 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.179409 18227 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:46.203824 18219 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:46.205767 18220 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.208547 18156 tablet_service.cc:1511] Processing CreateTablet for tablet 3304247386d14b18a8bdd4f25ae8630c (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:46.228170 18220 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.209892 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 3304247386d14b18a8bdd4f25ae8630c (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:46.229279 18156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3304247386d14b18a8bdd4f25ae8630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.230089 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3304247386d14b18a8bdd4f25ae8630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.209105 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:46.231209 18024 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:46.205523 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.234237 18256 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:46.232913 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:46.234302 18227 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.235203 18172 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:46.232899 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.256507 18162 tablet_service.cc:1511] Processing CreateTablet for tablet f0dc8b13a3294835a091ca099ee49b29 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:46.258200 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0dc8b13a3294835a091ca099ee49b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.267424 18232 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:46.268173 18219 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:46.268623 18219 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:46.268990 18219 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:46.247953 18136 leader_election.cc:343] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.273564 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:46.203992 18213 raft_consensus.cc:359] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.276170 18213 raft_consensus.cc:385] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.276541 18213 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.246114 18220 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.278321 18220 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:46.278728 18220 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:46.279078 18220 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.287810 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 52abd819c1c443e28c6c41a392a7ed6f (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:46.288570 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:46.289513 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52abd819c1c443e28c6c41a392a7ed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.300194 18261 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.301105 18261 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.304644 18261 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.277434 18213 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.327944 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:46.328795 18024 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.343016 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:46.344852 18160 tablet_service.cc:1511] Processing CreateTablet for tablet 52abd819c1c443e28c6c41a392a7ed6f (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:46.344868 18006 tablet_service.cc:1511] Processing CreateTablet for tablet f1f3da088a314a87a78898050dab39e9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:46.346668 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f3da088a314a87a78898050dab39e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.346530 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 52abd819c1c443e28c6c41a392a7ed6f (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:46.346668 18160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52abd819c1c443e28c6c41a392a7ed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.351490 18162 tablet_service.cc:1511] Processing CreateTablet for tablet f1f3da088a314a87a78898050dab39e9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:46.351490 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52abd819c1c443e28c6c41a392a7ed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.352957 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 7cd93736a3314dbbb5c62b5af8fbed11 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:46.354979 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd93736a3314dbbb5c62b5af8fbed11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.362478 18087 tablet_service.cc:1511] Processing CreateTablet for tablet f1f3da088a314a87a78898050dab39e9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:46.364187 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f3da088a314a87a78898050dab39e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.378825 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f3da088a314a87a78898050dab39e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.387210 18213 ts_tablet_manager.cc:1434] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.200s	user 0.005s	sys 0.003s
I20260709 12:20:46.388593 18213 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
W20260709 12:20:46.329969 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56523.3us
I20260709 12:20:46.395300 18258 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.322175 18012 tablet_service.cc:1511] Processing CreateTablet for tablet 3304247386d14b18a8bdd4f25ae8630c (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:46.408613 18009 tablet_service.cc:1511] Processing CreateTablet for tablet 7cd93736a3314dbbb5c62b5af8fbed11 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:46.413193 18256 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.453931 18258 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.453931 18256 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.477774 18214 tablet_bootstrap.cc:654] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:46.480316 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 316212us
I20260709 12:20:46.483111 18258 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.486073 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.486858 18099 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
W20260709 12:20:46.487699 18064 leader_election.cc:343] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.487778 18219 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.489378 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.491827 18213 tablet_bootstrap.cc:654] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:46.491676 18089 tablet_service.cc:1511] Processing CreateTablet for tablet 49a68a064bb44579af6ee5ad50121ffc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:46.483467 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260709 12:20:46.466660 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 130718us
I20260709 12:20:46.504890 18011 tablet_service.cc:1511] Processing CreateTablet for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:46.506830 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:46.507232 18005 tablet_service.cc:1511] Processing CreateTablet for tablet 49a68a064bb44579af6ee5ad50121ffc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:46.509059 18159 tablet_service.cc:1511] Processing CreateTablet for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:46.512859 18163 tablet_service.cc:1511] Processing CreateTablet for tablet 49a68a064bb44579af6ee5ad50121ffc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:46.514865 18007 tablet_service.cc:1511] Processing CreateTablet for tablet 583d61c844f048e4b53a66acc88cc1d9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:46.515987 18083 tablet_service.cc:1511] Processing CreateTablet for tablet 583d61c844f048e4b53a66acc88cc1d9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:46.519732 18256 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.520946 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.524036 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:46.524375 18012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3304247386d14b18a8bdd4f25ae8630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.521252 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.524825 18023 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:46.526634 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.528510 18215 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:46.529175 18215 ts_tablet_manager.cc:1403] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.448s	user 0.039s	sys 0.059s
I20260709 12:20:46.522624 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:46.530361 18024 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.530256 18260 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:46.534179 18260 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:46.534591 18260 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.533730 18011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.539526 18005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49a68a064bb44579af6ee5ad50121ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.541263 18261 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.528987 17921 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:46.541985 18261 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.549886 18215 raft_consensus.cc:359] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.550892 18215 raft_consensus.cc:385] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.542871 18007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583d61c844f048e4b53a66acc88cc1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.552243 18261 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.551216 18215 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.553887 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd93736a3314dbbb5c62b5af8fbed11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.555552 18232 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:46.493420 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49a68a064bb44579af6ee5ad50121ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.563310 18215 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.565982 18162 tablet_service.cc:1511] Processing CreateTablet for tablet 7cd93736a3314dbbb5c62b5af8fbed11 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:46.551191 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.570489 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:46.557603 18232 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:46.495056 18258 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:46.578027 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:46.582623 18215 ts_tablet_manager.cc:1434] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.053s	user 0.011s	sys 0.000s
I20260709 12:20:46.583856 18215 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:46.580523 18258 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:46.594089 18258 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.600243 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49a68a064bb44579af6ee5ad50121ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.602069 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583d61c844f048e4b53a66acc88cc1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.602684 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:46.606222 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.608319 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd93736a3314dbbb5c62b5af8fbed11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.583382 18232 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:46.625249 18214 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:46.631729 18215 tablet_bootstrap.cc:654] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:46.632937 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 0420fe6ea0ab490e922b7c35c62b6a19 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:46.634333 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0420fe6ea0ab490e922b7c35c62b6a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.616926 18080 tablet_service.cc:1511] Processing CreateTablet for tablet da76d5c707544d29a2f74d6015ec1cfc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:46.639087 18232 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:46.641713 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:46.643090 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:46.590734 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69595.7us
I20260709 12:20:46.606165 18219 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:46.637727 18214 ts_tablet_manager.cc:1403] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.473s	user 0.014s	sys 0.045s
I20260709 12:20:46.656239 18173 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:46.656544 18263 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:46.650171 17989 leader_election.cc:343] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.656983 18260 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.675115 18010 tablet_service.cc:1511] Processing CreateTablet for tablet ef30e67b67894670b0cf3980d6b0f624 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:46.675890 18260 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.676824 18010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef30e67b67894670b0cf3980d6b0f624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.658767 18014 tablet_service.cc:1511] Processing CreateTablet for tablet da76d5c707544d29a2f74d6015ec1cfc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:46.678364 18014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da76d5c707544d29a2f74d6015ec1cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.678671 18157 tablet_service.cc:1511] Processing CreateTablet for tablet da76d5c707544d29a2f74d6015ec1cfc (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:46.679296 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:46.679687 18013 tablet_service.cc:1511] Processing CreateTablet for tablet df3d21a581ef40feafcce4ab8b817645 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:46.680305 18157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da76d5c707544d29a2f74d6015ec1cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.681111 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d21a581ef40feafcce4ab8b817645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.659474 18024 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:46.671087 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da76d5c707544d29a2f74d6015ec1cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.672413 17989 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:46.673146 18263 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.680081 18022 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:46.636190 18136 leader_election.cc:343] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:46.696811 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68556.5us
I20260709 12:20:46.698177 18263 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:46.699684 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:46.696439 18161 tablet_service.cc:1511] Processing CreateTablet for tablet 583d61c844f048e4b53a66acc88cc1d9 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:46.701840 18161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583d61c844f048e4b53a66acc88cc1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.708093 18213 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:46.711362 18082 tablet_service.cc:1511] Processing CreateTablet for tablet ef30e67b67894670b0cf3980d6b0f624 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:46.713258 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef30e67b67894670b0cf3980d6b0f624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.716563 18213 ts_tablet_manager.cc:1403] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.328s	user 0.006s	sys 0.022s
I20260709 12:20:46.649298 18220 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.663012 18258 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.736992 18215 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:46.738137 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:46.742868 18214 raft_consensus.cc:359] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.746903 18088 tablet_service.cc:1511] Processing CreateTablet for tablet df3d21a581ef40feafcce4ab8b817645 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:46.748831 18085 tablet_service.cc:1511] Processing CreateTablet for tablet ff26d2026f1c4ff79de028b875b8ba96 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:46.749002 18213 raft_consensus.cc:359] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.717355 18172 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:46.759708 18215 ts_tablet_manager.cc:1403] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.176s	user 0.045s	sys 0.028s
I20260709 12:20:46.730711 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:46.769234 18214 raft_consensus.cc:385] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.783288 18155 tablet_service.cc:1511] Processing CreateTablet for tablet 0420fe6ea0ab490e922b7c35c62b6a19 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:46.784951 18155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0420fe6ea0ab490e922b7c35c62b6a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.787367 18258 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:46.789943 18136 leader_election.cc:343] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:46.796757 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 70685.2us
I20260709 12:20:46.774380 18260 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.799299 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:46.801932 18158 tablet_service.cc:1511] Processing CreateTablet for tablet ff26d2026f1c4ff79de028b875b8ba96 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:46.780944 18214 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.804435 18220 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:46.804658 18162 tablet_service.cc:1511] Processing CreateTablet for tablet df3d21a581ef40feafcce4ab8b817645 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:46.793519 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d21a581ef40feafcce4ab8b817645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.777560 18213 raft_consensus.cc:385] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.809616 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d21a581ef40feafcce4ab8b817645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.811949 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:46.812786 18098 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.814592 18215 raft_consensus.cc:359] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.815433 18215 raft_consensus.cc:385] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:46.815728 18215 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.825333 18215 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.818682 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff26d2026f1c4ff79de028b875b8ba96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.831902 18215 ts_tablet_manager.cc:1434] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.002s
I20260709 12:20:46.810496 18213 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:46.837909 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:46.818008 18008 tablet_service.cc:1511] Processing CreateTablet for tablet 0420fe6ea0ab490e922b7c35c62b6a19 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:46.840106 18008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0420fe6ea0ab490e922b7c35c62b6a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.842507 18213 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.845130 18213 ts_tablet_manager.cc:1434] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.119s	user 0.007s	sys 0.000s
I20260709 12:20:46.805117 18214 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.831003 17922 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:46.818691 18268 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:46.851744 18215 tablet_bootstrap.cc:492] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:46.851842 18013 tablet_service.cc:1511] Processing CreateTablet for tablet 8e0ff0e5252f4ace898a73a2268e5dc8 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:46.817723 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:46.864171 18097 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:46.804522 18154 tablet_service.cc:1511] Processing CreateTablet for tablet 8e0ff0e5252f4ace898a73a2268e5dc8 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:46.850986 18158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff26d2026f1c4ff79de028b875b8ba96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.866792 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:46.867163 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:46.867576 18173 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:46.828634 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:46.866781 18156 tablet_service.cc:1511] Processing CreateTablet for tablet ef30e67b67894670b0cf3980d6b0f624 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:46.872535 18156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef30e67b67894670b0cf3980d6b0f624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.873070 18215 tablet_bootstrap.cc:654] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:46.857589 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 23e4eb0b942f4c7e81dfe76f4e015618 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:46.858867 18012 tablet_service.cc:1511] Processing CreateTablet for tablet ff26d2026f1c4ff79de028b875b8ba96 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:46.859702 18009 tablet_service.cc:1511] Processing CreateTablet for tablet 23e4eb0b942f4c7e81dfe76f4e015618 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:46.860668 18268 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:46.881408 18268 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:46.884223 18064 leader_election.cc:343] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:46.862269 18214 ts_tablet_manager.cc:1434] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.195s	user 0.004s	sys 0.006s
I20260709 12:20:46.819501 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 8e0ff0e5252f4ace898a73a2268e5dc8 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:46.885965 18214 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:46.888933 18154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ff0e5252f4ace898a73a2268e5dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.856432 18013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ff0e5252f4ace898a73a2268e5dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.886004 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0ff0e5252f4ace898a73a2268e5dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.908684 18213 tablet_bootstrap.cc:492] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:46.923085 18098 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:46.935662 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:46.910211 18012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff26d2026f1c4ff79de028b875b8ba96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.936429 18011 tablet_service.cc:1511] Processing CreateTablet for tablet 5f144ec392b043d8b68e284ffecdee43 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:46.937390 18009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e4eb0b942f4c7e81dfe76f4e015618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.938850 18011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f144ec392b043d8b68e284ffecdee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.909885 18258 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:46.943008 18258 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:46.907406 18256 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:46.959770 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e4eb0b942f4c7e81dfe76f4e015618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:46.904480 18138 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:46.998023 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106093us
I20260709 12:20:46.998236 18268 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.000962 18213 tablet_bootstrap.cc:654] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.002296 18157 tablet_service.cc:1511] Processing CreateTablet for tablet 23e4eb0b942f4c7e81dfe76f4e015618 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:47.003958 18157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e4eb0b942f4c7e81dfe76f4e015618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:47.027362 18022 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:47.015121 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:47.014050 18220 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.028784 18220 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.029454 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:47.031158 18220 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.031728 18220 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.032157 18220 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:47.035115 18258 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.038360 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:47.039191 18098 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:47.040705 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:47.042209 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.050184 18256 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:47.050720 18256 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.051100 18256 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.053177 18277 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.053875 18277 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:47.054885 18215 tablet_bootstrap.cc:492] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:47.054800 18277 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.055429 18215 ts_tablet_manager.cc:1403] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.221s	user 0.013s	sys 0.029s
I20260709 12:20:47.056742 18258 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.059839 18258 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.060596 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:47.061551 18083 tablet_service.cc:1511] Processing CreateTablet for tablet 86bfc0ef6acd420b829a9a1df3376f48 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:47.062072 18261 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.063274 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86bfc0ef6acd420b829a9a1df3376f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:47.066962 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:47.067065 18159 tablet_service.cc:1511] Processing CreateTablet for tablet 5f144ec392b043d8b68e284ffecdee43 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:47.068774 18159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f144ec392b043d8b68e284ffecdee43. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:47.069041 17987 leader_election.cc:343] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:47.069237 17989 leader_election.cc:343] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.070040 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.071749 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 5f144ec392b043d8b68e284ffecdee43 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:47.072688 18214 tablet_bootstrap.cc:654] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.073278 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f144ec392b043d8b68e284ffecdee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:47.043043 18173 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:47.044960 18220 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.078018 18260 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.094410 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:47.095675 18260 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:47.095399 18006 tablet_service.cc:1511] Processing CreateTablet for tablet 86bfc0ef6acd420b829a9a1df3376f48 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:47.096194 18260 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.096032 18215 raft_consensus.cc:359] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.096634 18260 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.097106 18006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86bfc0ef6acd420b829a9a1df3376f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:47.096931 18215 raft_consensus.cc:385] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.099651 18215 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.100346 18215 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.108161 18163 tablet_service.cc:1511] Processing CreateTablet for tablet 86bfc0ef6acd420b829a9a1df3376f48 (DEFAULT_TABLE table=test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:47.109930 18163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86bfc0ef6acd420b829a9a1df3376f48. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:47.119210 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48721.5us
W20260709 12:20:47.108822 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:47.101413 18261 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:47.123942 18258 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.125960 18213 tablet_bootstrap.cc:492] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:47.192217 18213 ts_tablet_manager.cc:1403] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.302s	user 0.009s	sys 0.009s
I20260709 12:20:47.149224 17922 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.194841 18215 ts_tablet_manager.cc:1434] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.139s	user 0.014s	sys 0.059s
I20260709 12:20:47.196095 18215 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:47.167512 18268 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.142954 18261 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.208674 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.209128 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.209442 18173 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
W20260709 12:20:47.179133 18136 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:47.191386 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 90895.9us
I20260709 12:20:47.168988 18256 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.223605 18022 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:47.226171 18213 raft_consensus.cc:359] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.227586 18213 raft_consensus.cc:385] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.228070 18213 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.149087 18260 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.125033 18285 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.092s
I20260709 12:20:47.229321 18213 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.244419 18213 ts_tablet_manager.cc:1434] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.052s	user 0.010s	sys 0.001s
I20260709 12:20:47.244571 18282 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260709 12:20:47.260615 17989 leader_election.cc:343] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.274051 18268 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:47.274669 17922 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.263229 18214 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:47.279186 18214 ts_tablet_manager.cc:1403] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.394s	user 0.033s	sys 0.003s
I20260709 12:20:47.271024 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:47.282447 18099 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:47.271572 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:47.283710 18022 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:47.273847 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:47.284780 18172 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.261624 18256 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.289050 18214 raft_consensus.cc:359] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:47.289795 18214 raft_consensus.cc:385] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.290130 18214 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.296707 18215 tablet_bootstrap.cc:654] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.290946 18214 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:20:47.300683 18214 ts_tablet_manager.cc:1434] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.004s
I20260709 12:20:47.287261 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:47.287876 18260 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.312242 18213 tablet_bootstrap.cc:492] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:47.311640 18256 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:47.313645 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53087.1us
I20260709 12:20:47.322510 18213 tablet_bootstrap.cc:654] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.324883 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:47.331775 17989 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:47.333529 18256 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.341595 17922 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.356222 18214 tablet_bootstrap.cc:492] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:47.356751 18172 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:47.371471 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:47.375830 18261 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.376547 18261 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:47.381193 17921 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.389611 18214 tablet_bootstrap.cc:654] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.394663 18256 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.395901 18256 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.401253 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:47.403393 17987 leader_election.cc:343] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.406656 18256 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.407982 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:47.410866 17989 leader_election.cc:343] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.411609 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:47.413969 18256 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.432790 18215 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:47.433439 18215 ts_tablet_manager.cc:1403] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.238s	user 0.009s	sys 0.005s
I20260709 12:20:47.444005 18215 raft_consensus.cc:359] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.444976 18215 raft_consensus.cc:385] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.445328 18215 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.392931 18261 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.446235 18215 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.451300 18215 ts_tablet_manager.cc:1434] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.001s
I20260709 12:20:47.452898 18215 tablet_bootstrap.cc:492] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:47.464048 18215 tablet_bootstrap.cc:654] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.470404 18260 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:47.471819 18213 tablet_bootstrap.cc:492] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:47.472654 18213 ts_tablet_manager.cc:1403] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.161s	user 0.044s	sys 0.018s
I20260709 12:20:47.476838 18213 raft_consensus.cc:359] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.478340 18213 raft_consensus.cc:385] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.478569 18213 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.480688 18213 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.483958 18022 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.484038 18213 ts_tablet_manager.cc:1434] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20260709 12:20:47.486327 18213 tablet_bootstrap.cc:492] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:47.486327 18173 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:47.493010 18136 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:47.496215 18214 tablet_bootstrap.cc:492] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:47.496974 18214 ts_tablet_manager.cc:1403] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.141s	user 0.018s	sys 0.028s
I20260709 12:20:47.502004 18260 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:47.505673 18213 tablet_bootstrap.cc:654] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.516489 18214 raft_consensus.cc:359] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.517308 18214 raft_consensus.cc:385] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.517558 18214 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.519999 18256 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:47.521000 18214 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.522962 18232 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:47.523626 18214 ts_tablet_manager.cc:1434] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:20:47.525357 18214 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:47.533373 18232 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.534076 18232 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.538196 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.539340 18024 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:47.548840 18232 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:47.558314 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:47.557444 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:47.559464 18256 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:47.559669 18260 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.560467 18260 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
W20260709 12:20:47.561234 18136 leader_election.cc:343] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.561600 18232 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:47.562043 18232 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.562426 18232 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.566295 18260 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:20:47.572221 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.578116 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.583098 18215 tablet_bootstrap.cc:492] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:47.583639 18215 ts_tablet_manager.cc:1403] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.131s	user 0.012s	sys 0.008s
I20260709 12:20:47.589306 18064 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:47.590428 18220 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.591185 18214 tablet_bootstrap.cc:654] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.595259 18213 tablet_bootstrap.cc:492] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:47.595908 18213 ts_tablet_manager.cc:1403] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.110s	user 0.021s	sys 0.008s
I20260709 12:20:47.615934 17922 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.633037 18220 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260709 12:20:47.656576 18213 raft_consensus.cc:359] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.656446 18215 raft_consensus.cc:359] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.657707 18215 raft_consensus.cc:385] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.658095 18215 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.657367 18213 raft_consensus.cc:385] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.658797 18213 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.659920 18213 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:47.666172 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43645us
I20260709 12:20:47.667951 18215 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.671620 18215 ts_tablet_manager.cc:1434] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.088s	user 0.026s	sys 0.027s
I20260709 12:20:47.678046 18213 ts_tablet_manager.cc:1434] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.081s	user 0.011s	sys 0.000s
I20260709 12:20:47.679662 18213 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:47.679735 18215 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:47.685807 18098 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.700506 18213 tablet_bootstrap.cc:654] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.703163 18232 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.705991 18232 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:47.708052 18214 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:47.708596 18214 ts_tablet_manager.cc:1403] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.184s	user 0.013s	sys 0.006s
I20260709 12:20:47.709640 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:47.715484 18214 raft_consensus.cc:359] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.716182 18214 raft_consensus.cc:385] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.716460 18214 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.711125 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:47.711802 18024 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:47.719511 18136 leader_election.cc:343] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.723785 18220 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:47.717235 18214 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.735348 18024 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:47.737285 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:47.738626 18232 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.738924 18215 tablet_bootstrap.cc:654] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.739198 18232 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:47.740811 18214 ts_tablet_manager.cc:1434] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.000s
I20260709 12:20:47.742265 18214 tablet_bootstrap.cc:492] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:47.743846 18309 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.743777 18232 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.744550 18309 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.748720 18309 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:47.748998 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:47.750834 18136 leader_election.cc:343] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.753937 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.754972 18024 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:47.756443 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:47.757452 18309 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:47.757862 18309 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.758209 18309 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.761124 18214 tablet_bootstrap.cc:654] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.775302 18214 tablet_bootstrap.cc:492] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:47.775904 18214 ts_tablet_manager.cc:1403] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.007s
I20260709 12:20:47.779294 18214 raft_consensus.cc:359] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.780085 18214 raft_consensus.cc:385] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.780419 18214 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.781347 18214 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.784456 18214 ts_tablet_manager.cc:1434] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.003s
I20260709 12:20:47.785605 18214 tablet_bootstrap.cc:492] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:47.792923 18214 tablet_bootstrap.cc:654] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.792829 18220 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.793540 18220 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.797605 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.802928 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.805272 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:47.805962 18214 tablet_bootstrap.cc:492] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:47.806250 18260 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.806596 18214 ts_tablet_manager.cc:1403] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.014s
I20260709 12:20:47.806262 18220 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.808066 18215 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:47.808663 18215 ts_tablet_manager.cc:1403] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.129s	user 0.010s	sys 0.003s
I20260709 12:20:47.810824 18214 raft_consensus.cc:359] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.811759 18215 raft_consensus.cc:359] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.811712 18214 raft_consensus.cc:385] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.812922 18214 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
W20260709 12:20:47.813436 18315 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:20:47.813982 18215 raft_consensus.cc:385] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.814073 18214 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.814666 18215 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.815531 18215 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.816692 18214 ts_tablet_manager.cc:1434] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:20:47.817911 18214 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:47.818006 18220 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.818522 18220 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.821987 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.823016 18022 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:47.823570 18309 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.825727 18215 ts_tablet_manager.cc:1434] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.002s
I20260709 12:20:47.828061 18309 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:47.829640 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:47.830452 18098 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.831858 18213 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:47.832470 18213 ts_tablet_manager.cc:1403] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.153s	user 0.015s	sys 0.003s
I20260709 12:20:47.833961 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:47.834715 18022 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.836759 18213 raft_consensus.cc:359] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.812695 17921 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.837579 18213 raft_consensus.cc:385] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.837939 18213 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.832501 18214 tablet_bootstrap.cc:654] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.839540 18098 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:47.840678 18215 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:47.841337 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:47.842664 18309 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.843214 18309 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:47.843600 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.844162 18309 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.845517 18260 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.845933 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:47.846320 18260 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.847329 18283 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.849440 18260 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.850760 18022 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:47.842658 18213 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.855674 18213 ts_tablet_manager.cc:1434] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.002s
I20260709 12:20:47.856855 18213 tablet_bootstrap.cc:492] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:47.857498 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:47.858387 18024 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:47.861868 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:47.864126 18064 leader_election.cc:343] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.867739 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:47.870926 18260 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:47.871208 18220 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.871567 18260 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:47.871990 18260 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.870011 18214 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:47.873389 18214 ts_tablet_manager.cc:1403] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.056s	user 0.027s	sys 0.010s
I20260709 12:20:47.872870 18213 tablet_bootstrap.cc:654] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.875972 18215 tablet_bootstrap.cc:654] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.866616 18283 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:47.876655 18214 raft_consensus.cc:359] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.882732 18214 raft_consensus.cc:385] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.883373 18214 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.884524 18214 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.887930 18214 ts_tablet_manager.cc:1434] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.002s
I20260709 12:20:47.889210 18214 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:47.890357 18215 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:47.890981 18215 ts_tablet_manager.cc:1403] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.051s	user 0.017s	sys 0.003s
I20260709 12:20:47.896013 18215 raft_consensus.cc:359] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.896967 18215 raft_consensus.cc:385] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.897336 18215 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.898250 18215 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.896605 18214 tablet_bootstrap.cc:654] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.903177 18260 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.904023 18215 ts_tablet_manager.cc:1434] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:20:47.905177 18215 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:47.906682 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:47.907475 18024 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:47.908718 17921 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.917809 18215 tablet_bootstrap.cc:654] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:47.924275 18024 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:47.926568 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:47.928951 18283 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:47.929737 18283 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:47.931561 18215 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:47.931098 18283 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.932149 18215 ts_tablet_manager.cc:1403] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.003s
I20260709 12:20:47.936106 18215 raft_consensus.cc:359] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.936990 18215 raft_consensus.cc:385] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.937304 18215 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.939761 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:20:47.942420 18064 leader_election.cc:343] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:47.946007 18215 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.948639 18215 ts_tablet_manager.cc:1434] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:20:47.951983 18260 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.960085 18220 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:47.964720 18213 tablet_bootstrap.cc:492] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:47.965358 18213 ts_tablet_manager.cc:1403] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.109s	user 0.033s	sys 0.024s
I20260709 12:20:47.966910 18215 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:47.968809 18213 raft_consensus.cc:359] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.970041 18213 raft_consensus.cc:385] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:47.970451 18213 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:47.971592 18213 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.974485 18213 ts_tablet_manager.cc:1434] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.003s
I20260709 12:20:47.975648 18213 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:47.976758 18256 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:47.977406 18256 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:47.986495 18256 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:47.988677 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:47.986716 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:47.990964 18098 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
W20260709 12:20:47.994529 17989 leader_election.cc:343] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.004907 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:48.006177 18256 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.006661 18256 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.007086 18256 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.010753 18213 tablet_bootstrap.cc:654] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.013202 17921 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.021353 18214 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.022006 18214 ts_tablet_manager.cc:1403] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.133s	user 0.044s	sys 0.021s
I20260709 12:20:48.023595 18215 tablet_bootstrap.cc:654] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.034106 18309 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.034963 18309 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.037782 18098 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.042223 18283 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.047207 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.048080 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:48.052175 18213 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.052963 18213 ts_tablet_manager.cc:1403] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.078s	user 0.009s	sys 0.003s
I20260709 12:20:48.055621 18214 raft_consensus.cc:359] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.056761 18214 raft_consensus.cc:385] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.057143 18214 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.054551 18277 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.058509 18214 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.059686 18277 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.062675 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:48.063691 18268 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.064155 18268 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.064531 18268 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.066493 18277 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.067539 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.069299 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:48.072144 17989 leader_election.cc:343] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.073940 18256 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.076403 18256 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:48.082698 17987 leader_election.cc:343] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.083370 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:48.083217 18213 raft_consensus.cc:359] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.083997 18213 raft_consensus.cc:385] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.084278 18213 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.085011 18213 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.085788 18283 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.087885 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:48.088569 18099 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.088902 18213 ts_tablet_manager.cc:1434] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.036s	user 0.009s	sys 0.000s
I20260709 12:20:48.092015 18213 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:48.087898 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:48.086635 18283 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.098032 18256 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:48.098604 17989 leader_election.cc:343] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.100560 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.101636 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.117301 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.105906 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:48.102022 18309 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.119513 18097 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:48.123608 18064 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:48.126668 18261 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.130061 18261 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.130630 18261 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.137142 18214 ts_tablet_manager.cc:1434] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.115s	user 0.010s	sys 0.001s
I20260709 12:20:48.139261 18268 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.144330 18214 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.136905 18283 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.155867 18268 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.159192 18283 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.160072 18283 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.162374 18171 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:48.164422 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:48.166194 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.171485 18024 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.181201 18099 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:48.191968 18283 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.191968 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:48.193558 18256 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:48.194144 18256 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:48.194387 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.195089 18256 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.195403 18023 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.197799 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.198889 18172 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.220552 18215 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:48.223299 18215 ts_tablet_manager.cc:1403] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.257s	user 0.035s	sys 0.086s
I20260709 12:20:48.226400 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:48.228416 18260 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.228907 18260 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.229324 18260 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.229434 18213 tablet_bootstrap.cc:654] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:48.224068 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:48.228076 18261 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.236967 18096 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:48.239492 18215 raft_consensus.cc:359] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.240458 18215 raft_consensus.cc:385] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.240795 18215 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.241559 18215 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.242880 18214 tablet_bootstrap.cc:654] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.246364 18215 ts_tablet_manager.cc:1434] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260709 12:20:48.247272 18215 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:48.252610 18261 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.253093 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:48.253885 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.254570 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:48.255110 18023 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:48.255419 18172 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:48.254593 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:48.257162 18263 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:48.257718 18263 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:48.258806 18064 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.258740 18263 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.260175 18261 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.246990 18260 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.263558 18260 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.264635 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.268230 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:48.270252 18172 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.273471 18309 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.284004 18215 tablet_bootstrap.cc:654] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.274608 18023 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.291846 18213 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.292534 18213 ts_tablet_manager.cc:1403] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.201s	user 0.035s	sys 0.009s
I20260709 12:20:48.301172 18309 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.302014 18309 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.303316 18172 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:48.306067 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.310467 18024 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:48.310891 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:48.311929 18260 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:48.307520 18213 raft_consensus.cc:359] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.307732 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:48.313256 18213 raft_consensus.cc:385] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.313992 18213 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.309932 18309 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.316690 18213 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.320242 18213 ts_tablet_manager.cc:1434] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.027s	user 0.011s	sys 0.000s
I20260709 12:20:48.320147 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.315197 18260 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:48.321923 18214 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.321790 18260 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.322575 18214 ts_tablet_manager.cc:1403] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.179s	user 0.014s	sys 0.011s
I20260709 12:20:48.325089 18213 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:48.329066 18214 raft_consensus.cc:359] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.328611 17921 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.331099 18214 raft_consensus.cc:385] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.332010 18214 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.321169 18263 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.334914 18214 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.334852 17919 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:48.371249 18268 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.406685 18214 ts_tablet_manager.cc:1434] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.084s	user 0.008s	sys 0.030s
I20260709 12:20:48.408110 18214 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.413723 18268 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.416752 18268 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:48.417130 18214 tablet_bootstrap.cc:654] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.368381 18023 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:48.432662 18214 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.433292 18214 ts_tablet_manager.cc:1403] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.000s
I20260709 12:20:48.436601 18214 raft_consensus.cc:359] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.437408 18214 raft_consensus.cc:385] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.437705 18214 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.438444 18214 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.439023 18256 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.439600 18256 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.441030 18214 ts_tablet_manager.cc:1434] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:20:48.442467 18256 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.443770 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.444186 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:48.445329 17989 leader_election.cc:343] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:48.445915 17987 leader_election.cc:343] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.446400 17987 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:48.447327 18277 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.442209 18214 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.458369 18213 tablet_bootstrap.cc:654] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.463012 18263 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.470777 18260 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.473166 18214 tablet_bootstrap.cc:654] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.474962 18260 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.488108 18215 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:48.494976 18215 ts_tablet_manager.cc:1403] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.248s	user 0.051s	sys 0.029s
I20260709 12:20:48.493409 17922 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.516891 18263 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.520015 18263 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.525327 18213 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.526089 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.526402 18263 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.527235 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:48.527721 18138 leader_election.cc:343] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.526098 18213 ts_tablet_manager.cc:1403] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.201s	user 0.018s	sys 0.004s
W20260709 12:20:48.528708 18136 leader_election.cc:343] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.529374 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.533491 18215 raft_consensus.cc:359] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.533789 18268 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.534549 18215 raft_consensus.cc:385] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.535033 18215 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.536029 18215 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.536612 18214 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.537868 18214 ts_tablet_manager.cc:1403] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.096s	user 0.013s	sys 0.012s
I20260709 12:20:48.531859 18213 raft_consensus.cc:359] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.539496 18213 raft_consensus.cc:385] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.539914 18213 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.540722 18213 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.543426 18213 ts_tablet_manager.cc:1434] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:20:48.544538 18213 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:48.558089 18215 ts_tablet_manager.cc:1434] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.063s	user 0.009s	sys 0.000s
I20260709 12:20:48.560623 18214 raft_consensus.cc:359] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.561512 18214 raft_consensus.cc:385] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.561787 18214 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.563019 18215 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:48.562562 18214 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.564123 18213 tablet_bootstrap.cc:654] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.566361 18214 ts_tablet_manager.cc:1434] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:20:48.567673 18214 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.575042 18214 tablet_bootstrap.cc:654] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.594898 18214 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.595504 18214 ts_tablet_manager.cc:1403] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.006s
I20260709 12:20:48.602049 18214 raft_consensus.cc:359] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.603134 18214 raft_consensus.cc:385] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.603492 18214 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.604456 18214 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.607360 18214 ts_tablet_manager.cc:1434] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:20:48.610078 18213 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.610641 18213 ts_tablet_manager.cc:1403] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.066s	user 0.021s	sys 0.027s
I20260709 12:20:48.611253 18215 tablet_bootstrap.cc:654] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.611702 18214 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.613718 18213 raft_consensus.cc:359] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.614565 18213 raft_consensus.cc:385] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.614908 18213 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.615906 18213 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.618784 18213 ts_tablet_manager.cc:1434] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:48.618850 18214 tablet_bootstrap.cc:654] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.620076 18213 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:48.627096 18283 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.628010 18283 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.632673 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:48.634548 18064 leader_election.cc:343] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.634922 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.636263 18283 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:48.636840 18064 leader_election.cc:343] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.637841 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.639087 18214 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.639117 18283 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.639799 18214 ts_tablet_manager.cc:1403] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.011s
I20260709 12:20:48.640762 18268 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.642644 18215 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:48.643216 18215 ts_tablet_manager.cc:1403] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.081s	user 0.014s	sys 0.000s
I20260709 12:20:48.643421 18213 tablet_bootstrap.cc:654] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.643030 18214 raft_consensus.cc:359] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.643884 18214 raft_consensus.cc:385] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.644187 18214 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.644938 18214 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.647142 18214 ts_tablet_manager.cc:1434] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:48.646885 18215 raft_consensus.cc:359] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.647863 18215 raft_consensus.cc:385] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.648286 18215 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.651633 18268 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:48.658700 18213 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.659415 18213 ts_tablet_manager.cc:1403] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.040s	user 0.013s	sys 0.010s
I20260709 12:20:48.652474 18214 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.662851 18213 raft_consensus.cc:359] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.663710 18213 raft_consensus.cc:385] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.664032 18213 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.664764 18213 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.666939 18213 ts_tablet_manager.cc:1434] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:48.667352 18214 tablet_bootstrap.cc:654] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.670506 18213 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:48.649576 18215 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.673032 18215 ts_tablet_manager.cc:1434] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 12:20:48.675015 18309 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.674206 18215 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:48.675663 18309 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.683199 18309 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.683463 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.684085 18213 tablet_bootstrap.cc:654] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.684429 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:48.685289 18232 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.685854 18232 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.687841 18098 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:48.699615 18214 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.700184 18214 ts_tablet_manager.cc:1403] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.048s	user 0.012s	sys 0.004s
I20260709 12:20:48.698091 18171 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.701298 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.705363 18283 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.706017 18283 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.708417 18283 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.709229 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.710094 18023 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.711912 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:48.712445 18214 raft_consensus.cc:359] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.713260 18214 raft_consensus.cc:385] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.713578 18214 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.713793 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.714763 18214 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.715094 18172 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.715265 18283 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.716805 18283 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.717082 18214 ts_tablet_manager.cc:1434] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:20:48.717279 18283 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.718366 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.719550 18023 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:48.723814 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:48.724830 18309 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.725245 18309 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.725644 18309 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.728138 18283 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.718361 18214 tablet_bootstrap.cc:492] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.731643 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.732435 18023 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.737970 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:48.738823 18213 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.739224 18099 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:48.739434 18213 ts_tablet_manager.cc:1403] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.069s	user 0.017s	sys 0.012s
I20260709 12:20:48.742125 18232 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.748291 18283 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.756162 18263 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.757181 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:48.757968 18172 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.758059 18277 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.761200 18215 tablet_bootstrap.cc:654] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.762743 18256 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:48.766224 18023 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:48.767704 18309 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.769448 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:48.772147 18214 tablet_bootstrap.cc:654] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.773857 18261 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:48.774492 18261 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:48.775547 18261 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.743506 18213 raft_consensus.cc:359] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.778559 18213 raft_consensus.cc:385] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.778870 18213 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.779706 18213 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.784649 18213 ts_tablet_manager.cc:1434] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.004s
I20260709 12:20:48.785647 18309 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.786303 18260 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:48.785861 18213 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:48.790416 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:48.791122 18098 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.794157 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.795043 18023 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.789944 18309 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.798086 18309 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.801041 18309 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.782066 18172 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:48.801362 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.817081 18214 tablet_bootstrap.cc:492] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:48.818202 18214 ts_tablet_manager.cc:1403] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.100s	user 0.019s	sys 0.015s
I20260709 12:20:48.831511 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:48.834630 18096 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
W20260709 12:20:48.817673 18138 leader_election.cc:343] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.870307 18136 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.873391 18309 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.873855 18309 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.874218 18309 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.875916 18098 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:48.878257 18261 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.878870 18261 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.881393 18261 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.883898 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:48.886579 18064 leader_election.cc:343] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.872387 18213 tablet_bootstrap.cc:654] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:48.899044 18136 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:48.899832 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.902020 18232 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:48.902709 18232 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:48.903038 17921 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.904338 18232 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:48.903045 18064 leader_election.cc:343] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.906944 18064 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.907801 18261 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:48.914942 18309 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.926477 18215 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:48.927076 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.919387 18261 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:48.930033 18138 leader_election.cc:343] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:48.923174 18023 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:48.927987 18215 ts_tablet_manager.cc:1403] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.254s	user 0.017s	sys 0.030s
I20260709 12:20:48.934751 18309 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.934851 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:48.935884 18098 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.935639 18214 raft_consensus.cc:359] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.936774 18214 raft_consensus.cc:385] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.937208 18214 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.942963 18215 raft_consensus.cc:359] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.944044 18215 raft_consensus.cc:385] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.944489 18215 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.949630 18263 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.950395 18263 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.952594 18261 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:48.952839 18263 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:48.953214 18261 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.954046 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.954775 18022 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:48.955683 18261 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.946471 18214 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.956836 18213 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:48.957484 18213 ts_tablet_manager.cc:1403] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.172s	user 0.016s	sys 0.006s
I20260709 12:20:48.957423 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:48.958217 18022 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.958851 18214 ts_tablet_manager.cc:1434] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.133s	user 0.009s	sys 0.000s
I20260709 12:20:48.959865 18214 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:48.960430 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:48.960608 18213 raft_consensus.cc:359] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.961098 18170 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:48.961297 18213 raft_consensus.cc:385] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:48.945569 18215 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.962986 18138 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:48.963909 18263 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.964298 18263 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.954715 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:48.964658 18263 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.965058 18099 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:48.965322 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:48.965704 18098 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:48.966550 18260 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:48.966953 18260 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:48.967363 18260 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:48.967269 18136 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.961621 18213 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:48.968858 18268 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:48.968698 18213 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.969372 18268 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:48.970345 18268 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.984745 18215 ts_tablet_manager.cc:1434] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 12:20:48.987757 18260 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.990482 18260 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:48.995868 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:48.996457 18263 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:48.996668 18213 ts_tablet_manager.cc:1434] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.039s	user 0.011s	sys 0.004s
I20260709 12:20:49.013208 18256 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.017154 18256 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.000212 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:49.020026 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:49.020358 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.020684 18170 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:49.021108 18098 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:49.022516 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:49.023123 18263 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:49.023448 18364 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:49.004288 18214 tablet_bootstrap.cc:654] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.023977 18364 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.024468 18364 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.026329 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:49.027266 18098 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:49.028733 18136 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:49.028425 17921 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:49.030056 18268 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:49.030694 18256 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.017450 18213 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.004722 18215 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:49.006439 18261 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.038739 18261 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.039698 18213 tablet_bootstrap.cc:654] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.044929 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.045534 18261 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.045957 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:49.046942 18064 leader_election.cc:343] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:49.047837 18064 leader_election.cc:343] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.048436 18214 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.048329 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:48.996764 18024 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.048997 18214 ts_tablet_manager.cc:1403] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.089s	user 0.014s	sys 0.001s
I20260709 12:20:49.052335 18215 tablet_bootstrap.cc:654] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.052780 18261 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:49.052882 18214 raft_consensus.cc:359] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.059721 17921 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:49.063326 18214 raft_consensus.cc:385] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.063711 18214 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.064471 18214 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.064742 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.067159 18214 ts_tablet_manager.cc:1434] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
W20260709 12:20:49.067967 17989 leader_election.cc:343] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.069111 18214 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.071632 18364 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.076692 18214 tablet_bootstrap.cc:654] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.067858 18024 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:49.078984 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:49.080454 18261 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:49.081053 18261 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:49.082062 18261 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.084085 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:49.084827 18098 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.091367 18364 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.092262 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:49.092082 18213 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.094187 18213 ts_tablet_manager.cc:1403] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.077s	user 0.020s	sys 0.004s
W20260709 12:20:49.094817 17989 leader_election.cc:343] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.100566 18213 raft_consensus.cc:359] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.101413 18098 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:49.101877 18213 raft_consensus.cc:385] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.102401 18213 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.103152 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:49.104732 18364 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:49.105522 18364 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:49.106508 18364 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.111109 18213 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.112727 17922 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.114835 18213 ts_tablet_manager.cc:1434] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.020s	user 0.011s	sys 0.000s
I20260709 12:20:49.117038 18213 tablet_bootstrap.cc:492] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.125711 18215 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:49.126482 18215 ts_tablet_manager.cc:1403] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.122s	user 0.012s	sys 0.014s
I20260709 12:20:49.130268 18215 raft_consensus.cc:359] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.132392 18215 raft_consensus.cc:385] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.132818 18215 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.134004 18214 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.134714 18214 ts_tablet_manager.cc:1403] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.066s	user 0.044s	sys 0.019s
I20260709 12:20:49.134071 18215 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.136927 18215 ts_tablet_manager.cc:1434] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:20:49.138181 18215 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:49.140509 18214 raft_consensus.cc:359] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.141295 18214 raft_consensus.cc:385] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.141638 18214 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.142513 18214 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.144833 18214 ts_tablet_manager.cc:1434] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:20:49.146410 18214 tablet_bootstrap.cc:492] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.150228 18215 tablet_bootstrap.cc:654] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.149796 18213 tablet_bootstrap.cc:654] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.153472 18214 tablet_bootstrap.cc:654] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.152962 17922 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.165027 18213 tablet_bootstrap.cc:492] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.165405 18214 tablet_bootstrap.cc:492] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.165581 18213 ts_tablet_manager.cc:1403] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.049s	user 0.005s	sys 0.011s
I20260709 12:20:49.165948 18214 ts_tablet_manager.cc:1403] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.008s
I20260709 12:20:49.168774 18213 raft_consensus.cc:359] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.169564 18213 raft_consensus.cc:385] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.169334 18214 raft_consensus.cc:359] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.170068 18213 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.170148 18214 raft_consensus.cc:385] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.170485 18214 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.170881 18213 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.171284 18214 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.173413 18214 ts_tablet_manager.cc:1434] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:49.173414 18213 ts_tablet_manager.cc:1434] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:20:49.174747 18214 tablet_bootstrap.cc:492] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.175420 18213 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.183842 18213 tablet_bootstrap.cc:654] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.184258 18214 tablet_bootstrap.cc:654] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.186976 18261 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:49.204126 18214 tablet_bootstrap.cc:492] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.204779 18214 ts_tablet_manager.cc:1403] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.030s	user 0.010s	sys 0.008s
I20260709 12:20:49.208801 18214 raft_consensus.cc:359] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.209594 18214 raft_consensus.cc:385] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.209937 18214 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.214354 18215 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:49.214984 18215 ts_tablet_manager.cc:1403] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.077s	user 0.008s	sys 0.010s
I20260709 12:20:49.218341 18215 raft_consensus.cc:359] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.219157 18215 raft_consensus.cc:385] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.230337 18214 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.241055 18214 ts_tablet_manager.cc:1434] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260709 12:20:49.242256 18214 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.219444 18215 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.220748 18220 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.247047 18220 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.252264 18214 tablet_bootstrap.cc:654] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.254739 18364 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.255645 18364 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.246917 18215 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.258118 18215 ts_tablet_manager.cc:1434] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.043s	user 0.001s	sys 0.007s
I20260709 12:20:49.249673 18220 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.264135 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.265316 18170 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:49.230376 18261 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260709 12:20:49.273063 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:20:49.276602 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:49.280496 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.284886 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.285187 18364 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.285809 18098 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:49.286674 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.287456 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:49.287670 17987 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:49.282123 18024 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:49.294310 18364 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:49.294564 18220 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:49.298384 18364 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.298422 18220 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.299329 18220 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.310999 18215 tablet_bootstrap.cc:492] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:49.318791 18215 tablet_bootstrap.cc:654] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.321137 18277 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.321961 18277 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.325657 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.327634 18277 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.299403 18364 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.329483 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.330953 18213 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.331585 18213 ts_tablet_manager.cc:1403] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.156s	user 0.035s	sys 0.079s
I20260709 12:20:49.333351 18098 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:49.335034 18213 raft_consensus.cc:359] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.335927 18213 raft_consensus.cc:385] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.336236 18213 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.337237 18213 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.338953 18214 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.339602 18214 ts_tablet_manager.cc:1403] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.098s	user 0.017s	sys 0.027s
I20260709 12:20:49.340448 18213 ts_tablet_manager.cc:1434] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:49.341637 18213 tablet_bootstrap.cc:492] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.343327 18214 raft_consensus.cc:359] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.344211 18214 raft_consensus.cc:385] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.344511 18214 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.345736 18220 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.345342 18214 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.349423 18213 tablet_bootstrap.cc:654] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.350052 18214 ts_tablet_manager.cc:1434] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 12:20:49.351322 18214 tablet_bootstrap.cc:492] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.337966 17987 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:49.352635 18277 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.358543 18214 tablet_bootstrap.cc:654] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.359701 18215 tablet_bootstrap.cc:492] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:49.360249 18215 ts_tablet_manager.cc:1403] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.050s	user 0.015s	sys 0.004s
I20260709 12:20:49.360525 18364 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.359542 18220 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.363366 18215 raft_consensus.cc:359] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.364214 18215 raft_consensus.cc:385] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.364514 18215 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.367417 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:49.368134 18364 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.369767 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:49.370431 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:49.370713 18170 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.371095 18172 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.371261 18214 tablet_bootstrap.cc:492] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.371822 18214 ts_tablet_manager.cc:1403] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.007s
I20260709 12:20:49.368146 18024 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.373518 18213 tablet_bootstrap.cc:492] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.374099 18213 ts_tablet_manager.cc:1403] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.033s	user 0.004s	sys 0.016s
I20260709 12:20:49.375118 18214 raft_consensus.cc:359] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.375923 18214 raft_consensus.cc:385] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.376204 18214 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.377033 18214 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.378011 18213 raft_consensus.cc:359] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.378854 18213 raft_consensus.cc:385] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.379164 18213 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.379529 18214 ts_tablet_manager.cc:1434] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:49.379988 18213 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.380725 18214 tablet_bootstrap.cc:492] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.381174 18268 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:49.383358 18213 ts_tablet_manager.cc:1434] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:20:49.384549 18213 tablet_bootstrap.cc:492] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.388447 18214 tablet_bootstrap.cc:654] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.389648 18170 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:49.365314 18215 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.392627 18213 tablet_bootstrap.cc:654] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.392985 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:49.394505 18098 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.396461 18263 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.397205 18263 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.400588 18263 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:49.401137 18214 tablet_bootstrap.cc:492] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.401731 18214 ts_tablet_manager.cc:1403] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.014s
I20260709 12:20:49.403258 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.404520 18099 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:49.405051 18214 raft_consensus.cc:359] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.406001 18214 raft_consensus.cc:385] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.406327 18214 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.407015 18214 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.407224 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.408743 18263 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:49.409322 18263 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.409693 18214 ts_tablet_manager.cc:1434] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:49.409970 18263 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.410964 18214 tablet_bootstrap.cc:492] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.412352 18382 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.412999 18382 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.418160 18382 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.425614 18214 tablet_bootstrap.cc:654] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.425302 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.395881 18215 ts_tablet_manager.cc:1434] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
I20260709 12:20:49.440408 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.443051 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.444247 18283 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.450423 18283 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.451715 18268 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.452658 18268 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.454136 18382 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.460743 18382 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.464025 18382 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.470391 18024 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:49.475801 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.476990 18099 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:49.479497 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.481529 18368 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:49.482184 18368 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.482681 18368 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.453720 18283 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.445449 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.491094 18261 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:49.491987 18261 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:20:49.494019 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48741.5us
I20260709 12:20:49.493387 18261 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.502135 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.503317 18024 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
W20260709 12:20:49.456599 18138 leader_election.cc:343] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.456902 18215 tablet_bootstrap.cc:492] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
W20260709 12:20:49.508778 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52262.7us
I20260709 12:20:49.517086 18213 tablet_bootstrap.cc:492] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.520126 18260 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.520839 18213 ts_tablet_manager.cc:1403] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.137s	user 0.043s	sys 0.083s
I20260709 12:20:49.520998 18260 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.523384 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.524410 18213 raft_consensus.cc:359] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.525216 18213 raft_consensus.cc:385] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.523461 18260 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.498662 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.526197 18368 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.525583 18213 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.527724 18213 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.528434 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.529307 18215 tablet_bootstrap.cc:654] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.529670 18170 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:49.530254 18213 ts_tablet_manager.cc:1434] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260709 12:20:49.531505 18213 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.528360 18263 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.520988 18393 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.539309 18393 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.540907 18213 tablet_bootstrap.cc:654] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.542009 18393 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:49.546774 18368 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:49.545486 18263 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:49.454846 18277 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:49.554939 18213 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.557978 18213 ts_tablet_manager.cc:1403] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.013s
I20260709 12:20:49.562242 18213 raft_consensus.cc:359] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.563083 18213 raft_consensus.cc:385] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.563369 18213 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.564030 18213 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.568482 18213 ts_tablet_manager.cc:1434] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:20:49.569589 18213 tablet_bootstrap.cc:492] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.459621 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:49.576524 18213 tablet_bootstrap.cc:654] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.577370 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:49.581504 18277 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:49.583170 18277 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:49.584759 18277 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.587755 18215 tablet_bootstrap.cc:492] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:49.588949 18213 tablet_bootstrap.cc:492] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.592015 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.605568 18213 ts_tablet_manager.cc:1403] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.005s
I20260709 12:20:49.605531 18024 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:49.593299 18364 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:49.606925 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.607645 18215 ts_tablet_manager.cc:1403] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.151s	user 0.018s	sys 0.011s
I20260709 12:20:49.607921 18099 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:49.609419 18213 raft_consensus.cc:359] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.610478 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:49.610533 18213 raft_consensus.cc:385] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.611330 18096 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.611335 18213 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.612555 18213 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.615460 18213 ts_tablet_manager.cc:1434] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:20:49.618992 18098 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:49.620057 18213 tablet_bootstrap.cc:492] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:49.622017 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:49.622823 18095 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.595742 18268 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:49.596215 18064 leader_election.cc:343] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.600965 18368 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:49.631289 18213 tablet_bootstrap.cc:654] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.681538 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:49.678676 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82556.4us
I20260709 12:20:49.648967 18095 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:49.649653 18096 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:49.677604 18394 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:49.663823 18215 raft_consensus.cc:359] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.683900 18215 raft_consensus.cc:385] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.649154 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.642453 18392 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260709 12:20:49.650581 17921 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.682379 18022 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.693784 18214 tablet_bootstrap.cc:492] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:49.686228 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:49.685437 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.697937 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.684494 18215 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.660862 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:49.694607 18214 ts_tablet_manager.cc:1403] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.284s	user 0.094s	sys 0.138s
I20260709 12:20:49.699071 18394 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:49.699506 18392 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:49.699643 18394 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:49.700140 18392 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:49.701094 18393 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.700744 18394 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.701354 18392 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.701655 18393 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.702472 18214 raft_consensus.cc:359] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.703219 18214 raft_consensus.cc:385] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.703508 18214 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.703987 18393 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:49.704773 18064 leader_election.cc:343] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.706338 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.704270 18214 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.704660 18220 raft_consensus.cc:2764] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:20:49.710234 18022 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:49.710707 18214 ts_tablet_manager.cc:1434] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:20:49.711891 18214 tablet_bootstrap.cc:492] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:49.707142 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:49.714303 18213 tablet_bootstrap.cc:492] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:49.714900 18213 ts_tablet_manager.cc:1403] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.095s	user 0.014s	sys 0.003s
I20260709 12:20:49.716379 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.717598 18393 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:49.718086 18393 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:49.718482 18393 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.715179 18215 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:49.718287 18064 leader_election.cc:343] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.721982 18023 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.725498 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:49.787191 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68687.8us
W20260709 12:20:49.789404 18064 leader_election.cc:343] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.787513 17919 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.790333 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:49.800755 18213 raft_consensus.cc:359] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.802206 18213 raft_consensus.cc:385] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:49.802623 18213 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:49.803696 18213 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.804037 18138 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:49.807796 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.808598 18213 ts_tablet_manager.cc:1434] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.093s	user 0.011s	sys 0.002s
I20260709 12:20:49.742009 18393 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:49.812563 18244 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:20:49.812770 18368 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.834919 18368 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:49.809738 18172 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:49.818378 18260 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:49.819517 18214 tablet_bootstrap.cc:654] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:49.845175 18368 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:49.850833 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47379.1us
I20260709 12:20:49.826313 18220 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:49.854753 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.821555 18261 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260709 12:20:49.893468 18213 tablet_bootstrap.cc:492] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
W20260709 12:20:49.921299 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47627.9us
I20260709 12:20:49.925712 18393 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:49.941637 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89420.3us
I20260709 12:20:49.942159 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:49.943205 18261 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.943161 18170 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:49.944293 18261 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:49.949997 18138 leader_election.cc:343] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.952055 18261 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:49.963016 18023 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:49.971463 18220 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:49.974143 18064 leader_election.cc:343] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:49.975852 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.977146 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:49.981897 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:49.984889 18096 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:49.986383 18215 ts_tablet_manager.cc:1434] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.378s	user 0.016s	sys 0.005s
I20260709 12:20:49.988523 18215 tablet_bootstrap.cc:492] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:49.995965 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:49.998066 18368 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:49.998251 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:49.998916 18368 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.002264 18268 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.001461 17922 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.002882 18268 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.003302 18268 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.005640 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:50.008659 18368 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.009166 18169 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:50.009541 18213 tablet_bootstrap.cc:654] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.004227 17922 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.019984 18261 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.020805 18261 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:50.018675 18138 leader_election.cc:343] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.023788 18261 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:50.026625 18138 leader_election.cc:343] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.029392 18214 tablet_bootstrap.cc:492] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:50.030138 18214 ts_tablet_manager.cc:1403] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.319s	user 0.017s	sys 0.012s
I20260709 12:20:50.031806 18392 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:50.021018 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:50.032634 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.033131 18268 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.037060 18268 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.038965 18214 raft_consensus.cc:359] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.041504 18214 raft_consensus.cc:385] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.041800 18214 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.044776 18283 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.045303 18283 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.051036 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:50.052070 18096 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.045647 18283 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.055114 18215 tablet_bootstrap.cc:654] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.058768 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:50.060158 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:50.060231 18095 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:50.047593 18214 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.062500 18364 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:50.064239 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:50.067727 18261 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.070133 18261 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.072064 18261 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:50.079943 18064 leader_election.cc:343] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.084254 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:50.087432 18138 leader_election.cc:343] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.101274 18214 ts_tablet_manager.cc:1434] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.071s	user 0.004s	sys 0.004s
I20260709 12:20:50.102218 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.102880 18214 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:50.103844 18368 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.104477 18368 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.104513 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:50.105093 18368 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.121773 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.120865 18382 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:50.121883 18400 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:50.143321 18213 tablet_bootstrap.cc:492] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.144042 18213 ts_tablet_manager.cc:1403] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.253s	user 0.017s	sys 0.034s
W20260709 12:20:50.130626 18064 leader_election.cc:343] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.156361 18213 raft_consensus.cc:359] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.157358 18213 raft_consensus.cc:385] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.157727 18213 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.162976 18213 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.169808 18215 tablet_bootstrap.cc:492] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:50.170553 18215 ts_tablet_manager.cc:1403] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.183s	user 0.026s	sys 0.014s
W20260709 12:20:50.168954 18064 leader_election.cc:343] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.174546 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.191831 18213 ts_tablet_manager.cc:1434] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.047s	user 0.017s	sys 0.003s
I20260709 12:20:50.198150 18215 raft_consensus.cc:359] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.199095 18215 raft_consensus.cc:385] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.199432 18215 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.207216 18213 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:50.192507 18419 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:50.191985 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53184.9us
I20260709 12:20:50.230610 18215 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.239404 18215 ts_tablet_manager.cc:1434] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.064s	user 0.012s	sys 0.000s
I20260709 12:20:50.241496 18364 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:50.250238 18283 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.250617 18368 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.251747 18097 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:50.254299 18368 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:50.226720 18138 leader_election.cc:343] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.258963 18421 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:50.260186 18214 tablet_bootstrap.cc:654] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.255125 18283 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.265295 18400 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.270686 18400 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.270987 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.271904 18020 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.273113 18215 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:50.260604 18096 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.273793 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:50.281129 18400 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.277181 18097 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.290548 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.302487 18213 tablet_bootstrap.cc:654] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.303229 18261 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.306231 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:50.307775 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.309125 18020 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.289587 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:50.312598 18064 leader_election.cc:343] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.314469 18421 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:20:50.324560 18420 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:50.325150 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.326217 18420 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.326743 18420 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:50.327943 18170 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:50.329694 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:50.331048 18261 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.332549 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:50.334164 18364 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:50.334697 18097 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.335386 18421 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.335493 18393 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:50.336192 18421 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:50.374367 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.337278 18421 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.384900 18213 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.385600 18213 ts_tablet_manager.cc:1403] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.179s	user 0.031s	sys 0.022s
I20260709 12:20:50.392704 18268 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.393361 18268 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:50.394901 18064 leader_election.cc:343] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.395810 18268 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.399585 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:50.327859 18420 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.413030 18268 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:50.416167 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:50.417229 18172 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.422366 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:50.434998 18170 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:50.400993 18261 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
W20260709 12:20:50.450650 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:50.459715 18215 tablet_bootstrap.cc:654] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.461357 18214 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:50.461989 18214 ts_tablet_manager.cc:1403] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.359s	user 0.022s	sys 0.004s
W20260709 12:20:50.463218 18138 leader_election.cc:343] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.464617 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:50.465422 18099 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:50.465921 18213 raft_consensus.cc:359] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.466715 18213 raft_consensus.cc:385] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.467338 18213 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.468096 18213 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.468458 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.472273 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.474192 18213 ts_tablet_manager.cc:1434] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.088s	user 0.006s	sys 0.003s
I20260709 12:20:50.475562 18427 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:50.477097 18268 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.477564 18268 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.478025 18268 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.514883 18268 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.520342 18213 tablet_bootstrap.cc:492] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:50.520563 18268 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.523362 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.527606 18214 raft_consensus.cc:359] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.530895 18214 raft_consensus.cc:385] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.531275 18214 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.526450 18172 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
W20260709 12:20:50.535327 18138 leader_election.cc:343] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.537985 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:50.532330 18214 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.540297 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.538811 18097 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.546700 18220 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.547338 18220 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:50.548295 18220 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.554668 18097 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.562409 17921 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.567812 17921 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.570318 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.575387 18215 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:50.576233 18215 ts_tablet_manager.cc:1403] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.303s	user 0.025s	sys 0.030s
I20260709 12:20:50.575295 18214 ts_tablet_manager.cc:1434] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.113s	user 0.008s	sys 0.003s
I20260709 12:20:50.576531 18268 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.580068 18268 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:50.580763 18214 tablet_bootstrap.cc:492] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:50.582280 18213 tablet_bootstrap.cc:654] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.586948 18215 raft_consensus.cc:359] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.589596 18215 raft_consensus.cc:385] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.589998 18215 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.581964 18268 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.596854 17919 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:50.605715 18215 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.623811 18214 tablet_bootstrap.cc:654] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.624854 18215 ts_tablet_manager.cc:1434] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.048s	user 0.008s	sys 0.002s
I20260709 12:20:50.631855 18215 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:50.639768 18213 tablet_bootstrap.cc:492] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.640414 18213 ts_tablet_manager.cc:1403] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.120s	user 0.021s	sys 0.002s
I20260709 12:20:50.644649 18213 raft_consensus.cc:359] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.645448 18213 raft_consensus.cc:385] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.645736 18213 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.647965 17921 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.646610 18213 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.652371 18213 ts_tablet_manager.cc:1434] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.001s
I20260709 12:20:50.655483 18213 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:50.659677 18268 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.660384 18268 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.662940 18214 tablet_bootstrap.cc:492] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:50.663589 18214 ts_tablet_manager.cc:1403] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.083s	user 0.016s	sys 0.004s
I20260709 12:20:50.664620 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:50.665329 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:50.666580 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:50.667620 18420 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.668028 18420 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.668346 18420 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.671303 18268 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.671690 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:50.672444 18020 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:50.673622 18213 tablet_bootstrap.cc:654] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.676622 18215 tablet_bootstrap.cc:654] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.686014 18214 raft_consensus.cc:359] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.687023 18214 raft_consensus.cc:385] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.687381 18214 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.688355 18213 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.688242 18214 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.689159 18213 ts_tablet_manager.cc:1403] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.002s
I20260709 12:20:50.689240 18420 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.692725 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.693719 18020 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.694866 18213 raft_consensus.cc:359] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.695982 18213 raft_consensus.cc:385] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.696350 18213 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.697600 18213 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.703167 18214 ts_tablet_manager.cc:1434] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.000s
I20260709 12:20:50.705480 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:50.706255 18096 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.710523 18214 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:50.702994 18020 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.715046 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:50.716259 18213 ts_tablet_manager.cc:1434] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.001s
I20260709 12:20:50.716164 18268 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.714483 18420 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.719756 18213 tablet_bootstrap.cc:492] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:50.722610 18268 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:50.729146 18268 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.733664 18213 tablet_bootstrap.cc:654] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.734622 18214 tablet_bootstrap.cc:654] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.736089 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.751336 18215 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:50.752179 18215 ts_tablet_manager.cc:1403] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.121s	user 0.029s	sys 0.023s
I20260709 12:20:50.763525 18215 raft_consensus.cc:359] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.764356 18215 raft_consensus.cc:385] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.764643 18215 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.765347 18215 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.768002 18215 ts_tablet_manager.cc:1434] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.001s
I20260709 12:20:50.774407 18215 tablet_bootstrap.cc:492] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:50.775053 18214 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:50.775635 18214 ts_tablet_manager.cc:1403] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.065s	user 0.030s	sys 0.016s
I20260709 12:20:50.778187 17922 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.781509 18215 tablet_bootstrap.cc:654] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.784547 18214 raft_consensus.cc:359] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.785429 18214 raft_consensus.cc:385] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.785737 18214 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.786624 18214 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.798231 18213 tablet_bootstrap.cc:492] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.799032 18213 ts_tablet_manager.cc:1403] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.080s	user 0.021s	sys 0.020s
I20260709 12:20:50.801303 18215 tablet_bootstrap.cc:492] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:50.802058 18215 ts_tablet_manager.cc:1403] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.029s	user 0.016s	sys 0.009s
I20260709 12:20:50.802811 18214 ts_tablet_manager.cc:1434] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.001s
I20260709 12:20:50.802373 18213 raft_consensus.cc:359] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.803228 18213 raft_consensus.cc:385] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.803534 18213 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.804112 18214 tablet_bootstrap.cc:492] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:50.804363 18213 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.806653 18215 raft_consensus.cc:359] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.807183 18213 ts_tablet_manager.cc:1434] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:50.807687 18215 raft_consensus.cc:385] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.808081 18215 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.808907 18215 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.810930 18261 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.811460 18213 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:50.811554 18261 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.813933 18261 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.816998 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:50.819729 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:50.820565 18172 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:50.824927 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:50.826164 18215 ts_tablet_manager.cc:1434] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:20:50.827631 18215 tablet_bootstrap.cc:492] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:50.827708 18213 tablet_bootstrap.cc:654] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.829138 18214 tablet_bootstrap.cc:654] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.831292 18261 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:20:50.832098 18064 leader_election.cc:343] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.831781 18261 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.834261 18261 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.841610 18399 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.848027 18399 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.851938 18213 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.852581 18213 ts_tablet_manager.cc:1403] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.041s	user 0.012s	sys 0.013s
I20260709 12:20:50.852383 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:50.854399 18096 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:50.854779 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:50.855286 18268 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.855746 18172 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:50.855897 18399 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.855826 18268 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.860002 18268 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.858064 17989 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:50.861001 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:50.862987 18138 leader_election.cc:343] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.863404 18399 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.862874 18214 tablet_bootstrap.cc:492] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:50.866027 18214 ts_tablet_manager.cc:1403] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.062s	user 0.018s	sys 0.005s
I20260709 12:20:50.867125 18220 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.866909 18261 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.867604 18220 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.868665 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:50.869477 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:50.869753 18220 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.870004 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:50.870762 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.871906 18394 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.867727 18268 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:50.872399 18394 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:50.872013 18064 leader_election.cc:343] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.872821 18394 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.873157 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:50.873313 18213 raft_consensus.cc:359] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.874029 18213 raft_consensus.cc:385] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.874341 18213 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
W20260709 12:20:50.874745 18064 leader_election.cc:343] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.875090 18213 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.876272 18261 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.877410 18213 ts_tablet_manager.cc:1434] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:20:50.882166 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:50.882078 18394 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.883029 18172 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:50.884737 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.885260 18394 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:50.888043 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:50.888882 18173 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.891294 18399 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:50.887131 18427 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:50.893186 18427 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:50.893520 18427 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.899389 18215 tablet_bootstrap.cc:654] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.901211 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:50.901937 18097 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.887481 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:50.905113 18138 leader_election.cc:343] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.912058 18173 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:50.919226 18214 raft_consensus.cc:359] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.920090 18214 raft_consensus.cc:385] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.920397 18214 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.921175 18214 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.921715 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.922879 18220 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.923396 18220 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:50.923789 18214 ts_tablet_manager.cc:1434] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.057s	user 0.006s	sys 0.001s
I20260709 12:20:50.924304 18220 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.924940 18214 tablet_bootstrap.cc:492] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:50.926608 18213 tablet_bootstrap.cc:492] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:50.916584 18427 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.932118 18097 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:50.935726 18427 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.938982 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:50.940765 18064 leader_election.cc:343] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.942201 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:50.942936 18173 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.943742 18214 tablet_bootstrap.cc:654] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.953949 18024 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:50.959765 18213 tablet_bootstrap.cc:654] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:50.957219 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.963807 18173 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:50.964917 18215 tablet_bootstrap.cc:492] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:50.964432 17919 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:50.965451 18215 ts_tablet_manager.cc:1403] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.138s	user 0.012s	sys 0.005s
I20260709 12:20:50.966188 18394 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.966953 18394 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:50.967921 18394 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.974107 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:50.976117 18427 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:50.977339 18427 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:50.978811 18427 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:50.980849 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:50.981297 18220 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.982131 18220 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.986073 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:50.989698 18064 leader_election.cc:343] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:50.992509 18213 tablet_bootstrap.cc:492] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:50.993072 18213 ts_tablet_manager.cc:1403] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.067s	user 0.014s	sys 0.017s
I20260709 12:20:50.994009 18394 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:50.997673 18220 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:50.997514 18213 raft_consensus.cc:359] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:50.998380 18213 raft_consensus.cc:385] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:50.998705 18213 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:50.999388 18213 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.001533 18213 ts_tablet_manager.cc:1434] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260709 12:20:51.003803 18213 tablet_bootstrap.cc:492] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.003631 18215 raft_consensus.cc:359] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.004400 18215 raft_consensus.cc:385] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.005481 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:51.004621 17919 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.014340 18213 tablet_bootstrap.cc:654] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:51.014334 18064 leader_election.cc:343] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.015021 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.016553 18220 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:51.004678 18215 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.019053 18215 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.025717 18214 tablet_bootstrap.cc:492] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:51.026401 18214 ts_tablet_manager.cc:1403] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.102s	user 0.013s	sys 0.004s
I20260709 12:20:51.038635 18215 ts_tablet_manager.cc:1434] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.073s	user 0.011s	sys 0.000s
I20260709 12:20:51.043150 18215 tablet_bootstrap.cc:492] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.047369 18394 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:51.053685 18214 raft_consensus.cc:359] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.050050 18394 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:20:51.042846 18268 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:20:51.063443 18214 raft_consensus.cc:385] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.064522 18214 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.066551 18214 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.059824 17921 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:51.079840 18214 ts_tablet_manager.cc:1434] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.053s	user 0.013s	sys 0.000s
I20260709 12:20:51.104038 18214 tablet_bootstrap.cc:492] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:51.117892 18213 tablet_bootstrap.cc:492] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:51.119702 18213 ts_tablet_manager.cc:1403] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.116s	user 0.030s	sys 0.015s
I20260709 12:20:51.142072 18215 tablet_bootstrap.cc:654] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:51.142408 18064 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:51.157249 18213 raft_consensus.cc:359] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.158277 18213 raft_consensus.cc:385] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.158635 18213 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.162640 18214 tablet_bootstrap.cc:654] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.173373 18458 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:51.172806 18213 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.177017 18421 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.177615 18421 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.181945 18213 ts_tablet_manager.cc:1434] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.062s	user 0.012s	sys 0.001s
I20260709 12:20:51.186406 18421 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.202709 18213 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.203714 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:51.206624 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:51.208914 18136 leader_election.cc:343] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.217011 18213 tablet_bootstrap.cc:654] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:51.221562 18138 leader_election.cc:343] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.222219 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.223526 18394 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:51.245939 18214 tablet_bootstrap.cc:492] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:51.246832 18214 ts_tablet_manager.cc:1403] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.164s	user 0.019s	sys 0.004s
I20260709 12:20:51.254383 18214 raft_consensus.cc:359] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.255481 18214 raft_consensus.cc:385] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.255847 18214 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.256942 18214 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.260335 18214 ts_tablet_manager.cc:1434] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:20:51.262003 18214 tablet_bootstrap.cc:492] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:51.276667 18214 tablet_bootstrap.cc:654] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:51.282143 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60623.8us
I20260709 12:20:51.295050 18213 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:51.297909 18394 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.300966 18213 ts_tablet_manager.cc:1403] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.105s	user 0.029s	sys 0.019s
I20260709 12:20:51.301059 18394 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.303455 18394 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.306237 18215 tablet_bootstrap.cc:492] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.307542 18215 ts_tablet_manager.cc:1403] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.265s	user 0.022s	sys 0.014s
I20260709 12:20:51.311322 18215 raft_consensus.cc:359] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.312284 18215 raft_consensus.cc:385] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.312683 18215 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.313719 18215 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.316406 18215 ts_tablet_manager.cc:1434] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:51.319078 18215 tablet_bootstrap.cc:492] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.326589 18215 tablet_bootstrap.cc:654] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:51.327803 18064 consensus_peers.cc:597] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:51.337054 18470 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:51.338136 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:51.338917 18022 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:51.340651 18213 raft_consensus.cc:359] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.341488 18213 raft_consensus.cc:385] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.341753 18213 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.342607 18213 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.344802 18213 ts_tablet_manager.cc:1434] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.000s
I20260709 12:20:51.352123 18213 tablet_bootstrap.cc:492] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.351998 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:20:51.356348 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53486.5us
I20260709 12:20:51.358888 18463 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:51.360227 18463 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.360728 18463 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.374133 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:51.375255 18096 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:51.359571 18394 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:20:51.384039 18213 tablet_bootstrap.cc:654] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.398320 18213 tablet_bootstrap.cc:492] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:51.400228 18213 ts_tablet_manager.cc:1403] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.050s	user 0.015s	sys 0.004s
I20260709 12:20:51.400564 18463 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.403568 18024 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.406211 18427 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.406857 18427 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.409263 18427 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:51.411367 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:51.415858 18064 leader_election.cc:343] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:51.416834 18064 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:51.420166 18213 raft_consensus.cc:359] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.424038 18213 raft_consensus.cc:385] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.425030 18213 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.432727 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:51.433949 18173 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:51.435626 18261 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.436504 18261 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.439967 18261 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:51.443043 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.444226 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:51.452682 18213 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.466738 18427 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:51.467343 18427 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.467720 18427 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.470402 18213 ts_tablet_manager.cc:1434] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.067s	user 0.007s	sys 0.001s
I20260709 12:20:51.472437 18213 tablet_bootstrap.cc:492] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.478078 18427 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.479605 18463 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.480369 18427 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:51.482875 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:51.484642 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:51.484920 18064 leader_election.cc:343] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.485242 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:51.487428 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:51.488462 18213 tablet_bootstrap.cc:654] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:51.485723 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:51.489465 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:51.492244 18215 tablet_bootstrap.cc:492] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.508519 18024 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.509224 18215 ts_tablet_manager.cc:1403] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.192s	user 0.031s	sys 0.018s
I20260709 12:20:51.500841 18214 tablet_bootstrap.cc:492] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:51.506980 18096 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.510413 18214 ts_tablet_manager.cc:1403] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.249s	user 0.080s	sys 0.082s
I20260709 12:20:51.513669 18170 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:51.515805 18064 leader_election.cc:343] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.517414 18215 raft_consensus.cc:359] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.522936 18215 raft_consensus.cc:385] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.523233 18215 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.524730 18215 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.517710 18173 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:51.528231 18215 ts_tablet_manager.cc:1434] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.002s
I20260709 12:20:51.518096 18427 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:51.529783 18215 tablet_bootstrap.cc:492] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.532785 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.539491 18427 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:51.539990 18427 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.540386 18427 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.514420 18463 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260709 12:20:51.578776 18024 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
W20260709 12:20:51.579444 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:51.580459 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:51.581471 18463 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:51.582310 18463 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:51.583346 18463 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.580132 18214 raft_consensus.cc:359] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.590134 18214 raft_consensus.cc:385] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.590644 18214 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.591015 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:51.584041 18096 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:51.595669 18214 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.598379 18214 ts_tablet_manager.cc:1434] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.085s	user 0.008s	sys 0.001s
I20260709 12:20:51.600096 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.602348 18215 tablet_bootstrap.cc:654] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.602370 18458 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:51.603292 18458 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:51.604297 18427 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.604688 18458 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.600170 18214 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:51.615511 18213 tablet_bootstrap.cc:492] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:51.617054 18213 ts_tablet_manager.cc:1403] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.145s	user 0.018s	sys 0.033s
I20260709 12:20:51.618036 18260 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.618614 18260 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.620913 18260 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:51.622489 18427 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:51.624888 18214 tablet_bootstrap.cc:654] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.630987 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:51.629201 18260 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:51.639685 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:51.640559 18172 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.642073 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:51.642771 18024 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:51.643200 18064 leader_election.cc:343] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.644105 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:51.644829 18173 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:51.650810 18024 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:51.659617 18172 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:51.655390 18213 raft_consensus.cc:359] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.661712 18213 raft_consensus.cc:385] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.663026 18213 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.663810 18213 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.665500 17922 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.669965 18215 tablet_bootstrap.cc:492] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.670562 18215 ts_tablet_manager.cc:1403] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.141s	user 0.003s	sys 0.027s
I20260709 12:20:51.673887 18215 raft_consensus.cc:359] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.674714 18215 raft_consensus.cc:385] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.675025 18215 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.676112 18215 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.676901 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:51.669965 18214 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:51.691053 18458 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:51.692302 18458 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:51.693347 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.692477 18213 ts_tablet_manager.cc:1434] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.075s	user 0.004s	sys 0.008s
I20260709 12:20:51.696511 18427 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:51.696959 18427 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.697345 18427 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.702948 18458 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.705461 18215 ts_tablet_manager.cc:1434] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.003s
I20260709 12:20:51.710225 18215 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.712884 18213 tablet_bootstrap.cc:492] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.710467 18427 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.710942 18214 ts_tablet_manager.cc:1403] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.111s	user 0.010s	sys 0.011s
I20260709 12:20:51.722020 18214 raft_consensus.cc:359] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.723105 18214 raft_consensus.cc:385] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.727653 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:51.728230 18214 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.729588 18214 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.730641 18427 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:51.737298 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:51.738123 18173 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.740027 18214 ts_tablet_manager.cc:1434] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 12:20:51.744218 18215 tablet_bootstrap.cc:654] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.743300 18214 tablet_bootstrap.cc:492] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:51.727535 17919 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:51.754796 18064 leader_election.cc:343] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.756974 18463 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.757597 18463 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.766772 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:51.768959 18215 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.769553 18215 ts_tablet_manager.cc:1403] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.060s	user 0.010s	sys 0.004s
I20260709 12:20:51.763867 18213 tablet_bootstrap.cc:654] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.773702 18173 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
W20260709 12:20:51.774082 18138 leader_election.cc:343] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.776046 18463 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.778913 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:51.779786 18096 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:51.781661 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.782603 18268 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:51.783041 18268 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:51.783439 18268 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.783138 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.787051 18214 tablet_bootstrap.cc:654] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.791352 18261 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:51.792081 18261 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:51.792933 18268 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.793004 18261 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.796360 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:51.797909 18138 leader_election.cc:343] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.798026 18268 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.773121 18215 raft_consensus.cc:359] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.799360 18215 raft_consensus.cc:385] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.799688 18215 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.799726 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:51.800453 18096 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.800423 18215 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.807761 18215 ts_tablet_manager.cc:1434] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.000s
I20260709 12:20:51.809023 18215 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.809567 18096 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:51.811082 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.812363 18268 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:51.813211 18268 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:51.814109 18268 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.816640 18215 tablet_bootstrap.cc:654] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.841753 18213 tablet_bootstrap.cc:492] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:51.842136 18215 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.842388 18213 ts_tablet_manager.cc:1403] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.130s	user 0.016s	sys 0.003s
I20260709 12:20:51.842653 18215 ts_tablet_manager.cc:1403] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.034s	user 0.012s	sys 0.019s
I20260709 12:20:51.845758 18215 raft_consensus.cc:359] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.846654 18215 raft_consensus.cc:385] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.846987 18215 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.847726 18215 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.850183 18215 ts_tablet_manager.cc:1434] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20260709 12:20:51.849205 18213 raft_consensus.cc:359] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.851183 18213 raft_consensus.cc:385] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.851528 18213 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.853093 18215 tablet_bootstrap.cc:492] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.854946 18214 tablet_bootstrap.cc:492] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:51.855566 18214 ts_tablet_manager.cc:1403] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.113s	user 0.031s	sys 0.002s
I20260709 12:20:51.858685 18214 raft_consensus.cc:359] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.859483 18214 raft_consensus.cc:385] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.859748 18214 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.860700 17922 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.862726 18214 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.865725 18214 ts_tablet_manager.cc:1434] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:51.867280 18214 tablet_bootstrap.cc:492] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:51.868593 18213 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.870910 18215 tablet_bootstrap.cc:654] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.871729 18213 ts_tablet_manager.cc:1434] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.001s
I20260709 12:20:51.873037 18213 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.878046 17919 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:51.880411 17919 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:51.886066 18024 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.892795 18214 tablet_bootstrap.cc:654] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.915446 18215 tablet_bootstrap.cc:492] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.916163 18215 ts_tablet_manager.cc:1403] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.063s	user 0.033s	sys 0.015s
I20260709 12:20:51.916769 18214 tablet_bootstrap.cc:492] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:51.919785 18215 raft_consensus.cc:359] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.920729 18215 raft_consensus.cc:385] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.921088 18215 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.916433 18213 tablet_bootstrap.cc:654] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.928862 18215 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.931502 18215 ts_tablet_manager.cc:1434] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:20:51.933054 18215 tablet_bootstrap.cc:492] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.917294 18214 ts_tablet_manager.cc:1403] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.050s	user 0.016s	sys 0.003s
I20260709 12:20:51.936398 18213 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:51.937391 18213 ts_tablet_manager.cc:1403] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.065s	user 0.009s	sys 0.005s
I20260709 12:20:51.940676 18215 tablet_bootstrap.cc:654] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.941026 18213 raft_consensus.cc:359] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.941908 18213 raft_consensus.cc:385] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.936661 18214 raft_consensus.cc:359] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.942446 18213 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.942763 18214 raft_consensus.cc:385] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.943120 18214 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.943300 18213 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.945809 18213 ts_tablet_manager.cc:1434] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:20:51.943990 18214 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.948251 18213 tablet_bootstrap.cc:492] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:51.956897 18213 tablet_bootstrap.cc:654] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.957875 18215 tablet_bootstrap.cc:492] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:51.957674 18268 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.958422 18215 ts_tablet_manager.cc:1403] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.008s
I20260709 12:20:51.958351 18268 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.960952 18214 ts_tablet_manager.cc:1434] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.001s
I20260709 12:20:51.961122 18024 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:51.961431 18215 raft_consensus.cc:359] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.962188 18215 raft_consensus.cc:385] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.962512 18215 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.962452 18463 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:51.962846 18214 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:51.963199 18463 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.963352 18215 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.965886 18215 ts_tablet_manager.cc:1434] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:51.967068 18215 tablet_bootstrap.cc:492] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:51.967509 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:51.969535 18138 leader_election.cc:343] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.972668 18268 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.972919 18214 tablet_bootstrap.cc:654] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.980702 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:51.984654 18213 tablet_bootstrap.cc:492] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
W20260709 12:20:51.985554 18138 leader_election.cc:343] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.986344 18213 ts_tablet_manager.cc:1403] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.038s	user 0.018s	sys 0.002s
I20260709 12:20:51.989683 18213 raft_consensus.cc:359] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:51.990572 18213 raft_consensus.cc:385] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:51.990902 18213 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:51.965709 18463 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:51.974895 18215 tablet_bootstrap.cc:654] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:51.976116 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:51.993152 18136 leader_election.cc:343] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.993626 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.977380 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:51.995270 18268 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:51.996726 18136 leader_election.cc:343] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:51.997175 18136 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:51.997956 18268 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:51.991752 18213 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.002864 18213 ts_tablet_manager.cc:1434] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260709 12:20:52.005568 18213 tablet_bootstrap.cc:492] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:52.008145 18480 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:52.007719 18215 tablet_bootstrap.cc:492] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:52.009172 18215 ts_tablet_manager.cc:1403] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.042s	user 0.011s	sys 0.005s
I20260709 12:20:52.012288 18215 raft_consensus.cc:359] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.016114 18215 raft_consensus.cc:385] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.016822 18215 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:52.017851 18215 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.023851 18215 ts_tablet_manager.cc:1434] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.002s
I20260709 12:20:52.025511 18483 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.026160 18483 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.028440 18483 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:52.056849 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:52.066772 18215 tablet_bootstrap.cc:492] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:52.066846 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:52.067974 18172 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:52.079159 18020 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.084780 18213 tablet_bootstrap.cc:654] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:52.089095 18214 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:52.089774 18214 ts_tablet_manager.cc:1403] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.127s	user 0.026s	sys 0.058s
I20260709 12:20:52.109617 18024 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.115417 18215 tablet_bootstrap.cc:654] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:52.140071 18483 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.140354 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:52.141041 18483 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.150584 18458 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:52.151587 18512 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:52.200500 18480 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.201576 18480 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.215626 18213 tablet_bootstrap.cc:492] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:52.216401 18213 ts_tablet_manager.cc:1403] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.211s	user 0.018s	sys 0.035s
I20260709 12:20:52.160367 18483 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:52.222271 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:52.223440 18214 raft_consensus.cc:359] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.224579 18214 raft_consensus.cc:385] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.225126 18214 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:52.224042 18480 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:52.230967 18215 tablet_bootstrap.cc:492] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
W20260709 12:20:52.231671 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89812.1us
I20260709 12:20:52.233690 18214 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.234077 18480 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:52.231617 18215 ts_tablet_manager.cc:1403] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.165s	user 0.033s	sys 0.029s
I20260709 12:20:52.227442 18213 raft_consensus.cc:359] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.236163 18213 raft_consensus.cc:385] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.236470 18213 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:52.234898 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:52.237079 18463 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.239902 18095 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:52.240336 18463 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.242552 18463 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:52.245550 18214 ts_tablet_manager.cc:1434] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.155s	user 0.021s	sys 0.005s
I20260709 12:20:52.246227 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:52.247027 18214 tablet_bootstrap.cc:492] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:52.255751 18213 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.261250 18220 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:52.266566 18260 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:52.254451 18018 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.268669 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:52.275429 18268 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:52.276063 18268 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.276512 18268 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.278920 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:52.288084 18213 ts_tablet_manager.cc:1434] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.071s	user 0.000s	sys 0.008s
I20260709 12:20:52.314932 18215 raft_consensus.cc:359] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.315832 18215 raft_consensus.cc:385] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.316126 18215 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:52.313283 18268 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.319365 18215 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.325196 18213 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:52.331046 18215 ts_tablet_manager.cc:1434] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.096s	user 0.007s	sys 0.001s
I20260709 12:20:52.357837 18268 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:52.357734 18220 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.376984 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:52.377058 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:52.377909 18023 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:52.378197 18095 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:52.375111 18515 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.379976 18515 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.381781 18215 tablet_bootstrap.cc:492] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:52.375309 18220 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.395113 18214 tablet_bootstrap.cc:654] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:52.396843 18260 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:52.392468 18261 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:20:52.404217 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:52.407473 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:52.396941 18220 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:52.471442 18138 leader_election.cc:343] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:52.437325 18480 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:52.450137 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:52.450659 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:52.485226 18170 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:52.492565 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:52.493630 18213 tablet_bootstrap.cc:654] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:52.479910 18480 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:52.485231 18516 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:20:52.488942 18096 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.498046 18260 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:52.498932 18480 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.507529 18260 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.511623 18260 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.517637 17987 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:52.532280 18515 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:52.539748 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:52.556833 18064 leader_election.cc:343] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:52.525875 18483 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:52.572893 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76081.8us
I20260709 12:20:52.585796 18022 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.564869 18512 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.592911 18512 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.596174 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:52.602218 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45452.8us
I20260709 12:20:52.615900 18214 tablet_bootstrap.cc:492] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:52.616693 18214 ts_tablet_manager.cc:1403] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.370s	user 0.023s	sys 0.006s
I20260709 12:20:52.619043 18512 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:52.619232 18138 leader_election.cc:343] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:52.626405 18138 consensus_peers.cc:597] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:52.621045 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:52.630789 18095 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:52.633626 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:52.635571 18483 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:52.636029 18483 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:52.636416 18483 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.621906 18463 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.652153 18096 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:52.655820 18463 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.661034 18463 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:52.673264 18215 tablet_bootstrap.cc:654] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:52.655870 18260 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.693729 18214 raft_consensus.cc:359] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:52.701727 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74466.3us
I20260709 12:20:52.704820 18214 raft_consensus.cc:385] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.705447 18214 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:52.708171 18214 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.731446 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:52.735450 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:52.747578 18214 ts_tablet_manager.cc:1434] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.130s	user 0.008s	sys 0.003s
I20260709 12:20:52.775771 18521 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:52.776477 18521 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:20:52.777185 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 250841us
I20260709 12:20:52.788291 18517 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20260709 12:20:52.799975 18260 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:52.777637 18521 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:52.821977 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86682.7us
I20260709 12:20:52.823653 18524 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:52.820269 18480 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.834713 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:52.835887 18169 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
W20260709 12:20:52.840852 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:52.802161 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:52.851500 18173 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:52.851789 18213 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:52.852360 18213 ts_tablet_manager.cc:1403] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.528s	user 0.031s	sys 0.022s
I20260709 12:20:52.855479 18480 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:52.857445 18427 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.800395 18483 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.862149 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:52.862944 18017 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.865029 18483 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:52.866228 18023 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:52.858680 18427 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.870328 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:52.871021 18099 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.871268 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:52.884688 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:52.872110 18427 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:52.891680 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47729.2us
I20260709 12:20:52.895042 18215 tablet_bootstrap.cc:492] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:52.895654 18215 ts_tablet_manager.cc:1403] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.522s	user 0.036s	sys 0.049s
W20260709 12:20:52.894942 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 146419us
I20260709 12:20:52.910279 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:52.913244 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:52.914276 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:52.916939 18097 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.920202 18213 raft_consensus.cc:359] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.920879 18213 raft_consensus.cc:385] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.921340 18213 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:52.922082 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:52.922137 18213 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.923377 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:52.924248 18215 raft_consensus.cc:359] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.925113 18215 raft_consensus.cc:385] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:52.925503 18215 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
W20260709 12:20:52.927927 18064 leader_election.cc:343] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:52.928993 18215 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:52.915131 18529 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:52.920840 18420 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:52.941102 18099 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:52.956597 18213 ts_tablet_manager.cc:1434] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.104s	user 0.008s	sys 0.003s
I20260709 12:20:52.967872 18215 ts_tablet_manager.cc:1434] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.072s	user 0.009s	sys 0.001s
I20260709 12:20:52.946749 18529 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:20:52.978847 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:52.979394 18215 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:52.989460 18529 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.005604 18483 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:53.007798 18483 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:53.040635 18017 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:20:53.042299 18483 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:53.081045 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 162646us
I20260709 12:20:53.109567 18261 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.110458 18261 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.122030 18515 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.146173 18515 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.154510 18261 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:53.173415 18529 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.201728 18018 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:53.210480 18215 tablet_bootstrap.cc:654] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:53.247238 18024 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:53.253249 18515 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:53.253320 18214 tablet_bootstrap.cc:492] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:53.256290 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:53.260313 18064 leader_election.cc:343] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:53.265436 18024 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:53.281226 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:53.284118 18261 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:53.284768 18214 tablet_bootstrap.cc:654] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:53.284657 18261 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:53.287808 18261 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.292807 18215 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:53.293596 18215 ts_tablet_manager.cc:1403] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.315s	user 0.032s	sys 0.051s
W20260709 12:20:53.262089 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46456.2us
I20260709 12:20:53.294870 18529 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:53.305217 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:53.306071 18169 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:53.309751 18097 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:20:53.313213 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:53.314085 18169 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.313493 18215 raft_consensus.cc:359] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.314818 18215 raft_consensus.cc:385] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:53.315125 18215 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:53.315811 18215 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.316416 18517 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:53.317773 18138 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:20:53.318703 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 236014us
I20260709 12:20:53.299202 18420 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:53.322435 18542 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.323037 18542 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:53.324008 18542 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.335036 18215 ts_tablet_manager.cc:1434] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.001s
I20260709 12:20:53.352861 18427 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:53.355451 18170 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:53.353533 18427 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:53.381946 18213 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:53.386494 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:53.387372 18171 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:53.337026 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:20:53.390679 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69542.3us
I20260709 12:20:53.391887 18531 raft_consensus.cc:2764] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:20:53.414386 18215 tablet_bootstrap.cc:492] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:53.358264 18427 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.428388 18023 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:53.456020 18214 tablet_bootstrap.cc:492] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:53.445039 18427 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.486637 18169 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:20:53.507441 18168 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:20:53.508091 18022 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:53.517736 18214 ts_tablet_manager.cc:1403] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.267s	user 0.037s	sys 0.032s
I20260709 12:20:53.467891 18171 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:53.471519 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:53.526757 17920 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:53.527979 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:53.558954 18515 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.559590 18515 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:53.560585 18515 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.561932 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:53.562774 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:53.551344 18524 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.583045 18524 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.627231 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:53.629630 18166 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:53.631217 18214 raft_consensus.cc:359] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.632445 18214 raft_consensus.cc:385] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:53.632946 18214 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:53.636761 18214 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.643476 18268 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:53.651408 18213 tablet_bootstrap.cc:654] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:53.657470 18215 tablet_bootstrap.cc:654] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:53.664542 18524 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:53.682768 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:53.688686 18136 leader_election.cc:343] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:53.708519 18064 leader_election.cc:343] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:53.710724 18064 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:53.712116 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:53.715852 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71678.3us
W20260709 12:20:53.717401 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:53.719285 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:53.721400 18138 leader_election.cc:343] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:53.722115 18138 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:53.724565 18557 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.101s	user 0.005s	sys 0.000s
I20260709 12:20:53.763978 17919 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.808095 18562 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:53.808920 18562 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:53.809407 18562 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:53.814055 18561 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:53.728052 18524 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:53.836483 18214 ts_tablet_manager.cc:1434] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.318s	user 0.022s	sys 0.020s
I20260709 12:20:53.849602 18529 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:20:53.853562 18214 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:53.853720 18529 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.886610 18094 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:53.887184 18093 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:20:53.886664 18529 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:53.898339 18562 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.903491 18214 tablet_bootstrap.cc:654] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:53.909547 18524 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:53.914847 18215 tablet_bootstrap.cc:492] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:53.915494 18215 ts_tablet_manager.cc:1403] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.501s	user 0.019s	sys 0.009s
I20260709 12:20:53.921761 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:53.931900 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:53.933465 18562 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:53.943639 17989 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:53.953261 18483 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.953454 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:53.954813 18561 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.955425 18561 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:53.967888 18525 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:53.971911 18261 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:20:53.980186 18529 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:20:53.984906 18515 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:54.002563 18016 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:20:54.011591 18064 leader_election.cc:343] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.003772 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:54.034865 18215 raft_consensus.cc:359] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.035496 18215 raft_consensus.cc:385] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.035794 18215 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.036757 18215 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.041641 18215 ts_tablet_manager.cc:1434] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.126s	user 0.012s	sys 0.007s
I20260709 12:20:54.042366 18168 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.043572 18561 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:54.061980 18213 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:54.062984 18213 ts_tablet_manager.cc:1403] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.681s	user 0.065s	sys 0.037s
I20260709 12:20:54.063916 18542 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:20:54.069623 18023 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:54.096299 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:54.108084 18215 tablet_bootstrap.cc:492] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:54.111801 18098 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:54.082756 18524 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.126077 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:54.129422 18064 leader_election.cc:343] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.116350 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:54.154408 18529 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:54.131464 18524 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:54.157632 18064 leader_election.cc:343] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.165084 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.171689 18561 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:54.164063 18165 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:54.181777 17922 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:54.189108 18483 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:54.217080 18213 raft_consensus.cc:359] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.218344 18213 raft_consensus.cc:385] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.218606 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:54.218806 18213 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.219687 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.220022 18213 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:54.220892 18136 leader_election.cc:343] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.222455 18214 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:54.223251 18214 ts_tablet_manager.cc:1403] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.370s	user 0.039s	sys 0.032s
I20260709 12:20:54.230516 18524 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:20:54.221906 18138 leader_election.cc:343] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.231722 18138 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.250371 18213 ts_tablet_manager.cc:1434] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.187s	user 0.011s	sys 0.000s
I20260709 12:20:54.249670 18546 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:54.297691 18168 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:54.306807 18214 raft_consensus.cc:359] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.310452 18214 raft_consensus.cc:385] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.310931 18214 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.313692 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.331410 18561 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:54.331995 18561 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:54.332955 18561 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.339195 18214 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.348591 18213 tablet_bootstrap.cc:492] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:54.349956 18215 tablet_bootstrap.cc:654] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.356200 18214 ts_tablet_manager.cc:1434] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.132s	user 0.009s	sys 0.000s
I20260709 12:20:54.369176 18214 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:54.378161 18213 tablet_bootstrap.cc:654] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.415130 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 80) took 1106 ms. Trace:
I20260709 12:20:54.427013 18214 tablet_bootstrap.cc:654] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.432788 17919 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:54.446316 18215 tablet_bootstrap.cc:492] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:54.437716 17919 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 2, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:20:54.448155 18215 ts_tablet_manager.cc:1403] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.362s	user 0.026s	sys 0.007s
I20260709 12:20:54.458211 18213 tablet_bootstrap.cc:492] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:54.458878 18213 ts_tablet_manager.cc:1403] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.111s	user 0.033s	sys 0.015s
I20260709 12:20:54.416407 17920 rpcz_store.cc:276] 0709 12:20:53.309049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.309230 (+   181us) service_pool.cc:224] Handling call
0709 12:20:54.415023 (+1105793us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:53.646117 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:53.714772 (+ 68655us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:53.715375 (+   603us) write_op.cc:435] Acquired schema lock
0709 12:20:53.715969 (+   594us) tablet.cc:662] Decoding operations
0709 12:20:53.725363 (+  9394us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:53.725422 (+    59us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:53.725456 (+    34us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:20:53.725678 (+   222us) tablet.cc:701] Row locks acquired
0709 12:20:53.725696 (+    18us) write_op.cc:260] PREPARE: finished
0709 12:20:53.756176 (+ 30480us) write_op.cc:270] Start()
0709 12:20:53.784101 (+ 27925us) write_op.cc:276] Timestamp: P: 1783599653756068 usec, L: 0
0709 12:20:53.784143 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 12:20:53.785326 (+  1183us) log.cc:844] Serialized 537 byte log entry
0709 12:20:53.983736 (+198410us) op_driver.cc:464] REPLICATION: finished
0709 12:20:54.100458 (+116722us) write_op.cc:301] APPLY: starting
0709 12:20:54.111760 (+ 11302us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:54.122440 (+ 10680us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:54.135343 (+ 12903us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:54.140250 (+  4907us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:54.140437 (+   187us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:20:54.140487 (+    50us) write_op.cc:312] APPLY: finished
0709 12:20:54.158628 (+ 18141us) log.cc:844] Serialized 22 byte log entry
0709 12:20:54.174557 (+ 15929us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:54.174659 (+   102us) write_op.cc:454] Released schema lock
0709 12:20:54.174707 (+    48us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:54.175151 (+   444us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:54.182690 (+  7539us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":39039,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":101022,"apply.run_cpu_time_us":3103,"apply.run_wall_time_us":89996,"num_ops":1,"prepare.queue_time_us":111149,"prepare.run_cpu_time_us":8432,"prepare.run_wall_time_us":189432,"raft.queue_time_us":141211,"raft.run_cpu_time_us":76788,"raft.run_wall_time_us":132173,"replication_time_us":199440,"thread_start_us":238319,"threads_started":4,"wal-append.queue_time_us":61857}]]}
I20260709 12:20:54.481283 18213 raft_consensus.cc:359] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.482683 18213 raft_consensus.cc:385] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.482939 18213 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.484037 18213 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.496286 18215 raft_consensus.cc:359] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.509337 18215 raft_consensus.cc:385] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.509788 18215 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.512095 18215 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.507606 18214 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:54.513972 18215 ts_tablet_manager.cc:1434] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.065s	user 0.003s	sys 0.005s
I20260709 12:20:54.514235 18214 ts_tablet_manager.cc:1403] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.150s	user 0.019s	sys 0.022s
I20260709 12:20:54.515250 18215 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:54.508924 18213 ts_tablet_manager.cc:1434] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.002s
I20260709 12:20:54.517604 18214 raft_consensus.cc:359] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.518378 18214 raft_consensus.cc:385] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.518662 18214 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.519502 18214 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.523351 18215 tablet_bootstrap.cc:654] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.527871 18213 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:54.528143 18214 ts_tablet_manager.cc:1434] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20260709 12:20:54.535477 18214 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:54.535876 18213 tablet_bootstrap.cc:654] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.548027 18215 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:54.548782 18215 ts_tablet_manager.cc:1403] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.002s
I20260709 12:20:54.552188 18215 raft_consensus.cc:359] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.552879 18213 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:54.553226 18215 raft_consensus.cc:385] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.553488 18213 ts_tablet_manager.cc:1403] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.004s
I20260709 12:20:54.553588 18215 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.554657 18215 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.556479 18215 ts_tablet_manager.cc:1434] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:20:54.556831 18213 raft_consensus.cc:359] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.557652 18213 raft_consensus.cc:385] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.557840 18215 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:54.557991 18213 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.558873 18268 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.558787 18213 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.559425 18268 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.561941 18214 tablet_bootstrap.cc:654] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.566264 18215 tablet_bootstrap.cc:654] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.568619 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:54.566061 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.569515 18093 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:54.574122 18213 ts_tablet_manager.cc:1434] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
W20260709 12:20:54.574956 18138 leader_election.cc:343] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.575546 18213 tablet_bootstrap.cc:492] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:54.561738 18268 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.577751 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.579252 18524 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:54.579766 18524 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:54.579943 18214 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:54.580235 18524 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.580540 18214 ts_tablet_manager.cc:1403] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.000s
I20260709 12:20:54.583825 18213 tablet_bootstrap.cc:654] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.592305 18214 raft_consensus.cc:359] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.593199 18214 raft_consensus.cc:385] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.593493 18214 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.594287 18214 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.596145 18524 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.596702 18214 ts_tablet_manager.cc:1434] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260709 12:20:54.599134 18268 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.599748 18268 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.599152 18214 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:54.602888 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:54.603917 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:54.604306 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.604727 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:54.605036 18022 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:54.606503 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:20:54.606837 18138 leader_election.cc:343] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.607432 18546 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:54.607820 18546 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:54.608174 18546 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.609236 18524 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.616133 18215 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:54.616911 18214 tablet_bootstrap.cc:654] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.616940 18215 ts_tablet_manager.cc:1403] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.059s	user 0.018s	sys 0.002s
I20260709 12:20:54.618278 18213 tablet_bootstrap.cc:492] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:54.618062 18560 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.618822 18560 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.621251 18560 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:54.629308 18214 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:54.619459 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:54.625566 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:54.622280 18213 ts_tablet_manager.cc:1403] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.047s	user 0.016s	sys 0.001s
I20260709 12:20:54.630328 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:54.625736 18268 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.620972 18215 raft_consensus.cc:359] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.631975 18215 raft_consensus.cc:385] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.631778 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:54.632390 18215 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.632879 18560 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:54.633278 18560 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:54.633774 18560 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.633304 18215 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.629951 18214 ts_tablet_manager.cc:1403] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.012s
I20260709 12:20:54.634673 18213 raft_consensus.cc:359] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.622628 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.635593 18213 raft_consensus.cc:385] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.637173 18215 ts_tablet_manager.cc:1434] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.020s	user 0.003s	sys 0.003s
I20260709 12:20:54.638844 18213 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.639683 18213 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:54.642449 18064 leader_election.cc:343] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.648669 18546 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.650542 18215 tablet_bootstrap.cc:492] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:54.652320 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:54.653154 18023 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.652406 18213 ts_tablet_manager.cc:1434] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.002s
I20260709 12:20:54.664578 18023 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:54.630896 18096 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.659346 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:54.671904 18213 tablet_bootstrap.cc:492] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:54.670940 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:54.672713 18214 raft_consensus.cc:359] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.673406 18215 tablet_bootstrap.cc:654] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.673568 18214 raft_consensus.cc:385] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.674010 18214 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.677359 18214 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.676707 17919 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:54.678669 18268 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:54.679184 18268 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:54.680243 18214 ts_tablet_manager.cc:1434] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.045s	user 0.008s	sys 0.000s
I20260709 12:20:54.679992 18268 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.676961 18093 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.660306 18546 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.700327 18214 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:54.698673 18560 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.705303 18096 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:54.707019 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.708253 18268 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:54.708684 18560 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:54.708786 18268 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:54.709467 18560 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.710137 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:54.709970 18560 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.709801 18268 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:54.711936 18064 leader_election.cc:343] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.713546 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:54.714922 18172 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.722020 18093 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:54.729128 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:54.730826 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.733048 18561 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:54.733690 18561 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:54.734794 18561 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.730418 17920 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:54.738943 18214 tablet_bootstrap.cc:654] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.745440 18529 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.746301 18529 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.750483 18560 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:54.757319 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.761718 18215 tablet_bootstrap.cc:492] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:54.762611 18215 ts_tablet_manager.cc:1403] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.112s	user 0.011s	sys 0.026s
I20260709 12:20:54.763717 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:54.764565 18164 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:54.764783 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:54.765806 18017 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:54.770721 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:54.773434 18524 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.774003 18524 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:54.775348 17989 leader_election.cc:343] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:54.777160 17987 leader_election.cc:343] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.777805 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:54.778932 18483 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:54.759565 18529 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:54.784866 18213 tablet_bootstrap.cc:654] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:54.784327 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:54.789929 18560 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:54.797976 18560 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:54.798432 18560 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.803220 18524 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.806468 17920 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:54.812767 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:54.813607 18098 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:54.816879 18268 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:54.817472 18268 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.819235 18215 raft_consensus.cc:359] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.820031 18268 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.820319 18215 raft_consensus.cc:385] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.820417 18136 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:54.820653 18215 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.824132 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.814841 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:54.825412 18215 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:54.826767 18138 leader_election.cc:343] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.825881 18537 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:54.827888 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:54.828518 18458 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:54.829491 18537 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:54.832789 18537 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:54.837564 18215 ts_tablet_manager.cc:1434] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.075s	user 0.008s	sys 0.000s
I20260709 12:20:54.836971 17919 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:54.838905 18215 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
W20260709 12:20:54.851444 18138 leader_election.cc:343] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.870100 18022 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:54.916114 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51309.2us
W20260709 12:20:54.916320 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50005.7us
W20260709 12:20:54.917246 18136 leader_election.cc:343] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:54.924660 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:54.933640 18560 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.936954 18560 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:54.948748 18214 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:54.953447 18214 ts_tablet_manager.cc:1403] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.253s	user 0.020s	sys 0.019s
I20260709 12:20:54.966028 18213 tablet_bootstrap.cc:492] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:54.966889 18213 ts_tablet_manager.cc:1403] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.300s	user 0.030s	sys 0.003s
W20260709 12:20:54.971854 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54653.4us
I20260709 12:20:54.971177 18537 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.972095 18524 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:54.971216 18213 raft_consensus.cc:359] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.979153 18213 raft_consensus.cc:385] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:54.979482 18213 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:54.980446 18213 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:54.989959 18213 ts_tablet_manager.cc:1434] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.023s	user 0.010s	sys 0.000s
I20260709 12:20:54.992527 18213 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:54.997112 18537 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:54.998346 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:55.003546 18138 leader_election.cc:343] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:55.008160 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78743.1us
I20260709 12:20:55.010010 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:55.010696 18017 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.016664 18214 raft_consensus.cc:359] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.017889 18214 raft_consensus.cc:385] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.018221 18214 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.019098 18214 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.020819 18213 tablet_bootstrap.cc:654] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.027034 18213 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:55.027627 18213 ts_tablet_manager.cc:1403] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.035s	user 0.011s	sys 0.002s
I20260709 12:20:55.029973 18525 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:55.030965 18213 raft_consensus.cc:359] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.031724 18213 raft_consensus.cc:385] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.032027 18213 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.032827 18213 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.035053 18213 ts_tablet_manager.cc:1434] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:55.036640 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:55.036972 18213 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:55.037318 18164 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.044363 18213 tablet_bootstrap.cc:654] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.048302 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:55.049080 18099 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.055104 18017 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:55.060271 18213 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:55.060873 18213 ts_tablet_manager.cc:1403] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.012s
I20260709 12:20:55.064025 18213 raft_consensus.cc:359] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.064836 18213 raft_consensus.cc:385] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.065104 18213 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.065883 18213 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.069140 18213 ts_tablet_manager.cc:1434] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:55.069480 18099 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:55.070889 18214 ts_tablet_manager.cc:1434] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.111s	user 0.008s	sys 0.000s
I20260709 12:20:55.072072 18213 tablet_bootstrap.cc:492] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:55.078279 18214 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:55.078768 18164 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:55.084853 18215 tablet_bootstrap.cc:654] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.086922 18214 tablet_bootstrap.cc:654] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.097956 18525 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.098873 18525 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.104370 18525 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.107280 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:55.108495 18525 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:55.109113 18525 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:55.110706 18525 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.121282 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:55.123968 18136 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:55.134716 18064 leader_election.cc:343] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.136898 18524 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:55.138022 18524 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:55.139062 18524 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.152104 18537 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:55.178201 18213 tablet_bootstrap.cc:700] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent opening tablet: real 0.101s	user 0.010s	sys 0.000s
I20260709 12:20:55.185478 18213 tablet_bootstrap.cc:654] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.189870 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:55.205541 18064 leader_election.cc:343] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.206190 18064 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:55.207312 18215 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:55.208400 18215 ts_tablet_manager.cc:1403] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.370s	user 0.012s	sys 0.028s
I20260709 12:20:55.223448 18215 raft_consensus.cc:359] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.224609 18215 raft_consensus.cc:385] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.225016 18215 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.226029 18215 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.230262 18215 ts_tablet_manager.cc:1434] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.002s
I20260709 12:20:55.231654 18215 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:55.213703 18525 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:55.241014 18214 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:55.241691 18214 ts_tablet_manager.cc:1403] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.164s	user 0.022s	sys 0.043s
I20260709 12:20:55.250494 18560 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.265182 18560 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.266898 18214 raft_consensus.cc:359] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.267585 18560 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.267817 18214 raft_consensus.cc:385] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.268174 18214 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.268936 18214 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.271625 18214 ts_tablet_manager.cc:1434] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.000s
I20260709 12:20:55.272886 18214 tablet_bootstrap.cc:492] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
W20260709 12:20:55.272708 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45974.2us
I20260709 12:20:55.274717 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:55.277989 18064 consensus_peers.cc:597] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:55.278407 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:55.276309 17919 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:55.281433 18064 leader_election.cc:343] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.283488 18215 tablet_bootstrap.cc:654] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:55.286993 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:55.300431 18420 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.307160 18420 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.334302 18420 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.336253 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:55.337255 18024 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:55.339382 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:55.356580 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:55.350847 18420 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:55.385432 18268 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.386093 18268 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.387316 18546 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:55.359548 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:55.388168 18268 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.391194 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:55.393023 18138 leader_election.cc:343] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.397234 18215 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:55.397928 18215 ts_tablet_manager.cc:1403] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.167s	user 0.023s	sys 0.004s
W20260709 12:20:55.399410 18136 leader_election.cc:343] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:55.401474 18064 leader_election.cc:343] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.389776 18420 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:55.401959 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:55.403780 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:55.409394 18136 leader_election.cc:343] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.409987 18136 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:55.407503 18525 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:55.402803 18602 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:20:55.402884 18420 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.420074 18214 tablet_bootstrap.cc:654] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.414160 18561 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:20:55.419165 18215 raft_consensus.cc:359] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.428526 18215 raft_consensus.cc:385] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.428900 18215 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.413067 17919 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:55.414183 18603 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:55.433583 18213 tablet_bootstrap.cc:492] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:55.435424 18213 ts_tablet_manager.cc:1403] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.364s	user 0.018s	sys 0.007s
I20260709 12:20:55.429648 18215 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.458377 18215 ts_tablet_manager.cc:1434] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.060s	user 0.008s	sys 0.000s
I20260709 12:20:55.466110 18529 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.492010 18529 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.509902 18529 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.553745 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:55.556354 17987 leader_election.cc:343] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.524435 18420 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.572121 18213 raft_consensus.cc:359] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.586410 18213 raft_consensus.cc:385] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.588224 18213 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.589195 18213 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:55.590375 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 123379us
I20260709 12:20:55.598627 18215 tablet_bootstrap.cc:492] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:55.602146 18213 ts_tablet_manager.cc:1434] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.166s	user 0.009s	sys 0.006s
I20260709 12:20:55.603018 18617 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.629153 18214 tablet_bootstrap.cc:492] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:55.629752 18214 ts_tablet_manager.cc:1403] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.357s	user 0.023s	sys 0.008s
I20260709 12:20:55.628094 18561 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:55.601442 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135784us
I20260709 12:20:55.603577 18617 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.652189 18547 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.652873 18547 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.655555 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:55.671814 18214 raft_consensus.cc:359] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.672716 18214 raft_consensus.cc:385] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.673031 18214 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.673736 18214 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.678821 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:55.643131 18561 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.680362 18213 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:55.667160 18169 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:55.679284 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:55.681936 18017 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.691272 18215 tablet_bootstrap.cc:654] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.691602 18214 ts_tablet_manager.cc:1434] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.062s	user 0.008s	sys 0.000s
I20260709 12:20:55.690460 18547 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.693594 18561 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.698074 18260 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:55.700503 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:55.700557 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:55.703341 18617 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.706423 18525 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:55.709980 18064 leader_election.cc:343] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.713624 18214 tablet_bootstrap.cc:492] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:55.714339 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:55.715065 18096 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:55.722043 17989 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:20:55.724231 18064 leader_election.cc:343] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.724622 18017 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:55.727778 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:55.728236 18420 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.730033 18213 tablet_bootstrap.cc:654] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.730405 18618 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:55.730978 18618 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:55.731438 18618 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.737015 18215 tablet_bootstrap.cc:492] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:55.737702 18215 ts_tablet_manager.cc:1403] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.139s	user 0.015s	sys 0.017s
I20260709 12:20:55.724763 18064 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:55.739701 18525 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:55.741039 18215 raft_consensus.cc:359] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.742714 18215 raft_consensus.cc:385] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.743072 18215 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.743542 18483 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:55.744081 18483 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:55.744428 18483 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.743777 18215 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.745666 18622 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.746259 18622 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.747594 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:55.748664 18213 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:55.748951 18622 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.749210 18213 ts_tablet_manager.cc:1403] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.073s	user 0.006s	sys 0.015s
I20260709 12:20:55.751657 18215 ts_tablet_manager.cc:1434] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20260709 12:20:55.752931 18215 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:55.756413 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:55.757443 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:55.759047 18016 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.761499 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:55.758244 18136 leader_election.cc:343] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.758287 18098 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:55.762837 18215 tablet_bootstrap.cc:654] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.775851 18215 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:55.776427 18215 ts_tablet_manager.cc:1403] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.005s
I20260709 12:20:55.779451 18215 raft_consensus.cc:359] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.780256 18215 raft_consensus.cc:385] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.780750 18215 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.781476 18215 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.784242 18215 ts_tablet_manager.cc:1434] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:55.785408 18215 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
W20260709 12:20:55.786088 18138 leader_election.cc:343] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.787361 18618 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.791671 18213 raft_consensus.cc:359] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.795154 18213 raft_consensus.cc:385] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.795495 18213 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.793990 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:55.793668 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:20:55.796427 18136 leader_election.cc:343] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.800115 18215 tablet_bootstrap.cc:654] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.801146 18213 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.799499 18547 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:55.803820 18420 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:55.805091 18420 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:55.779928 18483 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.808725 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:55.809989 18483 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.810074 18529 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:55.811879 18019 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:55.812103 18214 tablet_bootstrap.cc:654] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.815714 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:55.816582 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:55.818462 18169 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.820465 18268 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.821040 18268 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:55.823567 17987 leader_election.cc:343] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.823022 18618 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.831209 18138 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:55.832494 18268 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.812987 18420 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.834265 18213 ts_tablet_manager.cc:1434] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.083s	user 0.009s	sys 0.001s
I20260709 12:20:55.837486 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:55.839084 18022 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:55.837689 18629 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:20:55.840855 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:55.841750 18099 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.845090 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:55.847407 18096 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:55.845388 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:55.852797 18268 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:55.853211 18268 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:55.853513 18268 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.862654 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:20:55.867576 18138 leader_election.cc:343] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:55.878100 18213 tablet_bootstrap.cc:492] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:55.877418 18268 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.899519 18268 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:55.905539 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:55.906370 18017 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.907171 18099 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:55.913666 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:55.914407 18099 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:55.915732 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:55.917943 18617 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:55.918483 18617 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:55.919405 18617 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.922258 18215 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:55.922870 18215 ts_tablet_manager.cc:1403] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.138s	user 0.012s	sys 0.015s
I20260709 12:20:55.930994 18169 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:55.938861 18627 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.932641 18618 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:55.939519 18627 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.967088 18099 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:55.972114 18136 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:55.972657 18213 tablet_bootstrap.cc:654] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:55.973451 18215 raft_consensus.cc:359] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.974684 18215 raft_consensus.cc:385] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:55.957873 17989 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:55.974947 18215 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:55.973156 18420 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:55.976744 18420 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:55.977671 18529 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:55.977799 18420 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.978672 18529 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:55.979933 18529 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:55.985330 18627 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:55.987061 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:55.994413 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:55.995610 18017 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:55.997306 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:55.998853 18622 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:56.002888 18215 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.006397 18215 ts_tablet_manager.cc:1434] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.083s	user 0.009s	sys 0.001s
I20260709 12:20:56.007885 18215 tablet_bootstrap.cc:492] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.027153 18213 tablet_bootstrap.cc:492] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.027805 18213 ts_tablet_manager.cc:1403] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.160s	user 0.011s	sys 0.011s
I20260709 12:20:56.030526 18214 tablet_bootstrap.cc:492] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:56.031121 18214 ts_tablet_manager.cc:1403] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.318s	user 0.018s	sys 0.040s
I20260709 12:20:56.034225 18213 raft_consensus.cc:359] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.035137 18213 raft_consensus.cc:385] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.035481 18213 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.031519 17920 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:56.036177 18213 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.037309 17920 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:56.038902 18214 raft_consensus.cc:359] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.040419 18213 ts_tablet_manager.cc:1434] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.003s
I20260709 12:20:56.040730 18214 raft_consensus.cc:385] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.041170 18214 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.041643 18213 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:56.042354 18214 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.043767 18215 tablet_bootstrap.cc:654] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.045452 18214 ts_tablet_manager.cc:1434] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.001s
I20260709 12:20:56.049747 18214 tablet_bootstrap.cc:492] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:56.053247 18213 tablet_bootstrap.cc:654] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.054857 18214 tablet_bootstrap.cc:654] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.076951 18213 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.077551 18213 ts_tablet_manager.cc:1403] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.036s	user 0.020s	sys 0.010s
I20260709 12:20:56.084046 18214 tablet_bootstrap.cc:492] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:56.084937 18214 ts_tablet_manager.cc:1403] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.035s	user 0.017s	sys 0.007s
I20260709 12:20:56.088639 18214 raft_consensus.cc:359] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.089406 18214 raft_consensus.cc:385] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.090445 18215 tablet_bootstrap.cc:492] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.091080 18215 ts_tablet_manager.cc:1403] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.084s	user 0.014s	sys 0.001s
I20260709 12:20:56.089699 18214 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.094959 18214 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.097309 18215 raft_consensus.cc:359] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.098223 18215 raft_consensus.cc:385] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.098517 18215 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.099241 18215 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.100145 18213 raft_consensus.cc:359] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.100875 18213 raft_consensus.cc:385] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.101150 18213 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.101601 18215 ts_tablet_manager.cc:1434] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.001s
I20260709 12:20:56.101904 18213 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.102795 18215 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.104535 18213 ts_tablet_manager.cc:1434] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.008s
I20260709 12:20:56.105651 17920 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:56.106865 18214 ts_tablet_manager.cc:1434] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.002s
I20260709 12:20:56.108598 18214 tablet_bootstrap.cc:492] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:56.110147 18215 tablet_bootstrap.cc:654] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.116711 18214 tablet_bootstrap.cc:654] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.119211 18213 tablet_bootstrap.cc:492] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:56.140544 18215 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.141167 18215 ts_tablet_manager.cc:1403] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.021s
I20260709 12:20:56.143338 18213 tablet_bootstrap.cc:654] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.144210 18215 raft_consensus.cc:359] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.144960 18215 raft_consensus.cc:385] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.145208 18215 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.145953 18215 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.148164 18215 ts_tablet_manager.cc:1434] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260709 12:20:56.149703 18215 tablet_bootstrap.cc:492] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.156760 18215 tablet_bootstrap.cc:654] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.172036 18215 tablet_bootstrap.cc:492] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.173219 18215 ts_tablet_manager.cc:1403] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.004s
I20260709 12:20:56.177959 18215 raft_consensus.cc:359] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.178784 18215 raft_consensus.cc:385] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.179095 18215 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.179975 18215 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.182610 18215 ts_tablet_manager.cc:1434] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:20:56.183740 18215 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.184235 18213 tablet_bootstrap.cc:492] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.185106 18213 ts_tablet_manager.cc:1403] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.066s	user 0.017s	sys 0.003s
I20260709 12:20:56.191594 18215 tablet_bootstrap.cc:654] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.206527 18215 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.207134 18215 ts_tablet_manager.cc:1403] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.003s
I20260709 12:20:56.210189 18215 raft_consensus.cc:359] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.210999 18215 raft_consensus.cc:385] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.211269 18215 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.211958 18215 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.213764 18215 ts_tablet_manager.cc:1434] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:56.215155 18215 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.219764 18019 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.219446 18213 raft_consensus.cc:359] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.221470 18213 raft_consensus.cc:385] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.222066 18213 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.223153 18213 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.227546 18213 ts_tablet_manager.cc:1434] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.042s	user 0.006s	sys 0.001s
I20260709 12:20:56.229445 18213 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:56.232664 18215 tablet_bootstrap.cc:654] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.244131 18213 tablet_bootstrap.cc:654] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.244613 18529 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.245183 18529 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.248823 18529 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.250063 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.251991 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:56.252938 18096 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:56.255234 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:56.256443 18529 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:56.256975 18529 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.257426 18529 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:56.263984 17989 leader_election.cc:343] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.274411 18622 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.275328 18622 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.290597 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:56.291643 18096 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:56.273897 18561 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.294976 17987 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:56.297297 18483 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:56.303395 18483 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.304008 18483 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.293718 18627 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.307307 18627 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.310350 18622 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.295979 18561 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.317159 18627 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.325779 18627 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.326820 18627 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.330435 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:56.337016 17987 leader_election.cc:343] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.343593 18483 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.359040 18260 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.359903 18483 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.359956 18260 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.369700 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.370983 18169 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:56.373211 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.374395 18168 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:56.378484 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:56.379515 18168 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.339082 18420 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.387544 18420 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.390403 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:56.391762 18099 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:20:56.392235 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.373580 18627 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.399696 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:56.400723 18096 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.402707 17989 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:56.397683 18529 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.412757 18660 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:56.413854 18660 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.414273 18420 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:56.414517 18660 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.439220 18213 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.442487 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:56.449671 18617 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:56.454056 18529 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.441725 18213 ts_tablet_manager.cc:1403] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.213s	user 0.010s	sys 0.013s
I20260709 12:20:56.459240 18622 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.432327 17920 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:56.471637 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:56.479565 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:56.484788 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:56.488078 18561 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.495442 18260 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:56.499051 17989 leader_election.cc:343] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.492412 18213 raft_consensus.cc:359] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.506776 18213 raft_consensus.cc:385] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.507169 18213 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.507964 18213 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.508771 18660 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.510378 18213 ts_tablet_manager.cc:1434] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.000s
I20260709 12:20:56.511605 18213 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:56.514120 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:56.514968 18170 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.518890 18213 tablet_bootstrap.cc:654] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.528084 18660 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.533365 18213 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.533531 18622 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:56.534031 18213 ts_tablet_manager.cc:1403] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.003s
I20260709 12:20:56.537150 18213 raft_consensus.cc:359] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.525197 18170 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:56.538018 18213 raft_consensus.cc:385] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.539117 18213 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.539710 17989 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:56.540176 18213 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.541153 18622 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:56.541986 18622 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:56.542732 18213 ts_tablet_manager.cc:1434] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:56.544116 18213 tablet_bootstrap.cc:492] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:56.543294 18622 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.552397 18215 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.553076 18215 ts_tablet_manager.cc:1403] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.338s	user 0.130s	sys 0.122s
I20260709 12:20:56.556351 18215 raft_consensus.cc:359] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.557194 18215 raft_consensus.cc:385] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.557480 18215 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
W20260709 12:20:56.558314 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:56.558213 18215 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.560890 18215 ts_tablet_manager.cc:1434] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:20:56.561357 18213 tablet_bootstrap.cc:654] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.562568 18215 tablet_bootstrap.cc:492] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.574494 18660 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:56.589860 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:56.593904 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:56.599467 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:56.600200 18099 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:56.612344 18138 leader_election.cc:343] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.623252 18215 tablet_bootstrap.cc:654] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.633348 18099 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:56.634433 18168 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:56.637993 18420 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.640647 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.654445 18165 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:56.654723 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:56.654320 18420 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.656334 18627 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:56.656876 18627 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:56.642231 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.658154 18164 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:56.657792 18627 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.661330 18096 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:56.658280 18420 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:56.665122 18214 tablet_bootstrap.cc:492] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:56.665912 18214 ts_tablet_manager.cc:1403] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.558s	user 0.164s	sys 0.244s
W20260709 12:20:56.655743 18136 leader_election.cc:343] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.669039 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:56.669234 18214 raft_consensus.cc:359] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.670004 18214 raft_consensus.cc:385] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.670284 18214 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
W20260709 12:20:56.670528 18136 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:56.670667 18213 tablet_bootstrap.cc:492] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.671327 18213 ts_tablet_manager.cc:1403] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.128s	user 0.041s	sys 0.030s
I20260709 12:20:56.671378 18617 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:56.671133 18214 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.677203 18215 tablet_bootstrap.cc:492] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.677991 18215 ts_tablet_manager.cc:1403] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.116s	user 0.019s	sys 0.003s
I20260709 12:20:56.680437 18627 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.680980 18627 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.681907 18215 raft_consensus.cc:359] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.682763 18215 raft_consensus.cc:385] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.683421 18627 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.684713 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.683831 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:20:56.686690 17989 leader_election.cc:343] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.683109 18215 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.687717 18214 ts_tablet_manager.cc:1434] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 12:20:56.688941 18214 tablet_bootstrap.cc:492] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:56.690865 18215 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.693739 18215 ts_tablet_manager.cc:1434] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260709 12:20:56.695086 18215 tablet_bootstrap.cc:492] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.701412 18601 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:56.704038 18213 raft_consensus.cc:359] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.704871 18213 raft_consensus.cc:385] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.705145 18213 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.705807 18213 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.710872 18213 ts_tablet_manager.cc:1434] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
W20260709 12:20:56.718530 18064 leader_election.cc:343] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.722630 18215 tablet_bootstrap.cc:654] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:56.723798 18064 leader_election.cc:343] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.725447 18213 tablet_bootstrap.cc:492] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Bootstrap starting.
I20260709 12:20:56.726486 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:56.730754 18019 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:56.731766 18214 tablet_bootstrap.cc:654] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:56.740975 17987 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:56.743084 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:56.744031 18627 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:56.744294 18420 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:56.744632 18627 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:20:56.749651 17987 leader_election.cc:343] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.751319 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:56.745572 18627 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:56.753479 17987 leader_election.cc:343] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:56.753983 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:20:56.754745 18660 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:56.759981 18618 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:56.764788 17922 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:56.766868 17922 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:56.768191 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:56.769097 18094 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:56.775977 18213 tablet_bootstrap.cc:654] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.782352 18215 tablet_bootstrap.cc:492] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:56.782907 18215 ts_tablet_manager.cc:1403] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.088s	user 0.012s	sys 0.002s
I20260709 12:20:56.767266 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:56.784174 18614 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:56.784631 18614 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.785054 18614 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.786120 18215 raft_consensus.cc:359] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.786901 18215 raft_consensus.cc:385] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.787194 18215 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.790824 18215 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.793321 18215 ts_tablet_manager.cc:1434] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:20:56.808228 18215 tablet_bootstrap.cc:492] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:56.825198 18213 tablet_bootstrap.cc:492] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: No bootstrap required, opened a new log
I20260709 12:20:56.826776 18213 ts_tablet_manager.cc:1403] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent bootstrapping tablet: real 0.102s	user 0.014s	sys 0.001s
I20260709 12:20:56.850052 18214 tablet_bootstrap.cc:492] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:56.850773 18214 ts_tablet_manager.cc:1403] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.162s	user 0.006s	sys 0.011s
I20260709 12:20:56.851254 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:56.852232 18525 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:56.852641 18525 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.852996 18525 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.858937 18136 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:56.864193 17922 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:56.861506 18547 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.836419 18614 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.872537 18213 raft_consensus.cc:359] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.874439 18213 raft_consensus.cc:385] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.874819 18213 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.881157 18213 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.897395 18261 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.898343 18261 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.908299 18261 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:56.918865 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54969.7us
W20260709 12:20:56.927134 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71988.1us
I20260709 12:20:56.874575 18420 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.041s
I20260709 12:20:56.929524 18214 raft_consensus.cc:359] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.934119 18214 raft_consensus.cc:385] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.912150 18524 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.934484 18214 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.934844 18213 ts_tablet_manager.cc:1434] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent starting tablet: real 0.107s	user 0.003s	sys 0.006s
I20260709 12:20:56.937402 18214 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.945233 18214 ts_tablet_manager.cc:1434] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.094s	user 0.008s	sys 0.000s
I20260709 12:20:56.944159 18483 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:56.952903 18614 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:56.942345 18524 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.955632 18214 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:56.951973 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:56.973744 18215 tablet_bootstrap.cc:654] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.977126 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:56.978013 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:56.965553 18483 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:56.994820 18016 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.978681 18019 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.997844 18483 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:56.998610 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52635.3us
I20260709 12:20:57.001104 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:57.009234 18170 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.009282 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:57.023733 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:57.004711 18524 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:57.040222 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:57.048700 17987 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:20:57.059128 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43986.9us
I20260709 12:20:57.001230 18525 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.061933 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:57.075114 18138 leader_election.cc:343] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.075295 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:57.078116 18682 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.095916 18622 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20260709 12:20:57.124867 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76358.8us
W20260709 12:20:57.125862 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64036.4us
I20260709 12:20:57.129470 18529 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.130234 18529 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.131739 18525 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:57.132253 18529 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:57.138473 18093 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.139722 18214 tablet_bootstrap.cc:654] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:57.144172 18136 leader_election.cc:343] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.144750 18136 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:57.145457 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:57.145576 18685 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.146404 18099 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:57.149766 18547 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.155727 18170 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:57.157284 18215 tablet_bootstrap.cc:492] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:57.158152 18215 ts_tablet_manager.cc:1403] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.350s	user 0.022s	sys 0.000s
I20260709 12:20:57.158835 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:57.159938 18164 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:57.161479 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:57.166011 18529 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.206149 18669 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:57.224292 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46199.4us
I20260709 12:20:57.184764 18529 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:57.236469 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50984.6us
W20260709 12:20:57.248270 17987 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.218667 18669 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.237286 18529 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.297927 18019 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:20:57.301755 18681 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.302548 18681 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.347759 18669 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:57.357069 18681 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:57.357187 18529 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.360270 18529 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:57.360841 18214 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:57.361479 18214 ts_tablet_manager.cc:1403] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.406s	user 0.021s	sys 0.008s
W20260709 12:20:57.350821 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 166953us
W20260709 12:20:57.349771 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44796.1us
I20260709 12:20:57.363639 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:57.369689 18214 raft_consensus.cc:359] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.378809 18214 raft_consensus.cc:385] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.379125 18214 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.379873 18214 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.380900 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:57.380874 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:57.424506 18261 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.579473 18261 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.439002 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:57.392974 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:57.395615 18214 ts_tablet_manager.cc:1434] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.004s
I20260709 12:20:57.415704 18215 raft_consensus.cc:359] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.585073 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:57.386824 18483 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.187s
I20260709 12:20:57.564241 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:57.596462 18098 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:57.596589 18167 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.586568 18525 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.588904 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:57.384527 18682 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.596617 18022 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:57.598917 18215 raft_consensus.cc:385] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.604964 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:57.608235 18215 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.609155 18261 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:57.610162 18215 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.617280 18214 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:57.618693 18215 ts_tablet_manager.cc:1434] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.457s	user 0.024s	sys 0.008s
I20260709 12:20:57.550395 18529 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.061s
W20260709 12:20:57.622318 18138 leader_election.cc:343] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.616411 18682 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:20:57.613030 18020 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 0.
I20260709 12:20:57.606720 18525 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.653321 18215 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:57.638067 18682 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:57.696784 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:20:57.708061 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59416.5us
I20260709 12:20:57.713025 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:57.723932 18679 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.743254 18096 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.735582 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:57.750247 18171 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 0.
I20260709 12:20:57.739552 18679 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.754027 18679 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.742231 18627 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.774211 18697 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.775184 18697 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:57.818404 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48836.2us
I20260709 12:20:57.820645 18697 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:57.844385 18136 leader_election.cc:343] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.831282 18525 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:57.851713 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:57.852605 18173 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
W20260709 12:20:57.855098 18136 leader_election.cc:343] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.855657 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:57.840670 18483 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:20:57.840888 18622 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:20:57.864384 18214 tablet_bootstrap.cc:654] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.868664 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:57.863878 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:57.866811 18685 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.875227 18702 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.875766 18702 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.876112 18702 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.886858 18215 tablet_bootstrap.cc:654] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.889340 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:57.890334 18261 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:20:57.900350 18695 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.191s	user 0.004s	sys 0.000s
I20260709 12:20:57.894506 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:57.910421 18167 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:57.917348 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:57.909783 18679 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.928692 18627 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.934206 18627 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:57.934382 18679 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:20:57.935415 18627 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.947926 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:20:57.948952 18702 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.971251 18702 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:57.965010 18017 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:57.978569 18214 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:57.979435 18214 ts_tablet_manager.cc:1403] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.362s	user 0.017s	sys 0.003s
I20260709 12:20:57.963770 18614 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.986866 18614 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.987375 18614 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.988373 18215 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:57.989039 18215 ts_tablet_manager.cc:1403] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.356s	user 0.028s	sys 0.009s
I20260709 12:20:57.992328 18215 raft_consensus.cc:359] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.993130 18215 raft_consensus.cc:385] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.993472 18215 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.994248 18215 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:57.999055 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:57.999912 18021 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.001762 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:58.001897 18214 raft_consensus.cc:359] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.002739 18167 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.003975 18214 raft_consensus.cc:385] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:58.004424 18214 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:58.005398 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:58.006580 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:58.005201 18214 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.011343 18215 ts_tablet_manager.cc:1434] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.022s	user 0.001s	sys 0.005s
I20260709 12:20:58.010566 17919 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.017767 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:58.019912 18215 tablet_bootstrap.cc:492] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Bootstrap starting.
I20260709 12:20:58.020869 18260 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:58.025156 18214 ts_tablet_manager.cc:1434] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.045s	user 0.008s	sys 0.000s
I20260709 12:20:58.029448 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:58.037654 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:58.067260 18214 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
W20260709 12:20:58.082260 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44542.2us
I20260709 12:20:58.121568 18529 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.124058 18529 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.141885 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:58.142117 18021 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:20:58.142714 18173 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:58.144358 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:58.146046 18529 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.151767 18697 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:58.152274 18697 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.152652 18697 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.154072 18214 tablet_bootstrap.cc:654] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:58.150331 18614 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.155057 18215 tablet_bootstrap.cc:654] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:58.156189 18095 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 1: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:20:58.156445 18167 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:58.160786 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:20:58.164950 18136 leader_election.cc:343] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:58.168130 18614 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.174850 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:58.181245 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:58.182549 18525 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.183918 18601 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:58.184558 18601 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:20:58.185513 18601 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.186369 18529 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:58.191823 18529 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:58.195827 18529 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.199265 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:20:58.209496 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:58.210682 18168 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:58.216502 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:20:58.222761 17987 leader_election.cc:343] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:58.230929 18697 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.250945 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:20:58.253417 17987 leader_election.cc:343] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:58.236402 18697 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.259713 18215 tablet_bootstrap.cc:492] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: No bootstrap required, opened a new log
I20260709 12:20:58.260344 18215 ts_tablet_manager.cc:1403] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent bootstrapping tablet: real 0.241s	user 0.022s	sys 0.009s
I20260709 12:20:58.249177 18614 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:58.254359 18682 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:20:58.257539 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:58.277072 18167 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.295218 18096 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.295634 18215 raft_consensus.cc:359] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.297662 18215 raft_consensus.cc:385] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:58.298274 18215 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Initialized, Role: FOLLOWER
I20260709 12:20:58.300184 18215 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.316047 18214 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:58.338163 18214 ts_tablet_manager.cc:1403] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.271s	user 0.034s	sys 0.019s
I20260709 12:20:58.347760 18215 ts_tablet_manager.cc:1434] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent starting tablet: real 0.087s	user 0.013s	sys 0.000s
W20260709 12:20:58.401484 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141358us
I20260709 12:20:58.436157 17920 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.459555 18627 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.462725 18167 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:58.446870 18214 raft_consensus.cc:359] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.489952 18214 raft_consensus.cc:385] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:58.490357 18214 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:58.491115 18214 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.492839 18214 ts_tablet_manager.cc:1434] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.154s	user 0.005s	sys 0.001s
I20260709 12:20:58.497679 18627 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.507426 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:58.507797 18214 tablet_bootstrap.cc:492] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:58.510908 18694 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:58.511601 18694 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:58.512650 18694 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.481128 17920 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.535986 18682 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:20:58.536618 18682 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.540539 18682 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.545558 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:58.545687 17922 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:58.548898 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:58.551891 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:58.552209 18214 tablet_bootstrap.cc:654] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:58.552814 18678 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.585935 18696 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.586553 18696 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.589031 18696 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.590359 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:58.591217 18165 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 0.
I20260709 12:20:58.592784 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:20:58.593807 18696 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:58.594292 18696 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:58.594709 18696 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.604630 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:20:58.615410 17987 leader_election.cc:343] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:58.616180 17919 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.614478 18696 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.620903 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:20:58.621666 18168 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.633890 18627 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.638983 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:20:58.644693 18696 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:58.655942 17987 leader_election.cc:343] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:58.659904 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:20:58.660102 18683 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.661736 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:20:58.663087 18696 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:58.663666 18696 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:20:58.664659 18696 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.664628 18214 tablet_bootstrap.cc:492] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:58.669492 18214 ts_tablet_manager.cc:1403] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.162s	user 0.030s	sys 0.034s
I20260709 12:20:58.672915 18214 raft_consensus.cc:359] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:20:58.678503 18136 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.678673 18214 raft_consensus.cc:385] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:58.679114 18214 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:58.681871 18214 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.688577 18214 ts_tablet_manager.cc:1434] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:58.708472 18214 tablet_bootstrap.cc:492] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Bootstrap starting.
I20260709 12:20:58.715641 18214 tablet_bootstrap.cc:654] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:58.720377 18696 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.736627 17920 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:58.750695 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57336.1us
I20260709 12:20:58.758893 18682 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.782923 18682 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.788419 18682 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.805835 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:20:58.818202 17970 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:58.816890 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:58.830284 18168 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:58.845485 18214 tablet_bootstrap.cc:492] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: No bootstrap required, opened a new log
I20260709 12:20:58.846166 18214 ts_tablet_manager.cc:1403] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent bootstrapping tablet: real 0.138s	user 0.025s	sys 0.030s
I20260709 12:20:58.860415 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:58.867165 18682 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.874782 18214 raft_consensus.cc:359] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.875685 18214 raft_consensus.cc:385] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:58.875984 18214 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Initialized, Role: FOLLOWER
I20260709 12:20:58.878146 18678 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.878958 18678 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.881695 18678 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:58.876715 18214 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:58.885574 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:58.886821 18168 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 1: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:20:58.889909 18214 ts_tablet_manager.cc:1434] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.002s
I20260709 12:20:58.895699 18682 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:58.900578 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:59.706440 18742 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.707065 18742 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:59.709446 18742 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:59.747350 18064 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:59.748780 18682 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:59.757737 18682 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:59.764477 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:59.781183 18627 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:20:59.782037 18627 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:59.784391 18627 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:59.800467 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260709 12:20:59.802165 18678 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:20:59.823654 18678 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:59.805558 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:59.826905 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:20:59.828933 18696 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:20:59.829629 18696 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:59.830576 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:59.832484 18678 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:59.832785 18682 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:59.844918 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:59.859716 18092 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:59.865881 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:59.857544 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:20:59.873608 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:59.874961 18739 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:59.878461 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:20:59.881414 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:20:59.882838 18745 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.885608 18694 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:20:59.886365 18694 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:20:59.832897 18696 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:20:59.896682 18694 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:20:59.982156 18749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.014979 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080003cae0 after lost signal to thread 16795
W20260709 12:21:00.032315 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159317us
W20260709 12:21:00.071910 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080003d1c0 after lost signal to thread 17956
I20260709 12:21:00.117674 18261 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.119776 18261 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.138110 18751 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.139422 18751 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.144910 18261 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.151111 18751 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.151842 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.156903 18696 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.158078 18696 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.162114 18758 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.163236 18758 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.161689 18696 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:00.172314 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 268397us
W20260709 12:21:00.173880 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c0dc0 after lost signal to thread 18048
W20260709 12:21:00.176620 17953 debug-util.cc:398] Leaking SignalData structure 0x7b0800159360 after lost signal to thread 18123
I20260709 12:21:00.181209 18696 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.182204 18696 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:00.191280 18754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.193737 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64986.4us
I20260709 12:21:00.204689 18757 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.205881 18757 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.208943 18757 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.185724 18696 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.212709 18758 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:00.235126 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2200 after lost signal to thread 18197
I20260709 12:21:00.233376 16792 server_base.cc:1061] running on GCE node
W20260709 12:21:00.242292 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c27a0 after lost signal to thread 18727
I20260709 12:21:00.244053 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.246066 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.247839 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.248332 18694 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.249066 18694 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.252146 18694 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.254499 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.258301 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.261381 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.263409 18739 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.264127 18739 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.267076 18739 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.267913 18739 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.268030 18694 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.268729 18739 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } }
I20260709 12:21:00.268976 18694 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:00.235970 18756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:00.270182 18761 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.270767 18758 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.280975 18739 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:00.280483 18761 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.287108 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:00.288107 18016 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:21:00.288369 18761 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.291186 18694 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.274017 18757 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:00.281211 18758 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:00.282886 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 194166us
I20260709 12:21:00.300239 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.301877 18739 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.301890 18702 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:21:00.305035 17987 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.306133 18702 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.306025 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.307633 18694 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:00.308039 18758 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.310228 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:00.310631 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:00.310940 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783599660310903 us; error 0 us; skew 500 ppm
I20260709 12:21:00.312218 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260709 12:21:00.314893 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75484.8us
I20260709 12:21:00.319290 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.335080 18694 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.335983 18694 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.336390 18757 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.341398 18739 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.341480 18751 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.342296 18739 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.344168 18751 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.348603 18694 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.349520 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.352406 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.356241 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.345645 18739 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.365244 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.365798 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.365334 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.367161 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.367542 18751 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.364501 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.368479 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.370256 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.372546 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.373013 18757 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.374130 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.375555 18696 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.376305 18696 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.377099 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.378930 18696 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.373948 18757 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.379114 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.381382 18757 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.374781 18751 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.375322 18739 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.385074 18696 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.384038 18739 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.385788 18696 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.387594 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.392908 18696 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.382274 18751 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.396128 18751 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.397147 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.398554 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.388839 18739 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.401966 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:00.405037 18682 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:00.405638 18682 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:00.406104 18682 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.415167 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.420308 18681 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:00.420490 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.421227 18681 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.423439 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.425969 18681 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:00.427819 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.428505 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.428907 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.428481 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.430475 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.432230 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:00.432315 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:00.442672 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.444336 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.459777 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.461447 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.464278 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.464753 16792 webserver.cc:533] Webserver started at http://127.16.102.4:44249/ using document root <none> and password file <none>
I20260709 12:21:00.465758 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:00.466208 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:00.466044 18682 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.466909 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:00.467556 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.468482 18682 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:00.474196 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-3-root/instance:
uuid: "910f8c8d787e4abbb092413c1f90ebf3"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-7xt3"
I20260709 12:21:00.474431 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.475615 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:00.492705 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:00.492901 17987 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.494014 18694 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.496423 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.497488 18757 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.498920 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.499758 18757 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.501386 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.502351 18757 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.525101 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.046s	user 0.009s	sys 0.000s
I20260709 12:21:00.527326 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.528692 17987 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.529733 17987 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.531211 18751 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.534732 18751 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.536597 17987 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.538530 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.540272 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.539744 18751 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.540337 18739 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.535984 18757 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.545619 17987 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.544147 18696 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.551195 18696 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.553781 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.554821 18696 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.555460 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.556881 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.557193 18694 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.558156 18694 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.560202 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.561131 18757 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.562644 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.563568 18757 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.565095 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.567987 18757 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.570806 17987 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.571854 18739 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.572484 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:00.582628 18739 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.587258 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:00.588557 18776 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.589197 18759 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.590039 16792 fs_manager.cc:730] Time spent opening block manager: real 0.025s	user 0.002s	sys 0.001s
I20260709 12:21:00.590651 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-3-root
uuid: "910f8c8d787e4abbb092413c1f90ebf3"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-7xt3"
I20260709 12:21:00.591257 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:00.672048 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:00.708761 18097 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.732324 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:00.734898 18682 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.762741 18748 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:00.764038 18759 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.764503 18759 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:00.765458 18759 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:00.767164 18759 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.786098 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:00.789486 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:00.796566 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:00.837261 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:00.837664 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.838121 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:00.838380 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:01.221458 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.4:46307
I20260709 12:21:01.221659 18839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.4:46307 every 8 connection(s)
I20260709 12:21:01.315681 16792 auto_leader_rebalancer-test.cc:439] add a tserver: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:01.343379 18840 heartbeater.cc:344] Connected to a master server at 127.16.102.62:42293
I20260709 12:21:01.344807 18840 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:01.347082 18840 heartbeater.cc:507] Master 127.16.102.62:42293 requested a full tablet report, sending...
I20260709 12:21:01.352254 17922 ts_manager.cc:194] Registered new tserver with Master: 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:01.361982 17922 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38984
I20260709 12:21:02.111929 18739 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:02.112661 18739 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.116026 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:02.117008 18098 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:02.123134 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:02.124001 18165 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:02.125730 17989 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:02.127122 18844 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:02.127576 18844 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:02.127977 18844 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:02.131726 18739 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:02.139902 18844 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.143479 18844 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:02.143966 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:02.145249 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:02.146260 18098 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:02.155074 18165 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:02.156701 17989 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:02.157649 18844 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:02.158293 18844 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:02.159199 18844 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.175792 17920 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:02.369789 18745 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.370214 18840 heartbeater.cc:499] Master 127.16.102.62:42293 was elected leader, sending a full tablet report...
I20260709 12:21:02.370486 18745 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.375021 18745 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:02.376121 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:02.379019 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:02.379818 18167 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:02.381048 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:02.382117 18850 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.382628 18850 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.383755 18850 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.393361 18850 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.395929 18850 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:02.397755 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:02.400234 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:02.401167 18165 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.413932 18165 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:02.415843 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:02.416851 18745 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.417486 18745 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:02.418517 18745 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.447847 17920 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:02.565356 18167 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:02.569298 18844 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:02.570899 18739 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:02.571369 18739 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:02.572425 18739 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:02.573776 18739 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:02.613749 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74691us
I20260709 12:21:02.640232 18020 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:02.649909 18097 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:02.649936 18165 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:02.652110 18739 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:02.656468 18857 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:02.672773 18844 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:02.676697 18097 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:02.721567 17920 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:02.830602 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88517.6us
I20260709 12:21:02.848132 18020 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:02.848644 18020 pending_rounds.cc:85] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 1
I20260709 12:21:02.851107 18020 pending_rounds.cc:107] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:02.857403 18020 raft_consensus.cc:2889] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:02.861871 18261 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:02.864097 18167 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:03.009485 17988 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df3d21a581ef40feafcce4ab8b817645. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:03.055012 18167 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:03.086006 18865 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:03.091770 18170 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.138141 18020 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:03.139495 18098 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:03.141031 18864 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:03.152813 18864 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:03.176614 18137 consensus_peers.cc:597] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18093241e4df42869e3d96cdf66a53e6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:03.182832 18884 ts_tablet_manager.cc:933] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:03.184289 18872 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.190526 18017 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.192270 18094 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.215428 18170 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.216626 17920 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:03.234937 18884 tablet_copy_client.cc:323] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:03.263365 18020 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:03.266788 18094 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:03.270983 18854 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:03.273095 18777 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:03.358908 18137 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53b6c1b05fff4ff4ab4260c2c231ed5e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:03.440900 18777 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.459901 18021 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.484200 18094 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.492429 17920 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.504614 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet df3d21a581ef40feafcce4ab8b817645 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:03.506620 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet df3d21a581ef40feafcce4ab8b817645 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-df3d21a581ef40feafcce4ab8b817645
I20260709 12:21:03.524914 18034 tablet_copy_source_session.cc:215] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:03.530301 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d21a581ef40feafcce4ab8b817645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:03.538178 18016 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.555985 18895 ts_tablet_manager.cc:933] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:03.556070 18094 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:03.557948 17988 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 578328e4e1fd478cbc58dd1048417480. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:03.560403 18165 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:03.564716 18895 tablet_copy_client.cc:323] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:03.573338 18892 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:03.597084 18892 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:03.593083 18884 tablet_copy_client.cc:806] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:03.601109 18884 tablet_copy_client.cc:670] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:03.614537 18884 tablet_copy_client.cc:538] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:03.637645 18739 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.645915 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet 18093241e4df42869e3d96cdf66a53e6 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:03.646728 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet 18093241e4df42869e3d96cdf66a53e6 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-18093241e4df42869e3d96cdf66a53e6
I20260709 12:21:03.648247 18094 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.652442 18884 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:03.696799 17920 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:03.705783 18183 tablet_copy_source_session.cc:215] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:03.712196 18170 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:03.719439 18895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18093241e4df42869e3d96cdf66a53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:03.749450 18900 ts_tablet_manager.cc:933] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:03.773128 18900 tablet_copy_client.cc:323] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:03.777580 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:03.778302 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet 53b6c1b05fff4ff4ab4260c2c231ed5e from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-53b6c1b05fff4ff4ab4260c2c231ed5e
I20260709 12:21:03.792111 18183 tablet_copy_source_session.cc:215] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:03.813956 18900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:03.826974 18884 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:03.829754 18895 tablet_copy_client.cc:806] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:03.838503 18895 tablet_copy_client.cc:670] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:03.831516 18884 tablet_bootstrap.cc:492] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:03.839943 18900 tablet_copy_client.cc:806] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:03.840189 18884 ts_tablet_manager.cc:1403] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.188s	user 0.022s	sys 0.015s
I20260709 12:21:03.842188 18900 tablet_copy_client.cc:670] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:03.848384 18884 raft_consensus.cc:359] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.849484 18884 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:03.849996 18895 tablet_copy_client.cc:538] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:03.849987 18900 tablet_copy_client.cc:538] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:03.852530 18884 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.867494 18900 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:03.873157 18884 ts_tablet_manager.cc:1434] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.031s	user 0.003s	sys 0.006s
I20260709 12:21:03.874301 18895 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:03.886622 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-df3d21a581ef40feafcce4ab8b817645 received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:03.887300 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-df3d21a581ef40feafcce4ab8b817645 on tablet df3d21a581ef40feafcce4ab8b817645 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:03.942993 18884 ts_tablet_manager.cc:933] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:03.956028 18884 tablet_copy_client.cc:323] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:03.960048 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet 578328e4e1fd478cbc58dd1048417480 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:03.960767 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet 578328e4e1fd478cbc58dd1048417480 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-578328e4e1fd478cbc58dd1048417480
I20260709 12:21:03.962194 18895 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:03.963320 18895 tablet_bootstrap.cc:492] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:03.964195 18895 ts_tablet_manager.cc:1403] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.096s	user 0.017s	sys 0.015s
I20260709 12:21:03.967978 18895 raft_consensus.cc:359] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.968950 18895 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:03.969689 18895 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:03.981640 18895 ts_tablet_manager.cc:1434] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260709 12:21:04.005909 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-18093241e4df42869e3d96cdf66a53e6 received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:04.006481 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-18093241e4df42869e3d96cdf66a53e6 on tablet 18093241e4df42869e3d96cdf66a53e6 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:04.021696 18815 raft_consensus.cc:1217] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:04.077307 18034 tablet_copy_source_session.cc:215] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:04.087754 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578328e4e1fd478cbc58dd1048417480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:04.104015 18900 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:04.115913 18815 raft_consensus.cc:1217] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:04.128465 18900 tablet_bootstrap.cc:492] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:04.129455 18900 ts_tablet_manager.cc:1403] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.262s	user 0.037s	sys 0.015s
I20260709 12:21:04.133404 18900 raft_consensus.cc:359] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:04.134436 18900 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:04.135228 18900 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:04.155735 18900 ts_tablet_manager.cc:1434] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:21:04.160157 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-53b6c1b05fff4ff4ab4260c2c231ed5e received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:04.160742 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-53b6c1b05fff4ff4ab4260c2c231ed5e on tablet 53b6c1b05fff4ff4ab4260c2c231ed5e with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:04.169540 18884 tablet_copy_client.cc:806] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:04.170539 18884 tablet_copy_client.cc:670] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:04.175800 18884 tablet_copy_client.cc:538] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:04.195465 18884 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:04.228466 18815 raft_consensus.cc:1217] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:04.276715 18884 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:04.278095 18884 tablet_bootstrap.cc:492] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:04.279065 18884 ts_tablet_manager.cc:1403] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.084s	user 0.026s	sys 0.017s
I20260709 12:21:04.283159 18884 raft_consensus.cc:359] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:04.284226 18884 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:04.285039 18884 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:04.288081 18884 ts_tablet_manager.cc:1434] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:21:04.290397 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-578328e4e1fd478cbc58dd1048417480 received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:04.290928 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-578328e4e1fd478cbc58dd1048417480 on tablet 578328e4e1fd478cbc58dd1048417480 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:04.330518 18815 raft_consensus.cc:1217] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:21:04.409232 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55808.7us
I20260709 12:21:04.573926 18854 raft_consensus.cc:1064] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:04.578665 18854 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:04.628692 18815 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.628260 18097 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.630978 18872 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:04.635811 18872 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:04.643313 18017 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.645728 18864 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:04.671046 18867 raft_consensus.cc:1064] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:04.674755 18871 raft_consensus.cc:1064] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:04.678560 18867 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:04.683153 18871 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:04.700219 18864 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:04.732019 18165 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.736068 18094 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.743034 18814 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.749212 18017 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.709491 18097 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:04.712522 18815 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.771399 18094 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:04.739114 18871 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:21:04.802579 18739 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:04.775030 18910 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20260709 12:21:04.826308 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52479us
I20260709 12:21:04.839620 18872 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:21:04.841536 18916 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:21:04.896682 18920 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:21:04.922747 18815 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:04.984608 17920 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:05.003238 18927 raft_consensus.cc:1064] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:05.008812 18927 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:05.030563 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143655us
W20260709 12:21:05.133332 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88707.2us
I20260709 12:21:05.141897 18924 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.191538 18024 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.214178 18170 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:05.229890 18918 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 12:21:05.237190 18814 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:05.199219 18813 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.251801 18929 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 12:21:05.230458 18919 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.299217 18090 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:05.314587 18925 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 12:21:05.335930 18170 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.315106 18096 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.313750 18099 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:05.403194 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 171356us
I20260709 12:21:05.408653 18918 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.417456 18166 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.423237 18097 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.427717 18814 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.452145 17922 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.459131 18024 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.461953 18813 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.488986 18171 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:05.518596 17920 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.523216 17921 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:05.525957 18814 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.528602 18949 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.553930 18167 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:05.562829 18814 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.566887 18024 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.573225 18926 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.579843 18024 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:05.579366 18016 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:05.594517 18919 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.603606 18814 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.610503 18096 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.611411 18928 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.615985 18815 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.623268 18739 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.623682 18925 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.626821 18097 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.627061 18814 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.641507 18739 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.649859 18812 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.655593 18096 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:05.665302 18948 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.667603 18933 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.678613 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet df3d21a581ef40feafcce4ab8b817645 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:05.680018 18925 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.698935 18815 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.701969 18813 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.718333 18739 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.718704 18918 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:05.720614 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 18093241e4df42869e3d96cdf66a53e6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:05.701969 18097 raft_consensus.cc:2955] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.708422 17921 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:05.790889 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63675.8us
W20260709 12:21:05.792464 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68594.3us
I20260709 12:21:05.803188 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 578328e4e1fd478cbc58dd1048417480 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:05.809036 17922 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:05.826054 18099 raft_consensus.cc:2955] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.852710 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:05.858197 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet df3d21a581ef40feafcce4ab8b817645 with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:05.858168 17919 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.852787 18090 raft_consensus.cc:2955] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.881698 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet 18093241e4df42869e3d96cdf66a53e6 with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:05.901435 17920 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.904289 18016 raft_consensus.cc:2955] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:05.919272 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 53b6c1b05fff4ff4ab4260c2c231ed5e with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:05.919633 18954 tablet_replica.cc:333] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:05.925166 18954 raft_consensus.cc:2243] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:05.967291 18954 raft_consensus.cc:2272] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:05.946382 18955 tablet_replica.cc:333] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:05.951798 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet 578328e4e1fd478cbc58dd1048417480 with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:05.973733 18955 raft_consensus.cc:2243] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:06.006717 18955 raft_consensus.cc:2272] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:06.009642 18956 tablet_replica.cc:333] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:06.011034 18954 ts_tablet_manager.cc:1916] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:06.016990 18956 raft_consensus.cc:2243] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:06.018357 18956 raft_consensus.cc:2272] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:06.022886 18955 ts_tablet_manager.cc:1916] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:06.024662 18956 ts_tablet_manager.cc:1916] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:06.036871 18954 ts_tablet_manager.cc:1929] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:06.037564 18954 log.cc:1199] T df3d21a581ef40feafcce4ab8b817645 P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/df3d21a581ef40feafcce4ab8b817645
I20260709 12:21:06.039602 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet df3d21a581ef40feafcce4ab8b817645 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:06.041111 18954 tablet_replica.cc:333] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:06.046115 18954 raft_consensus.cc:2243] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:06.051106 18954 raft_consensus.cc:2272] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:06.056066 18954 ts_tablet_manager.cc:1916] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:06.058751 18956 ts_tablet_manager.cc:1929] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:06.059396 18956 log.cc:1199] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/53b6c1b05fff4ff4ab4260c2c231ed5e
I20260709 12:21:06.067176 18955 ts_tablet_manager.cc:1929] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:06.067783 18955 log.cc:1199] T 18093241e4df42869e3d96cdf66a53e6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/18093241e4df42869e3d96cdf66a53e6
I20260709 12:21:06.069289 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 53b6c1b05fff4ff4ab4260c2c231ed5e (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:06.071211 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet 18093241e4df42869e3d96cdf66a53e6 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:06.079803 18954 ts_tablet_manager.cc:1929] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:06.080502 18954 log.cc:1199] T 578328e4e1fd478cbc58dd1048417480 P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/578328e4e1fd478cbc58dd1048417480
I20260709 12:21:06.082469 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet 578328e4e1fd478cbc58dd1048417480 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:07.121748 18171 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:07.133496 18137 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a83278d09c914df888a57be57eb08f8e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:07.134336 18099 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:07.136511 18016 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:07.136511 18928 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:07.138546 18872 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:07.165704 18919 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.171065 18017 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.173655 18099 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.206605 17921 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.227519 18099 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:07.250542 18167 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:07.252244 18921 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:07.283495 18017 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:07.303085 18941 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:07.313810 18971 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.316336 18171 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.365806 18017 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.376251 17921 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:07.405349 18063 consensus_peers.cc:597] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7a707b110a44c8c97606fdb8d7dece0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:07.421622 18166 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:07.440918 18137 consensus_peers.cc:597] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95f78d7b0f7f422eacbfa07bbc248bfa. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:07.444393 18017 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:07.445007 18099 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:07.447389 18928 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:07.460623 18872 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:07.474772 18919 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.497129 18016 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.503868 18099 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.516616 18166 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:07.534294 18137 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14d2c4b70e164856a205483966ca4963. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:07.546315 18024 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:07.548372 18919 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:07.552183 18090 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:07.554838 17918 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.573243 18959 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:07.589072 18872 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.591286 18017 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.640697 17918 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:07.645401 18099 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:07.693991 18983 ts_tablet_manager.cc:933] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:07.721470 18983 tablet_copy_client.cc:323] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:07.728272 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet a83278d09c914df888a57be57eb08f8e from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:07.728940 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet a83278d09c914df888a57be57eb08f8e from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-a83278d09c914df888a57be57eb08f8e
I20260709 12:21:07.742216 18183 tablet_copy_source_session.cc:215] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:07.761416 18983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83278d09c914df888a57be57eb08f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:07.766309 18986 ts_tablet_manager.cc:933] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:07.787943 18986 tablet_copy_client.cc:323] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:07.813094 18983 tablet_copy_client.cc:806] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:07.814193 18983 tablet_copy_client.cc:670] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:07.821215 18983 tablet_copy_client.cc:538] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:07.841099 18991 ts_tablet_manager.cc:933] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:07.847196 18983 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:07.874109 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet f7a707b110a44c8c97606fdb8d7dece0 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:07.874883 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet f7a707b110a44c8c97606fdb8d7dece0 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-f7a707b110a44c8c97606fdb8d7dece0
I20260709 12:21:07.889235 18109 tablet_copy_source_session.cc:215] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:07.894994 18986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a707b110a44c8c97606fdb8d7dece0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:07.904213 18983 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.905611 18983 tablet_bootstrap.cc:492] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:07.906585 18983 ts_tablet_manager.cc:1403] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.060s	user 0.021s	sys 0.012s
I20260709 12:21:07.950018 18991 tablet_copy_client.cc:323] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:07.952466 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet 95f78d7b0f7f422eacbfa07bbc248bfa from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:07.953132 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet 95f78d7b0f7f422eacbfa07bbc248bfa from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-95f78d7b0f7f422eacbfa07bbc248bfa
I20260709 12:21:07.962666 18986 tablet_copy_client.cc:806] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:07.963811 18986 tablet_copy_client.cc:670] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:07.968822 18183 tablet_copy_source_session.cc:215] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:07.971160 18986 tablet_copy_client.cc:538] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:07.980492 18994 ts_tablet_manager.cc:933] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:07.982167 18983 raft_consensus.cc:359] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:07.983163 18983 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:07.984232 18991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f78d7b0f7f422eacbfa07bbc248bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:07.983883 18983 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.000898 18994 tablet_copy_client.cc:323] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:08.005756 18983 ts_tablet_manager.cc:1434] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.099s	user 0.008s	sys 0.005s
I20260709 12:21:08.007336 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet 14d2c4b70e164856a205483966ca4963 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:08.007900 18182 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-a83278d09c914df888a57be57eb08f8e received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:08.008436 18182 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-a83278d09c914df888a57be57eb08f8e on tablet a83278d09c914df888a57be57eb08f8e with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:08.009284 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet 14d2c4b70e164856a205483966ca4963 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-14d2c4b70e164856a205483966ca4963
I20260709 12:21:08.022902 18183 tablet_copy_source_session.cc:215] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:08.027136 18994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d2c4b70e164856a205483966ca4963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:08.092896 18986 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:08.105571 18991 tablet_copy_client.cc:806] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:08.106616 18991 tablet_copy_client.cc:670] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:08.116236 18991 tablet_copy_client.cc:538] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:08.134068 18994 tablet_copy_client.cc:806] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:08.135071 18994 tablet_copy_client.cc:670] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:08.143101 18994 tablet_copy_client.cc:538] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:08.158521 18991 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:08.229336 18813 raft_consensus.cc:1217] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:08.272369 18994 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:08.288378 18986 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:08.289661 18986 tablet_bootstrap.cc:492] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:08.290623 18986 ts_tablet_manager.cc:1403] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.198s	user 0.023s	sys 0.023s
I20260709 12:21:08.295701 18986 raft_consensus.cc:359] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.300112 18986 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:08.301028 18986 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.320228 18986 ts_tablet_manager.cc:1434] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.004s
I20260709 12:21:08.349486 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-f7a707b110a44c8c97606fdb8d7dece0 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:08.350258 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-f7a707b110a44c8c97606fdb8d7dece0 on tablet f7a707b110a44c8c97606fdb8d7dece0 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:08.408926 18994 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:08.411024 18994 tablet_bootstrap.cc:492] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:08.411963 18994 ts_tablet_manager.cc:1403] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.140s	user 0.019s	sys 0.013s
I20260709 12:21:08.419337 18994 raft_consensus.cc:359] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.426201 18994 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:08.427070 18994 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.437481 18991 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:08.440683 18994 ts_tablet_manager.cc:1434] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.002s
I20260709 12:21:08.440836 18991 tablet_bootstrap.cc:492] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:08.453981 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-14d2c4b70e164856a205483966ca4963 received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:08.454629 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-14d2c4b70e164856a205483966ca4963 on tablet 14d2c4b70e164856a205483966ca4963 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:08.491070 18991 ts_tablet_manager.cc:1403] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.333s	user 0.016s	sys 0.022s
I20260709 12:21:08.495220 18991 raft_consensus.cc:359] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.496352 18991 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:08.497256 18991 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:08.533773 18991 ts_tablet_manager.cc:1434] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.001s
I20260709 12:21:08.536728 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-95f78d7b0f7f422eacbfa07bbc248bfa received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:08.537328 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-95f78d7b0f7f422eacbfa07bbc248bfa on tablet 95f78d7b0f7f422eacbfa07bbc248bfa with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:08.584231 18814 raft_consensus.cc:1217] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:08.665122 18814 raft_consensus.cc:1217] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:21:08.761629 18928 raft_consensus.cc:1064] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:08.764720 18928 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:08.785848 18814 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:08.787949 18016 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:08.788968 18099 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:08.789917 18928 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:08.791574 18919 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:08.793905 18919 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:08.839721 18919 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:08.839285 18099 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:08.867009 18813 raft_consensus.cc:1217] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:08.870123 18814 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:08.891613 17919 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:08.925400 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51207.3us
I20260709 12:21:08.946487 18021 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.054160 18166 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.069018 18021 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.071808 18959 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:09.072250 18814 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.074939 18959 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:09.083720 18978 raft_consensus.cc:1064] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:09.084527 18919 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.087666 18978 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.091451 18021 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.100884 18814 raft_consensus.cc:2955] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.117594 17922 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.147847 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a83278d09c914df888a57be57eb08f8e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.158190 18092 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:09.164033 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet a83278d09c914df888a57be57eb08f8e with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:09.158742 18814 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:09.159967 18021 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:09.167754 18872 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.172281 18959 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.176374 18959 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.178759 19012 tablet_replica.cc:333] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:09.180315 19012 raft_consensus.cc:2243] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:09.181509 19012 raft_consensus.cc:2272] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.192538 19012 ts_tablet_manager.cc:1916] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:09.195164 18872 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.203585 18813 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.206796 18921 raft_consensus.cc:1064] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:09.210655 18921 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.218783 19012 ts_tablet_manager.cc:1929] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:09.219475 19012 log.cc:1199] T a83278d09c914df888a57be57eb08f8e P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/a83278d09c914df888a57be57eb08f8e
I20260709 12:21:09.221449 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet a83278d09c914df888a57be57eb08f8e (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:09.248250 17922 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.249465 18814 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:09.253036 18921 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:09.257189 18021 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:09.258922 18171 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:09.266218 18171 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.235111 18016 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.289258 18980 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.295701 18997 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.232944 18092 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.331367 18812 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:09.335290 18919 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:09.347934 18872 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.355998 18812 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.370887 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 95f78d7b0f7f422eacbfa07bbc248bfa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.380481 17922 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:09.387387 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91961.8us
I20260709 12:21:09.399087 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet 95f78d7b0f7f422eacbfa07bbc248bfa with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:09.443945 19018 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.447958 19021 tablet_replica.cc:333] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:09.450884 18919 raft_consensus.cc:1064] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:09.453605 18919 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.462983 18813 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.467231 18171 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.474786 18099 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:09.479735 18959 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:09.496240 19021 raft_consensus.cc:2243] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:09.497622 19021 raft_consensus.cc:2272] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.499205 18021 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.502512 19021 ts_tablet_manager.cc:1916] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:09.502249 17922 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.530150 18092 raft_consensus.cc:2955] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.531469 18812 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:09.537143 18099 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.540069 18017 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:09.549178 18097 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:09.550655 18928 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.560349 18978 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.572126 18167 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.575526 18959 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:09.612564 18941 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:09.615156 18812 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.643828 19019 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.647892 19021 ts_tablet_manager.cc:1929] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:21:09.648531 19021 log.cc:1199] T 95f78d7b0f7f422eacbfa07bbc248bfa P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/95f78d7b0f7f422eacbfa07bbc248bfa
I20260709 12:21:09.649865 18941 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:09.661604 18166 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.662557 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet 95f78d7b0f7f422eacbfa07bbc248bfa (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:09.666195 18812 raft_consensus.cc:2955] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.676705 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f7a707b110a44c8c97606fdb8d7dece0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.678586 18928 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.686468 18813 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.688226 17921 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:09.705174 18021 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.706420 18092 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.717245 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet f7a707b110a44c8c97606fdb8d7dece0 with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:09.718534 19021 tablet_replica.cc:333] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:09.719785 19021 raft_consensus.cc:2243] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:09.720714 19021 raft_consensus.cc:2272] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.747622 17921 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.756901 19021 ts_tablet_manager.cc:1916] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:09.793983 18166 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:09.801783 18021 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.807468 19021 ts_tablet_manager.cc:1929] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:09.808060 19021 log.cc:1199] T f7a707b110a44c8c97606fdb8d7dece0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/f7a707b110a44c8c97606fdb8d7dece0
I20260709 12:21:09.813086 18812 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.813582 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet f7a707b110a44c8c97606fdb8d7dece0 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:09.821182 18959 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:21:09.824959 18928 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:21:09.843052 18812 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.838771 18928 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.862242 18021 raft_consensus.cc:2955] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.891671 17921 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:09.893236 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 14d2c4b70e164856a205483966ca4963 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.907631 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 14d2c4b70e164856a205483966ca4963 with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:09.909967 19028 tablet_replica.cc:333] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:09.912151 19028 raft_consensus.cc:2243] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:09.913692 19028 raft_consensus.cc:2272] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.918143 19028 ts_tablet_manager.cc:1916] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:09.940971 19028 ts_tablet_manager.cc:1929] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:09.941471 19028 log.cc:1199] T 14d2c4b70e164856a205483966ca4963 P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/14d2c4b70e164856a205483966ca4963
I20260709 12:21:09.943090 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 14d2c4b70e164856a205483966ca4963 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:10.998500 18021 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.006316 18166 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:11.007695 17988 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef30e67b67894670b0cf3980d6b0f624. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.007750 18090 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.008195 19032 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.009611 18918 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.025067 18166 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.022818 19032 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.029141 18092 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.065491 18017 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.067934 17921 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:11.115137 17988 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f144ec392b043d8b68e284ffecdee43. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.135458 18166 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.139598 18918 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:11.143813 18092 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.158888 18918 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.169690 19038 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.172531 18171 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.195416 18092 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.198204 17919 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.199941 18090 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:11.243829 18063 consensus_peers.cc:597] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7fd19d9bf374e6da56b7ead11c666e9. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.247504 18171 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.249300 18017 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.251773 19030 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.253059 19035 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.274210 18171 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.276139 19039 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.278877 18015 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.298130 17919 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.301333 18090 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:11.322055 18063 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e793f93c4f740b78f31d2f0dd643095. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.324620 18015 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.326395 18171 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.326753 19049 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.333014 19049 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.365612 18941 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.385385 18017 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.395658 18170 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:11.401017 17919 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.598868 19053 ts_tablet_manager.cc:933] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:11.602473 19053 tablet_copy_client.cc:323] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:11.604844 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet ef30e67b67894670b0cf3980d6b0f624 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:11.605679 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet ef30e67b67894670b0cf3980d6b0f624 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-ef30e67b67894670b0cf3980d6b0f624
I20260709 12:21:11.616037 18034 tablet_copy_source_session.cc:215] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:11.624071 19053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef30e67b67894670b0cf3980d6b0f624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:11.630214 19055 ts_tablet_manager.cc:933] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:11.633580 19055 tablet_copy_client.cc:323] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:11.636797 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet 5f144ec392b043d8b68e284ffecdee43 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:11.637542 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet 5f144ec392b043d8b68e284ffecdee43 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-5f144ec392b043d8b68e284ffecdee43
I20260709 12:21:11.649009 18034 tablet_copy_source_session.cc:215] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:11.662565 19055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f144ec392b043d8b68e284ffecdee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:11.666802 19053 tablet_copy_client.cc:806] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:11.667811 19053 tablet_copy_client.cc:670] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:11.673646 19053 tablet_copy_client.cc:538] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:11.686410 19053 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:11.699610 19055 tablet_copy_client.cc:806] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:11.700371 19055 tablet_copy_client.cc:670] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:11.705312 19055 tablet_copy_client.cc:538] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:11.718267 19055 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:11.730872 19053 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:11.732244 19053 tablet_bootstrap.cc:492] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:11.737641 19053 ts_tablet_manager.cc:1403] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.052s	user 0.022s	sys 0.006s
I20260709 12:21:11.742452 19058 ts_tablet_manager.cc:933] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:11.744199 19053 raft_consensus.cc:359] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.745505 19053 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:11.746624 19053 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.773568 19058 tablet_copy_client.cc:323] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:11.776288 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet 7e793f93c4f740b78f31d2f0dd643095 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:11.776994 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet 7e793f93c4f740b78f31d2f0dd643095 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-7e793f93c4f740b78f31d2f0dd643095
I20260709 12:21:11.779300 19053 ts_tablet_manager.cc:1434] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.041s	user 0.014s	sys 0.003s
I20260709 12:21:11.785694 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-ef30e67b67894670b0cf3980d6b0f624 received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:11.787878 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-ef30e67b67894670b0cf3980d6b0f624 on tablet ef30e67b67894670b0cf3980d6b0f624 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:11.799832 18109 tablet_copy_source_session.cc:215] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:11.804617 19058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e793f93c4f740b78f31d2f0dd643095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:11.815196 19055 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:11.816390 19055 tablet_bootstrap.cc:492] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:11.817276 19055 ts_tablet_manager.cc:1403] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.099s	user 0.023s	sys 0.018s
I20260709 12:21:11.820783 19055 raft_consensus.cc:359] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.821837 19055 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:11.822629 19055 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.861990 19055 ts_tablet_manager.cc:1434] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.044s	user 0.010s	sys 0.025s
I20260709 12:21:11.865566 19058 tablet_copy_client.cc:806] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:11.866679 19058 tablet_copy_client.cc:670] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:11.869415 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-5f144ec392b043d8b68e284ffecdee43 received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:11.869942 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-5f144ec392b043d8b68e284ffecdee43 on tablet 5f144ec392b043d8b68e284ffecdee43 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:11.874858 19053 ts_tablet_manager.cc:933] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:11.886528 19058 tablet_copy_client.cc:538] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:11.899286 19058 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:11.911348 19053 tablet_copy_client.cc:323] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:11.942018 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet b7fd19d9bf374e6da56b7ead11c666e9 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:11.942850 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet b7fd19d9bf374e6da56b7ead11c666e9 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-b7fd19d9bf374e6da56b7ead11c666e9
I20260709 12:21:11.958738 19058 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:11.958877 18109 tablet_copy_source_session.cc:215] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:11.960074 19058 tablet_bootstrap.cc:492] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:11.960980 19058 ts_tablet_manager.cc:1403] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.062s	user 0.023s	sys 0.015s
I20260709 12:21:11.969192 19053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fd19d9bf374e6da56b7ead11c666e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:11.977873 19058 raft_consensus.cc:359] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.978971 19058 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:11.979774 19058 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:11.990878 19058 ts_tablet_manager.cc:1434] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.030s	user 0.010s	sys 0.001s
I20260709 12:21:11.993300 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-7e793f93c4f740b78f31d2f0dd643095 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:11.993880 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-7e793f93c4f740b78f31d2f0dd643095 on tablet 7e793f93c4f740b78f31d2f0dd643095 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:11.999805 19053 tablet_copy_client.cc:806] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:12.000571 19053 tablet_copy_client.cc:670] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:12.004935 19053 tablet_copy_client.cc:538] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:12.014845 19053 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:12.043143 19053 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:12.044361 19053 tablet_bootstrap.cc:492] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:12.045265 19053 ts_tablet_manager.cc:1403] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.031s	user 0.019s	sys 0.008s
I20260709 12:21:12.052218 19053 raft_consensus.cc:359] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:12.053328 19053 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:12.054163 19053 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:12.056977 19053 ts_tablet_manager.cc:1434] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:21:12.059330 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-b7fd19d9bf374e6da56b7ead11c666e9 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:12.059850 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-b7fd19d9bf374e6da56b7ead11c666e9 on tablet b7fd19d9bf374e6da56b7ead11c666e9 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:12.124739 18812 raft_consensus.cc:1217] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:12.140153 18812 raft_consensus.cc:1217] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:12.177521 18812 raft_consensus.cc:1217] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:12.316192 18812 raft_consensus.cc:1217] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:12.508029 19038 raft_consensus.cc:1064] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:12.511682 19038 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:12.522248 19040 raft_consensus.cc:1064] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:12.525290 19040 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:12.549117 18813 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.553336 18090 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.553934 18813 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.554045 19066 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.556439 19066 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.561618 18165 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.562683 18918 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.572660 19049 raft_consensus.cc:1064] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
W20260709 12:21:12.619454 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63138.5us
I20260709 12:21:12.620956 19049 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:12.629096 18166 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.636471 19073 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.640062 19040 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.650187 18090 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.661082 19072 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.668871 18090 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.661670 19070 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.718061 18918 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.719869 18812 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.721484 18165 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.730700 17921 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.726850 18170 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.748236 18166 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.748425 18015 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.750774 19049 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.769549 18941 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.809980 18813 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.818804 17922 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:12.821717 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53769.5us
I20260709 12:21:12.832487 18941 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.863322 18814 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.870810 19049 raft_consensus.cc:1064] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:12.873606 19049 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:12.883564 18090 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.886370 18815 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.888198 19030 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.891227 18165 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.910017 18015 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.921963 19030 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.931252 19067 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.932433 18941 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.939219 18015 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.939947 18165 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:12.978669 18812 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.002614 19030 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.011485 18015 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.033541 18812 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.040371 18165 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.053131 17920 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:13.093482 18092 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:13.096016 17918 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.106642 18813 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.108847 19082 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.120445 18941 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.124435 18812 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.130198 18015 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.140861 19067 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:13.156241 18015 raft_consensus.cc:2955] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.170169 18017 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:13.170969 18016 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:13.175588 17919 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.192310 18092 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:13.187320 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7e793f93c4f740b78f31d2f0dd643095 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:13.188640 18812 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.189945 18097 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.197675 19038 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.199782 19070 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.204468 18812 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.205273 18165 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.210676 19070 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.212419 18812 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.212492 18171 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.215317 19030 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.216332 19066 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.218608 19030 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.219153 19073 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.228524 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet 7e793f93c4f740b78f31d2f0dd643095 with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:13.235271 18918 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.235797 19095 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.238708 18814 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.253664 18165 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.257356 18171 raft_consensus.cc:2955] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.266731 18812 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.271950 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ef30e67b67894670b0cf3980d6b0f624 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:13.273272 18813 raft_consensus.cc:2955] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.284636 18097 raft_consensus.cc:2955] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.284636 17919 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.301657 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b7fd19d9bf374e6da56b7ead11c666e9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:13.303917 19097 tablet_replica.cc:333] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:13.308892 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f144ec392b043d8b68e284ffecdee43 with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:13.309801 17922 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:13.329141 17918 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:13.360415 19098 tablet_replica.cc:333] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:13.363183 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5f144ec392b043d8b68e284ffecdee43 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:13.364190 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet b7fd19d9bf374e6da56b7ead11c666e9 with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:13.365955 19097 raft_consensus.cc:2243] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.373720 19097 raft_consensus.cc:2272] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.374154 19099 tablet_replica.cc:333] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:13.375685 19099 raft_consensus.cc:2243] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.376685 19099 raft_consensus.cc:2272] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.381985 19099 ts_tablet_manager.cc:1916] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:13.399094 19097 ts_tablet_manager.cc:1916] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:13.407516 19099 ts_tablet_manager.cc:1929] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:13.408317 19099 log.cc:1199] T b7fd19d9bf374e6da56b7ead11c666e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/b7fd19d9bf374e6da56b7ead11c666e9
I20260709 12:21:13.412859 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet b7fd19d9bf374e6da56b7ead11c666e9 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:13.412968 19098 raft_consensus.cc:2243] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.415536 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet ef30e67b67894670b0cf3980d6b0f624 with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:13.416565 19098 raft_consensus.cc:2272] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.420083 19098 ts_tablet_manager.cc:1916] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:13.433669 19097 ts_tablet_manager.cc:1929] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:13.434330 19097 log.cc:1199] T 7e793f93c4f740b78f31d2f0dd643095 P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/7e793f93c4f740b78f31d2f0dd643095
I20260709 12:21:13.436487 19097 tablet_replica.cc:333] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:13.436456 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet 7e793f93c4f740b78f31d2f0dd643095 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:13.438834 19097 raft_consensus.cc:2243] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.440310 19097 raft_consensus.cc:2272] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.440631 19098 ts_tablet_manager.cc:1929] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:13.441298 19098 log.cc:1199] T 5f144ec392b043d8b68e284ffecdee43 P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/5f144ec392b043d8b68e284ffecdee43
I20260709 12:21:13.443325 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 5f144ec392b043d8b68e284ffecdee43 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:13.443883 19097 ts_tablet_manager.cc:1916] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:13.462181 19097 ts_tablet_manager.cc:1929] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:13.462623 19097 log.cc:1199] T ef30e67b67894670b0cf3980d6b0f624 P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/ef30e67b67894670b0cf3980d6b0f624
I20260709 12:21:13.464715 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet ef30e67b67894670b0cf3980d6b0f624 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:14.504652 18097 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:14.512778 18063 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acbc4a634f954522904f2b865531f456. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:14.512903 18171 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:14.513302 18017 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:14.514497 19102 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:14.515529 19030 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:14.526249 19095 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.532145 18171 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.531291 18016 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.569602 17918 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.586598 18016 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:14.600764 17988 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1236f612fda5419089c0c9aa244bb5bc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:14.600955 18171 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:14.606171 18097 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:14.604938 19073 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:21:14.612850 19073 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:14.636152 19115 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.638265 18165 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.656167 18097 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:14.673343 17920 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:14.684983 18063 consensus_peers.cc:597] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e84e056b63c34d8da15d2fde3877279b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:14.686774 18090 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.692662 18015 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:14.693399 18170 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:14.694938 19030 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:14.698704 19095 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:14.725272 19095 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.728899 18016 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.728642 18171 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.744228 18015 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:14.759851 17921 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:14.784584 17988 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da76d5c707544d29a2f74d6015ec1cfc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:14.785276 18090 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:14.787674 19114 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:14.788151 18165 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:14.790329 19073 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:14.818509 19073 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.825647 18092 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.825647 18165 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:14.844852 17921 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:14.979642 19125 ts_tablet_manager.cc:933] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:14.982147 19125 tablet_copy_client.cc:323] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:14.984359 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet acbc4a634f954522904f2b865531f456 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:14.984992 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet acbc4a634f954522904f2b865531f456 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-acbc4a634f954522904f2b865531f456
I20260709 12:21:14.993674 18109 tablet_copy_source_session.cc:215] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:14.997982 19125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acbc4a634f954522904f2b865531f456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:15.018606 19125 tablet_copy_client.cc:806] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:15.019371 19125 tablet_copy_client.cc:670] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:15.025723 19125 tablet_copy_client.cc:538] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:15.038069 19125 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:15.063618 19125 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:15.064796 19125 tablet_bootstrap.cc:492] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:15.065653 19125 ts_tablet_manager.cc:1403] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.028s	user 0.023s	sys 0.004s
I20260709 12:21:15.070123 19125 raft_consensus.cc:359] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.071118 19125 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:15.072140 19125 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.103521 19125 ts_tablet_manager.cc:1434] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.037s	user 0.010s	sys 0.012s
I20260709 12:21:15.116078 19130 ts_tablet_manager.cc:933] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:15.126911 19129 ts_tablet_manager.cc:933] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:15.171959 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-acbc4a634f954522904f2b865531f456 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:15.172566 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-acbc4a634f954522904f2b865531f456 on tablet acbc4a634f954522904f2b865531f456 with peer 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:15.176012 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51872.6us
I20260709 12:21:15.178769 19129 tablet_copy_client.cc:323] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:15.183035 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet 1236f612fda5419089c0c9aa244bb5bc from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:15.183769 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet 1236f612fda5419089c0c9aa244bb5bc from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-1236f612fda5419089c0c9aa244bb5bc
I20260709 12:21:15.206846 18034 tablet_copy_source_session.cc:215] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:15.216270 19129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1236f612fda5419089c0c9aa244bb5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:15.258745 19130 tablet_copy_client.cc:323] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:15.262956 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet e84e056b63c34d8da15d2fde3877279b from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:15.263628 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet e84e056b63c34d8da15d2fde3877279b from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-e84e056b63c34d8da15d2fde3877279b
I20260709 12:21:15.287930 19129 tablet_copy_client.cc:806] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:15.288851 19129 tablet_copy_client.cc:670] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:15.304236 18109 tablet_copy_source_session.cc:215] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:15.307780 19129 tablet_copy_client.cc:538] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:15.323881 19129 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:15.335064 19130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84e056b63c34d8da15d2fde3877279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:15.369308 19130 tablet_copy_client.cc:806] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:15.369987 19130 tablet_copy_client.cc:670] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:15.374502 19130 tablet_copy_client.cc:538] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:15.375289 19129 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:15.376360 19129 tablet_bootstrap.cc:492] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:15.377156 19129 ts_tablet_manager.cc:1403] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.054s	user 0.018s	sys 0.008s
I20260709 12:21:15.380702 19129 raft_consensus.cc:359] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.381688 19129 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:15.382450 19129 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.386715 19130 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:15.391885 19129 ts_tablet_manager.cc:1434] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.000s
I20260709 12:21:15.394142 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-1236f612fda5419089c0c9aa244bb5bc received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:15.394665 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-1236f612fda5419089c0c9aa244bb5bc on tablet 1236f612fda5419089c0c9aa244bb5bc with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:15.394284 19125 ts_tablet_manager.cc:933] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:15.410205 19125 tablet_copy_client.cc:323] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:15.412369 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet da76d5c707544d29a2f74d6015ec1cfc from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:15.413008 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet da76d5c707544d29a2f74d6015ec1cfc from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-da76d5c707544d29a2f74d6015ec1cfc
I20260709 12:21:15.462822 18034 tablet_copy_source_session.cc:215] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:15.491497 18813 raft_consensus.cc:1217] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:15.491585 19125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da76d5c707544d29a2f74d6015ec1cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:15.521728 19125 tablet_copy_client.cc:806] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:15.523214 19125 tablet_copy_client.cc:670] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:15.529944 19125 tablet_copy_client.cc:538] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:15.531010 19130 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:15.534093 19130 tablet_bootstrap.cc:492] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:15.535108 19130 ts_tablet_manager.cc:1403] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.149s	user 0.043s	sys 0.061s
I20260709 12:21:15.539086 19130 raft_consensus.cc:359] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.541347 19130 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:15.542294 19130 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.548420 18813 raft_consensus.cc:1217] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 12:21:15.550100 19130 ts_tablet_manager.cc:1434] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:21:15.553074 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-e84e056b63c34d8da15d2fde3877279b received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:15.553646 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-e84e056b63c34d8da15d2fde3877279b on tablet e84e056b63c34d8da15d2fde3877279b with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:15.555995 19125 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:15.598305 19125 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:15.599534 19125 tablet_bootstrap.cc:492] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:15.600447 19125 ts_tablet_manager.cc:1403] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.045s	user 0.012s	sys 0.012s
I20260709 12:21:15.613374 19125 raft_consensus.cc:359] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.615682 19125 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:15.616489 19125 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:15.619364 19125 ts_tablet_manager.cc:1434] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 12:21:15.621491 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-da76d5c707544d29a2f74d6015ec1cfc received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:15.622012 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-da76d5c707544d29a2f74d6015ec1cfc on tablet da76d5c707544d29a2f74d6015ec1cfc with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:15.691602 18813 raft_consensus.cc:1217] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:15.782887 18813 raft_consensus.cc:1217] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:15.856686 19095 raft_consensus.cc:1064] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:15.860700 19095 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:15.868665 18813 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:15.869216 18171 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:15.870361 19095 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:15.870915 18017 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:15.871631 19100 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:15.872753 19102 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:15.909240 19100 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:15.912684 18165 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:15.935809 18017 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:15.941915 17918 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:15.964741 18813 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.032094 18099 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:16.040128 18813 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:16.040128 18017 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:16.042023 19102 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:16.043365 19100 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:16.053642 19030 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.055727 18813 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.060638 18017 raft_consensus.cc:2955] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.084789 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet acbc4a634f954522904f2b865531f456 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:16.086442 17921 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.106003 19095 raft_consensus.cc:1064] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:16.109302 19095 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:16.117666 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet acbc4a634f954522904f2b865531f456 with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:16.141235 18813 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:16.143344 19100 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:16.150447 18024 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:16.154547 19095 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:16.160869 19149 tablet_replica.cc:333] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:16.163410 19149 raft_consensus.cc:2243] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.168778 19149 raft_consensus.cc:2272] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.173484 19149 ts_tablet_manager.cc:1916] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:16.178820 18166 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:16.193221 19095 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:16.207239 19073 raft_consensus.cc:1064] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:16.213460 19073 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:16.223099 19095 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.225003 18024 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.236620 18813 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.238878 18166 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.240029 19115 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:16.242398 19115 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:16.244668 18097 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.253645 19115 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:16.277910 17920 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.292593 19149 ts_tablet_manager.cc:1929] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:16.293206 19149 log.cc:1199] T acbc4a634f954522904f2b865531f456 P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/acbc4a634f954522904f2b865531f456
I20260709 12:21:16.281641 19096 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.284123 18812 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.302712 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet acbc4a634f954522904f2b865531f456 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:16.309226 18170 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.323902 17920 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.280817 18813 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:16.348204 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85697.4us
I20260709 12:21:16.352610 18024 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:16.356801 18099 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.364907 18814 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:16.367108 18170 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:16.367206 19115 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:16.376608 19073 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:16.380851 19115 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.385654 18814 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.394901 19096 raft_consensus.cc:1064] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:16.396559 18090 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:16.398283 19096 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:16.421018 18812 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:16.423476 19121 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:16.442646 19102 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.444119 18813 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.450083 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1236f612fda5419089c0c9aa244bb5bc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:16.449128 17920 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.459889 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e84e056b63c34d8da15d2fde3877279b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:16.487753 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 1236f612fda5419089c0c9aa244bb5bc with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:16.484622 18167 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.475538 17921 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 4 to 5, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.539577 18165 raft_consensus.cc:2955] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.553412 19161 tablet_replica.cc:333] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:16.554951 19161 raft_consensus.cc:2243] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.555904 19161 raft_consensus.cc:2272] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:16.558305 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47502.5us
I20260709 12:21:16.560873 18813 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.561652 18170 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.565371 19115 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:16.575129 19073 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:16.589278 19161 ts_tablet_manager.cc:1916] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:16.589732 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet e84e056b63c34d8da15d2fde3877279b with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:16.625392 19161 ts_tablet_manager.cc:1929] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:16.625731 18171 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:16.630606 19095 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:16.634433 19161 log.cc:1199] T 1236f612fda5419089c0c9aa244bb5bc P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/1236f612fda5419089c0c9aa244bb5bc
I20260709 12:21:16.636771 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 1236f612fda5419089c0c9aa244bb5bc (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:16.642283 19166 tablet_replica.cc:333] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:16.644621 18171 raft_consensus.cc:2955] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.653065 19166 raft_consensus.cc:2243] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.649010 18090 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.652701 19073 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.673895 19166 raft_consensus.cc:2272] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.678236 19166 ts_tablet_manager.cc:1916] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:16.654649 18813 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.691395 18167 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:16.708788 17920 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:16.715920 19160 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:16.733510 19166 ts_tablet_manager.cc:1929] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260709 12:21:16.734251 19166 log.cc:1199] T e84e056b63c34d8da15d2fde3877279b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/e84e056b63c34d8da15d2fde3877279b
I20260709 12:21:16.755285 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet e84e056b63c34d8da15d2fde3877279b (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:16.762611 18097 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:17.310039 18016 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:17.317633 18097 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:17.318137 18813 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:17.319773 19160 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:17.324659 19160 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:17.334055 19160 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:17.343559 18097 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:17.342772 18813 raft_consensus.cc:2955] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:17.377595 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet da76d5c707544d29a2f74d6015ec1cfc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:17.392593 17921 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:17.415457 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet da76d5c707544d29a2f74d6015ec1cfc with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:17.419332 19172 tablet_replica.cc:333] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:17.420593 19172 raft_consensus.cc:2243] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:17.421397 19172 raft_consensus.cc:2272] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:17.429874 19172 ts_tablet_manager.cc:1916] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:17.448493 19172 ts_tablet_manager.cc:1929] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:17.448894 19172 log.cc:1199] T da76d5c707544d29a2f74d6015ec1cfc P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/da76d5c707544d29a2f74d6015ec1cfc
I20260709 12:21:17.450831 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet da76d5c707544d29a2f74d6015ec1cfc (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:18.500149 18097 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:18.509352 18063 consensus_peers.cc:597] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 666440ca642c4715bd2f211bb5f14d29. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:18.510393 18167 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:18.510767 18024 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:18.512008 19102 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:18.514905 19100 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:18.523437 19100 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.525513 18167 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.536335 18024 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.539812 18167 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:18.544858 17921 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.573139 18016 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:18.574919 19173 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:18.576773 18137 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c289d2c3e9fc40fbb7102b5788f74aab. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:18.577510 18097 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:18.583627 19173 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:18.616313 18090 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.612448 19173 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.626420 18024 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.647291 18090 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:18.662436 17919 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.668497 18167 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:18.669034 18063 consensus_peers.cc:597] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1f3da088a314a87a78898050dab39e9. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:18.670773 19177 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:18.673406 18017 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:18.678972 19177 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:18.686386 19182 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.695705 18167 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.700400 18017 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.710638 17919 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.704787 18090 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:18.739176 18171 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:18.740983 19102 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:18.742029 18017 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:18.742816 18063 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c53769058ca3443f805983b61a69328e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:18.748602 19102 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:18.764412 19177 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.767553 18171 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.770133 18017 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:18.789484 17921 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:19.190618 19196 ts_tablet_manager.cc:933] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:19.232185 19196 tablet_copy_client.cc:323] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:19.233211 19195 ts_tablet_manager.cc:933] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:19.235960 19195 tablet_copy_client.cc:323] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:19.460769 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet c289d2c3e9fc40fbb7102b5788f74aab from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:19.461486 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet c289d2c3e9fc40fbb7102b5788f74aab from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-c289d2c3e9fc40fbb7102b5788f74aab
I20260709 12:21:19.479786 18183 tablet_copy_source_session.cc:215] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:19.485585 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet 666440ca642c4715bd2f211bb5f14d29 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:19.492901 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet 666440ca642c4715bd2f211bb5f14d29 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-666440ca642c4715bd2f211bb5f14d29
I20260709 12:21:19.494243 19195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c289d2c3e9fc40fbb7102b5788f74aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:19.515105 19209 ts_tablet_manager.cc:933] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:19.548097 19207 ts_tablet_manager.cc:933] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:19.564273 19209 tablet_copy_client.cc:323] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:19.558063 18109 tablet_copy_source_session.cc:215] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:19.668190 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet f1f3da088a314a87a78898050dab39e9 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:19.669097 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet f1f3da088a314a87a78898050dab39e9 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-f1f3da088a314a87a78898050dab39e9
I20260709 12:21:19.680387 19196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 666440ca642c4715bd2f211bb5f14d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:19.683612 19195 tablet_copy_client.cc:806] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:19.684474 19195 tablet_copy_client.cc:670] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:19.686061 19207 tablet_copy_client.cc:323] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:19.694243 18108 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet c53769058ca3443f805983b61a69328e from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:19.694964 18108 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet c53769058ca3443f805983b61a69328e from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-c53769058ca3443f805983b61a69328e
I20260709 12:21:19.698750 19195 tablet_copy_client.cc:538] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:19.708541 18108 tablet_copy_source_session.cc:215] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:19.713523 19195 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:19.718393 19207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c53769058ca3443f805983b61a69328e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:21:19.726442 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53130us
I20260709 12:21:19.752815 19196 tablet_copy_client.cc:806] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:19.769999 19196 tablet_copy_client.cc:670] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:19.777411 19196 tablet_copy_client.cc:538] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:19.816711 19207 tablet_copy_client.cc:806] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:19.821348 19195 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:19.821879 19196 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:19.822448 19207 tablet_copy_client.cc:670] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:19.824249 19195 tablet_bootstrap.cc:492] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:19.825733 19195 ts_tablet_manager.cc:1403] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.112s	user 0.028s	sys 0.037s
I20260709 12:21:19.827762 19207 tablet_copy_client.cc:538] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:19.829989 19195 raft_consensus.cc:359] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:19.831159 19195 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:19.832240 19195 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:19.841231 19207 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:19.843211 19195 ts_tablet_manager.cc:1434] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.007s
I20260709 12:21:19.844063 18109 tablet_copy_source_session.cc:215] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:19.848830 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-c289d2c3e9fc40fbb7102b5788f74aab received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:19.869794 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-c289d2c3e9fc40fbb7102b5788f74aab on tablet c289d2c3e9fc40fbb7102b5788f74aab with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:19.876694 19209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f3da088a314a87a78898050dab39e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:19.895517 19207 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:19.896960 19207 tablet_bootstrap.cc:492] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:19.897958 19207 ts_tablet_manager.cc:1403] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.057s	user 0.021s	sys 0.016s
I20260709 12:21:19.901657 19207 raft_consensus.cc:359] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:19.902806 19207 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:19.903689 19207 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:19.907510 19207 ts_tablet_manager.cc:1434] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:21:19.909956 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-c53769058ca3443f805983b61a69328e received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:19.910660 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-c53769058ca3443f805983b61a69328e on tablet c53769058ca3443f805983b61a69328e with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:19.952814 19196 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:19.954154 19196 tablet_bootstrap.cc:492] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:19.955063 19196 ts_tablet_manager.cc:1403] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.148s	user 0.021s	sys 0.017s
I20260709 12:21:19.968365 19196 raft_consensus.cc:359] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:19.969485 19196 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:19.970333 19196 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:19.973222 19196 ts_tablet_manager.cc:1434] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 12:21:19.978327 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-666440ca642c4715bd2f211bb5f14d29 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:19.978888 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-666440ca642c4715bd2f211bb5f14d29 on tablet 666440ca642c4715bd2f211bb5f14d29 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:19.980712 18814 raft_consensus.cc:1217] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:20.024098 18812 raft_consensus.cc:1217] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:20.037472 19209 tablet_copy_client.cc:806] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:20.038398 19209 tablet_copy_client.cc:670] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:20.064318 19209 tablet_copy_client.cc:538] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:20.081480 19209 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:20.122396 19209 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:20.123651 19209 tablet_bootstrap.cc:492] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:20.124545 19209 ts_tablet_manager.cc:1403] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.043s	user 0.021s	sys 0.008s
I20260709 12:21:20.124852 18812 raft_consensus.cc:1217] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:20.133983 19209 raft_consensus.cc:359] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:20.137181 19209 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:20.138389 19209 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:20.141284 19209 ts_tablet_manager.cc:1434] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.002s
I20260709 12:21:20.147078 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-f1f3da088a314a87a78898050dab39e9 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:20.147630 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-f1f3da088a314a87a78898050dab39e9 on tablet f1f3da088a314a87a78898050dab39e9 with peer 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:20.203994 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c28c0 after lost signal to thread 16795
W20260709 12:21:20.205322 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2180 after lost signal to thread 17956
W20260709 12:21:20.206735 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080003d160 after lost signal to thread 18048
W20260709 12:21:20.207835 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2a80 after lost signal to thread 18123
W20260709 12:21:20.208962 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080028a220 after lost signal to thread 18197
W20260709 12:21:20.210173 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080019bf20 after lost signal to thread 18839
I20260709 12:21:20.579324 18812 raft_consensus.cc:1217] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:20.597193 19202 raft_consensus.cc:1064] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:20.601734 19202 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:20.660948 19105 raft_consensus.cc:1064] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:20.664294 19105 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:20.692979 18812 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.697319 18092 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.697702 19147 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.701296 19175 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:21:20.751331 19224 raft_consensus.cc:1064] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:20.758159 18019 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.762816 19212 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:21:20.766127 18812 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.776360 19202 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.760511 19224 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:20.784705 18024 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.798121 18167 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.804837 19202 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.812749 19175 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.825173 18024 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.826267 18092 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.846721 19225 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.887939 19182 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.904394 18019 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.911396 18812 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.914040 19205 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.921415 18167 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.916635 17921 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.926280 18024 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:20.935412 19220 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.938194 19225 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.937521 18813 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.959489 18812 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.963284 19225 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.968667 18015 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:20.984182 18170 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.023548 18813 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.057214 18167 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.177423 18167 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:21.184970 19182 raft_consensus.cc:1064] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:21.188315 19182 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:21.191062 18015 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:21.191429 18813 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:21.192562 19212 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:21.195429 19208 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:21.207508 19208 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.218068 17921 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.220528 17921 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.223096 18812 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.210436 18015 raft_consensus.cc:2955] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.246368 18167 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:21.247941 18024 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:21.254032 19182 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:21.255631 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c289d2c3e9fc40fbb7102b5788f74aab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:21.258554 19224 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:21.264688 17922 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.267098 18812 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:21.270499 19224 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:21.291025 18092 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:21.299693 18098 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:21.303187 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet c289d2c3e9fc40fbb7102b5788f74aab with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:21.365773 18813 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:21.369443 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:21.380748 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70231us
I20260709 12:21:21.387662 18170 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:21.388876 19224 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.395161 19225 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:21.405995 19243 tablet_replica.cc:333] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:21.414553 18021 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:21.415848 19243 raft_consensus.cc:2243] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:21.417725 18814 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:21.417034 18021 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.426965 19225 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:21.428297 19220 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:21.429800 19205 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.433439 19243 raft_consensus.cc:2272] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:21.462828 18813 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.462153 19225 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:21:21.471870 19182 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.488580 18815 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.497678 18167 raft_consensus.cc:2955] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.495042 19243 ts_tablet_manager.cc:1916] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:21.526826 17920 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:21.528826 17922 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.561409 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet c53769058ca3443f805983b61a69328e with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:21.561754 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c53769058ca3443f805983b61a69328e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:21.564741 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 666440ca642c4715bd2f211bb5f14d29 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:21.592229 17920 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:21.527338 18015 raft_consensus.cc:2955] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.622857 19243 ts_tablet_manager.cc:1929] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:21.629441 19243 log.cc:1199] T c289d2c3e9fc40fbb7102b5788f74aab P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/c289d2c3e9fc40fbb7102b5788f74aab
I20260709 12:21:21.632462 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet c289d2c3e9fc40fbb7102b5788f74aab (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:21.635453 19245 tablet_replica.cc:333] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
W20260709 12:21:21.636621 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93803.3us
I20260709 12:21:21.652626 18814 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:21.656955 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet 666440ca642c4715bd2f211bb5f14d29 with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:21.670249 19245 raft_consensus.cc:2243] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:21.671252 19245 raft_consensus.cc:2272] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:21.741546 19249 tablet_replica.cc:333] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:21.755075 18165 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:21.819032 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 104370us
W20260709 12:21:21.824623 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97504.6us
I20260709 12:21:21.842772 19245 ts_tablet_manager.cc:1916] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:21.899093 19249 raft_consensus.cc:2243] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:21.900152 19249 raft_consensus.cc:2272] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:21.901239 19242 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.144s	user 0.001s	sys 0.011s
W20260709 12:21:21.901970 19228 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.196s	user 0.011s	sys 0.011s
W20260709 12:21:21.914750 19232 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.164s	user 0.006s	sys 0.014s
W20260709 12:21:21.915632 19241 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.085s	user 0.016s	sys 0.013s
W20260709 12:21:21.938095 19239 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.066s	user 0.002s	sys 0.008s
W20260709 12:21:21.944849 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 124418us
W20260709 12:21:21.945773 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 118127us
W20260709 12:21:21.958386 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121897us
I20260709 12:21:21.985625 19249 ts_tablet_manager.cc:1916] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:22.093287 19249 ts_tablet_manager.cc:1929] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:22.093936 19249 log.cc:1199] T 666440ca642c4715bd2f211bb5f14d29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/666440ca642c4715bd2f211bb5f14d29
I20260709 12:21:22.099617 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet 666440ca642c4715bd2f211bb5f14d29 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:22.105600 18092 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.113524 19245 ts_tablet_manager.cc:1929] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:22.147794 18812 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:22.151165 19250 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:22.162170 19224 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:22.170006 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.140810 19245 log.cc:1199] T c53769058ca3443f805983b61a69328e P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/c53769058ca3443f805983b61a69328e
I20260709 12:21:22.173991 18812 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:22.182224 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet c53769058ca3443f805983b61a69328e (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:22.225263 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f1f3da088a314a87a78898050dab39e9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:22.226408 18171 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:22.232360 17921 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:22.249519 19266 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:22.266613 18171 raft_consensus.cc:2955] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:22.307552 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet f1f3da088a314a87a78898050dab39e9 with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:22.309221 19249 tablet_replica.cc:333] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:22.310680 19249 raft_consensus.cc:2243] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:22.311812 19249 raft_consensus.cc:2272] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:22.313755 18092 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:22.314631 18092 raft_consensus.cc:606] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:22.317215 19249 ts_tablet_manager.cc:1916] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:22.336745 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:22.337482 18171 raft_consensus.cc:606] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:22.340862 18092 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:22.341470 18092 raft_consensus.cc:606] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:22.344583 18092 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:22.345307 18092 raft_consensus.cc:606] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:22.348361 18092 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:22.349002 18092 raft_consensus.cc:606] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:22.351002 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:22.353165 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:22.354454 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:22.356886 19249 ts_tablet_manager.cc:1929] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:22.357417 19249 log.cc:1199] T f1f3da088a314a87a78898050dab39e9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/f1f3da088a314a87a78898050dab39e9
I20260709 12:21:22.362507 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet f1f3da088a314a87a78898050dab39e9 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:22.424669 19251 raft_consensus.cc:993] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:22.425359 19251 raft_consensus.cc:1081] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:22.427327 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:22.427978 18812 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:22.428392 18812 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.438359 18812 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.440995 18812 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:22.442456 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:22.442555 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:22.443538 18171 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.443653 18096 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:22.444133 18096 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:22.445067 18096 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.446568 18096 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.457175 18096 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:22.459012 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:21:22.460481 19274 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.451556 18171 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:22.467396 19274 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:22.468514 19266 raft_consensus.cc:993] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:22.468643 19274 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.470381 19251 raft_consensus.cc:1081] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:22.472747 18019 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:22.474571 18019 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:22.474973 18019 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.483048 18019 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.482694 17921 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:22.486116 18019 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:22.487897 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:22.488665 18171 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.494601 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:22.499279 18096 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:22.499676 18096 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:22.501421 18096 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.504652 18096 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.505064 18171 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:22.509200 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:22.510660 19263 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.511269 19263 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:22.512302 19263 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.535569 18096 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:22.570976 19208 raft_consensus.cc:993] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:22.571568 19208 raft_consensus.cc:1081] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:22.575170 18092 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:22.575896 18092 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:22.576332 18092 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.577560 19204 raft_consensus.cc:993] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:22.578176 19224 raft_consensus.cc:1081] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:22.588174 18813 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:22.589030 18813 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:22.589548 18813 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.617791 18813 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.625461 18092 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.632203 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:22.633083 18019 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.633687 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:22.634419 18024 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.635979 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:22.637184 18171 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:22.637529 18171 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:22.638793 18171 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.640183 18171 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.647305 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:22.648600 18090 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:22.648967 18090 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:22.647895 18024 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:22.649914 18090 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.651752 18090 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.646544 18813 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:22.668952 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:22.670118 19251 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.670750 19251 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:22.657079 18092 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:22.672031 19251 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.683506 18019 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:22.685238 18171 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:22.687022 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:22.686817 17921 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.687866 18090 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:22.688237 19274 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.688807 19274 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:22.689802 19274 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:22.697698 17922 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.715500 17921 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:22.785722 19224 raft_consensus.cc:993] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:22.786372 19224 raft_consensus.cc:1081] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:22.788115 18019 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:22.788781 18019 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:22.789175 18019 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.800745 18019 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.803822 18019 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:22.804095 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:22.804631 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:22.805315 18171 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.805807 18090 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:22.806253 18090 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:22.807235 18090 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.809785 18090 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.822578 18171 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:22.824189 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:22.825313 19257 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.825945 19257 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:22.826999 19257 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:22.827842 18090 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:22.840917 17921 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.871902 18090 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:22.873360 19274 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:22.896055 18171 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:22.897724 18090 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:22.899416 19257 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:22.901554 18171 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:22.902518 19274 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:22.903558 19257 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:21:23.206970 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153354us
I20260709 12:21:23.209996 18090 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:23.334626 18019 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:23.336686 19274 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:23.333467 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120063us
I20260709 12:21:23.349706 18166 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:23.357681 19274 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:23.366671 18024 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:23.381994 19266 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:23.384794 19266 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 12:21:23.433629 18098 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:23.435303 18167 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.017s	sys 0.006s
W20260709 12:21:23.435952 18167 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.017s	sys 0.006s
I20260709 12:21:23.438310 19301 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:23.439400 18167 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:23.483601 19265 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:23.496954 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:23.516989 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59673.2us
I20260709 12:21:23.817682 18171 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:23.858143 18137 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4843c80e9d9348a397ce35e196deb9bd. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:23.867165 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:23.867977 18171 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:23.870187 18015 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:23.873019 19212 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:23.919087 18098 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:23.981920 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54908.7us
W20260709 12:21:23.989018 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50039.2us
I20260709 12:21:23.991650 19270 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20260709 12:21:24.139624 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 216537us
W20260709 12:21:24.144448 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153574us
I20260709 12:21:24.156948 18019 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:24.157773 18019 raft_consensus.cc:606] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:24.163735 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:24.164568 18169 raft_consensus.cc:606] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:24.177153 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:24.178330 18170 raft_consensus.cc:606] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:24.183796 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:24.184461 18170 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:24.200874 19317 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.210142 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:24.211139 18096 raft_consensus.cc:606] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:24.214690 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:21:24.216711 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002e3060 after lost signal to thread 16795
I20260709 12:21:24.217000 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:21:24.218111 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002e3740 after lost signal to thread 17956
I20260709 12:21:24.218353 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:24.219718 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800329000 after lost signal to thread 18048
W20260709 12:21:24.221259 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08003154a0 after lost signal to thread 18123
W20260709 12:21:24.222618 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002760c0 after lost signal to thread 18197
W20260709 12:21:24.224018 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800280940 after lost signal to thread 18839
W20260709 12:21:24.224411 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800244740 after lost signal to thread 19204
I20260709 12:21:24.229638 18020 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.237564 19323 raft_consensus.cc:993] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.238364 19296 raft_consensus.cc:1081] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.241780 19323 raft_consensus.cc:993] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.242463 19270 raft_consensus.cc:1081] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.258720 19208 raft_consensus.cc:993] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:24.259297 19296 raft_consensus.cc:1081] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:24.261245 18019 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49542
I20260709 12:21:24.261900 18019 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:24.262319 18019 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.263456 18096 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:24.264195 18096 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:24.264637 18096 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.269078 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:24.269809 18090 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:24.270207 18090 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.305402 18019 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.308164 18019 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:24.310012 18169 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:24.316720 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:24.329378 18092 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.363662 18092 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:24.372969 18098 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.395761 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:24.440820 18171 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:24.441256 18171 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:24.442276 18171 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.443764 18171 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.452596 18090 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.455025 18090 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:24.324368 17919 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:24.465798 19326 ts_tablet_manager.cc:933] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:24.475750 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:24.476924 19259 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:24.477484 19259 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:24.478485 19259 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.483572 18096 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.486284 18096 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:24.496057 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:24.496940 18021 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.497800 19296 raft_consensus.cc:993] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.498751 19208 raft_consensus.cc:1081] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.500025 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:24.500789 18166 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:24.501120 18166 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:24.502055 18166 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.503507 18166 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.509114 18021 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.514245 19326 tablet_copy_client.cc:323] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
I20260709 12:21:24.521559 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:24.522465 18017 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.528100 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet 4843c80e9d9348a397ce35e196deb9bd from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:24.528924 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet 4843c80e9d9348a397ce35e196deb9bd from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-4843c80e9d9348a397ce35e196deb9bd
I20260709 12:21:24.545068 18183 tablet_copy_source_session.cc:215] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:24.548316 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:24.549952 18165 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:24.550302 18165 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:24.551236 18165 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.552657 18165 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.558164 19326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4843c80e9d9348a397ce35e196deb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:24.546119 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:24.563082 18090 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:24.515551 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.589210 18090 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.514504 18171 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:24.584664 18137 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3304247386d14b18a8bdd4f25ae8630c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:24.585453 18092 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:24.587518 18017 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.593870 19212 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:24.601357 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.611248 18021 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:24.609089 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:24.648401 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:24.648875 19322 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:24.649441 19321 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:24.649557 19322 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:24.650100 19321 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:24.651269 19322 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.651185 19321 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.671025 19212 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:24.674381 19265 raft_consensus.cc:993] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.677994 19265 raft_consensus.cc:1081] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:24.682122 18096 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:21:24.682835 18096 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:24.683210 18096 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.674461 19326 tablet_copy_client.cc:806] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:24.706982 18096 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.734840 18096 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:24.715071 19326 tablet_copy_client.cc:670] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:24.768399 19326 tablet_copy_client.cc:538] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:24.770890 19296 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.773221 18096 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.816980 18021 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.829064 17922 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:24.842841 18090 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.844156 18098 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:24.847820 18090 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:24.866034 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128581us
I20260709 12:21:24.867750 19296 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:24.869078 19323 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:24.869510 19323 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:24.870533 19323 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.873150 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:24.873886 18020 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:24.874207 18020 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:24.875061 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:24.875794 18166 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.878983 18092 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:24.880863 17920 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:24.887874 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:24.891299 18024 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.888300 17919 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:24.888995 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:24.875082 18020 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:24.901530 18020 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.903121 18090 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.900619 17919 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:24.907977 18024 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.910861 19323 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.928730 18166 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.945261 18020 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:24.932433 18171 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.949776 19301 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:24.951363 19259 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:24.977926 19320 raft_consensus.cc:993] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:24.980019 19322 raft_consensus.cc:1081] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
W20260709 12:21:24.983111 18063 consensus_peers.cc:597] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5240f7371cc74cb48dffa1dadcbeec49. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:25.002403 18815 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:25.003364 18815 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:25.003947 18815 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:25.006677 19326 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:25.048771 18165 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:25.048889 18815 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:25.062816 18815 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:25.076151 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:25.076963 18166 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:25.088487 18166 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:25.090786 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:25.095400 19274 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:25.096266 19274 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:25.101416 19341 ts_tablet_manager.cc:933] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:25.097644 19274 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:25.149750 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 187738us
W20260709 12:21:25.151207 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60528.6us
I20260709 12:21:25.172035 19326 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:25.173283 19326 tablet_bootstrap.cc:492] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:25.174247 19326 ts_tablet_manager.cc:1403] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.231s	user 0.021s	sys 0.010s
I20260709 12:21:25.200122 19341 tablet_copy_client.cc:323] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.3:39905
W20260709 12:21:25.206590 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51354.4us
I20260709 12:21:25.207707 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:25.210582 18183 tablet_copy_service.cc:140] P ae9641d5c3184572a102d51a75f670ee: Received BeginTabletCopySession request for tablet 3304247386d14b18a8bdd4f25ae8630c from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:60850)
I20260709 12:21:25.212276 18183 tablet_copy_service.cc:161] P ae9641d5c3184572a102d51a75f670ee: Beginning new tablet copy session on tablet 3304247386d14b18a8bdd4f25ae8630c from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:60850: session id = 910f8c8d787e4abbb092413c1f90ebf3-3304247386d14b18a8bdd4f25ae8630c
I20260709 12:21:25.213557 19339 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:25.213693 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:25.214336 19339 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:25.214893 19224 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:25.215517 19224 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:25.215397 19339 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:25.217300 19224 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:25.222915 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:25.223851 18090 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:25.224257 18090 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:25.225347 18090 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:25.228363 18090 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:25.234122 19326 raft_consensus.cc:359] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:25.242269 19326 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:25.243072 17922 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:25.244668 18165 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:25.247660 18020 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:25.267073 18183 tablet_copy_source_session.cc:215] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:25.272770 19326 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:25.277307 19326 ts_tablet_manager.cc:1434] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.102s	user 0.009s	sys 0.000s
W20260709 12:21:25.281579 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48435.9us
I20260709 12:21:25.286844 19341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3304247386d14b18a8bdd4f25ae8630c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:25.313795 17919 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:25.321094 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-4843c80e9d9348a397ce35e196deb9bd received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:25.321772 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-4843c80e9d9348a397ce35e196deb9bd on tablet 4843c80e9d9348a397ce35e196deb9bd with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:25.340584 19326 ts_tablet_manager.cc:933] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:25.345006 19322 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.345739 19341 tablet_copy_client.cc:806] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:25.346424 19341 tablet_copy_client.cc:670] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:25.362337 19341 tablet_copy_client.cc:538] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:25.365293 19326 tablet_copy_client.cc:323] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:25.364794 17922 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:25.381773 18171 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.386700 18020 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.389017 18020 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.393747 18090 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:25.399886 19341 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:25.423926 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet 5240f7371cc74cb48dffa1dadcbeec49 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:25.424713 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet 5240f7371cc74cb48dffa1dadcbeec49 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-5240f7371cc74cb48dffa1dadcbeec49
I20260709 12:21:25.430042 19224 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.456650 19319 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:21:25.475442 18812 raft_consensus.cc:1217] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:25.487306 18165 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.490324 18109 tablet_copy_source_session.cc:215] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:25.498762 19224 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.501722 19319 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:21:25.520748 19319 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:25.555810 19326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5240f7371cc74cb48dffa1dadcbeec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:25.560878 18019 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:25.594012 19322 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.639698 19326 tablet_copy_client.cc:806] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:25.649657 19326 tablet_copy_client.cc:670] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:25.658178 17922 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:25.690922 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52481.2us
I20260709 12:21:25.716470 18169 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:25.718554 19274 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:25.744045 18171 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:25.761318 18092 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:25.774178 19341 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.073s	user 0.003s	sys 0.000s
I20260709 12:21:25.779273 19341 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:21:25.802043 18063 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5eedbd3c223402caf9a71dac2fcaa98. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:25.817400 19341 tablet_bootstrap.cc:492] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:25.818538 19341 ts_tablet_manager.cc:1403] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.419s	user 0.034s	sys 0.028s
I20260709 12:21:25.829766 18092 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:25.831017 18170 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.853266 18017 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.855443 19341 raft_consensus.cc:359] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:25.856508 19341 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:25.859342 18169 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.862607 18169 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:25.865651 18021 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:25.866134 19321 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:25.868780 18020 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:25.896656 19321 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.914408 19322 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.857347 19341 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:25.922320 19326 tablet_copy_client.cc:538] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:25.922930 19367 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:25.958163 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:25.961494 19224 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.969712 19321 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.973342 19319 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:25.985492 19341 ts_tablet_manager.cc:1434] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.161s	user 0.008s	sys 0.002s
I20260709 12:21:26.000975 18183 tablet_copy_service.cc:342] P ae9641d5c3184572a102d51a75f670ee: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-3304247386d14b18a8bdd4f25ae8630c received from {username='slave'} at 127.0.0.1:60850
I20260709 12:21:26.001567 18183 tablet_copy_service.cc:434] P ae9641d5c3184572a102d51a75f670ee: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-3304247386d14b18a8bdd4f25ae8630c on tablet 3304247386d14b18a8bdd4f25ae8630c with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:26.001010 19326 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:26.005048 19208 raft_consensus.cc:1064] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:26.036816 19208 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.118722 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:26.119640 18017 raft_consensus.cc:606] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:26.126674 18815 raft_consensus.cc:1217] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:26.154507 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:26.155287 18165 raft_consensus.cc:606] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:26.168984 18812 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:26.176721 19323 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:26.205158 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47693.6us
I20260709 12:21:26.229574 19391 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:26.238464 18090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:26.267367 18090 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:26.254856 19385 raft_consensus.cc:993] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:26.257995 19326 tablet_bootstrap.cc:700] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent opening tablet: real 0.154s	user 0.000s	sys 0.002s
I20260709 12:21:26.279871 19351 raft_consensus.cc:1081] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:26.286602 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:58848
I20260709 12:21:26.287451 18812 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:26.287950 18812 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.288043 18092 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:26.292393 18017 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:26.306512 19380 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:26.317106 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75933.8us
I20260709 12:21:26.328871 19380 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:26.355854 18812 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.355469 18017 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:26.358146 18812 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:26.359305 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:26.360157 18020 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:26.360569 18020 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:26.361656 18020 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.363196 18020 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.372483 19380 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:26.374181 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43784.9us
I20260709 12:21:26.375669 19318 raft_consensus.cc:993] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:26.377312 19318 raft_consensus.cc:1081] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:26.377395 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:26.385555 18167 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:26.392407 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60741.3us
I20260709 12:21:26.399211 18020 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:26.401067 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:26.402534 19367 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:26.403488 19367 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:26.401621 18096 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:26.404691 19367 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.409238 18171 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.419857 18815 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:26.422297 17919 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:26.425559 18098 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:26.428593 18098 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:26.429071 18098 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.427445 19326 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:21:26.444613 18171 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:26.451785 17922 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:26.478374 19341 ts_tablet_manager.cc:933] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:26.473342 18098 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.490047 18098 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:26.502822 19326 tablet_bootstrap.cc:492] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:26.513461 19326 ts_tablet_manager.cc:1403] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.513s	user 0.014s	sys 0.018s
I20260709 12:21:26.508916 17920 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:26.535388 19397 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:21:26.539134 18097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:26.540058 18097 raft_consensus.cc:606] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:26.545146 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:26.546257 18017 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.569455 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:26.570391 18169 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:26.570798 18169 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:26.571745 18169 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.576408 19382 raft_consensus.cc:993] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:26.577204 19382 raft_consensus.cc:1081] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:26.585160 19391 raft_consensus.cc:993] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:26.586026 19391 raft_consensus.cc:1081] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:26.589205 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:26.590114 18812 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:26.590528 18812 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.612869 18169 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.562405 19326 raft_consensus.cc:359] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:26.620195 19326 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:26.580737 18021 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:26.638074 18017 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
W20260709 12:21:26.639775 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157848us
W20260709 12:21:26.642750 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61558.9us
I20260709 12:21:26.649973 18812 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.653668 18019 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:26.639412 19341 tablet_copy_client.cc:323] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:26.675944 18019 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:26.676451 18019 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.678787 18064 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:26.663746 18169 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:26.698172 18812 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:26.700764 19386 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:26.701462 19386 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:26.697747 18019 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.707471 18019 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:26.709184 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:26.705713 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:26.740365 18170 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.702602 19386 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.734299 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c53769058ca3443f805983b61a69328e"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:26.734862 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet b5eedbd3c223402caf9a71dac2fcaa98 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:26.734288 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:26.736501 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57238.2us
I20260709 12:21:26.746963 18096 raft_consensus.cc:606] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:26.747536 18097 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:26.747851 18097 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:26.709942 18098 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:26.748756 18097 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.749362 18098 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:26.750056 18170 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:26.750289 18097 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.753491 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:26.746963 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet b5eedbd3c223402caf9a71dac2fcaa98 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-b5eedbd3c223402caf9a71dac2fcaa98
I20260709 12:21:26.754827 19274 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:26.755400 19274 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:26.757576 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:26.758278 18170 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.750381 18098 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.775377 18098 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.680302 19326 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.783632 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:26.801924 18096 raft_consensus.cc:606] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:26.802619 19326 ts_tablet_manager.cc:1434] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.289s	user 0.010s	sys 0.001s
I20260709 12:21:26.819375 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:26.821745 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:21:26.823448 18108 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-5240f7371cc74cb48dffa1dadcbeec49 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:26.823997 18108 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-5240f7371cc74cb48dffa1dadcbeec49 on tablet 5240f7371cc74cb48dffa1dadcbeec49 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:26.824074 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:26.826027 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:26.818018 18109 tablet_copy_source_session.cc:215] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:26.786077 18170 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:26.789929 18098 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:26.792714 19274 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.832154 17989 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:26.833145 19398 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:26.833668 19398 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:26.834635 19398 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:26.840811 19341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5eedbd3c223402caf9a71dac2fcaa98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:26.881660 19319 raft_consensus.cc:993] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:26.882381 19319 raft_consensus.cc:1081] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:26.884980 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c53769058ca3443f805983b61a69328e"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:26.886348 18812 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:26.886888 18812 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.936815 18812 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:26.957472 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:26.958276 18019 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.995070 19341 tablet_copy_client.cc:806] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:26.995913 19341 tablet_copy_client.cc:670] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:27.003091 19341 tablet_copy_client.cc:538] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:27.014730 18021 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:27.015331 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:27.015995 18098 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:27.016301 18098 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:27.017205 18098 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:27.018436 18098 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.027464 18812 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:27.028537 19274 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:27.024602 17920 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:27.029006 17922 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.044116 18170 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:27.047649 19274 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:27.057415 19341 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:27.063271 17919 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:27.069379 19224 raft_consensus.cc:993] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:27.078538 19224 raft_consensus.cc:1081] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:27.079658 18097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:27.080425 18097 raft_consensus.cc:606] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:27.084203 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:27.044955 18019 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:27.099076 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:27.099431 18098 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:27.098163 18812 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.117337 18020 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:27.128187 18020 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:27.130045 19274 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:27.130813 18812 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.143795 19274 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:27.144831 18167 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.159494 19274 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:27.165495 19386 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.187032 18098 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:27.188079 18098 raft_consensus.cc:606] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:27.188928 19224 raft_consensus.cc:993] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:27.195640 19224 raft_consensus.cc:1081] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:27.202080 18169 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50510
I20260709 12:21:27.222899 18169 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.223598 18169 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.226929 18020 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:21:27.231153 17906 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 3304247386d14b18a8bdd4f25ae8630c already in progress [suppressed 5 similar messages]
I20260709 12:21:27.237298 18814 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Advancing to term 2
I20260709 12:21:27.249146 18812 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:27.250561 18169 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:27.263489 19224 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.261508 18814 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.267470 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:27.268180 18167 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.271088 18812 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:27.273496 19386 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.274698 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:27.275405 18098 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:27.275758 18098 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:27.276588 18098 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:27.278322 18098 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.268497 18170 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:27.285400 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54315us
I20260709 12:21:27.294250 18167 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:27.298310 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:27.320084 18169 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:27.365103 18098 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:27.381052 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:27.321969 19412 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:21:27.374725 19381 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:21:27.419319 18098 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:27.420269 18098 raft_consensus.cc:606] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership
W20260709 12:21:27.454464 17970 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 12:21:27.455219 17970 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 3304247386d14b18a8bdd4f25ae8630c already in progress
W20260709 12:21:27.455596 17970 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 3304247386d14b18a8bdd4f25ae8630c already in progress
I20260709 12:21:27.481720 18019 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:27.484660 19341 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:27.504839 18167 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:27.519363 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:27.519513 18814 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:27.520460 18098 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:27.520967 18098 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:27.522797 18098 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:27.524721 18098 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.530529 19341 tablet_bootstrap.cc:492] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:27.531879 19341 ts_tablet_manager.cc:1403] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.475s	user 0.052s	sys 0.075s
I20260709 12:21:27.544553 19367 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:27.515997 19430 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:21:27.553141 19208 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:27.558651 18814 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Advancing to term 2
I20260709 12:21:27.560626 18096 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:27.563690 18098 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:27.577386 19412 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:27.591691 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:27.592463 18017 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:27.593405 18814 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.627743 19435 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:27.630303 19412 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:27.641137 18090 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:21:27.646777 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56507.9us
I20260709 12:21:27.657626 19341 raft_consensus.cc:359] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:27.710805 19341 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:27.745539 18021 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:27.746381 18021 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:27.745628 19341 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:27.750777 19341 ts_tablet_manager.cc:1434] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.218s	user 0.013s	sys 0.002s
W20260709 12:21:27.713882 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.721174 19437 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:21:27.769374 19424 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:27.771509 17970 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 12:21:27.772120 17970 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 5240f7371cc74cb48dffa1dadcbeec49 already in progress
W20260709 12:21:27.772426 17970 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 5240f7371cc74cb48dffa1dadcbeec49 already in progress
I20260709 12:21:27.773707 18811 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:27.793009 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-b5eedbd3c223402caf9a71dac2fcaa98 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:27.705624 18136 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:27.793738 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-b5eedbd3c223402caf9a71dac2fcaa98 on tablet b5eedbd3c223402caf9a71dac2fcaa98 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:27.794384 19421 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:27.795017 19421 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:27.796324 19421 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:27.806205 19430 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:27.739244 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159339us
I20260709 12:21:27.819041 18097 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:21:27.822028 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116496us
I20260709 12:21:27.836967 19398 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:27.840535 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50989.9us
I20260709 12:21:27.842813 18017 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:27.851362 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.851999 18169 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.900349 18813 raft_consensus.cc:1217] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:27.903263 19437 raft_consensus.cc:993] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:27.906222 19434 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:21:27.908171 19437 raft_consensus.cc:1081] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:27.920576 18097 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:21:27.921237 18097 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.921645 18097 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.947875 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4843c80e9d9348a397ce35e196deb9bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:28.141315 18097 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:28.144397 18097 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:28.169557 18096 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:28.212847 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 307401us
I20260709 12:21:28.240033 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:28.243090 18021 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:28.254326 19427 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260709 12:21:28.267885 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 356396us
I20260709 12:21:28.274479 17922 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.300305 17920 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:28.301896 18019 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:28.305367 18814 raft_consensus.cc:2955] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.340698 18814 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Advancing to term 3
I20260709 12:21:28.349488 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:28.382506 18170 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:28.386270 19274 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:28.413162 18814 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:28.417876 19434 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:28.423164 19423 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:28.463671 19421 raft_consensus.cc:1064] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
W20260709 12:21:28.464213 19421 raft_consensus.cc:1068] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Unable to promote non-voter 910f8c8d787e4abbb092413c1f90ebf3: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:21:28.465973 18019 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:28.467303 18020 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:28.467748 18020 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:28.468587 18020 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:28.471310 18092 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:28.476925 19412 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:28.505072 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48002.9us
I20260709 12:21:28.507987 18020 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:28.509281 19429 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:28.510119 19429 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:28.514616 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:28.524955 18020 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:28.525646 19429 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:28.536113 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:28.539438 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:28.540491 18096 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:28.540624 18064 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:28.544641 19322 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:28.545384 19453 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:28.547812 19322 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:28.549222 19322 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:28.568971 19317 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:28.578176 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50638.7us
I20260709 12:21:28.583529 19471 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:28.602259 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:28.603075 18167 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:28.611738 18167 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:28.611856 17921 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:28.657241 19423 raft_consensus.cc:1064] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:28.663877 19423 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:28.675740 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 4843c80e9d9348a397ce35e196deb9bd with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:28.688795 18812 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:28.688786 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:28.690835 19471 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:28.721967 18019 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:28.744213 19430 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:28.755026 17920 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:28.757550 17920 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.744941 19475 tablet_replica.cc:333] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:28.808413 19224 raft_consensus.cc:1064] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:28.811501 19224 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:28.814869 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14d2c4b70e164856a205483966ca4963"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:28.815675 18165 raft_consensus.cc:606] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:28.823951 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:28.824688 18165 raft_consensus.cc:606] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:28.828685 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:28.829419 18165 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:28.851580 18096 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:28.853456 18090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "acbc4a634f954522904f2b865531f456"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:28.854301 18090 raft_consensus.cc:606] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:28.860816 19269 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.887069 18814 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.891021 18019 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.892906 18090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:28.893637 18090 raft_consensus.cc:606] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:28.894042 18090 raft_consensus.cc:614] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:28.910707 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:28.911445 18017 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:28.923756 19208 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:28.925709 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:21:28.928198 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:28.929242 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:28.944144 19430 raft_consensus.cc:993] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:28.941066 19208 raft_consensus.cc:993] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:28.944955 19317 raft_consensus.cc:1081] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:28.945182 19208 raft_consensus.cc:1081] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:28.957110 18814 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:28.960723 19386 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:28.962365 18017 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:28.967219 19322 raft_consensus.cc:993] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:28.967789 19422 raft_consensus.cc:1081] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:28.977919 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "acbc4a634f954522904f2b865531f456"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:28.978580 18812 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:28.978943 18812 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:28.979902 17921 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 2 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:28.988682 18170 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:21:28.989883 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080030a160 after lost signal to thread 16795
W20260709 12:21:28.991181 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1900 after lost signal to thread 17956
W20260709 12:21:28.992713 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080013c7a0 after lost signal to thread 18048
I20260709 12:21:28.995609 19475 raft_consensus.cc:2243] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:28.996655 19475 raft_consensus.cc:2272] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:28.997920 18812 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:29.000957 18099 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:29.001540 18099 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:29.001932 18099 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.002652 18094 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:29.003158 18094 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:29.003417 18094 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.006506 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:29.007611 18017 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:29.009302 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2480 after lost signal to thread 18123
I20260709 12:21:29.011844 18094 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:29.014875 18167 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260709 12:21:29.033835 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080033dde0 after lost signal to thread 18197
I20260709 12:21:29.020015 19475 ts_tablet_manager.cc:1916] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:29.024488 19451 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:29.035701 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:29.040594 18092 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:29.040867 18017 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:29.015820 18811 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Advancing to term 3
I20260709 12:21:29.041078 18092 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:29.041759 18020 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:29.048206 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08000552a0 after lost signal to thread 18839
I20260709 12:21:29.043955 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:29.031430 18812 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:29.049424 18099 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:29.052680 18094 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:29.055383 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:29.056113 18017 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.058425 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:29.059549 18167 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:29.060083 18167 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:29.061513 18167 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:29.063624 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:29.064617 18019 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.071830 18167 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.079535 18099 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:29.075052 18092 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:29.081656 18092 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.085639 18017 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:29.085922 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:29.086962 18170 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:29.087493 18170 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:29.087733 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:29.095615 18019 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:29.097781 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:29.100680 18170 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:29.079665 19386 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:21:29.077984 19427 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:29.119707 18167 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:29.126233 19453 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:29.112722 19422 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:29.142102 19427 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:29.070152 19322 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:21:29.155176 18092 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:29.150185 18170 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.177232 18098 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:29.178877 18170 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:29.167105 19453 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:29.169090 19422 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:29.208274 19453 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:29.209379 19422 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:29.213402 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 115736us
I20260709 12:21:29.217720 18019 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:29.247718 19224 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:29.269963 18172 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:29.351730 18814 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:29.373485 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:29.380993 19454 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:21:29.267261 19427 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:29.369050 19430 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:21:29.426194 18811 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260709 12:21:29.426724 19488 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.074s	user 0.000s	sys 0.002s
W20260709 12:21:29.451933 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 157744us
I20260709 12:21:29.517132 18019 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:29.528102 19490 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 12:21:29.568643 18812 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:29.605698 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65083us
W20260709 12:21:29.635406 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70571.9us
I20260709 12:21:29.605306 19423 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:21:29.440563 19322 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:29.656600 19208 raft_consensus.cc:993] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:29.657299 19208 raft_consensus.cc:1081] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
W20260709 12:21:29.084862 19319 consensus_queue.cc:1175] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:21:29.675733 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.741616 19495 raft_consensus.cc:993] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
W20260709 12:21:29.747495 19503 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:21:29.748572 19475 ts_tablet_manager.cc:1929] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:29.763178 19475 log.cc:1199] T 4843c80e9d9348a397ce35e196deb9bd P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/4843c80e9d9348a397ce35e196deb9bd
I20260709 12:21:29.795395 18016 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14d2c4b70e164856a205483966ca4963"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49542
I20260709 12:21:29.796578 18016 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:29.797087 18016 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.783355 19495 raft_consensus.cc:1081] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:29.814828 18167 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:29.844660 19475 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45024 (request call id 76) took 1169 ms. Trace:
I20260709 12:21:29.845314 19475 rpcz_store.cc:276] 0709 12:21:28.675369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.675506 (+   137us) service_pool.cc:224] Handling call
0709 12:21:28.744297 (+ 68791us) ts_tablet_manager.cc:1151] Deleting tablet 4843c80e9d9348a397ce35e196deb9bd
0709 12:21:28.744375 (+    78us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:21:29.368307 (+623932us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:29.748382 (+380075us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:29.844530 (+ 96148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":224524,"tablet-delete.queue_time_us":67846,"thread_start_us":67499,"threads_started":1}
I20260709 12:21:29.856199 18016 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:29.859095 18016 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:29.867919 18020 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:29.895182 18020 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:29.872236 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 4843c80e9d9348a397ce35e196deb9bd (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:29.908196 19323 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:29.913165 17920 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:29.926664 17919 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:29.928586 17919 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:29.973639 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150039us
I20260709 12:21:29.974923 18015 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:29.984157 18020 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:29.986553 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79791.3us
I20260709 12:21:29.989634 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:21:29.990389 18090 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:29.990792 18090 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:29.987658 19208 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:29.997051 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:29.999189 18016 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:30.001892 18171 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:30.002452 18171 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:30.004086 18171 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:30.005630 18171 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:30.022396 19492 raft_consensus.cc:1064] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:30.037185 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:30.038046 18812 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:30.040961 18815 raft_consensus.cc:2955] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:30.066382 18169 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:30.067221 18165 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:30.077783 18090 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:30.082144 19493 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:21:30.085139 19491 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:21:30.097290 18171 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:30.115404 19430 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:30.119063 18090 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:30.121515 18812 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:30.025455 19492 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:30.139744 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:30.141916 19504 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:30.142666 19504 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:30.144096 19504 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:30.134028 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3304247386d14b18a8bdd4f25ae8630c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:21:30.189060 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.197620 17922 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:30.243125 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82654.8us
W20260709 12:21:30.257656 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122132us
I20260709 12:21:30.262756 19224 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:30.265445 19490 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:30.270330 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:30.271121 18024 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:30.271448 18024 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:30.272432 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:30.273157 18166 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:30.272311 18024 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:30.297747 18024 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:30.301208 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
W20260709 12:21:30.362851 19518 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20260709 12:21:30.447100 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 191082us
I20260709 12:21:30.785840 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:30.879354 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 507254us
W20260709 12:21:30.934998 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53883.7us
W20260709 12:21:30.941419 19531 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.059s	user 0.001s	sys 0.001s
W20260709 12:21:30.968117 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 519432us
I20260709 12:21:30.973218 19386 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:30.974668 19386 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:31.035598 19386 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:31.009565 19451 raft_consensus.cc:1027] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: attempt to promote peer 910f8c8d787e4abbb092413c1f90ebf3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:21:31.081797 18814 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:31.084065 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:31.098131 18024 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.649s	user 0.014s	sys 0.011s
I20260709 12:21:31.111013 19490 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:21:31.121016 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:31.145184 19490 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:31.168164 19490 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:31.198938 19541 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:21:31.211915 19490 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:31.232077 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:31.186950 19531 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:21:31.255373 18166 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:21:31.257458 19529 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.292s	user 0.002s	sys 0.006s
I20260709 12:21:31.269991 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:31.290630 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 217) took 1381 ms. Trace:
I20260709 12:21:31.292866 18016 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:31.294872 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:31.299646 17919 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 224) took 1562 ms. Trace:
I20260709 12:21:31.308537 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1394) took 1040 ms. Trace:
I20260709 12:21:31.309096 18015 rpcz_store.cc:276] 0709 12:21:30.268135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.268703 (+   568us) service_pool.cc:224] Handling call
0709 12:21:30.269064 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.286494 (+ 17430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.286535 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.296658 (+ 10123us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.472693 (+176035us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:30.472772 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.472895 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:31.256403 (+783508us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:21:31.300814 (+ 44411us) raft_consensus.cc:1685] finished
0709 12:21:31.300836 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.308299 (+  7463us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13152,"raft.queue_time_us":80440,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":2744,"spinlock_wait_cycles":34253312,"thread_start_us":119314,"threads_started":1,"wal-append.queue_time_us":621310}
I20260709 12:21:31.314235 19512 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.316027 19512 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:31.328788 19485 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.328910 19509 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.333633 19491 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:31.334349 19491 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:31.304366 17920 rpcz_store.cc:276] 0709 12:21:29.908828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.909390 (+   562us) service_pool.cc:224] Handling call
0709 12:21:31.290253 (+1380863us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:30.263095 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:30.365093 (+101998us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:30.365133 (+    40us) write_op.cc:435] Acquired schema lock
0709 12:21:30.365155 (+    22us) tablet.cc:662] Decoding operations
0709 12:21:30.365587 (+   432us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:30.365674 (+    87us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:30.365713 (+    39us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:30.365951 (+   238us) tablet.cc:701] Row locks acquired
0709 12:21:30.365972 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:21:30.366130 (+   158us) write_op.cc:270] Start()
0709 12:21:30.366282 (+   152us) write_op.cc:276] Timestamp: P: 1783599690366089 usec, L: 0
0709 12:21:30.366308 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:21:30.367104 (+   796us) log.cc:844] Serialized 595 byte log entry
0709 12:21:31.205207 (+838103us) op_driver.cc:464] REPLICATION: finished
0709 12:21:31.274828 (+ 69621us) write_op.cc:301] APPLY: starting
0709 12:21:31.275140 (+   312us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:31.275339 (+   199us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:31.275496 (+   157us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:31.275872 (+   376us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:31.276196 (+   324us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:31.276269 (+    73us) write_op.cc:312] APPLY: finished
0709 12:21:31.276613 (+   344us) log.cc:844] Serialized 23 byte log entry
0709 12:21:31.276839 (+   226us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:31.276955 (+   116us) write_op.cc:454] Released schema lock
0709 12:21:31.277026 (+    71us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:31.277574 (+   548us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:31.277604 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":69188,"apply.run_cpu_time_us":2740,"apply.run_wall_time_us":2740,"num_ops":1,"prepare.queue_time_us":336980,"prepare.run_cpu_time_us":61869,"prepare.run_wall_time_us":586682,"raft.queue_time_us":179352,"raft.run_cpu_time_us":8755,"raft.run_wall_time_us":81037,"replication_time_us":838669,"thread_start_us":597389,"threads_started":4,"wal-append.queue_time_us":514711}]]}
I20260709 12:21:31.331023 19509 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:31.337541 19491 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:31.351970 19485 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:31.408722 19495 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.305565 17919 rpcz_store.cc:276] 0709 12:21:29.737220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.737380 (+   160us) service_pool.cc:224] Handling call
0709 12:21:29.780059 (+ 42679us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16621090 00007f6c166bff3a 00007f6c1c475e9c 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:21:31.299393 (+1519334us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:30.858662 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:30.859883 (+  1221us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:30.859976 (+    93us) write_op.cc:435] Acquired schema lock
0709 12:21:30.860038 (+    62us) tablet.cc:662] Decoding operations
0709 12:21:30.860813 (+   775us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:30.860921 (+   108us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:30.860975 (+    54us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:21:30.861645 (+   670us) tablet.cc:701] Row locks acquired
0709 12:21:30.861709 (+    64us) write_op.cc:260] PREPARE: finished
0709 12:21:30.883322 (+ 21613us) write_op.cc:270] Start()
0709 12:21:30.915104 (+ 31782us) write_op.cc:276] Timestamp: P: 1783599690872000 usec, L: 0
0709 12:21:30.915167 (+    63us) op_driver.cc:348] REPLICATION: starting
0709 12:21:30.916936 (+  1769us) log.cc:844] Serialized 923 byte log entry
0709 12:21:31.273787 (+356851us) op_driver.cc:464] REPLICATION: finished
0709 12:21:31.279837 (+  6050us) write_op.cc:301] APPLY: starting
0709 12:21:31.279926 (+    89us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:31.280051 (+   125us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:31.280160 (+   109us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:31.280627 (+   467us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:31.285348 (+  4721us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:21:31.285431 (+    83us) write_op.cc:312] APPLY: finished
0709 12:21:31.295465 (+ 10034us) log.cc:844] Serialized 29 byte log entry
0709 12:21:31.295704 (+   239us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:31.295795 (+    91us) write_op.cc:454] Released schema lock
0709 12:21:31.295843 (+    48us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":8275200,"child_traces":[["op",{"apply.queue_time_us":5738,"apply.run_cpu_time_us":2269,"apply.run_wall_time_us":16416,"num_ops":2,"prepare.queue_time_us":917220,"prepare.run_cpu_time_us":6461,"prepare.run_wall_time_us":58844,"replication_time_us":358430,"thread_start_us":84272,"threads_started":1}]]}
I20260709 12:21:31.415797 19512 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:31.417853 19509 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:31.427596 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:31.432032 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:31.440828 18064 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:31.444228 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68056.4us
I20260709 12:21:31.409603 19495 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:31.503667 19451 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:21:31.507112 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66625.5us
I20260709 12:21:31.511991 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:31.514780 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:31.515811 18015 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:31.504462 19451 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:31.540971 17922 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 127) took 1417 ms. Trace:
I20260709 12:21:31.530406 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:31.544504 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:31.544756 19318 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:31.546175 19552 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:31.546669 19552 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.547096 19552 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.520738 19490 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:31.562603 19494 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:31.581985 19549 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:21:31.563627 19494 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:31.599107 19453 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 12:21:31.541512 17922 rpcz_store.cc:276] 0709 12:21:30.123915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.124554 (+   639us) service_pool.cc:224] Handling call
0709 12:21:31.540726 (+1416172us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:30.945558 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:31.099495 (+153937us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:31.103951 (+  4456us) write_op.cc:435] Acquired schema lock
0709 12:21:31.105442 (+  1491us) tablet.cc:662] Decoding operations
0709 12:21:31.108315 (+  2873us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:31.108379 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:31.108438 (+    59us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:31.108685 (+   247us) tablet.cc:701] Row locks acquired
0709 12:21:31.108705 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:21:31.109119 (+   414us) write_op.cc:270] Start()
0709 12:21:31.109304 (+   185us) write_op.cc:276] Timestamp: P: 1783599691109048 usec, L: 0
0709 12:21:31.109336 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:21:31.110349 (+  1013us) log.cc:844] Serialized 533 byte log entry
0709 12:21:31.403702 (+293353us) op_driver.cc:464] REPLICATION: finished
0709 12:21:31.420038 (+ 16336us) write_op.cc:301] APPLY: starting
0709 12:21:31.420152 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:31.420273 (+   121us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:31.427189 (+  6916us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:31.427614 (+   425us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:31.427855 (+   241us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:31.427926 (+    71us) write_op.cc:312] APPLY: finished
0709 12:21:31.428272 (+   346us) log.cc:844] Serialized 23 byte log entry
0709 12:21:31.449881 (+ 21609us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:31.454782 (+  4901us) write_op.cc:454] Released schema lock
0709 12:21:31.458584 (+  3802us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:31.530915 (+ 72331us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:31.534185 (+  3270us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":2944,"child_traces":[["op",{"apply.queue_time_us":7986,"apply.run_cpu_time_us":3501,"apply.run_wall_time_us":99520,"num_ops":1,"prepare.queue_time_us":677405,"prepare.run_cpu_time_us":5039,"prepare.run_wall_time_us":192997,"replication_time_us":287379}]]}
I20260709 12:21:31.605487 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:31.603063 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:31.616791 18024 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:31.607060 19451 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:31.640733 19494 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:31.661156 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:31.657568 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:31.663012 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:31.670847 19540 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.671801 19540 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:31.672930 19550 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.175s	user 0.000s	sys 0.004s
I20260709 12:21:31.676394 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:31.678664 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:31.646083 19485 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:31.676206 19540 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:31.692595 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 219) took 1506 ms. Trace:
I20260709 12:21:31.696589 17921 rpcz_store.cc:276] 0709 12:21:30.195200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.195379 (+   179us) service_pool.cc:224] Handling call
0709 12:21:31.692306 (+1496927us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 12:21:31.707444 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:31.708755 19495 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:31.710745 17989 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:31.719415 19552 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:31.722774 19495 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.725680 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:31.731161 19495 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.733281 19495 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.771776 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:31.772311 17922 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 5 to 6, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:31.774541 17922 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:31.775955 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:31.778365 19540 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:31.778342 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:31.784777 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56312.2us
I20260709 12:21:31.797791 18138 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:31.800395 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:31.799757 19509 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.802492 19509 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:31.803210 19509 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.805297 19552 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:31.816392 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:31.817276 18024 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.824649 19501 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:21:31.838259 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:31.838354 18173 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:31.841871 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:31.843350 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:31.862560 18021 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:31.863492 18021 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:31.868741 18024 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:31.869937 18015 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:31.888127 19520 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:31.888895 19520 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:31.890389 17921 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.893150 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:31.894621 19520 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:31.891017 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63927.5us
I20260709 12:21:31.900784 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:31.887683 19495 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:31.905352 19502 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:31.906406 19495 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:31.908948 18020 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:31.911831 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:31.923547 19502 raft_consensus.cc:993] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:31.923856 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 3304247386d14b18a8bdd4f25ae8630c with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:31.924314 19502 raft_consensus.cc:1081] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:31.933629 18170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:31.934552 18170 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:31.934538 19509 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:31.935549 18170 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.936669 18165 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:31.938606 19509 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:31.945250 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:31.959300 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:31.960397 19430 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:31.960960 19430 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:31.961915 19430 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:31.965718 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:31.972411 19562 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:21:31.983816 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:31.987185 18024 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:31.992101 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:31.999117 18170 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.013638 19422 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 12:21:32.020292 18170 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:32.019273 17987 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:32.018232 19453 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:32.036223 19453 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:32.034384 19495 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:32.038568 19453 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.040071 19453 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.057726 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:32.063280 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:32.070583 18095 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.071187 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:32.072412 19453 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:32.078140 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:32.033447 19520 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:21:32.090034 19480 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:32.093281 19224 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:32.101429 19563 tablet_replica.cc:333] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:32.111042 19520 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:32.111557 19520 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:32.112643 19520 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:32.115250 18138 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:32.102169 18097 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:32.110100 18095 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:32.119433 19509 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:32.119644 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:32.121382 18020 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:32.121719 18020 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:32.118468 19491 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:21:32.131083 18020 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.140048 18020 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.132302 17919 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:32.142496 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:32.158316 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:32.159077 18096 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:32.160192 18169 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:32.164680 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.166323 19553 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:32.170127 19553 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:32.171407 19553 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.211314 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:32.212153 18165 raft_consensus.cc:606] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:32.220147 18812 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:32.223963 19319 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:32.217573 19520 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:32.235306 18020 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:32.242055 19563 raft_consensus.cc:2243] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:32.243772 19563 raft_consensus.cc:2272] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:32.252485 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58101.7us
W20260709 12:21:32.259449 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66932.5us
I20260709 12:21:32.263806 19422 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:21:32.271649 19563 ts_tablet_manager.cc:1916] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:32.275571 18016 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:32.310819 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:32.311587 18169 raft_consensus.cc:606] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:32.319478 19552 raft_consensus.cc:993] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:32.320043 19552 raft_consensus.cc:1081] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:32.320165 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:21:32.322671 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:32.323827 18016 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49542
I20260709 12:21:32.324502 18016 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:32.324898 18016 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.323827 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:32.345302 18016 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:32.348019 19422 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:32.354020 18020 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:32.354521 18016 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:32.365926 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:32.366801 18170 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:32.367234 18170 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:32.374469 18170 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:32.381772 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114741us
I20260709 12:21:32.385524 18170 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.417505 19491 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:32.424397 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:32.425151 18811 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.420653 18812 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:32.398505 17921 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:32.439229 18170 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:32.447458 18811 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:32.460856 18095 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:32.463850 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:32.467777 19494 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:32.468395 19494 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:32.469494 19494 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:32.480093 19551 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:21:32.497473 19422 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:32.520365 19501 raft_consensus.cc:993] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:32.523161 19501 raft_consensus.cc:1081] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:32.534135 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b5eedbd3c223402caf9a71dac2fcaa98 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:32.541280 18020 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:32.547437 19553 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:32.551738 17921 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:32.552006 17919 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.590574 18021 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:32.591331 18021 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:32.591792 18021 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.609079 19553 raft_consensus.cc:993] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:32.609737 19553 raft_consensus.cc:1081] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:32.610602 18021 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.616128 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:32.616956 18097 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:32.617369 18097 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:32.618212 18097 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.619594 18097 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.628554 18096 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:32.629350 18096 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:32.629765 18096 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.639783 18096 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.643591 18097 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:32.645525 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:32.647507 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet b5eedbd3c223402caf9a71dac2fcaa98 with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:32.657198 18021 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:32.657586 19502 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:32.657497 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:32.658326 19502 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:32.658377 18173 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.656127 18096 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:32.667400 19502 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.695557 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:32.703275 18165 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:32.703765 18165 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:32.709870 18173 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:32.721082 19580 tablet_replica.cc:333] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:32.682148 19563 ts_tablet_manager.cc:1929] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 12:21:32.722101 19563 log.cc:1199] T 3304247386d14b18a8bdd4f25ae8630c P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/3304247386d14b18a8bdd4f25ae8630c
I20260709 12:21:32.725039 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 3304247386d14b18a8bdd4f25ae8630c (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:32.734108 18015 raft_consensus.cc:2955] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:32.739957 18021 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:32.736860 18165 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.747866 19459 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:21:32.762739 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:32.763612 18019 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.779419 18019 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:32.799607 18165 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.813865 19580 raft_consensus.cc:2243] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:32.815227 19580 raft_consensus.cc:2272] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:32.809294 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:32.819137 17919 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.834175 18090 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:32.837116 19317 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:32.844100 19580 ts_tablet_manager.cc:1916] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:21:32.862462 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53231.9us
I20260709 12:21:32.863974 19501 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:32.887465 19501 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:32.888667 19501 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:32.896631 18165 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:32.899832 18812 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:32.901582 19583 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:21:32.943136 19582 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.073s	user 0.004s	sys 0.000s
W20260709 12:21:32.953440 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61434.5us
W20260709 12:21:32.956079 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63446.9us
I20260709 12:21:32.970058 18168 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:33.026238 19589 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:33.048465 17921 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:33.102448 19580 ts_tablet_manager.cc:1929] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:33.103214 19580 log.cc:1199] T b5eedbd3c223402caf9a71dac2fcaa98 P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/b5eedbd3c223402caf9a71dac2fcaa98
I20260709 12:21:33.105871 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet b5eedbd3c223402caf9a71dac2fcaa98 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:33.105865 18090 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:33.119848 19589 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:33.143877 18170 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:33.149500 19583 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:33.237210 19586 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:33.237895 19586 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:33.242902 19586 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:33.245651 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:33.246641 18165 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:33.247727 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:33.248636 18020 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:33.248809 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:33.249574 18090 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:33.257189 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:33.258626 19565 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.270251 19565 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.322633 18020 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:33.324523 19501 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:33.399622 18168 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:33.421989 19480 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:33.524188 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:33.595959 19585 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.596710 19585 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:33.600472 19585 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:33.601022 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:33.602257 18095 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 12:21:33.603464 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:33.604293 18168 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:21:33.610646 17989 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:33.612668 19351 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:33.613138 19351 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.613509 19351 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.631093 18019 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:33.632038 18019 raft_consensus.cc:606] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:33.634106 19351 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:33.641570 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:33.642529 18168 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.642694 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:33.645596 18097 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.653680 19351 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:33.655441 18168 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:21:33.658324 17989 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:33.659724 19351 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:33.660406 19351 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:33.661543 19351 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:33.698776 17922 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 5, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:33.735235 18095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:33.736450 18095 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:33.805984 18097 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 12:21:33.938251 19453 raft_consensus.cc:993] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:33.939221 19224 raft_consensus.cc:1081] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:33.949673 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:33.950554 18165 raft_consensus.cc:606] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:33.952387 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:33.954802 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:33.955823 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:33.957895 18020 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:33.958634 18020 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:33.959082 18020 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:34.010721 19583 raft_consensus.cc:993] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:34.011472 19583 raft_consensus.cc:1081] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:34.021124 18165 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:34.022156 18165 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:34.022766 18165 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:34.043387 18020 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:34.070833 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53018.8us
I20260709 12:21:34.075127 18165 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.077276 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:34.077157 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:34.075927 18020 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:34.078152 18168 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:34.078293 18095 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:34.079838 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 63390.4us
I20260709 12:21:34.079355 18095 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:34.083388 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:34.084218 18019 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:34.084672 18019 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:34.086802 18095 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.088726 18095 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:34.088846 18168 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:34.090998 18019 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.092789 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:34.101944 19585 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:34.103300 18019 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:34.108796 18168 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260709 12:21:34.112005 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150810us
I20260709 12:21:34.113127 18165 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:34.118439 17970 auto_rebalancer.cc:629] Failed to schedule move for tablet a6ac73d7138d4200bbfcbf2f84c2c775: Remote error: Service unavailable: leader transfer in progress
I20260709 12:21:34.111212 19585 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:34.123713 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:34.124568 18098 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:34.124583 19585 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.126154 18019 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:34.129009 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:34.115301 19583 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:21:34.130121 19598 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:34.130671 19598 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:34.131574 19598 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.180718 18090 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260709 12:21:34.186281 18090 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:21:34.186748 18090 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.5
I20260709 12:21:34.187705 18090 raft_consensus.cc:2917] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Aborting config change with OpId 3.5: Aborted: Op aborted by new leader
I20260709 12:21:34.184865 18098 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:34.190146 18090 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:34.194170 18812 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.205616 18019 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:34.203410 19583 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:21:34.223682 18095 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:34.242595 17920 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:34.238260 17922 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:34.247377 18812 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260709 12:21:34.268056 18812 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 4
I20260709 12:21:34.276057 18812 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.5
I20260709 12:21:34.276533 18812 raft_consensus.cc:2917] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Aborting config change with OpId 3.5: Aborted: Op aborted by new leader
I20260709 12:21:34.277094 18812 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:34.280690 19551 raft_consensus.cc:993] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:34.281257 19551 raft_consensus.cc:1081] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:34.282486 19351 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:34.284198 18097 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:34.284906 18097 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:34.285328 18097 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:34.304219 18097 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.319583 18097 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:34.323410 18165 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:34.323187 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:34.324182 18173 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:34.324522 18173 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:34.330880 18097 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:34.332235 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:34.333170 18019 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:34.325403 18173 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.335486 18173 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:34.339947 19502 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:34.348031 17988 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0420fe6ea0ab490e922b7c35c62b6a19. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:34.352770 19502 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:34.375075 18173 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:34.379046 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:34.383903 19224 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:34.384508 19224 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:34.385468 19224 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:34.409917 18019 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:34.458339 19502 raft_consensus.cc:1064] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:34.473028 19502 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:34.486865 19583 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:34.492812 18165 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:34.492774 18098 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:34.499492 18168 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:34.505734 18811 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:34.512543 19494 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:34.515630 19351 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:34.526134 17922 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:34.554761 18090 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:34.563316 18095 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:34.539383 17921 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:34.564116 19598 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:34.568111 19494 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:34.571400 19502 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 2 to 6, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:34.580767 18811 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 2 to 6, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:34.621371 18019 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:34.621114 18098 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 2 to 6, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:34.645978 19598 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:34.612283 18095 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:34.707576 18168 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:34.709100 17921 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 2 to 6, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:34.724084 19623 ts_tablet_manager.cc:933] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:34.725772 19583 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:34.739557 18063 consensus_peers.cc:597] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a29e00b535474906948e66659110db3b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:34.741858 18165 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 2 to 6, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:34.751386 19623 tablet_copy_client.cc:323] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:34.764216 18168 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:34.777215 18098 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:34.781457 19583 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:34.793177 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet 0420fe6ea0ab490e922b7c35c62b6a19 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:34.794005 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet 0420fe6ea0ab490e922b7c35c62b6a19 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-0420fe6ea0ab490e922b7c35c62b6a19
I20260709 12:21:34.794384 18019 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:34.813176 18034 tablet_copy_source_session.cc:215] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:34.818826 19501 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:34.827639 19501 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:34.831398 19623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0420fe6ea0ab490e922b7c35c62b6a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:34.872032 19623 tablet_copy_client.cc:806] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:34.872908 19623 tablet_copy_client.cc:670] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:34.886548 19623 tablet_copy_client.cc:538] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:34.913162 19224 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:34.917929 18019 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:34.933779 19623 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:34.940259 18095 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:34.962764 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:34.988137 18019 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:34.997036 18165 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:35.013793 18063 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cc94b1c76be48d6813e7d1c89a4d66c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:35.016096 19603 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:35.026201 18020 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:35.032563 18165 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:35.050031 19603 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:35.052340 18170 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:35.059397 19603 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:35.063665 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:35.087170 19224 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:35.107369 19623 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:35.109872 19623 tablet_bootstrap.cc:492] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:35.110953 19623 ts_tablet_manager.cc:1403] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.177s	user 0.073s	sys 0.034s
I20260709 12:21:35.114928 19623 raft_consensus.cc:359] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:35.126050 19623 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:35.127069 19623 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:35.130275 19623 ts_tablet_manager.cc:1434] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.002s
I20260709 12:21:35.147181 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-0420fe6ea0ab490e922b7c35c62b6a19 received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:35.147785 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-0420fe6ea0ab490e922b7c35c62b6a19 on tablet 0420fe6ea0ab490e922b7c35c62b6a19 with peer 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:35.184625 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98227us
I20260709 12:21:35.186139 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:35.242903 18165 raft_consensus.cc:606] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:35.187428 19623 ts_tablet_manager.cc:933] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:35.250418 19623 tablet_copy_client.cc:323] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:35.251777 18019 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:35.264860 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet a29e00b535474906948e66659110db3b from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:35.265656 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet a29e00b535474906948e66659110db3b from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-a29e00b535474906948e66659110db3b
I20260709 12:21:35.202853 18811 raft_consensus.cc:1217] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:35.288892 18019 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:35.296195 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:35.296947 18170 raft_consensus.cc:606] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:35.301342 18016 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:35.302107 18016 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:35.306320 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:21:35.309626 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:35.314484 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:35.311954 19459 raft_consensus.cc:993] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:35.309330 19453 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:35.319010 19430 raft_consensus.cc:1081] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:35.317413 19583 raft_consensus.cc:993] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:35.322418 19583 raft_consensus.cc:1081] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:35.334709 18098 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:35.337970 18173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:35.338246 18098 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:35.351579 18098 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.345162 18019 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:35.353266 18173 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:35.363234 18173 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.344506 18109 tablet_copy_source_session.cc:215] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:35.382009 18173 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:35.390175 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:35.390931 18090 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.396718 18165 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:35.398290 17920 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:35.399505 19623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e00b535474906948e66659110db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:35.413371 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:35.414166 18016 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:35.414462 18016 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:35.415313 18016 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:35.417508 18016 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.419564 18090 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:21:35.437103 18173 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:35.438772 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:35.439571 18811 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.465129 18811 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:21:35.467166 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:35.455618 18098 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:35.472209 19459 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:35.472849 19459 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:35.475322 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:35.476152 18019 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.480006 18016 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:21:35.483596 18098 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:35.487867 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:35.488576 18173 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:35.488912 18173 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:35.489732 18173 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:35.497001 18173 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.502964 19459 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:35.520823 19623 tablet_copy_client.cc:806] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:35.521652 19623 tablet_copy_client.cc:670] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:35.581640 19623 tablet_copy_client.cc:538] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:35.655064 18173 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:35.578563 17919 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 5 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.704319 18019 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:35.709609 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:35.710748 19604 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:35.711323 19604 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:35.712320 19604 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:35.704263 19626 raft_consensus.cc:1064] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
W20260709 12:21:35.721122 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50450.7us
I20260709 12:21:35.742974 19619 raft_consensus.cc:993] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:35.743590 19619 raft_consensus.cc:1081] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:35.785405 19459 raft_consensus.cc:993] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:35.788980 19459 raft_consensus.cc:1081] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:35.792287 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:58850
I20260709 12:21:35.792999 18812 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:35.793349 18812 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.800352 18812 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:35.802801 18812 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:35.804325 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:35.805099 18168 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:35.805439 18168 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:35.806406 18168 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:35.808738 18168 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.805469 17919 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.822146 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:35.824307 19640 ts_tablet_manager.cc:933] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:35.830353 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98141.1us
W20260709 12:21:35.831246 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 100543us
I20260709 12:21:35.834048 18168 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:35.834689 18168 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:35.835039 18168 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.854547 18168 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:35.855522 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:35.856294 18019 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.859256 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:35.860256 18090 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.865713 18019 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:35.768973 19626 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:35.877909 18168 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:35.904047 19623 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:35.906577 18090 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:35.923628 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:35.925204 19598 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:35.926183 19598 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:35.927615 19598 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:35.966365 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:35.982599 19586 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:36.034025 19640 tablet_copy_client.cc:323] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:36.011724 19586 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:36.040521 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet 4cc94b1c76be48d6813e7d1c89a4d66c from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:36.040433 19586 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:36.041219 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet 4cc94b1c76be48d6813e7d1c89a4d66c from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-4cc94b1c76be48d6813e7d1c89a4d66c
W20260709 12:21:36.069710 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87650.1us
I20260709 12:21:36.077328 18813 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:36.080040 19645 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:36.081878 18165 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:36.083560 19655 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:36.082939 18109 tablet_copy_source_session.cc:215] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:36.096112 19640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc94b1c76be48d6813e7d1c89a4d66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:36.100529 18095 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:36.103911 18095 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:36.114632 18173 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:36.118019 18016 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:36.120148 19655 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:36.121250 19655 consensus_queue.cc:1243] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:21:36.122190 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:36.122927 18016 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:36.123308 18016 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:36.124164 18016 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:36.125501 18016 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:36.128849 17920 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 3 to 4, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:36.133491 17919 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 3 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:36.137930 18095 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:36.144107 18022 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:36.145059 19643 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:21:36.155872 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.177480 19598 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:21:36.220957 19502 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.223742 19623 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:36.224892 19623 tablet_bootstrap.cc:492] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:36.231122 18812 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:36.225731 19623 ts_tablet_manager.cc:1403] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.322s	user 0.034s	sys 0.017s
I20260709 12:21:36.233515 18173 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.240373 19623 raft_consensus.cc:359] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:36.243247 18016 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:36.245150 19351 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:36.272401 19459 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:36.277740 17920 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.243023 19623 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:36.284325 19623 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:36.291815 19623 ts_tablet_manager.cc:1434] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.058s	user 0.009s	sys 0.000s
I20260709 12:21:36.286965 18811 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.339083 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-a29e00b535474906948e66659110db3b received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:36.339713 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-a29e00b535474906948e66659110db3b on tablet a29e00b535474906948e66659110db3b with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:36.400242 18090 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.420329 18015 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:36.463090 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:36.482674 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92564.6us
I20260709 12:21:36.511397 19640 tablet_copy_client.cc:806] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:36.512295 19640 tablet_copy_client.cc:670] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:36.529479 18015 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:36.535580 19643 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:36.551237 18165 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:36.555655 18811 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:36.557694 18090 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:36.561669 19649 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:36.559840 19459 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:36.578148 19502 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:36.591842 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 103792us
I20260709 12:21:36.599488 18168 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:36.600310 18173 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.608189 18811 raft_consensus.cc:1217] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:36.611802 19626 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:36.617893 19636 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:36.621452 18811 raft_consensus.cc:2955] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:36.631134 19640 tablet_copy_client.cc:538] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:36.647123 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:36.647897 18165 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:36.652792 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:36.653797 18165 raft_consensus.cc:606] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:36.655748 18015 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:36.657792 18090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:36.658615 18090 raft_consensus.cc:606] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:36.667287 18015 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:36.668059 18015 raft_consensus.cc:606] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:36.670118 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:21:36.694624 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:21:36.695411 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88789.8us
I20260709 12:21:36.695648 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:36.836249 19640 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:36.905280 19604 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
W20260709 12:21:36.920258 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 221756us
W20260709 12:21:36.921319 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65955.2us
W20260709 12:21:36.933270 17908 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leader transfer in progress [suppressed 1 similar messages]
I20260709 12:21:36.947876 18020 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:36.969758 18173 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:21:36.981541 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106433us
I20260709 12:21:36.979120 17919 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:37.005630 19648 raft_consensus.cc:993] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:37.006263 19648 raft_consensus.cc:1081] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:37.011830 19650 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:21:37.014019 19583 raft_consensus.cc:993] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:37.014652 19583 raft_consensus.cc:1081] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:37.014539 19678 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:21:36.995007 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61787.2us
W20260709 12:21:37.032930 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75416.2us
I20260709 12:21:37.060920 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0420fe6ea0ab490e922b7c35c62b6a19 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:37.061709 19453 raft_consensus.cc:993] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:37.067432 19224 raft_consensus.cc:1081] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:37.086792 18020 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:37.087708 18020 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:37.088912 18020 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:37.094647 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98209.3us
W20260709 12:21:37.126847 17908 catalog_manager.cc:4755] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): ChangeConfig:REMOVE_PEER RPC failed for tablet 5240f7371cc74cb48dffa1dadcbeec49: Remote error: Service unavailable: leader transfer in progress
W20260709 12:21:37.161706 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 135683us
W20260709 12:21:37.164105 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141106us
I20260709 12:21:37.157663 18020 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:37.171782 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:37.172678 18090 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:37.174906 18090 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.182416 18166 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:37.204408 18166 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:37.206708 18166 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:37.217476 18020 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:37.227907 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67511.6us
I20260709 12:21:37.254012 18090 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:37.269330 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:37.284442 18165 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:37.303328 18165 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:37.304059 18090 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:37.304955 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:37.312485 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:37.313475 18811 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.319617 18166 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:37.366672 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61789.1us
W20260709 12:21:37.366710 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45034.1us
I20260709 12:21:37.375164 18166 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:37.410585 18811 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:21:37.393026 19583 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:37.378648 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:37.438810 18091 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:37.451099 18091 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:37.452107 18091 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:37.453362 18091 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:37.461594 19583 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:37.494081 18091 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:37.507722 19677 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.104s	user 0.003s	sys 0.000s
I20260709 12:21:37.508221 18813 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:37.509562 19683 raft_consensus.cc:993] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:37.519822 19683 raft_consensus.cc:1081] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:37.493809 19583 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:37.524130 19459 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:37.525542 19459 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:37.526082 19459 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:37.527191 19459 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:37.531368 19459 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.532596 19640 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:21:37.538379 19689 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.303s	user 0.004s	sys 0.001s
I20260709 12:21:37.551736 19640 tablet_bootstrap.cc:492] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:37.513026 19690 raft_consensus.cc:1064] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:37.554816 18166 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:37.560614 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 164341us
I20260709 12:21:37.574677 19640 ts_tablet_manager.cc:1403] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.739s	user 0.039s	sys 0.029s
W20260709 12:21:37.585318 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98317.1us
I20260709 12:21:37.588862 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:37.560527 19502 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:21:37.606204 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:37.607012 18090 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:37.641301 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:21:37.642722 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56657.5us
I20260709 12:21:37.646194 18169 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:37.642853 18017 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49542
I20260709 12:21:37.647811 18017 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:37.648213 18017 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:37.649008 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57873.6us
I20260709 12:21:37.659312 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:37.660127 18019 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.662618 18016 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:37.671041 18021 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:37.671561 18021 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:37.672410 18021 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:37.674566 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.682296 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:37.695374 18017 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:37.702967 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48058us
I20260709 12:21:37.666077 19690 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:37.720323 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:37.730453 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:37.731577 18170 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:37.732009 18170 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:37.737259 18019 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:21:37.737673 18017 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:37.842379 18021 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:37.844163 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93047.8us
W20260709 12:21:37.847985 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 95223.9us
I20260709 12:21:37.848495 19646 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:37.849193 19646 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:37.850486 19646 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:37.881132 19702 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:21:37.894290 18021 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
W20260709 12:21:37.896721 19626 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:37.903084 19640 raft_consensus.cc:359] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:37.904630 19640 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
W20260709 12:21:37.911047 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 160321us
I20260709 12:21:37.885891 19706 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:37.925155 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:37.925969 18812 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:37.919963 19706 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:37.946990 19706 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:37.970278 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:37.976984 19706 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:37.977669 19706 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:37.978060 18812 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:37.982275 18024 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:37.917500 19640 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:38.008880 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:37.994817 19706 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:38.018443 19655 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:38.026013 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:38.028368 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:37.995572 19683 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
W20260709 12:21:38.035518 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 118524us
I20260709 12:21:38.037753 19640 ts_tablet_manager.cc:1434] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.457s	user 0.008s	sys 0.004s
I20260709 12:21:38.038028 19655 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:38.043042 17919 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 147) took 1088 ms. Trace:
I20260709 12:21:38.045666 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:38.060335 19655 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:38.046299 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:38.064134 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60716.9us
I20260709 12:21:38.073045 18090 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:38.081013 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:21:38.086472 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 0420fe6ea0ab490e922b7c35c62b6a19 with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45024
I20260709 12:21:38.092470 18064 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:38.064066 17919 rpcz_store.cc:276] 0709 12:21:36.954182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.954461 (+   279us) service_pool.cc:224] Handling call
0709 12:21:37.037245 (+ 82784us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5400040ca8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1a021161 00007f6c1a0ab3af 00007f6c1a0acb11 00007f6c1c5acfa1 00007f6c1c5ad7ff 00007f6c1c46ca1b 00007f6c1c57490e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc
0709 12:21:38.042964 (+1005719us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:37.056173 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:37.109909 (+ 53736us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:37.111674 (+  1765us) write_op.cc:435] Acquired schema lock
0709 12:21:37.118552 (+  6878us) tablet.cc:662] Decoding operations
0709 12:21:37.224020 (+105468us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:37.229407 (+  5387us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:37.245536 (+ 16129us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:37.275741 (+ 30205us) tablet.cc:701] Row locks acquired
0709 12:21:37.275767 (+    26us) write_op.cc:260] PREPARE: finished
0709 12:21:37.280816 (+  5049us) write_op.cc:270] Start()
0709 12:21:37.280999 (+   183us) write_op.cc:276] Timestamp: P: 1783599697275898 usec, L: 0
0709 12:21:37.281025 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:21:37.337113 (+ 56088us) log.cc:844] Serialized 533 byte log entry
0709 12:21:37.937876 (+600763us) op_driver.cc:464] REPLICATION: finished
0709 12:21:38.034824 (+ 96948us) write_op.cc:301] APPLY: starting
0709 12:21:38.034931 (+   107us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:38.035049 (+   118us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:38.038243 (+  3194us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:38.038625 (+   382us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:38.038812 (+   187us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:38.038862 (+    50us) write_op.cc:312] APPLY: finished
0709 12:21:38.039119 (+   257us) log.cc:844] Serialized 23 byte log entry
0709 12:21:38.039349 (+   230us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:38.039456 (+   107us) write_op.cc:454] Released schema lock
0709 12:21:38.039500 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:38.039928 (+   428us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:38.039965 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":2785408,"child_traces":[["op",{"apply.queue_time_us":95011,"apply.run_cpu_time_us":2222,"apply.run_wall_time_us":5077,"num_ops":1,"prepare.queue_time_us":12143,"prepare.run_cpu_time_us":38416,"prepare.run_wall_time_us":595579,"raft.queue_time_us":212518,"raft.run_cpu_time_us":27336,"raft.run_wall_time_us":78985,"replication_time_us":645692,"thread_start_us":315381,"threads_started":4,"wal-append.queue_time_us":336556}]]}
I20260709 12:21:38.090652 19688 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:21:38.114236 19673 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:38.141743 18167 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:38.154526 18812 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:38.158327 18170 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:38.160847 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 5240f7371cc74cb48dffa1dadcbeec49 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:38.166002 18170 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:38.157712 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-4cc94b1c76be48d6813e7d1c89a4d66c received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:38.175278 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-4cc94b1c76be48d6813e7d1c89a4d66c on tablet 4cc94b1c76be48d6813e7d1c89a4d66c with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:38.180928 18169 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:38.185142 18024 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:38.169981 19673 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:21:38.204258 19604 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:38.218927 19224 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:38.281791 19723 tablet_replica.cc:333] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:21:38.296017 19696 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:38.303007 18812 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:38.288689 18170 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:38.306249 17922 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 248) took 1323 ms. Trace:
I20260709 12:21:38.306708 17922 rpcz_store.cc:276] 0709 12:21:36.982885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.983031 (+   146us) service_pool.cc:224] Handling call
0709 12:21:38.306174 (+1323143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1348,"reported_tablets":1}
I20260709 12:21:38.319897 19723 raft_consensus.cc:2243] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.330826 19655 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:38.350153 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45607.7us
W20260709 12:21:38.356675 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53408.6us
I20260709 12:21:38.356489 18170 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:21:38.359095 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50812.2us
I20260709 12:21:38.380791 19723 raft_consensus.cc:2272] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.382741 17922 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 7, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.387786 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 248) took 1389 ms. Trace:
I20260709 12:21:38.388335 17921 rpcz_store.cc:276] 0709 12:21:36.998001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.998227 (+   226us) service_pool.cc:224] Handling call
0709 12:21:38.387707 (+1389480us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33872,"reported_tablets":4}
I20260709 12:21:38.385135 17922 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:21:38.442128 19723 ts_tablet_manager.cc:1916] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:38.476274 18812 raft_consensus.cc:1217] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:38.476377 18022 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:38.484496 18024 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:38.486099 19646 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:38.510319 18096 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:38.544349 18812 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:38.550487 19453 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:38.583841 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:38.516863 19700 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260709 12:21:38.637380 18022 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:38.648729 18814 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:38.658690 19697 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:38.666544 18811 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:38.695309 19224 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:38.705896 18173 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:38.747458 18173 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:21:38.786954 19734 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:38.760306 19680 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:21:38.832195 18096 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:38.772791 19710 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
W20260709 12:21:38.859210 19733 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.144s	user 0.004s	sys 0.000s
I20260709 12:21:38.877962 19723 ts_tablet_manager.cc:1929] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:38.880638 19723 log.cc:1199] T 0420fe6ea0ab490e922b7c35c62b6a19 P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/0420fe6ea0ab490e922b7c35c62b6a19
W20260709 12:21:38.911760 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49117.5us
W20260709 12:21:38.928942 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48899.5us
I20260709 12:21:38.931022 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:38.931875 18165 raft_consensus.cc:606] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:38.933182 19680 raft_consensus.cc:1064] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:38.919024 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet 0420fe6ea0ab490e922b7c35c62b6a19 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:38.937994 17919 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.941633 18024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:38.942708 18024 raft_consensus.cc:606] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:38.943166 18024 raft_consensus.cc:614] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:38.982126 19680 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:39.002483 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:39.004611 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:39.042914 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:39.069002 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72847us
W20260709 12:21:39.076530 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66713.1us
W20260709 12:21:39.122433 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86383.6us
I20260709 12:21:39.239988 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.242439 17921 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:39.266952 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 227771us
W20260709 12:21:39.320895 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 152304us
W20260709 12:21:39.378325 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66620.9us
W20260709 12:21:39.453107 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47537.1us
I20260709 12:21:39.490311 19651 raft_consensus.cc:993] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:39.491081 19651 raft_consensus.cc:1081] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
W20260709 12:21:39.514531 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58096us
W20260709 12:21:39.530562 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49294.2us
I20260709 12:21:39.534443 18096 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:39.535426 18096 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:39.535956 18096 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.562994 18812 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:21:39.564064 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 93203.8us
I20260709 12:21:39.568269 18096 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:39.573381 19224 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:39.577085 18169 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:39.584358 18096 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:39.601527 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:39.602356 18169 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:39.602685 18169 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:39.603524 18169 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:39.622362 18169 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.622176 19636 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:39.639824 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60762.8us
I20260709 12:21:39.715476 18016 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:39.717159 18095 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:39.723302 18169 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:39.725123 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:39.726521 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:39.727384 18016 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.732856 18813 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:21:39.734872 19706 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:39.740346 19680 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:39.741012 19680 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:39.761027 18173 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:21:39.763190 19751 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Committing config change with OpId 7.9: config changed from index 6 to 9, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.766705 18016 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:39.781526 18813 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 6 to 9, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.797082 19720 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:39.749567 19680 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:39.800889 19636 raft_consensus.cc:993] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:39.801608 19636 raft_consensus.cc:1081] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:39.824144 18017 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:39.825136 18017 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:39.825569 18017 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.826839 18814 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.823776 19720 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.836974 18017 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:39.850791 19762 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:39.852751 18017 leader_election.cc:290] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:39.854175 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:39.854915 18812 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.857857 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:39.858603 18169 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.874289 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5240f7371cc74cb48dffa1dadcbeec49 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
W20260709 12:21:39.877187 17906 catalog_manager.cc:4755] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): ChangeConfig:REMOVE_PEER RPC failed for tablet a29e00b535474906948e66659110db3b: Remote error: Service unavailable: leader transfer in progress
I20260709 12:21:39.880672 18812 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:39.861037 17919 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.891238 18165 raft_consensus.cc:2955] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 6 to 9, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.910223 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:39.911082 18095 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:39.911429 18095 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:39.912732 18095 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:39.941895 18095 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.944605 18165 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.946094 17921 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 6 to 9, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:39.967828 17921 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:39.983704 18024 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:39.920701 18169 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:39.988255 17989 leader_election.cc:304] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:39.989441 19753 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:39.990883 19753 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:39.992090 19753 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:40.051837 18095 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:40.057294 17906 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a29e00b535474906948e66659110db3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 768723648a9444fda53659e6a8cc3091 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:40.085743 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.141122 18095 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:40.143431 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet 5240f7371cc74cb48dffa1dadcbeec49 with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:40.167900 18812 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:40.170200 19720 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:40.172389 19779 tablet_replica.cc:333] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:40.180155 19779 raft_consensus.cc:2243] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:40.183543 19779 raft_consensus.cc:2272] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:40.187286 18019 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:40.194038 19680 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:40.205309 19762 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:40.206476 19779 ts_tablet_manager.cc:1916] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:40.215471 18812 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:40.242013 18024 raft_consensus.cc:2955] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:40.244681 17906 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4cc94b1c76be48d6813e7d1c89a4d66c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:40.256529 17922 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 4, VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3) evicted. New cstate: current_term: 1 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:40.262933 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:40.263716 18017 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:40.282871 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:40.283640 18017 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership
I20260709 12:21:40.287153 18154 tablet_service.cc:1558] Processing DeleteTablet for tablet 4cc94b1c76be48d6813e7d1c89a4d66c with delete_type TABLET_DATA_TOMBSTONED (TS ae9641d5c3184572a102d51a75f670ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50498
I20260709 12:21:40.289534 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:40.290367 18017 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership
W20260709 12:21:40.295850 17970 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 12:21:40.296344 17970 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for a29e00b535474906948e66659110db3b already in progress
W20260709 12:21:40.296671 17970 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for a29e00b535474906948e66659110db3b already in progress
I20260709 12:21:40.307341 18165 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:40.317796 19751 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:40.350533 18017 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:40.359115 19680 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:40.368626 19780 tablet_replica.cc:333] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:21:40.371968 19780 raft_consensus.cc:2243] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:40.373211 19780 raft_consensus.cc:2272] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:40.433975 18090 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:40.435429 18166 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:40.437245 19649 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:21:40.443269 19783 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:40.447494 19780 ts_tablet_manager.cc:1916] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:40.471946 19779 ts_tablet_manager.cc:1929] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.8
I20260709 12:21:40.506352 19779 log.cc:1199] T 5240f7371cc74cb48dffa1dadcbeec49 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/5240f7371cc74cb48dffa1dadcbeec49
I20260709 12:21:40.526401 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet 5240f7371cc74cb48dffa1dadcbeec49 (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:40.528937 18812 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:40.529441 19649 raft_consensus.cc:993] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:40.530069 19649 raft_consensus.cc:1081] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:40.531867 18095 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:21:40.532487 18095 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:40.532970 18095 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:40.536968 19783 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:40.551285 18095 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:40.555886 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:40.556676 18812 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:40.565529 18812 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:40.579708 18090 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:40.584787 18812 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:21:40.591634 19649 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:40.592362 19780 ts_tablet_manager.cc:1929] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:40.592870 19780 log.cc:1199] T 4cc94b1c76be48d6813e7d1c89a4d66c P ae9641d5c3184572a102d51a75f670ee: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/wals/4cc94b1c76be48d6813e7d1c89a4d66c
I20260709 12:21:40.592646 19649 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:40.595777 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:40.596511 18019 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:40.596868 18019 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:40.597767 18019 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:40.599470 18019 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:40.595777 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:40.602630 18166 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:40.572757 18095 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:40.623090 17908 catalog_manager.cc:5028] TS ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): tablet 4cc94b1c76be48d6813e7d1c89a4d66c (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:40.630398 18166 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:40.647138 18019 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:40.679931 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:40.680941 19756 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:40.681505 19756 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:40.684770 19756 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:40.701689 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.746698 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:41.129326 18170 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:41.132797 18095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:41.134819 18095 raft_consensus.cc:606] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:41.217694 18022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:41.218578 18022 raft_consensus.cc:606] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:41.238096 18024 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:21:41.255328 17906 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a29e00b535474906948e66659110db3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:21:41.256562 19735 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:41.274674 19756 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:41.276638 19756 raft_consensus.cc:993] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:41.277277 19756 raft_consensus.cc:1081] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:41.279131 18813 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:41.281840 19756 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:41.295287 19692 raft_consensus.cc:993] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:41.296051 19692 raft_consensus.cc:1081] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:41.302939 18098 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:21:41.304020 18098 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:41.304622 18098 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:41.328001 18095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:41.329095 18095 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:41.386257 18166 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50510
I20260709 12:21:41.402837 18166 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:41.404489 18166 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:41.406569 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:41.413161 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:41.414933 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:41.484814 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91804.7us
I20260709 12:21:41.494719 18098 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:41.507421 19756 raft_consensus.cc:993] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:41.508141 19756 raft_consensus.cc:1081] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:41.535847 18098 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:41.637588 18170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50510
I20260709 12:21:41.638491 18170 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:41.638944 18170 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:41.627694 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:41.643316 18019 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:41.646448 18019 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:41.648993 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:41.649748 18169 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:41.656044 18019 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:41.656916 18170 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:41.658025 18019 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:41.660849 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:41.661484 18170 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:41.662815 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:41.663578 18813 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:41.673921 18813 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:41.544054 18166 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:41.678942 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:41.679713 18095 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:41.680047 18095 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:41.680944 18095 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:41.683521 18095 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:41.691532 18169 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:41.702941 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:41.704453 18024 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:41.706405 18166 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:41.708506 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:41.710721 18098 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:41.711072 18098 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:41.711974 18098 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:41.713210 18098 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:41.735553 18095 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:41.737840 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:41.739213 19795 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:41.739840 19795 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:41.740180 18098 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:41.741040 19795 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:41.747891 18019 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:41.750268 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:41.752038 19795 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:41.753021 19795 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:41.754380 19795 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:41.777370 18024 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:41.779654 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:41.806432 19756 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:41.807340 19756 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:41.808449 19756 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:41.889427 18170 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:41.894963 17922 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 3 to 4, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.910123 17922 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:41.927522 18095 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:41.949602 17920 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:41.960484 18063 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85764dcebf974f7cbec322be36e3dd5f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:41.965945 18166 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:41.967626 18024 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:42.028324 19690 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.037202 19690 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.147282 19756 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.156584 18024 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.161407 18019 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:42.165937 18091 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:42.166229 19651 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:42.172376 19795 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:42.184597 18166 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.193207 18091 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:42.195549 19791 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:42.240947 17920 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.267383 18090 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:42.305699 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a29e00b535474906948e66659110db3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:21:42.314280 18063 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6ac73d7138d4200bbfcbf2f84c2c775. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:42.317288 18170 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:42.318801 18812 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:42.320806 19791 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:42.323091 18019 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:42.328043 19793 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.329236 19690 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.366703 18017 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:42.394405 18166 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:42.421165 19756 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:42.431849 18017 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:42.433765 19791 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:42.446084 18019 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:42.448390 19762 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.472224 18017 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.483217 18019 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:42.500391 19690 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:42.504814 18166 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.526847 19821 ts_tablet_manager.cc:933] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:42.530637 18170 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:42.533015 19747 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:42.489006 17922 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:42.544708 17988 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aef65d70106f4f458174493864b4ba3d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:42.547626 19821 tablet_copy_client.cc:323] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:42.559719 19747 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.579337 18166 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.594269 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet 85764dcebf974f7cbec322be36e3dd5f from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:42.595346 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet 85764dcebf974f7cbec322be36e3dd5f from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-85764dcebf974f7cbec322be36e3dd5f
I20260709 12:21:42.617604 18095 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:42.629113 19649 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:42.640048 18109 tablet_copy_source_session.cc:215] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:42.632707 18095 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:42.672952 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72174us
I20260709 12:21:42.679236 17922 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.689870 19821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85764dcebf974f7cbec322be36e3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:42.689232 18090 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:21:42.750058 18063 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 -> Peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Couldn't send request to peer 910f8c8d787e4abbb092413c1f90ebf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9731c63bc7f406db2c77f5ef0415a78. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:42.755108 18017 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:42.755609 18166 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:42.757891 19690 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.763571 19690 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.845265 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:42.867156 19835 ts_tablet_manager.cc:933] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:42.868366 19823 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.874161 18019 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.885079 18166 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:21:42.876415 19821 tablet_copy_client.cc:806] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:42.888768 19821 tablet_copy_client.cc:670] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:42.900020 19821 tablet_copy_client.cc:538] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:42.914254 17918 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index -1 to 3, NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) added. New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:42.953359 19821 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:42.963096 19835 tablet_copy_client.cc:323] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:42.977164 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet a6ac73d7138d4200bbfcbf2f84c2c775 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:42.977931 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet a6ac73d7138d4200bbfcbf2f84c2c775 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-a6ac73d7138d4200bbfcbf2f84c2c775
I20260709 12:21:43.018554 18109 tablet_copy_source_session.cc:215] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:43.030968 19838 ts_tablet_manager.cc:933] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:43.036810 19838 tablet_copy_client.cc:323] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.1:45965
I20260709 12:21:43.039062 18034 tablet_copy_service.cc:140] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Received BeginTabletCopySession request for tablet aef65d70106f4f458174493864b4ba3d from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:36156)
I20260709 12:21:43.039659 18034 tablet_copy_service.cc:161] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Beginning new tablet copy session on tablet aef65d70106f4f458174493864b4ba3d from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:36156: session id = 910f8c8d787e4abbb092413c1f90ebf3-aef65d70106f4f458174493864b4ba3d
I20260709 12:21:43.044925 19835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ac73d7138d4200bbfcbf2f84c2c775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.051322 18034 tablet_copy_source_session.cc:215] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:43.056084 19821 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:43.059417 19838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef65d70106f4f458174493864b4ba3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.063032 19821 tablet_bootstrap.cc:492] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:43.063967 19821 ts_tablet_manager.cc:1403] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.111s	user 0.036s	sys 0.001s
I20260709 12:21:43.096551 18017 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:43.097393 18017 raft_consensus.cc:606] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:43.104140 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:43.104915 18170 raft_consensus.cc:606] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:43.106601 19835 tablet_copy_client.cc:806] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:43.107475 19835 tablet_copy_client.cc:670] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:43.112388 19835 tablet_copy_client.cc:538] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:43.113312 19838 tablet_copy_client.cc:806] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:43.114162 19838 tablet_copy_client.cc:670] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:43.118978 19838 tablet_copy_client.cc:538] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:43.112952 19821 raft_consensus.cc:359] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:43.120718 18095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:43.121722 19821 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:43.121948 18095 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:43.122686 19821 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:43.128120 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:43.129015 18170 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:43.130272 19821 ts_tablet_manager.cc:1434] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.042s	user 0.005s	sys 0.005s
I20260709 12:21:43.134476 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-85764dcebf974f7cbec322be36e3dd5f received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:43.135110 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-85764dcebf974f7cbec322be36e3dd5f on tablet 85764dcebf974f7cbec322be36e3dd5f with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:43.146709 18095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:43.147471 18095 raft_consensus.cc:606] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:43.149415 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:43.151875 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:43.152908 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:43.161751 19835 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:43.195744 19838 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:43.237629 19551 raft_consensus.cc:993] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:43.238250 19551 raft_consensus.cc:1081] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:43.248867 19649 raft_consensus.cc:993] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:43.249454 19831 raft_consensus.cc:1081] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:43.249903 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:43.250587 18090 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:43.251464 18166 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:43.258015 18166 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:43.258726 18166 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:43.257683 18090 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:43.273046 18166 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.276201 19835 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:43.277508 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:43.278358 18017 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:43.278692 18017 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:43.281042 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:43.277748 18166 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:43.282049 18813 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:43.286099 19821 ts_tablet_manager.cc:933] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:43.277479 19835 tablet_bootstrap.cc:492] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:43.281148 18017 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.301479 18090 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.306282 18017 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:43.336313 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:43.337400 18812 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:43.336342 18090 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:43.342159 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46749.6us
I20260709 12:21:43.348775 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:43.349506 18166 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:43.349798 18166 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:43.350785 18166 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.352053 18166 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:43.355832 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45610.3us
I20260709 12:21:43.362109 18166 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:43.373039 19835 ts_tablet_manager.cc:1403] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.212s	user 0.023s	sys 0.009s
I20260709 12:21:43.374141 18813 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:43.381448 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:43.382571 19551 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:43.383142 19551 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:43.384148 19551 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.394776 19835 raft_consensus.cc:359] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:43.442132 19835 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:43.448673 19835 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:43.430459 19821 tablet_copy_client.cc:323] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.102.2:40047
I20260709 12:21:43.438198 18017 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:43.437203 17922 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:43.409624 18812 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:43.412672 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:43.475145 19841 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:43.477650 19841 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:43.478919 19841 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:43.484508 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71961.7us
I20260709 12:21:43.486575 19835 ts_tablet_manager.cc:1434] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.113s	user 0.007s	sys 0.001s
I20260709 12:21:43.489604 19812 raft_consensus.cc:993] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:43.490410 19842 raft_consensus.cc:1081] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:21:43.492681 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:21:43.493362 18090 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:43.493734 18090 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.494663 19838 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:43.495810 19838 tablet_bootstrap.cc:492] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:43.496721 19838 ts_tablet_manager.cc:1403] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 0.301s	user 0.023s	sys 0.010s
I20260709 12:21:43.524210 19838 raft_consensus.cc:359] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:43.525262 19838 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:43.535418 19838 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:43.538848 19838 ts_tablet_manager.cc:1434] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.042s	user 0.012s	sys 0.000s
I20260709 12:21:43.538473 18090 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:43.539327 18812 raft_consensus.cc:1217] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:43.540864 18090 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:43.550810 18034 tablet_copy_service.cc:342] P cab4a2f5d3ca48a2a803ee8ec3906e1d: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-aef65d70106f4f458174493864b4ba3d received from {username='slave'} at 127.0.0.1:36156
I20260709 12:21:43.551388 18034 tablet_copy_service.cc:434] P cab4a2f5d3ca48a2a803ee8ec3906e1d: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-aef65d70106f4f458174493864b4ba3d on tablet aef65d70106f4f458174493864b4ba3d with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:43.529387 17918 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:43.578886 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62926us
I20260709 12:21:43.628311 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:43.629201 18017 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.640851 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:43.645294 18166 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:43.645805 18166 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:43.647668 18166 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:43.651276 18166 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.674100 19756 raft_consensus.cc:993] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:43.674870 19756 raft_consensus.cc:1081] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:21:43.677579 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:35286
I20260709 12:21:43.678325 18812 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:43.678726 18812 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.743716 18017 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:21:43.757249 18813 raft_consensus.cc:1217] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:21:43.774955 18109 tablet_copy_service.cc:140] P 768723648a9444fda53659e6a8cc3091: Received BeginTabletCopySession request for tablet f9731c63bc7f406db2c77f5ef0415a78 from peer 910f8c8d787e4abbb092413c1f90ebf3 ({username='slave'} at 127.0.0.1:40710)
I20260709 12:21:43.790367 18108 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-a6ac73d7138d4200bbfcbf2f84c2c775 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:43.803129 18108 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-a6ac73d7138d4200bbfcbf2f84c2c775 on tablet a6ac73d7138d4200bbfcbf2f84c2c775 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:43.804679 18109 tablet_copy_service.cc:161] P 768723648a9444fda53659e6a8cc3091: Beginning new tablet copy session on tablet f9731c63bc7f406db2c77f5ef0415a78 from peer 910f8c8d787e4abbb092413c1f90ebf3 at {username='slave'} at 127.0.0.1:40710: session id = 910f8c8d787e4abbb092413c1f90ebf3-f9731c63bc7f406db2c77f5ef0415a78
I20260709 12:21:43.796665 18812 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.816247 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:43.816998 18165 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.829427 18813 raft_consensus.cc:1217] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:21:43.831349 18812 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:43.847378 19762 raft_consensus.cc:993] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:43.848057 19762 raft_consensus.cc:1081] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:21:43.842345 18166 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:21:43.854820 18165 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:21:43.857273 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:43.859154 19703 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:43.859756 19703 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:43.860841 19703 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.866899 18109 tablet_copy_source_session.cc:215] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:43.871553 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:43.873071 19823 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:43.912184 17918 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 7 to 8, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 8 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:43.913996 19823 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:43.917794 19823 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:43.943789 18019 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49528
I20260709 12:21:43.944577 18019 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:43.946955 19821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9731c63bc7f406db2c77f5ef0415a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.949703 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:43.950531 18096 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:43.950901 18096 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:43.951741 18096 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:43.953054 18096 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.944958 18019 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:43.981036 18812 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:43.983137 19795 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:43.996387 18017 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:43.998754 18166 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:43.999436 19795 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:44.005956 18813 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:44.007984 19850 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:44.015769 19831 raft_consensus.cc:1064] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:44.027180 19831 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:44.056159 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46865.4us
I20260709 12:21:44.060909 18096 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:21:44.098405 18019 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:44.146239 18019 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:44.148811 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:44.149596 18813 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:44.200153 18813 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:44.203972 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:44.205276 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 134393us
I20260709 12:21:44.206779 18096 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:44.206715 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:44.207432 18096 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:44.203117 17918 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:44.241178 18096 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:44.243273 18096 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:44.245561 18165 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:44.253445 19649 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:44.254200 19649 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:21:44.273722 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 214625us
W20260709 12:21:44.277420 19858 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
W20260709 12:21:44.279464 19855 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
W20260709 12:21:44.316931 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 110328us
I20260709 12:21:44.331147 19649 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:44.377014 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 101963us
I20260709 12:21:44.338343 18096 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:44.385602 18090 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:44.387676 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:44.444928 18170 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:44.538417 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.571146 19873 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.127s
W20260709 12:21:44.591045 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51015.5us
I20260709 12:21:44.659394 18165 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:21:44.707015 18813 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:44.747962 19882 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:44.798322 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 352623us
I20260709 12:21:44.921734 19880 raft_consensus.cc:1064] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
W20260709 12:21:44.922444 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.935456 19821 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.506s	user 0.005s	sys 0.014s
I20260709 12:21:44.924996 19880 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:44.936954 19821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:35286 (request call id 364) took 1652 ms. Trace:
I20260709 12:21:44.957669 19802 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
W20260709 12:21:44.959952 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.010s	sys 0.009s
W20260709 12:21:44.968889 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.012s	sys 0.010s
W20260709 12:21:45.027998 19649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.016s	sys 0.021s
W20260709 12:21:45.043962 19649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.017s	sys 0.022s
I20260709 12:21:45.035256 18020 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:45.047717 18017 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:45.046475 18097 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
W20260709 12:21:45.078466 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 388632us
I20260709 12:21:45.094195 18090 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:45.095624 19893 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:45.096352 19893 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.100382 19842 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:45.101166 19842 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.104130 19842 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:44.967605 19821 rpcz_store.cc:276] 0709 12:21:43.284922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.285084 (+   162us) service_pool.cc:224] Handling call
0709 12:21:43.295806 (+ 10722us) ts_tablet_manager.cc:934] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Initiating tablet copy from peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
0709 12:21:43.945464 (+649658us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:21:44.386489 (+441025us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:44.389451 (+  2962us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:21:44.936063 (+546612us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:21:44.936438 (+   375us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:44.936779 (+   341us) ts_tablet_manager.cc:989] Replica registered.
0709 12:21:44.936846 (+    67us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":295,"spinlock_wait_cycles":840448,"tablet-copy.queue_time_us":400,"thread_start_us":118511,"threads_started":1}
I20260709 12:21:45.144982 18169 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:21:45.146473 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 146515us
I20260709 12:21:45.152863 19831 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:45.153638 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 103374us
I20260709 12:21:45.150672 19821 tablet_copy_client.cc:806] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:45.204486 18814 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:21:45.231971 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:45.233541 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:45.235291 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87847.3us
I20260709 12:21:45.241636 19821 tablet_copy_client.cc:670] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Starting download of 1 WAL segments...
W20260709 12:21:45.274978 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.276527 19906 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:45.277357 19906 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:45.270779 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 109394us
I20260709 12:21:45.285053 19881 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:45.287629 19893 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:45.291065 18167 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:45.269619 19878 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:45.307659 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 142525us
I20260709 12:21:45.313155 19867 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:45.313967 19867 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:45.316606 19867 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:45.319192 19906 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:45.342562 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:45.357894 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:45.357532 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:45.322981 19878 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:45.374502 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 218556us
I20260709 12:21:45.382441 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:45.382550 19878 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:45.305001 19881 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.408241 19881 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:45.421391 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 286) took 1223 ms. Trace:
I20260709 12:21:45.423359 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.425477 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.423727 19852 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:45.425524 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.443974 19881 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:45.444226 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.444003 18017 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:45.445317 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:45.446411 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86811us
I20260709 12:21:45.447479 19908 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:45.449208 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.464910 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:45.438700 17918 rpcz_store.cc:276] 0709 12:21:44.198345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.198723 (+   378us) service_pool.cc:224] Handling call
0709 12:21:45.421254 (+1222531us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:44.263712 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:44.412320 (+148608us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:44.413723 (+  1403us) write_op.cc:435] Acquired schema lock
0709 12:21:44.416250 (+  2527us) tablet.cc:662] Decoding operations
0709 12:21:44.456804 (+ 40554us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:44.480538 (+ 23734us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:44.483446 (+  2908us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:44.537542 (+ 54096us) tablet.cc:701] Row locks acquired
0709 12:21:44.537602 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:21:44.566348 (+ 28746us) write_op.cc:270] Start()
0709 12:21:44.578135 (+ 11787us) write_op.cc:276] Timestamp: P: 1783599704566231 usec, L: 0
0709 12:21:44.578181 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:21:44.800391 (+222210us) log.cc:844] Serialized 538 byte log entry
0709 12:21:44.975123 (+174732us) op_driver.cc:464] REPLICATION: finished
0709 12:21:44.977434 (+  2311us) write_op.cc:301] APPLY: starting
0709 12:21:44.977583 (+   149us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:44.977762 (+   179us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:45.015914 (+ 38152us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:45.024512 (+  8598us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:45.071167 (+ 46655us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:45.073906 (+  2739us) write_op.cc:312] APPLY: finished
0709 12:21:45.162466 (+ 88560us) log.cc:844] Serialized 23 byte log entry
0709 12:21:45.169662 (+  7196us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:45.169764 (+   102us) write_op.cc:454] Released schema lock
0709 12:21:45.169808 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:45.279748 (+109940us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:45.282841 (+  3093us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1932,"apply.run_cpu_time_us":15687,"apply.run_wall_time_us":281940,"num_ops":1,"prepare.queue_time_us":1069,"prepare.run_cpu_time_us":21271,"prepare.run_wall_time_us":712675,"replication_time_us":396741,"spinlock_wait_cycles":2304,"thread_start_us":6151,"threads_started":1}]]}
I20260709 12:21:45.495745 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:45.507865 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:45.510360 19912 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:45.502852 19878 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:45.501624 19875 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:45.527963 19875 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.499686 19910 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:45.465927 19908 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:45.547453 19908 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:45.562278 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99450.1us
I20260709 12:21:45.569346 19883 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:45.573061 18813 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:45.558358 19910 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.578333 19910 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:45.583990 19902 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.305s	user 0.002s	sys 0.000s
I20260709 12:21:45.587459 19894 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:45.588264 19894 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:45.615413 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.708963 18097 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:45.742760 19882 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 12:21:45.751185 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:45.752215 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:45.753006 18167 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:45.753142 19914 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:45.753963 19914 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:45.640162 19892 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:45.620724 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49109.4us
I20260709 12:21:45.741509 19812 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:21:45.646792 19907 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.347s	user 0.005s	sys 0.000s
I20260709 12:21:45.647907 19875 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:45.698757 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:45.797151 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:45.800510 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:21:45.802955 18167 raft_consensus.cc:606] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:21:45.809393 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:45.793870 19867 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:45.819258 19868 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:45.758390 19892 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.835186 19812 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.891819 19812 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:45.834398 19867 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:45.617753 19894 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:45.857132 19868 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:45.647083 19852 raft_consensus.cc:1064] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:45.648869 19903 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.201s
I20260709 12:21:45.869791 19892 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:46.264477 19892 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:46.265218 19892 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.190902 17920 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.956336 19914 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:46.173426 17922 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 275) took 1058 ms. Trace:
I20260709 12:21:46.268989 17922 rpcz_store.cc:276] 0709 12:21:45.115308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.115490 (+   182us) service_pool.cc:224] Handling call
0709 12:21:46.173305 (+1057815us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 12:21:46.162945 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:46.198055 19911 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.693s	user 0.001s	sys 0.018s
I20260709 12:21:46.191656 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:46.272097 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1485) took 1302 ms. Trace:
I20260709 12:21:46.272576 18096 rpcz_store.cc:276] 0709 12:21:44.969909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.970310 (+   401us) service_pool.cc:224] Handling call
0709 12:21:44.978637 (+  8327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.979150 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.979199 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.980417 (+  1218us) log.cc:844] Serialized 556 byte log entry
0709 12:21:45.364090 (+383673us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:45.364177 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.364280 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.164124 (+799844us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:21:46.271724 (+107600us) raft_consensus.cc:1685] finished
0709 12:21:46.271753 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.271991 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43073920,"thread_start_us":379856,"threads_started":1,"wal-append.queue_time_us":474403}
W20260709 12:21:46.274818 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67951.7us
I20260709 12:21:46.263438 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:45.922654 19867 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:45.879208 19875 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.300766 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:46.209636 19913 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.159476 19887 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:46.311518 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 571424us
I20260709 12:21:46.314613 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:46.316464 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:46.318148 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:46.319726 18136 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:46.328447 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:46.326340 19908 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:46.341701 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:46.356626 19918 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:46.287420 19875 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.277908 19912 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:21:46.326795 19891 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:46.326581 19867 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:46.332871 19892 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:46.377880 19875 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:46.406414 18024 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:21:46.334568 19887 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:46.423895 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 109305us
W20260709 12:21:46.451256 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114199us
W20260709 12:21:46.453116 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 119516us
I20260709 12:21:46.460721 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:46.466499 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:46.467693 18170 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:21:46.478359 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:46.440203 19703 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.487057 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:46.187467 19872 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:46.490208 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:46.491053 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:46.504808 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48472.8us
I20260709 12:21:46.466324 18811 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:46.494288 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:46.494405 19905 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.446494 19895 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:21:46.396214 19885 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:21:46.501053 19887 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:46.491523 18167 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:46.361440 19913 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.524237 19921 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:46.528774 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:46.520596 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:46.554280 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114446us
I20260709 12:21:46.556716 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:46.562033 19931 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.515741 19910 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:46.562814 19931 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.581992 18170 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:46.556365 19892 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:46.593266 19868 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:46.514320 19703 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.581104 19875 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:46.661639 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:46.690820 19931 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:46.692900 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 131969us
I20260709 12:21:46.610920 19821 tablet_copy_client.cc:538] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:46.687016 19875 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:46.703574 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146458us
I20260709 12:21:46.619186 19928 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.641749 19842 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:46.661536 19933 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:46.684090 19881 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.721897 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:46.704797 19892 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.726491 19892 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:46.730723 19892 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:46.663712 19913 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:46.734984 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:46.699556 19868 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.704660 19931 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.746443 19868 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:46.705590 19875 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.711866 19925 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.749094 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:46.759043 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54653.7us
I20260709 12:21:46.744652 19926 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.744481 19917 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.780490 18095 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:46.802734 19924 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.803632 19886 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.804751 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:46.751978 19892 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.661639 19852 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.559976 19905 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.733132 19842 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.764859 19910 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.761427 19841 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.749603 19923 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.734758 19881 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:46.778173 19868 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:46.839192 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:46.840299 19938 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:46.841602 19937 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:46.842212 19884 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.759760 19931 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.806912 19913 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.740947 19928 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.865231 19922 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:46.865489 19869 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:46.876698 19934 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:21:46.845221 19878 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:46.791587 19925 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.844388 19924 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.886863 19937 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.924309 19937 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:46.925074 19937 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.859592 19912 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:46.836710 19892 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.945297 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:46.865309 19928 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:46.883255 19842 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:46.944662 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:46.859109 19910 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.858798 19841 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.968344 19924 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:46.886099 19884 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.883659 19938 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.983888 19938 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:46.939657 19912 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:46.820147 19926 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:46.875245 19913 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.925707 19937 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.846402 19886 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.866250 19922 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:46.937922 19878 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.837509 19917 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:46.928771 19869 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.039243 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.082623 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.084342 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:46.881706 19923 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.010303 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:46.991982 19912 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.082073 19703 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:46.988917 19884 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.111505 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.117477 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:47.054576 19946 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.130549 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.137686 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.138947 18095 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:47.118705 19881 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.146678 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.098349 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.144114 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.134775 19931 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.155256 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.146565 19882 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.161388 19917 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.175402 19923 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.161803 19905 leader_election.cc:290] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.178697 19841 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.186136 19869 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:47.186916 19951 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:21:47.190172 19913 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.192008 19910 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.167367 19892 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.200153 19812 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.206986 19926 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.208508 19812 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.146317 19946 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.198709 19868 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:47.217064 19948 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.184s	user 0.001s	sys 0.001s
I20260709 12:21:47.210858 19869 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.221684 18812 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:47.224336 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.224997 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:47.204339 19878 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.210669 19812 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.231978 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.229141 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.238401 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.205348 19905 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.242466 18094 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:47.244750 18094 raft_consensus.cc:606] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:47.214807 19841 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.219914 19926 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.254024 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.261272 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:47.264556 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.219498 19922 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.256577 19908 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.242626 19958 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.264603 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.265777 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.271150 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.286424 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.288457 19925 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.228700 19868 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.259126 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.234354 19886 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.249023 19878 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:47.295751 19868 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.278688 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:47.270520 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:47.279078 19953 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.280220 19812 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.266577 19926 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.247903 19869 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.284451 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.260090 19841 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.268596 19852 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:21:47.289242 18172 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:47.269426 19875 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.294430 19959 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.296341 19894 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:47.249420 19905 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.314188 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.281983 19908 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.267254 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.418414 18170 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:47.299808 19882 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.307096 19867 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:47.393901 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.308651 19913 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.421809 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.433023 18020 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260709 12:21:47.433506 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101735us
I20260709 12:21:47.443367 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.414323 19886 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.461683 19946 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.330799 19917 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260709 12:21:47.432140 19867 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:21:47.452982 19872 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.782s	user 0.014s	sys 0.002s
I20260709 12:21:47.322683 19953 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.416514 19894 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.453147 19957 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:47.484126 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.486553 18024 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:47.303349 19958 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.431424 19959 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.443543 19913 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.444314 19882 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.467083 19886 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.593451 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.592695 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.477607 19867 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:47.595832 18022 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:47.595909 18024 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:47.516047 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.492633 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.633630 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.634516 18015 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:47.537668 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.533885 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.641539 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.573421 19926 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.582985 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.646297 19886 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.647017 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.587607 19869 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.538458 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.494791 19946 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.161s
I20260709 12:21:47.658870 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.571661 19918 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.661633 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.586635 19894 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.533529 19841 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.666025 19884 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:47.676681 19884 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.631803 19854 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.679912 19884 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.674345 19908 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.674430 19918 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:47.682554 19912 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.570s	user 0.007s	sys 0.005s
I20260709 12:21:47.683136 19912 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.707154 18170 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:21:47.708606 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:47.755935 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.764657 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:47.763522 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46471us
W20260709 12:21:47.770397 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 330698us
I20260709 12:21:47.775847 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:47.777223 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:47.774981 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51490us
I20260709 12:21:47.706686 19912 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.778519 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.779263 18172 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:47.745329 19821 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap starting.
I20260709 12:21:47.777899 17988 leader_election.cc:304] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.786522 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.786931 19930 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.787312 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.761250 19908 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.778947 19939 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.791234 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.734496 19918 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:47.745250 19913 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.785138 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.793094 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.744038 19875 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.738868 18093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:21:47.831265 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.800339 18021 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:47.792631 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:47.745047 19882 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.790390 19905 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:47.727548 19958 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.839435 18022 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:47.783474 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:47.842200 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49634us
I20260709 12:21:47.839736 19551 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.854390 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:21:47.839766 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54707.4us
I20260709 12:21:47.839886 19885 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:47.856877 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:47.855835 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:47.862753 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 85013.3us
I20260709 12:21:47.863243 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.845885 18093 raft_consensus.cc:606] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:47.864521 19918 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:47.865053 19918 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.865479 19918 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:47.865787 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.870540 19872 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:47.853180 19912 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.860189 19551 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.845496 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.884852 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:47.887938 19885 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.838238 19891 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:47.874431 19908 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:47.888468 19931 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:47.861567 19930 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:47.900882 19551 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:47.898734 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.845580 19939 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.893582 19891 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.909528 19867 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:47.910175 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:47.910398 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.918141 18021 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:47.915257 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:47.902609 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:47.899477 19908 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.921523 19869 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:47.922047 19869 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:47.922231 19649 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:47.922573 19869 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:47.922822 19649 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.923251 19968 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:47.923823 19968 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:47.921277 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:47.916337 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:47.935447 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:47.942549 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.919852 19958 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:47.926672 19908 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:47.957913 19551 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:47.956076 19649 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:47.946157 19964 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:21:47.946478 19885 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:47.997520 19551 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:48.009244 19551 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:47.942040 19939 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:48.011520 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.020355 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.998920 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.013540 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:48.024178 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:47.988173 19881 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:47.981633 19912 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:21:48.007506 19948 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:47.990595 19886 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:47.990046 19958 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:48.438565 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1870) took 1018 ms. Trace:
I20260709 12:21:48.142097 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.028683 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.215000 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.400738 19551 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:48.214466 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:48.136922 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.161167 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.029052 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.034521 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:21:48.054967 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94992.7us
I20260709 12:21:48.578027 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:48.036559 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.579867 18019 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:48.041471 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.105597 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.573283 18022 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:21:48.568300 18170 rpcz_store.cc:276] 0709 12:21:47.419997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.421984 (+  1987us) service_pool.cc:224] Handling call
0709 12:21:47.422339 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.718450 (+296111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.438125 (+719675us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:48.438473 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":654310272}
I20260709 12:21:48.011941 19968 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:48.638675 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.639423 19551 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:48.135231 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:21:48.203850 19972 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:21:48.647481 18016 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:21:48.388620 19918 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.000s
I20260709 12:21:48.217404 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2141) took 1500 ms. Trace:
W20260709 12:21:48.523590 19869 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.600s	user 0.014s	sys 0.000s
I20260709 12:21:48.133841 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.948041 19959 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:48.649946 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.792757 19926 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:48.949956 19926 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:48.950217 19869 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:48.951267 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:48.952492 19926 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:48.210210 19891 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:48.952795 19869 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:48.663955 19872 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:48.953540 18091 rpcz_store.cc:276] 0709 12:21:46.716773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.721016 (+  4243us) service_pool.cc:224] Handling call
0709 12:21:46.838827 (+117811us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.985356 (+146529us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.986773 (+  1417us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:47.292617 (+305844us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b4c004c6280. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14983a20 00007f6c14994a29 00007f6c1495a644 00007f6c14981f2d 00007f6c14981b93 00007f6c166b218b 00007f6c166b2cb5 00007f6c166b4004 00007f6c166b3729 00007f6c166b8d96 00007f6c166adadb
0709 12:21:47.458222 (+165605us) log.cc:844] Serialized 494 byte log entry
0709 12:21:47.507697 (+ 49475us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:47.507771 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.216892 (+709121us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x7b5400410fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:48.216954 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:48.217022 (+    68us) raft_consensus.cc:1685] finished
0709 12:21:48.217052 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.217305 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":225545344,"thread_start_us":21725,"threads_started":1,"wal-append.queue_time_us":198378,"wal-append.run_cpu_time_us":2394,"wal-append.run_wall_time_us":1060684}
I20260709 12:21:48.954850 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:48.955741 19841 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:48.955888 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1532) took 1057 ms. Trace:
I20260709 12:21:48.955596 19831 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:48.956382 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:48.948658 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 914219us
I20260709 12:21:48.959096 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1802) took 2288 ms. Trace:
I20260709 12:21:48.729875 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.946318 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.097350 19939 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:48.965967 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 272) took 3603 ms. Trace:
I20260709 12:21:48.946774 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.428913 19867 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:48.142761 19914 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:48.297390 19868 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:48.960299 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:48.978411 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.730494 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.451946 19937 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:49.001904 19854 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:48.648607 19918 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:48.824687 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.093454 19912 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:48.416704 19924 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:48.159235 19973 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:21:49.002935 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44046.7us
I20260709 12:21:48.109440 19933 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.126271 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:48.562840 19821 tablet_bootstrap.cc:700] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent opening tablet: real 0.417s	user 0.006s	sys 0.007s
W20260709 12:21:49.005132 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.012s	sys 0.059s
I20260709 12:21:49.113770 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:49.155689 18019 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:21:49.156267 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1815) took 1011 ms. Trace:
W20260709 12:21:49.156307 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 255229us
I20260709 12:21:49.035575 17919 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 276) took 2749 ms. Trace:
I20260709 12:21:49.157325 18020 rpcz_store.cc:276] 0709 12:21:46.592881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.593326 (+   445us) service_pool.cc:224] Handling call
0709 12:21:46.713620 (+120294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.444717 (+731097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.165288 (+720571us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54003485d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:48.881152 (+715864us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54003485d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:48.881460 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1888482816}
I20260709 12:21:49.046859 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 287) took 3498 ms. Trace:
W20260709 12:21:49.153435 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.012s	sys 0.066s
I20260709 12:21:48.898653 19943 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:48.898116 19922 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:48.898898 19864 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:48.135632 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:49.278674 18020 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:49.156720 18019 rpcz_store.cc:276] 0709 12:21:48.144909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.710901 (+565992us) service_pool.cc:224] Handling call
0709 12:21:49.156198 (+445297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.244972 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:49.208508 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1814) took 1070 ms. Trace:
I20260709 12:21:49.305800 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1820) took 1159 ms. Trace:
I20260709 12:21:49.311322 18019 rpcz_store.cc:276] 0709 12:21:48.146534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.304889 (+1158355us) service_pool.cc:224] Handling call
0709 12:21:49.305264 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.305456 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.305485 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.305530 (+    45us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:49.305561 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.305615 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.305733 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:48.902047 19977 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.208092 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2162) took 1198 ms. Trace:
W20260709 12:21:49.219931 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67177.3us
I20260709 12:21:49.306879 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:49.238910 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:49.261857 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:48.901729 19852 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.229048 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1813) took 1091 ms. Trace:
I20260709 12:21:49.035574 19882 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.304606 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1819) took 1158 ms. Trace:
I20260709 12:21:49.249248 19959 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:49.342226 19979 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.342589 19959 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:49.343053 19959 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.342952 19979 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:49.252640 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.005s	sys 0.004s
I20260709 12:21:49.278378 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
W20260709 12:21:49.344444 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.007s	sys 0.004s
I20260709 12:21:49.346642 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:49.348047 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:49.348827 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:49.350536 19551 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.351218 19551 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.157917 17919 rpcz_store.cc:276] 0709 12:21:46.286386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.286592 (+   206us) service_pool.cc:224] Handling call
0709 12:21:49.035344 (+2748752us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39715,"reported_tablets":1}
I20260709 12:21:49.299229 19928 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.372394 18165 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:21:49.247342 19973 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.329356 19977 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.319317 19922 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.381124 19922 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:49.382018 18092 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:21:49.072408 19908 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.056s	user 0.013s	sys 0.000s
I20260709 12:21:49.311400 19864 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.312040 19986 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.160519 19841 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:49.311087 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 733183us
I20260709 12:21:49.333323 19910 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:49.247606 19924 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.158828 17918 rpcz_store.cc:276] 0709 12:21:45.548829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.549046 (+   217us) service_pool.cc:224] Handling call
0709 12:21:49.046755 (+3497709us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82448,"reported_tablets":2}
I20260709 12:21:49.166006 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2165) took 1067 ms. Trace:
I20260709 12:21:49.305967 18016 rpcz_store.cc:276] 0709 12:21:48.137738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.207121 (+1069383us) service_pool.cc:224] Handling call
0709 12:21:49.207678 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.207969 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.208019 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.208094 (+    75us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:49.208144 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.208227 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.208407 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:48.901448 19917 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:49.306223 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 180086us
I20260709 12:21:48.902526 19895 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:49.241441 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85985.4us
I20260709 12:21:49.320251 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1821) took 1173 ms. Trace:
I20260709 12:21:48.978411 19884 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.058409 19930 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:48.826802 19968 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:49.235939 19983 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.087s	user 0.002s	sys 0.000s
I20260709 12:21:49.127800 18090 rpcz_store.cc:276] 0709 12:21:47.898269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.732872 (+834603us) service_pool.cc:224] Handling call
0709 12:21:48.733184 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:48.955475 (+222291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.955512 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:48.955555 (+    43us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:48.955587 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.955671 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.955815 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.320439 18094 rpcz_store.cc:276] 0709 12:21:47.953415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.091516 (+138101us) service_pool.cc:224] Handling call
0709 12:21:49.151304 (+1059788us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.329349 18015 rpcz_store.cc:276] 0709 12:21:48.137454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.162106 (+1024652us) service_pool.cc:224] Handling call
0709 12:21:49.228130 (+ 66024us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.228475 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.228518 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.228589 (+    71us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:49.228637 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.228744 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.228950 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:48.731128 19946 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.279888 19891 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:48.826282 19953 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:49.523312 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:49.308362 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:49.300696 19958 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:49.336126 19649 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:49.525300 19649 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.306756 19881 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:49.336354 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:49.337433 19943 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.524339 19891 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:49.548271 19891 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.544406 19884 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.261627 19974 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.299278 19984 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.278838 19893 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:49.341507 19981 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:49.152204 19939 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:49.227874 19875 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.359506 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:49.160406 18093 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:49.360788 19918 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:49.160218 19868 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.125784 17920 rpcz_store.cc:276] 0709 12:21:45.308141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.308395 (+   254us) service_pool.cc:224] Handling call
0709 12:21:45.402757 (+ 94362us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16621090 00007f6c166bff3a 00007f6c1c475e9c 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:21:48.911696 (+3508939us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:46.473349 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:46.664156 (+190807us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:46.664205 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:21:46.664228 (+    23us) tablet.cc:662] Decoding operations
0709 12:21:46.664673 (+   445us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:46.664789 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:46.664829 (+    40us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:46.665095 (+   266us) tablet.cc:701] Row locks acquired
0709 12:21:46.665117 (+    22us) write_op.cc:260] PREPARE: finished
0709 12:21:46.665315 (+   198us) write_op.cc:270] Start()
0709 12:21:46.665455 (+   140us) write_op.cc:276] Timestamp: P: 1783599706665257 usec, L: 0
0709 12:21:46.665477 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:21:46.744414 (+ 78937us) log.cc:844] Serialized 533 byte log entry
0709 12:21:47.666153 (+921739us) op_driver.cc:464] REPLICATION: finished
0709 12:21:47.841603 (+175450us) write_op.cc:301] APPLY: starting
0709 12:21:47.841734 (+   131us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:47.846369 (+  4635us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:47.875230 (+ 28861us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:47.880772 (+  5542us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:47.891222 (+ 10450us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:47.891277 (+    55us) write_op.cc:312] APPLY: finished
0709 12:21:47.918335 (+ 27058us) log.cc:844] Serialized 23 byte log entry
0709 12:21:47.918575 (+   240us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:47.918697 (+   122us) write_op.cc:454] Released schema lock
0709 12:21:47.918741 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:47.940229 (+ 21488us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:47.940274 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20675,"reported_tablets":1,"spinlock_wait_cycles":4472576,"child_traces":[["op",{"apply.queue_time_us":175107,"apply.run_cpu_time_us":3882,"apply.run_wall_time_us":98559,"num_ops":1,"prepare.queue_time_us":163176,"prepare.run_cpu_time_us":46529,"prepare.run_wall_time_us":432860,"raft.queue_time_us":420310,"raft.run_cpu_time_us":8707,"raft.run_wall_time_us":140146,"replication_time_us":1000442,"thread_start_us":503500,"threads_started":4,"wal-append.queue_time_us":259342}]]}
I20260709 12:21:49.248993 19933 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.367049 19852 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:48.638958 19886 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.234894 19872 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.126771 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2275) took 1069 ms. Trace:
I20260709 12:21:49.161890 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1811) took 1025 ms. Trace:
I20260709 12:21:49.241484 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:49.043917 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 172) took 3692 ms. Trace:
I20260709 12:21:49.207434 19914 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.248905 19854 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.397362 19551 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:49.520548 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:49.521574 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1530) took 1624 ms. Trace:
I20260709 12:21:49.405917 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:49.443698 18168 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:49.396894 19928 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.466733 18167 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:49.519706 18019 rpcz_store.cc:276] 0709 12:21:48.160990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.318369 (+1157379us) service_pool.cc:224] Handling call
0709 12:21:49.319637 (+  1268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.319878 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.319908 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.319958 (+    50us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:49.319990 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.320049 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.320185 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.334457 18020 rpcz_store.cc:276] 0709 12:21:48.146243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.277918 (+1131675us) service_pool.cc:224] Handling call
0709 12:21:49.278272 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.304275 (+ 26003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.304529 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.521060 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:21:49.523402 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61150.1us
I20260709 12:21:49.520663 18095 rpcz_store.cc:276] 0709 12:21:48.098946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.729331 (+630385us) service_pool.cc:224] Handling call
0709 12:21:49.165902 (+436571us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.544708 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:49.417184 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:49.547255 18813 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:21:49.570942 19974 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.545043 19930 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.442519 19936 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.460569 19977 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:49.499575 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:49.545835 19908 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:49.426481 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:49.466226 19980 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 3 to 7, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:49.386025 19882 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:49.660625 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1812) took 1523 ms. Trace:
I20260709 12:21:49.702723 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:49.677305 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 351956us
I20260709 12:21:50.294900 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.318569 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:49.758765 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.299175 18017 rpcz_store.cc:276] 0709 12:21:48.136790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.997182 (+860392us) service_pool.cc:224] Handling call
0709 12:21:49.136694 (+139512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.136931 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.136957 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.137135 (+   178us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:49.137173 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.137236 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.161607 (+ 24371us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540010a9d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:49.161776 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319510272}
W20260709 12:21:49.575285 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1538) took 1370 ms (client timeout 1809 ms). Trace:
W20260709 12:21:49.571148 18136 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.614s (SENT) [suppressed 5 similar messages]
I20260709 12:21:50.306339 18015 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
W20260709 12:21:50.320115 18092 rpcz_store.cc:269] 0709 12:21:48.181176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.639933 (+458757us) service_pool.cc:224] Handling call
0709 12:21:49.551559 (+911626us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b5400410ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:49.551814 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527584640}
I20260709 12:21:50.307543 18166 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:50.308024 19551 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:50.321764 19551 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:50.322279 19551 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:49.706182 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:50.298485 18097 rpcz_store.cc:276] 0709 12:21:47.897448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.729581 (+832133us) service_pool.cc:224] Handling call
0709 12:21:48.729968 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.376047 (+646079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.376082 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.376132 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:49.376168 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.498094 (+121926us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:49.498174 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.521504 (+ 23330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1485177344}
I20260709 12:21:50.304847 18173 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:49.582551 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1887) took 1913 ms. Trace:
I20260709 12:21:50.325453 18169 rpcz_store.cc:276] 0709 12:21:47.645541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.012442 (+366901us) service_pool.cc:224] Handling call
0709 12:21:48.092267 (+ 79825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.306174 (+1213907us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.306213 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.306260 (+    47us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:49.306293 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.558444 (+252151us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b540029dbd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:49.558521 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.558858 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1126996736}
I20260709 12:21:50.331350 19551 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.527093 19968 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:50.295794 18021 rpcz_store.cc:276] 0709 12:21:48.137162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.080480 (+943318us) service_pool.cc:224] Handling call
0709 12:21:49.080994 (+   514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.520013 (+439019us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.520046 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.520093 (+    47us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:49.520126 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.660254 (+140128us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:49.660317 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.660535 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1009044352}
I20260709 12:21:50.296641 17921 rpcz_store.cc:276] 0709 12:21:45.351104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.351675 (+   571us) service_pool.cc:224] Handling call
0709 12:21:49.043674 (+3691999us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":697644,"reported_tablets":3}
I20260709 12:21:49.460275 19912 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:50.338518 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1543) took 1276 ms. Trace:
I20260709 12:21:50.339031 18095 rpcz_store.cc:276] 0709 12:21:49.062607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.337218 (+1274611us) service_pool.cc:224] Handling call
0709 12:21:50.337572 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.337779 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.337808 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.338202 (+   394us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:50.338244 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.338311 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.338451 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:50.297624 18023 rpcz_store.cc:276] 0709 12:21:48.057130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.133283 (+ 76153us) service_pool.cc:224] Handling call
0709 12:21:49.126692 (+993409us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.549767 19881 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:49.498818 19989 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.349866 19551 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:49.549435 19925 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:50.373342 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.008s	sys 0.010s
I20260709 12:21:50.356729 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:50.359804 19649 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.372994 18170 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:21:50.384150 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1544) took 1281 ms. Trace:
I20260709 12:21:50.384881 18094 rpcz_store.cc:276] 0709 12:21:49.102778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.382768 (+1279990us) service_pool.cc:224] Handling call
0709 12:21:50.383320 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.383612 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.383654 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.383739 (+    85us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:50.383788 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.383871 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.384046 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:49.985888 19841 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:49.807826 19864 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:49.631352 19926 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:49.546070 19910 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.983130 19884 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:50.019454 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:49.861320 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:49.631014 19965 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:50.391978 19649 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.335115 19939 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:49.592932 19990 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.302609 19979 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:50.428880 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 543) took 1098 ms. Trace:
I20260709 12:21:49.729372 19958 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:50.431934 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 542) took 1844 ms. Trace:
I20260709 12:21:50.409396 19841 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:49.980397 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2170) took 1265 ms. Trace:
W20260709 12:21:50.349717 19959 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.006s	user 0.026s	sys 0.029s
I20260709 12:21:50.343523 19977 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:49.631657 19973 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:50.390514 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 413133us
I20260709 12:21:50.417088 19991 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:50.340592 19968 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:50.343012 18093 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1537) took 2153 ms (client timeout 1982 ms). Trace:
I20260709 12:21:49.545908 19917 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:50.404893 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.005s	sys 0.021s
I20260709 12:21:49.571182 19895 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.439767 19953 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:50.463299 19995 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.463389 19953 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.358891 19918 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:49.571641 19975 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.854s
I20260709 12:21:49.545023 19946 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:49.546209 19986 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.469038 19975 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:50.469470 19975 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:50.470120 19986 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:50.408046 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2175) took 1101 ms. Trace:
I20260709 12:21:50.471035 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1541) took 1413 ms. Trace:
I20260709 12:21:50.471304 18092 rpcz_store.cc:276] 0709 12:21:49.307169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.321272 (+1014103us) service_pool.cc:224] Handling call
0709 12:21:50.357921 (+ 36649us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.363018 (+  5097us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.363071 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.363145 (+    74us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:50.363210 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.407590 (+ 44380us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b540011dd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.407682 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.407930 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11018496}
W20260709 12:21:50.419502 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 101037us
I20260709 12:21:50.473728 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.423430 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:50.343678 19875 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.425001 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1890) took 1076 ms. Trace:
I20260709 12:21:50.475095 18170 rpcz_store.cc:276] 0709 12:21:49.348594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.348725 (+   131us) service_pool.cc:224] Handling call
0709 12:21:49.349284 (+   559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.376198 (+1026914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.424628 (+ 48430us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54000438d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.424931 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2323675904}
I20260709 12:21:50.333119 19893 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.421415 19649 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.429296 18024 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:50.429574 18812 rpcz_store.cc:276] 0709 12:21:49.324268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.324438 (+   170us) service_pool.cc:224] Handling call
0709 12:21:49.324791 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.374941 (+1050150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.374971 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.375016 (+    45us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:50.375056 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.422602 (+ 47546us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.422663 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.422797 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2414569856}
I20260709 12:21:50.333158 19936 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.404691 19999 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:50.435515 18815 rpcz_store.cc:276] 0709 12:21:48.579719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.580517 (+   798us) service_pool.cc:224] Handling call
0709 12:21:49.245567 (+665050us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:21:49.245842 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.374181 (+1128339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.374213 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.374255 (+    42us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:50.374287 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.424044 (+ 49757us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540059fe54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.424104 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.424228 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2642511616}
I20260709 12:21:50.415669 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 405) took 1030 ms. Trace:
I20260709 12:21:50.440984 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:50.412736 18018 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:21:50.392993 19925 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:50.394079 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 873004us
W20260709 12:21:50.415294 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 397) took 2609 ms (client timeout 2184 ms). Trace:
I20260709 12:21:50.340883 19981 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.418969 19924 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.342183 19984 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:50.399466 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 851304us
I20260709 12:21:50.430866 19992 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.423786 19943 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.447748 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:21:50.442610 18091 rpcz_store.cc:276] 0709 12:21:48.715615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.277539 (+561924us) service_pool.cc:224] Handling call
0709 12:21:49.571092 (+293553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:49.576847 (+  5755us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.576878 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:49.576936 (+    58us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:49.576990 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.577072 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.980020 (+402948us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400410ad0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:49.980270 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627216640}
W20260709 12:21:50.515218 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67591.1us
I20260709 12:21:50.447043 19993 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:50.394981 19989 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.456214 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:50.453195 18093 rpcz_store.cc:269] 0709 12:21:48.180839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.720512 (+539673us) service_pool.cc:224] Handling call
0709 12:21:50.333719 (+1613207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:50.457630 19884 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:50.455828 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:50.459888 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.005s	sys 0.021s
I20260709 12:21:50.413141 19926 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:50.490258 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 401) took 1520 ms (client timeout 1831 ms). Trace:
I20260709 12:21:50.479524 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.471433 18099 rpcz_store.cc:276] 0709 12:21:49.057852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.336053 (+278201us) service_pool.cc:224] Handling call
0709 12:21:49.336438 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.470077 (+1133639us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.470115 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.470165 (+    50us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:50.470205 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.470781 (+   576us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400296dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.470819 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.470959 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606564096}
I20260709 12:21:50.434557 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 395) took 2631 ms. Trace:
I20260709 12:21:50.487598 19872 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.487917 19930 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.488452 19974 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.538750 19910 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.522416 19841 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:50.457546 20001 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:50.487986 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2272) took 2441 ms (client timeout 2431 ms). Trace:
W20260709 12:21:50.546877 18024 rpcz_store.cc:269] 0709 12:21:48.047128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.049387 (+  2259us) service_pool.cc:224] Handling call
0709 12:21:50.487740 (+2438353us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b5400251a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:50.487874 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079363968}
I20260709 12:21:50.490599 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 544) took 1166 ms. Trace:
I20260709 12:21:50.433332 19996 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:50.496202 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:50.500135 19649 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:50.500754 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43770.3us
I20260709 12:21:50.442672 19997 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:50.464155 19995 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.441856 19958 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.457253 19908 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:50.504341 18022 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:21:50.505191 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2174) took 1270 ms. Trace:
W20260709 12:21:50.516319 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 757801us
I20260709 12:21:50.557808 18090 rpcz_store.cc:276] 0709 12:21:49.235278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.340907 (+1105629us) service_pool.cc:224] Handling call
0709 12:21:50.351757 (+ 10850us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.493570 (+141813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.493623 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.493706 (+    83us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:50.493769 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.504901 (+ 11132us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54004123d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.504957 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.505111 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":325459840}
W20260709 12:21:50.519340 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.10214e+06us
I20260709 12:21:50.451112 19959 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.520921 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:50.470468 19975 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:50.525091 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 830325us
I20260709 12:21:50.571835 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:50.466658 19895 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.543557 19910 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:50.531491 19891 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.983s	user 0.017s	sys 0.000s
I20260709 12:21:50.489537 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1827) took 1138 ms. Trace:
I20260709 12:21:50.541564 20006 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:50.532882 18807 rpcz_store.cc:269] 0709 12:21:48.958253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.964894 (+  6641us) service_pool.cc:224] Handling call
0709 12:21:49.631436 (+666542us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:50.471242 (+839806us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11825024}
I20260709 12:21:50.542481 19936 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.542887 19926 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.484421 18806 rpcz_store.cc:276] 0709 12:21:49.382128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.383233 (+  1105us) service_pool.cc:224] Handling call
0709 12:21:50.412384 (+1029151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:50.548034 18814 rpcz_store.cc:276] 0709 12:21:49.324622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.324764 (+   142us) service_pool.cc:224] Handling call
0709 12:21:49.339318 (+ 14554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.472898 (+1133580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.472934 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.472991 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:50.473023 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.490350 (+ 17327us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54006f2cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.490399 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.490537 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606421760}
I20260709 12:21:50.591146 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:50.531746 19864 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:50.543618 19882 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.547292 19841 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:21:50.528550 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49124.7us
W20260709 12:21:50.577502 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56679us
I20260709 12:21:50.446064 19990 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.518559 19992 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.579843 18015 rpcz_store.cc:276] 0709 12:21:49.350675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.320878 (+970203us) service_pool.cc:224] Handling call
0709 12:21:50.488106 (+167228us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.488346 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.488378 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.488429 (+    51us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:50.488465 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.488535 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.489363 (+   828us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b5400251a50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:50.489468 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":383755008}
I20260709 12:21:50.494123 19999 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.568259 19914 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.620525 19910 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.565485 19997 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.565577 19996 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:50.534108 19884 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.568776 19852 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:50.664371 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43938.5us
I20260709 12:21:50.459631 19991 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.664459 20008 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:50.620828 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44027.2us
I20260709 12:21:50.524933 19993 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:50.439400 19965 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:50.620812 19928 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.621090 19891 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:50.746106 19917 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.750433 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:50.760607 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1818) took 2614 ms (client timeout 2113 ms). Trace:
I20260709 12:21:50.756459 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:50.752775 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:50.724613 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79871.7us
I20260709 12:21:50.747567 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:50.692142 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 121902us
I20260709 12:21:50.868821 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:50.819235 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:50.815729 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:50.755391 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:50.748370 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:50.719331 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:50.673445 18814 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:50.862069 18813 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
W20260709 12:21:50.729278 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1807) took 2635 ms (client timeout 1798 ms). Trace:
I20260709 12:21:50.829612 19872 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:50.866621 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:50.728477 19893 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.758737 19984 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.886296 18092 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:50.888532 18022 rpcz_store.cc:269] 0709 12:21:48.145927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.666070 (+520143us) service_pool.cc:224] Handling call
0709 12:21:50.760375 (+2094305us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:50.760512 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":454503168}
I20260709 12:21:50.890379 18173 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:50.859659 20012 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:50.892382 19996 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.728099 19995 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.827023 19841 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:50.728037 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 245) took 1381 ms. Trace:
I20260709 12:21:50.733991 19994 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.754145 19981 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:50.893410 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:50.950311 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56982.1us
I20260709 12:21:50.756677 19989 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:50.907114 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87971.9us
I20260709 12:21:50.952389 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:50.620195 19933 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.894706 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.532107 18810 rpcz_store.cc:276] 0709 12:21:47.794351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.796915 (+  2564us) service_pool.cc:224] Handling call
0709 12:21:48.454744 (+657829us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:21:48.455134 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.455911 (+   777us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.455967 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:48.457674 (+  1707us) log.cc:844] Serialized 556 byte log entry
0709 12:21:49.003447 (+545773us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:49.058554 (+ 55107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.058650 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:50.403450 (+1344800us) raft_consensus.cc:1685] finished
0709 12:21:50.403478 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.422427 (+ 18949us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29959424,"thread_start_us":318472,"threads_started":1,"wal-append.queue_time_us":679437}
I20260709 12:21:50.947575 19852 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:21:50.954381 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59766.1us
I20260709 12:21:50.910586 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.955411 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:50.660938 19975 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:50.916740 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:50.502020 18808 rpcz_store.cc:269] 0709 12:21:47.795155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.796791 (+  1636us) service_pool.cc:224] Handling call
0709 12:21:48.898432 (+1101641us) spinlock_profiling.cc:243] Waited 9.49 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:49.492204 (+593772us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:50.404293 (+912089us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36612608}
I20260709 12:21:50.859072 19875 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.748250 19886 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.922722 18171 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:50.926250 18170 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:50.890935 19999 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.921581 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:50.938247 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:50.868316 20008 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:50.941437 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 125823us
I20260709 12:21:50.918000 19974 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:50.959895 19893 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:50.906247 19925 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:50.989611 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 82872.7us
W20260709 12:21:50.958892 19881 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.561s	user 0.012s	sys 0.002s
W20260709 12:21:50.994087 18780 connection.cc:256] RPC call timeout handler was delayed by -1.8328809229997205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
W20260709 12:21:51.025290 18780 connection.cc:256] RPC call timeout handler was delayed by -1.7563809229995968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
W20260709 12:21:51.025807 18780 connection.cc:256] RPC call timeout handler was delayed by -1.5071162459998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20260709 12:21:50.905619 19943 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:50.954952 19852 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:50.955700 19917 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.040189 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:50.993754 19986 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:51.043615 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.000949 19984 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:51.030834 18167 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:50.985021 19959 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:50.945096 19872 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:51.030464 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 108992us
I20260709 12:21:51.053187 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:50.995849 19882 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.041301 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:51.001211 19908 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:51.006932 19928 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:51.042500 19852 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:51.005324 19958 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.020151 19974 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:51.013741 20008 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:50.947583 18164 rpcz_store.cc:276] 0709 12:21:49.346276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.356339 (+ 10063us) service_pool.cc:224] Handling call
0709 12:21:49.360802 (+  4463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.520424 (+1159622us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.524052 (+  3628us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.524166 (+   114us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:50.524233 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.727737 (+203504us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540007c454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:50.727789 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.727954 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2645594880}
I20260709 12:21:51.083407 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:51.059237 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1891) took 1709 ms. Trace:
I20260709 12:21:51.059497 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:51.062211 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:50.963713 20012 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.063704 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:51.064525 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:51.064736 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:50.915005 18018 rpcz_store.cc:269] 0709 12:21:48.093809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.135018 (+ 41209us) service_pool.cc:224] Handling call
0709 12:21:50.729069 (+2594051us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001d1354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:50.729189 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2510635776}
W20260709 12:21:51.045641 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 172817us
I20260709 12:21:51.071741 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:51.045784 20009 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:51.031373 19881 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:51.071662 19968 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.599s	user 0.013s	sys 0.000s
W20260709 12:21:51.085660 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99573.2us
I20260709 12:21:51.084623 18171 rpcz_store.cc:276] 0709 12:21:49.349751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.349913 (+   162us) service_pool.cc:224] Handling call
0709 12:21:49.366247 (+ 16334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.997203 (+1630956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.058982 (+ 61779us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:51.059156 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3566930688}
I20260709 12:21:51.161594 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:51.070995 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:51.169039 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83154.7us
I20260709 12:21:51.067643 19908 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:51.170076 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.170310 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.171343 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.166571 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:51.171968 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.172494 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.044938 19943 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:51.046928 19917 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.083547 19974 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:51.080701 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:51.165545 19924 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.074767 19991 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.186100 18098 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 3 to 7, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:51.173475 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.170794 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:51.171139 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.175418 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.167861 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:51.070333 19882 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.164597 20009 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:51.087247 19965 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.174374 19908 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:51.180444 19926 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:51.267681 19993 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:51.273104 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 231605us
I20260709 12:21:51.275322 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.175631 19980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50498 (request call id 85) took 6731 ms. Trace:
I20260709 12:21:51.023977 19994 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.083746 20006 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20260709 12:21:51.280037 19997 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.215345 19891 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:51.213284 19875 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:51.172371 19968 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:51.323546 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.227854 20009 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:51.226447 19953 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.273218 19926 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.336421 19872 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.200809 19924 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.321916 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:51.354341 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 169673us
I20260709 12:21:51.333920 19990 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.369207 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.341517 19926 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:51.342002 19946 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.373055 20023 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.382777 19992 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.375361 20023 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.332553 20018 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.392702 20018 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.285791 19980 rpcz_store.cc:276] 0709 12:21:44.432029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.432217 (+   188us) service_pool.cc:224] Handling call
0709 12:21:44.448461 (+ 16244us) log.cc:844] Serialized 494 byte log entry
0709 12:21:45.069766 (+621305us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b18001e6e80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c16610b0b 00007f6c16695b63 00007f6c166bd752 00007f6c166b59fa
0709 12:21:51.163045 (+6093279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":403917,"raft.queue_time_us":402374,"raft.run_cpu_time_us":1861,"raft.run_wall_time_us":15110,"spinlock_wait_cycles":10441984,"thread_start_us":209917,"threads_started":1,"wal-append.queue_time_us":543237,"wal-append.run_cpu_time_us":2466,"wal-append.run_wall_time_us":1057811}
I20260709 12:21:51.402814 19943 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:51.403214 19924 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.439412 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:51.282333 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114543us
I20260709 12:21:51.478643 19917 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:51.454595 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 285630us
I20260709 12:21:51.347513 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.463079 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.469043 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.402616 19882 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.405584 20012 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.484619 18164 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:51.409344 20023 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.485196 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.482352 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:51.559182 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76831.5us
W20260709 12:21:51.708184 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46219.2us
W20260709 12:21:51.502530 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 227607us
I20260709 12:21:51.487206 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.527384 18171 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:51.875197 19989 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:51.876585 19989 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.551415 19981 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:51.708393 19994 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.882725 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.883445 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:51.490998 20019 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:51.633592 20027 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:51.635134 19854 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:51.891670 18098 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.691s	user 0.016s	sys 0.016s
I20260709 12:21:51.707041 20018 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.501675 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a29e00b535474906948e66659110db3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 9)
I20260709 12:21:51.881623 19981 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:51.912139 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:51.908546 19868 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:51.906327 19854 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:51.930222 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 428840us
W20260709 12:21:51.933211 19926 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.001s
I20260709 12:21:51.933956 19926 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:51.935751 19908 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.010s
I20260709 12:21:51.936964 19926 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.950541 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.953127 19943 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:51.955206 19917 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:51.955745 18092 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:51.955821 19943 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.956140 19917 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.959630 19943 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.960033 19917 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:51.945657 19989 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:51.963436 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58478.5us
I20260709 12:21:51.950954 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:51.948669 19975 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.964s	user 0.018s	sys 0.022s
W20260709 12:21:51.951753 20008 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.792s	user 0.011s	sys 0.009s
W20260709 12:21:51.950043 20009 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.605s	user 0.010s	sys 0.007s
I20260709 12:21:51.918035 20019 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:51.967419 19939 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.498s	user 0.009s	sys 0.013s
I20260709 12:21:51.982252 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:51.982847 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 546) took 1525 ms. Trace:
I20260709 12:21:51.924388 20027 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:51.964521 19908 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:51.983481 18811 rpcz_store.cc:276] 0709 12:21:50.455619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.456478 (+   859us) service_pool.cc:224] Handling call
0709 12:21:50.456814 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.862366 (+405552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.862401 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.862447 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:50.862476 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.976028 (+1113552us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b54004b41d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:51.976162 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.980675 (+  4513us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":932086144}
W20260709 12:21:51.986528 19974 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.691s	user 0.013s	sys 0.011s
I20260709 12:21:51.987748 19926 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:51.991422 19975 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:51.975375 19854 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.001631 18166 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.000643 18172 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:51.978346 20009 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:51.973748 20008 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.004186 19975 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:52.015547 19981 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:51.986760 19939 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:52.020581 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.021577 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.020604 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.025470 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.026726 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.038583 18092 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:52.026726 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.040335 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.040704 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1423) took 1126 ms. Trace:
I20260709 12:21:52.027480 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.028797 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.041987 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.029182 19908 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.002242 19974 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.029979 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.015683 19884 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.033120 19928 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.035235 19975 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.054749 19928 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.002044 19926 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.058504 19926 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:52.034116 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83507.9us
I20260709 12:21:52.038273 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:52.045367 18097 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:52.040359 18099 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:52.046207 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.046869 18024 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:52.041203 18167 rpcz_store.cc:276] 0709 12:21:50.914435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.915306 (+   871us) service_pool.cc:224] Handling call
0709 12:21:52.040202 (+1124896us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:52.040470 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63388416}
I20260709 12:21:52.076392 18017 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:21:52.076601 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.078449 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.080365 19939 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.076862 18170 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:21:52.074280 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 201747us
I20260709 12:21:52.057703 19928 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:52.077572 19841 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.130s	user 0.005s	sys 0.024s
I20260709 12:21:52.086331 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:21:52.086728 19943 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:52.090260 19884 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:52.093039 20019 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.095299 19968 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.095467 19854 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.109539 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.110100 19884 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.110877 18091 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:52.111670 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1550) took 1366 ms (client timeout 1744 ms). Trace:
W20260709 12:21:52.112252 18091 rpcz_store.cc:269] 0709 12:21:50.745530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.750627 (+  5097us) service_pool.cc:224] Handling call
0709 12:21:51.315327 (+564700us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:52.109079 (+793752us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592e9c 00007f6c13592a5b 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:52.111543 (+  2464us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318388864}
I20260709 12:21:52.114157 19884 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.115463 19917 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:52.115253 19943 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.116087 19917 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.116689 19917 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.116729 19841 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.097404 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.119169 19943 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.119593 19841 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.080513 18099 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.142707 20009 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:52.143967 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47046.1us
I20260709 12:21:52.145099 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.146224 20019 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:52.148291 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:52.149364 20019 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:52.149921 20019 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:52.150373 20019 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.150930 19821 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:52.152468 19821 tablet_bootstrap.cc:492] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Bootstrap complete.
I20260709 12:21:52.156394 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.159830 19974 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.164166 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.165307 18022 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.166491 19881 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.166596 20027 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.171125 19821 ts_tablet_manager.cc:1403] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent bootstrapping tablet: real 4.426s	user 0.045s	sys 0.052s
I20260709 12:21:52.174719 19910 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.179540 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1828) took 2828 ms. Trace:
I20260709 12:21:52.190023 18017 rpcz_store.cc:276] 0709 12:21:49.351086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.323027 (+971941us) service_pool.cc:224] Handling call
0709 12:21:50.323795 (+   768us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.077102 (+1753307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.178836 (+101734us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:52.179384 (+   548us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4012284160}
I20260709 12:21:52.196563 19975 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.201984 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.193305 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 202) took 1130 ms. Trace:
I20260709 12:21:52.202082 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.193358 18019 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.182612 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1428) took 1094 ms. Trace:
I20260709 12:21:52.184444 19926 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.201508 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.202886 18811 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
W20260709 12:21:52.202049 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 86077.9us
I20260709 12:21:52.203233 18024 rpcz_store.cc:276] 0709 12:21:51.063280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.063430 (+   150us) service_pool.cc:224] Handling call
0709 12:21:52.192938 (+1129508us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:52.193192 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2013864192}
I20260709 12:21:52.203843 18170 rpcz_store.cc:276] 0709 12:21:51.088145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.170852 (+ 82707us) service_pool.cc:224] Handling call
0709 12:21:52.182287 (+1011435us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:52.182525 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1990754048}
I20260709 12:21:52.211026 18099 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.211673 20008 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.224386 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.225943 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2197) took 1055 ms. Trace:
I20260709 12:21:52.226425 18090 rpcz_store.cc:276] 0709 12:21:51.170545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.173553 (+  3008us) service_pool.cc:224] Handling call
0709 12:21:52.205193 (+1031640us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:52.225850 (+ 20657us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":300316544}
W20260709 12:21:52.228602 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 79040.8us
I20260709 12:21:52.235162 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.247810 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.248842 18810 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:52.251456 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 140820us
I20260709 12:21:52.255174 20008 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:52.256126 20008 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:52.256978 19852 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.187s	user 0.011s	sys 0.010s
I20260709 12:21:52.257551 19852 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.258272 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:52.259240 20008 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.261862 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.262550 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.262892 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.263661 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.263919 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.264341 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.261935 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.265045 18167 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:52.266322 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.266419 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.262130 19928 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:52.267716 19928 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.268253 19928 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.268603 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:52.270485 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77293.3us
I20260709 12:21:52.274058 18098 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:52.274562 18022 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:52.273429 19841 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:52.278501 20033 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.279023 19841 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.279568 19841 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.279394 20033 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.281505 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.281673 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.282068 20033 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.282691 18164 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:52.285895 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.286780 18809 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.278354 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.289110 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.289736 19975 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.294384 19854 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:52.295435 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.295982 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.296468 18020 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:52.296803 18017 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.298933 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.300127 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.301568 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.302592 18168 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.297235 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.297747 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.298085 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:52.298172 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.304466 18016 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:52.304829 20008 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:52.297694 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:21:52.305076 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54455.2us
I20260709 12:21:52.305269 20008 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:52.305675 18021 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.305873 20008 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.260143 19852 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.308794 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.303640 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.298333 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.315429 18016 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.315943 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.316993 18164 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:52.315143 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.320466 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:52.320734 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.322039 18016 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 2: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:52.322201 19936 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:52.316713 19975 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:52.324473 19975 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:52.325013 19975 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:52.325452 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.326196 19975 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.327096 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.328676 19975 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.328751 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.326254 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.316005 20019 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.332795 19821 raft_consensus.cc:359] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:52.334751 20019 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.335918 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:52.337136 19936 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:52.315727 18808 raft_consensus.cc:2955] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 3 to 7, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:52.334949 19821 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Initialized, Role: LEARNER
I20260709 12:21:52.345592 19841 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.346935 19926 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.345288 19821 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:52.348191 19841 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.348374 18098 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.349285 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.366389 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.367208 18023 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.368944 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.370970 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.355244 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.372092 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.372517 18098 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:21:52.358409 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.359534 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.365981 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.374150 18015 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.376529 18017 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:52.358300 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.377647 18019 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.373449 18090 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:52.376652 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.371224 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.386402 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.353592 19821 ts_tablet_manager.cc:1434] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent starting tablet: real 0.182s	user 0.009s	sys 0.003s
W20260709 12:21:52.389899 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54079.6us
I20260709 12:21:52.363840 18021 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:52.398360 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.391870 18015 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:21:52.413836 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 113478us
I20260709 12:21:52.381548 19917 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.430505 18019 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:52.452286 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:52.476959 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59796.7us
I20260709 12:21:52.459636 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.478350 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.427109 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:52.430423 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.424535 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:52.492775 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.495920 18024 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:52.496729 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.452353 17921 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 2 to 3, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:21:52.502219 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.502897 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.504885 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.505626 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.506887 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.509148 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.450393 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.512138 20012 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:52.515318 18807 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:52.516350 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.518157 18815 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.510614 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:52.517088 18168 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.526521 18015 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.527223 18166 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.475522 19917 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.529176 19910 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:52.536383 19910 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.536934 19910 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.452286 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.533413 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:52.353672 20008 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:52.541419 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114519us
I20260709 12:21:52.453512 19928 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.550657 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.558779 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.549656 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.560453 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.564359 20008 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.566202 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.566761 19928 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:52.567271 18168 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.566951 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.568289 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.542426 18809 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.573011 18166 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.569119 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.575790 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.576701 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.578037 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.555426 20012 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:52.586282 18098 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.587327 18170 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.588805 18812 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.590119 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.590929 18167 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:21:52.559213 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.587272 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.592015 20012 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.598387 18815 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:52.608922 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.594765 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.609035 20012 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.609357 18166 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.610940 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:21:52.613274 19852 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:52.613756 19852 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:52.598456 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.578953 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.616461 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
W20260709 12:21:52.618278 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.618419 18165 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.622011 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.621852 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.624266 18165 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.624152 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.628620 18167 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.630568 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.631382 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:21:52.632262 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.632529 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:21:52.632936 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72173.5us
I20260709 12:21:52.633651 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.634465 18167 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.634438 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.637873 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.638593 18099 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.639664 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.642802 18097 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.646602 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.648012 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.652030 18165 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:52.653436 20001 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:52.654083 20001 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:52.661382 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.662551 18097 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:21:52.667742 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.668605 18171 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:52.669629 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.671005 18015 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:52.672135 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.673938 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.676452 18166 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.678891 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:52.646003 18810 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.682951 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.673137 19831 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.681102 18812 raft_consensus.cc:1217] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:21:52.686467 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.687428 18015 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.678738 20019 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:52.694204 18168 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.695632 18097 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.623095 19852 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.697525 19831 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.699012 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.697340 18099 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:52.705191 19831 leader_election.cc:290] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.705216 19881 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:52.706583 19881 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:52.707989 20027 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:52.708453 20027 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:52.693447 20019 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.709676 20027 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.710412 19910 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.711261 20027 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.711898 18023 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.712080 20019 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:52.714788 19910 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.716461 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.718986 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.721930 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.723865 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.727756 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.728957 19884 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:52.730415 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.730687 18164 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.738772 20032 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:52.739238 20001 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.740335 19975 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:52.740782 19975 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:52.741194 18171 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.741087 20001 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.741909 19975 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.743340 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.743708 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.744318 18023 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:52.744480 18019 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:21:52.749440 18015 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.751878 19975 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:52.754344 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:52.756163 20033 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:52.756740 19884 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:52.757563 19841 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:52.757017 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:52.758412 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.761195 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.767684 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.772766 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.769018 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.781518 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:52.782814 18166 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:21:52.774562 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69350us
I20260709 12:21:52.789788 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.792356 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.796824 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.812067 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:52.759064 20012 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:52.807555 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.884344 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.892206 18165 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.885330 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.885224 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.799029 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.877931 18166 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.805363 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.902083 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.796705 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:52.807555 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.908692 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.906599 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.907296 18169 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:52.907191 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.910090 18167 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:21:52.911679 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.911069 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.915191 18092 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.906380 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:52.918390 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.919281 18016 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:21:52.919942 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.921221 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.922554 18016 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:21:52.922952 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.923076 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:52.809669 19854 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:52.928138 18096 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.927829 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:52.924356 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150947us
I20260709 12:21:52.871500 19928 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:21:52.930794 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 118887us
I20260709 12:21:52.932040 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.931802 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.934520 19928 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:21:52.935490 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 177204us
I20260709 12:21:52.936578 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:52.937381 19551 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:52.939064 18023 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:52.939998 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:52.940564 19926 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:52.944293 20001 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:52.934571 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.808892 20020 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:52.931393 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.936365 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:52.906817 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.949380 18172 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:52.951361 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:52.954105 18165 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:52.963541 19551 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:52.965303 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:52.970750 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47970.6us
W20260709 12:21:52.974283 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68008.1us
I20260709 12:21:52.931324 19936 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:52.977530 19936 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.978190 19936 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:52.977967 19551 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:52.979990 20040 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:52.933099 20019 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:52.985152 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.986320 18023 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:52.987363 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:52.987782 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.988518 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:52.949210 20012 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:52.990226 20020 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:52.990401 20012 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:52.990718 20020 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.990886 20012 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:52.991187 20020 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:52.991379 20012 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:52.928483 19854 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:52.946465 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:52.973299 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:52.989625 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:21:52.978346 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46691.1us
I20260709 12:21:52.985462 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:52.975809 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.007586 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:53.008028 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.003660 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.011054 20027 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.014017 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:52.998728 19854 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.020607 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47408.5us
I20260709 12:21:53.017102 18164 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:53.021898 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.017292 19841 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.023779 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.024576 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.025391 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:53.056733 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120301us
I20260709 12:21:53.025920 19831 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:53.026212 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.026304 18109 tablet_copy_service.cc:342] P 768723648a9444fda53659e6a8cc3091: Request end of tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-f9731c63bc7f406db2c77f5ef0415a78 received from {username='slave'} at 127.0.0.1:40710
I20260709 12:21:53.010043 19910 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:53.027109 19926 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:53.017558 20027 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.030244 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.038807 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.039041 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:53.039573 19928 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:53.046396 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71080.3us
I20260709 12:21:53.049278 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.052491 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.062402 19831 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:53.062917 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.050086 19936 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.063194 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.063565 18109 tablet_copy_service.cc:434] P 768723648a9444fda53659e6a8cc3091: ending tablet copy session 910f8c8d787e4abbb092413c1f90ebf3-f9731c63bc7f406db2c77f5ef0415a78 on tablet f9731c63bc7f406db2c77f5ef0415a78 with peer 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.064288 18019 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:53.066419 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102777us
I20260709 12:21:53.070580 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.082072 18015 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:53.082072 18022 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:53.067700 19926 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:53.086658 19926 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.087075 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.088574 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.059239 19910 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:53.056552 19841 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:21:53.090041 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59916.2us
I20260709 12:21:53.090617 19551 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.094300 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:53.097875 18016 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:53.103441 20019 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:53.105304 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66578.2us
I20260709 12:21:53.089469 19936 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.097142 19910 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.098289 19928 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.118443 20027 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.118870 19925 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:53.119434 19925 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:53.119716 20027 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.119908 19925 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.120200 20027 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.123292 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:53.115343 18022 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:53.123633 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.097585 20043 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:53.106325 19881 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:21:53.129544 20008 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:53.130168 20008 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:53.130548 19881 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:53.131095 19881 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.130807 19831 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.132107 19881 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.123114 19995 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:53.117863 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.134372 19995 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:53.134881 19995 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.115576 18015 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:53.124620 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72246.9us
W20260709 12:21:53.125325 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67509.2us
I20260709 12:21:53.097391 20038 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:53.106956 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:53.146849 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:53.129566 20001 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:21:53.145975 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.155983 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.137501 19884 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:21:53.162957 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:53.164970 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.142123 19881 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.169597 19551 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:53.149161 20032 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.121349 20027 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.170198 19551 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:53.176429 19551 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.130558 20008 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:53.151188 20038 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:53.184654 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.187397 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.123047 19841 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.170151 20033 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:53.192482 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86205.7us
I20260709 12:21:53.193976 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:53.198504 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46243.9us
I20260709 12:21:53.210491 20053 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.183262 19852 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.218866 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:53.257076 20046 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:21:53.196420 20033 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.193228 18810 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:53.293123 19910 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.190274 20045 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.288833 19854 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.248976 20012 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.295521 19926 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.296686 20019 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:53.285274 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 94777.6us
I20260709 12:21:53.299935 19884 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:53.363256 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 274763us
I20260709 12:21:53.192265 20040 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.318017 19925 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:53.318508 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 142433us
I20260709 12:21:53.187804 20038 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.328641 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 110039us
I20260709 12:21:53.305344 19551 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:53.333701 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 210572us
I20260709 12:21:53.374015 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.360344 20027 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.378022 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:53.334846 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.379871 20040 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:53.380338 20040 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:53.380709 19841 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.361977 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143261us
I20260709 12:21:53.301514 20001 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.361315 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 173) took 1012 ms. Trace:
I20260709 12:21:53.383813 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.364849 19884 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:53.366600 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 3 to 7, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:53.367744 20012 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.369602 19926 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.330291 20033 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.358517 19936 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:53.387609 19551 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:53.389678 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.393381 18171 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:53.395267 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:53.396353 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.391484 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.365631 20020 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.398478 18165 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:53.399801 20020 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.369349 20008 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.403154 19854 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.403465 20057 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.380734 20040 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.405339 18172 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:53.406126 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 212312us
I20260709 12:21:53.407346 19925 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:53.383806 20001 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:53.416213 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:21:53.417448 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53373.6us
I20260709 12:21:53.428153 20008 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:53.429191 20054 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.432230 20054 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:53.432828 18018 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:53.394779 19936 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:53.434165 19936 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.436290 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 255791us
I20260709 12:21:53.438532 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.449848 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:53.447027 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.454304 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.454676 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.455554 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:53.457495 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.468350 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.465925 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.466213 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.466560 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.472488 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.440670 19854 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:53.475397 18094 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:53.405055 19852 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:21:53.476900 19854 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:53.484982 20057 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:53.486840 17987 leader_election.cc:304] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.409714 20019 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
W20260709 12:21:53.491536 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72440us
I20260709 12:21:53.414034 19884 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.499459 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.501611 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:53.499423 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108151us
W20260709 12:21:53.502218 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121012us
I20260709 12:21:53.505857 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.420616 19995 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.495942 19854 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:53.508303 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.509224 19910 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.477929 20020 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:53.513171 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74771.4us
I20260709 12:21:53.514135 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.463384 20056 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.455406 20059 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.487156 20052 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:53.515815 19831 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.522843 19925 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.491643 20049 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:53.506716 20054 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:53.498286 20057 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:53.527443 19551 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:53.526999 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.530761 18006 tablet_service.cc:1558] Processing DeleteTablet for tablet a29e00b535474906948e66659110db3b with delete_type TABLET_DATA_TOMBSTONED (TS cab4a2f5d3ca48a2a803ee8ec3906e1d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49522
I20260709 12:21:53.384003 17921 rpcz_store.cc:276] 0709 12:21:52.318868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.319227 (+   359us) service_pool.cc:224] Handling call
0709 12:21:53.330879 (+1011652us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:52.723434 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:52.724235 (+   801us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:52.724293 (+    58us) write_op.cc:435] Acquired schema lock
0709 12:21:52.724331 (+    38us) tablet.cc:662] Decoding operations
0709 12:21:52.724842 (+   511us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:52.724945 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:52.724983 (+    38us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:52.725359 (+   376us) tablet.cc:701] Row locks acquired
0709 12:21:52.725415 (+    56us) write_op.cc:260] PREPARE: finished
0709 12:21:52.725614 (+   199us) write_op.cc:270] Start()
0709 12:21:52.726103 (+   489us) write_op.cc:276] Timestamp: P: 1783599712725537 usec, L: 0
0709 12:21:52.726133 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 12:21:52.872834 (+146701us) log.cc:844] Serialized 595 byte log entry
0709 12:21:53.086577 (+213743us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b2e28 00007f6c1a0ed581 00007f6c1a0ecd58 00007f6c1a0f19f1 00007f6c1a0f19a9 00007f6c1a0f1939 00007f6c1a0f1901 00007f6c1a0f0b2d 0000000000388634 0000000000388479
0709 12:21:53.158432 (+ 71855us) op_driver.cc:464] REPLICATION: finished
0709 12:21:53.214087 (+ 55655us) write_op.cc:301] APPLY: starting
0709 12:21:53.214241 (+   154us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:53.214429 (+   188us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:53.214482 (+    53us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:53.214962 (+   480us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:53.215343 (+   381us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:53.215423 (+    80us) write_op.cc:312] APPLY: finished
0709 12:21:53.215876 (+   453us) log.cc:844] Serialized 23 byte log entry
0709 12:21:53.246928 (+ 31052us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:53.305530 (+ 58602us) write_op.cc:454] Released schema lock
0709 12:21:53.305631 (+   101us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:53.313732 (+  8101us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:53.313767 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":55227,"apply.run_cpu_time_us":6837,"apply.run_wall_time_us":99608,"num_ops":1,"prepare.queue_time_us":173111,"prepare.run_cpu_time_us":67284,"prepare.run_wall_time_us":402023,"raft.queue_time_us":140436,"raft.run_cpu_time_us":6375,"raft.run_wall_time_us":62359,"replication_time_us":432079,"spinlock_wait_cycles":87786240,"thread_start_us":300377,"threads_started":4,"wal-append.queue_time_us":135198}]]}
I20260709 12:21:53.531195 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.530988 20059 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:53.553758 19551 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.523603 19975 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:53.540155 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.547935 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.548374 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:53.550554 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59690.2us
I20260709 12:21:53.512987 20020 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:53.558997 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:21:53.565549 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.575877 20056 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.577657 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.578478 20045 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.578944 20045 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.553758 19928 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.579258 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.579954 19881 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:53.584980 18015 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.581810 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.560146 17920 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 2 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:53.581796 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:53.556995 20052 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.574383 20020 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.611477 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63639.3us
I20260709 12:21:53.579602 20008 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:53.618360 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87305us
I20260709 12:21:53.516389 20001 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.611083 18022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14d2c4b70e164856a205483966ca4963"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:21:53.632165 18022 raft_consensus.cc:606] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.579970 20045 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.593185 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:53.600392 19925 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:53.581336 20062 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:53.626545 17918 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 2, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:53.627352 20057 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.588990 19551 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:53.635952 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.585057 20061 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:53.651975 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 140233us
W20260709 12:21:53.659010 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79845.4us
I20260709 12:21:53.662490 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.660611 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:53.669888 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.669517 20062 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.670329 20057 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.672750 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.658962 19928 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20260709 12:21:53.706012 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.673679 19852 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:21:53.709492 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.681346 20061 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:53.716825 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.718231 20051 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.720346 20051 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.706322 19854 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.721633 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.723075 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:53.694614 19831 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.696724 20001 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.731443 19854 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.732199 18807 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:53.714502 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:21:53.738574 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106537us
I20260709 12:21:53.697897 19975 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:21:53.712795 19995 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.665170 20056 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 12:21:53.704402 19841 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:21:53.726502 20055 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:53.704038 20053 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 12:21:53.764685 19854 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.780496 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.762111 20061 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.771346 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:53.776819 17919 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:53.771938 20055 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:21:53.815943 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 99199.3us
I20260709 12:21:53.816138 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:53.819911 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:53.781471 20072 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.821324 20008 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.794768 20048 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:53.846031 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 124535us
I20260709 12:21:53.846686 20073 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:53.847266 20073 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:53.847366 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.847779 20073 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:53.816320 19881 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.816046 20043 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:53.853552 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 223569us
I20260709 12:21:53.774724 20038 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:53.828691 20072 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:53.872296 19936 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:53.869468 20068 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20260709 12:21:53.759007 20052 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:53.871649 20069 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:21:53.852360 20056 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.872447 20067 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:53.874221 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.843832 19841 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:53.873986 20072 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.875002 20055 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.876386 20032 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:53.858268 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:53.877954 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.868818 20066 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.841019 19854 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:53.885782 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 223392us
I20260709 12:21:53.868629 20061 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.855000 19975 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:53.892391 20070 tablet_replica.cc:333] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:21:53.917385 19841 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:53.934829 20070 raft_consensus.cc:2243] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:53.934597 19841 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.892860 20065 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.876619 20062 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:53.936218 19841 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.896001 20077 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:53.891117 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 230640us
I20260709 12:21:53.930750 19975 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:53.919790 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.942823 19975 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.944339 19975 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:53.944850 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.945897 19854 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.947371 19854 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.952102 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93961.9us
I20260709 12:21:53.953233 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.954021 20062 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:53.948596 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:53.915933 20067 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:53.860508 20079 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:53.927975 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:53.888548 20056 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:53.923488 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.960947 20077 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:53.962383 20065 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:53.965624 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:53.967489 20033 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.566s	user 0.013s	sys 0.000s
I20260709 12:21:53.968564 19936 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:53.974349 19881 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.982003 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.982268 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:53.980450 20067 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:53.989893 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62003.6us
W20260709 12:21:53.998469 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.998982 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:53.990713 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:53.992832 18094 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:54.000191 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:21:53.979348 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159563us
I20260709 12:21:53.936204 20070 raft_consensus.cc:2272] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:54.004603 18166 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:54.003474 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:54.005537 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:54.008695 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.976316 20072 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:54.013607 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60461.5us
I20260709 12:21:54.016418 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:54.018479 18167 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:21:54.026501 20063 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:53.986352 20065 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:54.002483 20066 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:54.004117 20062 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:53.929188 20032 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.043113 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:54.059794 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43497.8us
I20260709 12:21:54.023530 20055 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:54.003901 20033 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.035969 20065 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:54.076763 20076 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:54.077076 20085 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:53.998425 20073 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:54.007831 20043 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260709 12:21:54.077574 20040 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.004s
I20260709 12:21:53.997454 20079 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:53.987581 20008 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:54.122258 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98021.7us
W20260709 12:21:54.124491 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45391.7us
I20260709 12:21:54.068153 20066 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260709 12:21:54.127768 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98046.1us
W20260709 12:21:54.129657 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 86745.8us
I20260709 12:21:54.078455 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:54.055583 20020 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:54.130697 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 125582us
I20260709 12:21:54.128525 20012 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:54.134975 20088 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.134557 20040 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:54.160485 18165 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:54.135910 20078 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:54.136449 20080 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.140069 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:54.077447 20074 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 12:21:54.136861 20064 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:54.154399 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123841us
I20260709 12:21:54.154307 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:54.158442 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:54.172570 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.120374 20053 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:21:54.182106 20078 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:21:54.188376 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48413.5us
I20260709 12:21:54.191828 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:54.194487 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.195457 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.161134 20038 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:54.161507 20049 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:54.197314 20081 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:54.198192 18022 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:54.198431 20078 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.197034 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:54.207461 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:54.211097 19551 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:54.211788 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:54.213965 19925 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:54.217343 20042 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:21:54.217756 20032 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:54.217343 20073 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:54.166638 20012 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.194168 20088 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:54.205178 20049 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:54.220249 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63408.9us
I20260709 12:21:54.229954 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:54.250870 20082 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:21:54.252589 20095 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.252837 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:54.253235 18806 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:21:54.245146 20057 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.574s	user 0.016s	sys 0.000s
I20260709 12:21:54.244002 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:54.255859 18018 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:54.256122 19551 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.250494 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:54.257095 18172 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:54.258257 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:54.253394 20095 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:54.129202 20077 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:54.264958 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:54.267086 18092 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:54.218953 20033 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:54.266959 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:54.271723 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:54.227883 20038 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.275933 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:54.295979 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.296351 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:54.301786 18095 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:54.217799 20098 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.296955 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:54.226071 20088 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:54.303471 18169 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.303561 20088 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:54.218977 20020 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:54.298589 19925 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:21:54.327168 20073 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:54.273566 19854 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.330626 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:54.337443 18168 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:54.292371 20052 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:54.345057 20105 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:54.350375 18169 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
W20260709 12:21:54.350240 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74469.6us
W20260709 12:21:54.357540 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92296.9us
I20260709 12:21:54.219929 20064 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260709 12:21:54.365129 20070 ts_tablet_manager.cc:1916] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:54.342219 20038 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:21:54.367614 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67190.8us
I20260709 12:21:54.361156 18806 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20260709 12:21:54.377100 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:54.378278 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:54.378448 18015 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:54.366912 20033 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:54.373301 18806 pending_rounds.cc:85] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 3
W20260709 12:21:54.389624 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 94953.6us
I20260709 12:21:54.391618 18022 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:54.371057 20038 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:54.372828 20052 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:54.656307 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.392911 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:54.751648 18016 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.754226 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:54.474954 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.747913 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 362210us
W20260709 12:21:54.759569 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.389045 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.760121 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:54.794090 18171 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:54.759568 20062 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.692s	user 0.011s	sys 0.033s
I20260709 12:21:54.797219 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:54.474265 18018 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:54.526479 19975 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.003s
I20260709 12:21:54.804801 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:54.732831 19841 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.772s	user 0.008s	sys 0.005s
I20260709 12:21:54.795853 20116 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:54.471378 20093 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:54.687271 20067 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.656s	user 0.013s	sys 0.001s
I20260709 12:21:54.946306 20067 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.947831 18173 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:54.949321 20067 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:54.735095 20055 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.663s	user 0.014s	sys 0.000s
I20260709 12:21:54.390437 20012 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:54.609334 19884 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:54.392445 20040 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:54.392925 18093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
W20260709 12:21:54.817600 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57590.1us
I20260709 12:21:54.981709 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:54.425199 20064 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:54.674518 20060 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:54.987885 20060 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:54.750144 18806 pending_rounds.cc:107] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20260709 12:21:54.992995 18806 raft_consensus.cc:2917] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260709 12:21:54.993193 20097 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.993839 18806 raft_consensus.cc:2889] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:54.994000 20097 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.996814 20097 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:54.997972 18093 raft_consensus.cc:606] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:54.998593 18093 raft_consensus.cc:614] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:54.942437 20089 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:54.943859 19928 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:55.017089 19841 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:55.017768 19841 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.018286 19841 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:54.989295 20093 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:54.742841 20095 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:54.403934 20033 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:54.975620 18091 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:54.819279 20062 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:54.983824 20102 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:54.984690 20064 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:54.991420 20060 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:54.790436 20099 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:54.736099 20107 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 12:21:55.030876 18016 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:21:55.045598 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1930) took 1040 ms. Trace:
W20260709 12:21:55.053540 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71998.7us
W20260709 12:21:55.067478 20088 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.763s	user 0.011s	sys 0.003s
W20260709 12:21:55.069414 20065 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.947s	user 0.016s	sys 0.000s
I20260709 12:21:55.021296 20093 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.072618 18168 rpcz_store.cc:276] 0709 12:21:54.004656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.007496 (+  2840us) service_pool.cc:224] Handling call
0709 12:21:54.026959 (+ 19463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.340805 (+313846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.045142 (+704337us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b540029dbd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:55.045475 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713414400}
I20260709 12:21:55.091293 20064 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.092763 20115 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.093540 20115 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.073217 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:55.096971 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.017132 19928 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:21:55.079512 20072 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.056s	user 0.012s	sys 0.004s
I20260709 12:21:55.079778 20052 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.080113 20088 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.080849 20061 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.112118 20061 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.112792 20062 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.113478 20062 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.114771 20061 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.081338 20078 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.081928 20113 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:21:55.084350 20105 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.721s	user 0.009s	sys 0.002s
I20260709 12:21:55.084844 20055 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.117921 20113 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.121007 20088 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.088814 20033 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.121361 20113 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:54.986101 20038 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.074882 20099 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.089476 20065 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.118335 20055 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.085641 18164 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:55.114781 19928 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.123458 20072 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:55.117952 20078 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:55.077471 20060 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:55.089318 20049 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.804s	user 0.009s	sys 0.004s
W20260709 12:21:55.126624 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53518.4us
W20260709 12:21:55.115952 18807 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.081s	sys 0.077s
I20260709 12:21:55.126893 20105 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:55.138693 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 341558us
W20260709 12:21:55.137334 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.000s	sys 0.012s
W20260709 12:21:55.146000 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.000s	sys 0.013s
I20260709 12:21:55.147209 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:55.147243 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 775537us
I20260709 12:21:55.142988 19928 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:55.151218 20038 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:55.152891 20108 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:55.155380 20064 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:55.155915 18815 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:55.137943 20033 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:55.144794 18807 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.081s	sys 0.078s
I20260709 12:21:55.195039 20027 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.195876 20027 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.196966 20113 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.198331 20093 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.153627 20108 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.146158 20115 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:55.204437 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.212081 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:55.213310 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.212441 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:55.160392 20089 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:55.228645 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.219091 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.168893 20062 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.199774 20072 raft_consensus.cc:1064] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:55.145718 20049 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.242954 20122 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.243938 20119 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:55.255589 20065 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.168000 20033 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.253561 20055 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.260979 20117 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:21:55.199185 20027 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.262470 20055 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:55.262948 20055 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:55.234530 19975 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.252419 20083 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.232929 20113 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.255435 20078 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.199108 20093 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.241252 20097 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.249637 20115 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.263602 20033 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.280392 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:55.265195 19884 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.281183 19975 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.288442 18811 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:55.301525 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1925) took 1296 ms. Trace:
I20260709 12:21:55.302084 18171 rpcz_store.cc:276] 0709 12:21:54.000977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.001104 (+   127us) service_pool.cc:224] Handling call
0709 12:21:54.026659 (+ 25555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.810427 (+783768us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.286646 (+476219us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:55.297449 (+ 10803us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1764349696}
I20260709 12:21:55.304066 20045 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:55.304251 20027 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.304687 20099 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.286410 18808 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:55.297788 20033 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:55.302901 20123 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.305627 20078 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:55.322809 20099 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.302842 19884 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.320578 20052 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.300431 20115 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.284391 20097 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.271729 20122 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.282514 20055 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.282483 20083 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.311977 20027 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.391396 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:55.404850 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.168s (SENT) [suppressed 33 similar messages]
I20260709 12:21:55.381219 18168 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:55.391993 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:55.367827 20048 raft_consensus.cc:993] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:55.427891 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:55.349687 20121 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.458393 20113 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:55.479135 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 306633us
I20260709 12:21:55.335498 18810 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:55.366910 20093 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:55.345969 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.475107 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:55.489490 20121 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.491837 20083 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.480166 19926 raft_consensus.cc:1081] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:21:55.520113 19831 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:55.520320 20102 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:21:55.519743 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.521555 20088 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.370664 20096 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:55.392362 20108 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.367623 19975 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.349227 20078 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.508301 19854 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.499858 20012 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.530619 20083 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.464836 20099 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.518280 20095 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.530248 18809 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:55.373970 20056 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.555704 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:55.518805 20049 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:55.570122 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.508s (SENT)
W20260709 12:21:55.534333 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.356271 20123 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.536118 20125 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.530074 18806 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:55.502444 20052 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.546356 20088 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.535667 19831 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.551501 20096 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:55.578750 18094 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:55.581554 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.558768 19854 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.541160 20102 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.567433 20122 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.567561 20083 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.563536 20012 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:55.685487 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 473268us
I20260709 12:21:55.678056 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.725224 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.843950 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:55.793633 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.686285 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 281483us
I20260709 12:21:55.583381 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:55.727459 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.845898 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.559158 20075 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:55.844285 18097 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:55.853602 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:55.848546 18099 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:55.854671 20140 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:21:55.744565 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.856206 18015 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:55.856592 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.856248 18022 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:55.709417 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.849689 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:55.858208 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.858541 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:55.846344 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 263062us
I20260709 12:21:55.650281 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.850607 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.855437 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.860713 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.861136 18099 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:21:55.861719 18092 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:55.778676 20080 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.655699 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 439) took 1287 ms. Trace:
I20260709 12:21:55.675729 20130 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.864693 20080 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.865701 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.867350 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:55.867944 20080 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.868637 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.868923 18015 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
W20260709 12:21:55.712395 20038 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.561s	user 0.013s	sys 0.000s
I20260709 12:21:55.656724 20056 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:55.707409 20084 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:55.869426 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.857309 18097 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:55.877312 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:55.877164 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.871485 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:21:55.839974 20045 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.005s
W20260709 12:21:55.715704 19841 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.697s	user 0.010s	sys 0.002s
I20260709 12:21:55.865000 20130 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.888617 18092 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:55.719673 19884 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.721062 20097 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.717854 20088 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.720053 20063 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
I20260709 12:21:55.879657 18168 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:55.875097 20060 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:21:55.875712 20084 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:55.882463 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.812947 18809 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:55.755990 19926 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.922070 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.853591 20075 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:55.874038 20132 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:55.714133 20078 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.719058 20121 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.570884 20095 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.944322 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:55.916301 20088 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:55.946727 20088 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.949208 20088 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.920531 20045 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:21:55.723505 20123 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.991171 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 274) took 2482 ms. Trace:
I20260709 12:21:55.901644 20140 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260709 12:21:55.995325 18811 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:55.670334 20027 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.713321 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45562 (request call id 21) took 1349 ms. Trace:
I20260709 12:21:56.013695 18093 rpcz_store.cc:276] 0709 12:21:54.373942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.388516 (+ 14574us) service_pool.cc:224] Handling call
0709 12:21:55.713059 (+1324543us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:55.713237 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12102400}
I20260709 12:21:55.864629 18807 rpcz_store.cc:276] 0709 12:21:54.366509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.366669 (+   160us) service_pool.cc:224] Handling call
0709 12:21:54.397637 (+ 30968us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.441567 (+ 43930us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:55.181474 (+739907us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.578078 (+396604us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.607002 (+ 28924us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:55.646425 (+ 39423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.651525 (+  5100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.653883 (+  2358us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1181,"prepare.queue_time_us":653065,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":6590,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":31809,"thread_start_us":1016686,"threads_started":2,"wal-append.queue_time_us":573899}
I20260709 12:21:56.016048 18807 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:55.646165 20049 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:56.016852 20049 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.897799 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:56.018499 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120849us
I20260709 12:21:56.018713 20132 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:56.019289 20132 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:55.676047 20115 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.903278 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.887804 19841 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.907569 20083 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.887501 20134 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:55.909106 19884 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:55.660760 20126 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.712585 20070 ts_tablet_manager.cc:1929] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20260709 12:21:56.031620 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.033552 18090 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:55.875252 19831 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.917428 18167 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:55.872937 20122 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:55.921527 20108 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.928980 20073 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.581789 20125 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.932993 19926 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.940496 20130 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:55.950227 18168 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:55.950233 20063 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:55.962399 20075 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:55.963549 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 438) took 1691 ms. Trace:
I20260709 12:21:55.932904 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:55.972499 20079 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:55.974262 20095 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:55.978096 20068 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:55.982870 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:55.984124 20096 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:55.965315 20102 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.015825 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:55.821046 20117 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:56.019479 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.020362 18095 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:55.991752 17920 rpcz_store.cc:276] 0709 12:21:53.508850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.509164 (+   314us) service_pool.cc:224] Handling call
0709 12:21:55.991032 (+2481868us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:53.666597 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:53.773936 (+107339us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:53.777795 (+  3859us) write_op.cc:435] Acquired schema lock
0709 12:21:53.780416 (+  2621us) tablet.cc:662] Decoding operations
0709 12:21:53.841524 (+ 61108us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:53.850184 (+  8660us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:53.852975 (+  2791us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:53.882440 (+ 29465us) tablet.cc:701] Row locks acquired
0709 12:21:53.884353 (+  1913us) write_op.cc:260] PREPARE: finished
0709 12:21:53.893414 (+  9061us) write_op.cc:270] Start()
0709 12:21:53.893594 (+   180us) write_op.cc:276] Timestamp: P: 1783599713893351 usec, L: 0
0709 12:21:53.893621 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:21:53.950111 (+ 56490us) log.cc:844] Serialized 533 byte log entry
0709 12:21:54.181629 (+231518us) op_driver.cc:464] REPLICATION: finished
0709 12:21:54.302601 (+120972us) write_op.cc:301] APPLY: starting
0709 12:21:54.302756 (+   155us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:54.302917 (+   161us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:54.303098 (+   181us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:54.343849 (+ 40751us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:54.344308 (+   459us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:54.344400 (+    92us) write_op.cc:312] APPLY: finished
0709 12:21:54.344756 (+   356us) log.cc:844] Serialized 23 byte log entry
0709 12:21:54.345034 (+   278us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:54.345133 (+    99us) write_op.cc:454] Released schema lock
0709 12:21:54.345183 (+    50us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:54.352021 (+  6838us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:54.353405 (+  1384us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":114825,"apply.run_cpu_time_us":2923,"apply.run_wall_time_us":43085,"num_ops":1,"prepare.queue_time_us":20516,"prepare.run_cpu_time_us":5437,"prepare.run_wall_time_us":287958,"replication_time_us":264755,"thread_start_us":101693,"threads_started":1}]]}
I20260709 12:21:56.024796 18170 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:21:56.026129 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 138146us
I20260709 12:21:56.029289 18023 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:56.030215 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:55.977599 18806 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:56.019438 20049 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.034330 20080 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.399931 18018 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:56.398782 20056 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:56.400604 19926 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.401321 20061 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.136880 18167 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:56.402078 20097 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:56.403126 20076 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.403292 18168 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:56.403584 20076 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.401999 20061 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.411233 18172 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:56.411716 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2235) took 2429 ms. Trace:
I20260709 12:21:56.412309 18091 rpcz_store.cc:276] 0709 12:21:53.982399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.983308 (+   909us) service_pool.cc:224] Handling call
0709 12:21:53.983642 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.046434 (+1062792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.411184 (+1364750us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54004119d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:56.411511 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270052096}
W20260709 12:21:56.413380 18810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 501) took 2700 ms (client timeout 2168 ms). Trace:
I20260709 12:21:56.414006 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.315495 20127 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.299312 20054 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.034034 20088 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:56.276459 19831 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.409005 20144 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:56.043112 20115 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.019811 20132 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:56.412858 18813 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:56.414518 18814 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:56.190119 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:56.398902 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 379513us
I20260709 12:21:56.415153 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 277) took 1034 ms. Trace:
I20260709 12:21:56.440706 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:56.441440 18091 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:21:56.415867 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 506) took 1150 ms (client timeout 1504 ms). Trace:
I20260709 12:21:56.417969 20125 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:56.418329 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1924) took 2417 ms. Trace:
W20260709 12:21:56.251863 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 365764us
W20260709 12:21:56.365239 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.029747 20084 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.399608 20080 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:56.217033 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.337217 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 457618us
W20260709 12:21:56.490921 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.491475 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.492003 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.492417 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51839.3us
I20260709 12:21:56.026417 20020 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.484464 18165 rpcz_store.cc:276] 0709 12:21:54.000622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.000769 (+   147us) service_pool.cc:224] Handling call
0709 12:21:54.023292 (+ 22523us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.161193 (+137901us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.418006 (+2256813us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400382f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:56.418211 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":314739968}
I20260709 12:21:56.036307 20070 log.cc:1199] T a29e00b535474906948e66659110db3b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/wals/a29e00b535474906948e66659110db3b
I20260709 12:21:56.446483 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.440448 18169 rpcz_store.cc:276] 0709 12:21:55.381087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.381562 (+   475us) service_pool.cc:224] Handling call
0709 12:21:55.381880 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.572374 (+190494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.572406 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.572453 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:55.572493 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.414910 (+842417us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:56.414960 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.415088 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437431168}
I20260709 12:21:56.486138 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.429760 20141 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:56.316552 20051 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.489462 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:56.184448 20099 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.444769 18095 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:56.406946 20068 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:56.413370 18172 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14d2c4b70e164856a205483966ca4963"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:21:56.498412 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:56.424794 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 1922) took 2424 ms (client timeout 2214 ms). Trace:
I20260709 12:21:56.483345 18093 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:56.501214 19551 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:56.531046 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.531500 19551 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.531939 19551 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.502122 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:56.428022 20052 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.491261 19852 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:56.538247 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:56.506114 20141 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:21:56.562629 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64384.5us
I20260709 12:21:56.126498 19841 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.176783 20143 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.518648 18022 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:56.526257 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:56.493407 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.401458 20117 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.529991 18172 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:56.573630 18172 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.438730 20063 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:56.399011 18808 rpcz_store.cc:276] 0709 12:21:54.272039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.272242 (+   203us) service_pool.cc:224] Handling call
0709 12:21:54.272578 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.324837 (+1052259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.963055 (+638218us) spinlock_profiling.cc:243] Waited 1e+03 ms on lock 0x7b54005e01d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:55.963432 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2299496192}
I20260709 12:21:56.504257 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.556497 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:56.557650 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.578855 18168 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:56.400643 20073 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:56.558586 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56567us
I20260709 12:21:56.578569 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.584537 18165 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:21:56.584931 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:56.184938 19854 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:56.564491 17988 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.037s	sys 0.031s
I20260709 12:21:56.446187 19831 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.566833 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.563560 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.513669 20075 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:56.571383 18812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 590) took 2500 ms (client timeout 2037 ms). Trace:
I20260709 12:21:56.131568 20126 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.571938 19551 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.486009 20115 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:56.414070 18810 rpcz_store.cc:269] 0709 12:21:53.712682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.716869 (+  4187us) service_pool.cc:224] Handling call
0709 12:21:56.412185 (+2695316us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54005f1c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:56.413268 (+  1083us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2952201472}
I20260709 12:21:56.455056 20144 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.467610 20108 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.579869 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:56.577996 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73867.9us
I20260709 12:21:56.612994 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.594151 20095 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:56.481344 20122 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.447122 20127 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.531939 18809 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:56.120194 20134 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.577924 20143 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.528674 20051 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.566663 20077 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:56.566931 20079 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:56.567735 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:21:56.483839 18806 rpcz_store.cc:269] 0709 12:21:55.265708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.266272 (+   564us) service_pool.cc:224] Handling call
0709 12:21:56.414158 (+1147886us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400557154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:56.415776 (+  1618us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":542720896}
W20260709 12:21:56.592231 17988 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.037s	sys 0.031s
I20260709 12:21:56.573060 20056 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.594327 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.595129 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.507014 20020 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:56.653396 20043 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:56.660290 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.660609 20043 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:56.660871 20020 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.446847 20054 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.661278 18023 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:56.661849 18807 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3. Re-sending same reply.
I20260709 12:21:56.663095 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.604992 20141 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.536469 20060 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:21:56.599335 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.587558 19995 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:21:56.531184 18164 rpcz_store.cc:269] 0709 12:21:54.000023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.010880 (+ 10857us) service_pool.cc:224] Handling call
0709 12:21:56.424495 (+2413615us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:56.424655 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1917406976}
I20260709 12:21:56.592200 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.594959 19852 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:56.614753 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.580981 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:56.599716 20130 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:56.612452 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.618726 19925 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:56.623574 20096 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.624715 20098 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:56.704766 20098 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:21:56.627331 20061 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.631227 20117 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.558303 20076 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:21:56.636475 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.717200 20134 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.717442 18165 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:56.561980 20099 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.718212 20099 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.720743 20099 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:56.643805 19551 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.651132 18172 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:56.653059 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 141119us
W20260709 12:21:56.596946 18812 rpcz_store.cc:269] 0709 12:21:54.071686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.081898 (+ 10212us) service_pool.cc:224] Handling call
0709 12:21:56.538003 (+2456105us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:56.571281 (+ 33278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87475456}
W20260709 12:21:56.655306 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60280.8us
I20260709 12:21:56.658906 20051 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:56.721899 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 1921) took 2722 ms (client timeout 1555 ms). Trace:
I20260709 12:21:56.722409 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.723430 18168 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:56.723562 20115 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.723979 18020 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:56.724126 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.724594 20054 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.724818 19551 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:56.725252 19551 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.725455 20073 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.725718 19551 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.725651 20061 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:56.726298 18166 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:56.727054 20085 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:56.727501 20085 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.657326 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.728144 20085 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.728410 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.663903 18017 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:56.598448 18021 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:56.730283 18015 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:56.659291 18016 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:56.660007 18169 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.655164 18811 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
W20260709 12:21:56.678283 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79009.2us
W20260709 12:21:56.654825 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 590728us
W20260709 12:21:56.685969 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105303us
I20260709 12:21:56.684432 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.732313 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:56.684492 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69826.1us
I20260709 12:21:56.733420 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.734781 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.735070 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:56.735623 19551 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.700350 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.694375 20020 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:56.673985 19831 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.681252 20079 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.686107 20130 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.703473 20108 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.706002 20098 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.754308 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.765806 18015 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:56.766309 18811 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:21:56.711987 20055 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:56.764145 20130 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:56.713939 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.655906 20056 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:56.770524 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.447021 20088 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.727609 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1931) took 2717 ms. Trace:
I20260709 12:21:56.771799 18166 rpcz_store.cc:276] 0709 12:21:54.010891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.011449 (+   558us) service_pool.cc:224] Handling call
0709 12:21:56.725936 (+2714487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.726700 (+   764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.727373 (+   673us) spinlock_profiling.cc:243] Waited 2.71 s on lock 0x7b54000325d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:56.727531 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6241772416}
I20260709 12:21:56.732522 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:56.655499 20093 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:56.736191 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.781591 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.699204 18808 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:56.641884 20095 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.808209 20095 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.725291 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.749931 20054 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:56.738889 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.740438 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.740809 18017 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:56.813061 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72725.2us
I20260709 12:21:56.744287 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:56.810681 18020 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
W20260709 12:21:56.746073 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.722009 20144 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.694968 18018 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:56.820715 20052 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.707851 20141 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:56.821456 20052 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.822505 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.823472 18093 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:56.823802 20052 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:56.824337 20125 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:56.764956 19551 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:56.762270 18172 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.749652 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:21:56.826457 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76930.5us
I20260709 12:21:56.761884 20115 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:56.774593 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:56.775950 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 123876us
I20260709 12:21:56.778939 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:56.780570 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:56.782526 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46442.5us
I20260709 12:21:56.831461 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.832160 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:56.739749 20078 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260709 12:21:56.801637 20108 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:56.753961 19841 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:56.810122 20098 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.763154 20126 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:21:56.753708 20134 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:21:56.811419 18023 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:56.727430 20061 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:21:56.753937 20073 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:56.762964 20097 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.865111 20097 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.867967 20097 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.869248 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1929) took 2865 ms. Trace:
W20260709 12:21:56.814630 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 70469.1us
I20260709 12:21:56.869745 18173 rpcz_store.cc:276] 0709 12:21:54.004205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.004330 (+   125us) service_pool.cc:224] Handling call
0709 12:21:54.023085 (+ 18755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.948432 (+925347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.868934 (+1920502us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:56.869169 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2125890944}
I20260709 12:21:56.871217 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:56.830196 20115 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.820614 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.694787 19884 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:56.825903 18812 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:21:56.827915 18169 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:56.753878 20117 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:56.820614 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.897150 18092 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:56.833498 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.834511 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.898674 18166 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:56.835538 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:56.782052 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:56.900947 18806 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.902001 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.903056 18092 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:21:56.904656 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.905602 20097 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:56.906075 20020 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:56.906235 20097 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:56.906747 20020 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:21:56.722514 18170 rpcz_store.cc:269] 0709 12:21:53.999636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.999837 (+   201us) service_pool.cc:224] Handling call
0709 12:21:56.721640 (+2721803us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:56.721781 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4396417280}
I20260709 12:21:56.907739 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.907903 20020 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.646008 20063 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.836670 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.802259 20093 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:56.839622 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:56.912981 18811 raft_consensus.cc:2376] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2. Re-sending same reply.
I20260709 12:21:56.782379 18097 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:56.846313 18091 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:56.914820 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.847807 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.763586 20059 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260709 12:21:56.915477 18091 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:56.831084 20095 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:56.829422 20125 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:56.839747 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.857167 20126 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.856848 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:21:56.819366 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.873723 18017 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:56.896728 20117 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.835966 20144 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:56.836783 20060 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.956103 20144 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:56.956580 20144 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:56.956357 20060 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.837878 20054 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:56.755877 20143 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:56.898900 18168 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:56.899012 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:56.899891 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64456.3us
I20260709 12:21:56.961180 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:56.906814 20097 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:56.910414 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73882.8us
I20260709 12:21:56.838109 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:56.911552 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.966141 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.966807 18095 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:56.916539 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:56.917906 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.852819 18020 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:56.967412 18099 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.892954 20132 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.983598 19841 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:56.984234 19841 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:56.984757 19841 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.967602 19926 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:56.985692 20132 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.986129 19926 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:56.879058 20115 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.912552 19854 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:56.861379 20130 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:56.919436 20063 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.924203 18164 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:56.925748 18018 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:56.926903 20125 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.874908 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.935498 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.936560 20070 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49522 (request call id 84) took 3435 ms. Trace:
I20260709 12:21:56.938784 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:21:57.036190 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97495.4us
I20260709 12:21:57.038647 18020 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:57.039373 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.039882 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.040906 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.940623 20078 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:57.042389 20078 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.043393 18023 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:57.044593 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.045327 18023 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:21:56.994381 19854 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.046232 19854 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.005328 19926 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:57.050321 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.052184 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:21:56.944432 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.765s (SENT)
I20260709 12:21:56.946704 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 278) took 3419 ms. Trace:
W20260709 12:21:57.055119 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 110714us
I20260709 12:21:56.947036 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.950582 18810 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.951103 20049 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.070417 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:56.951651 20067 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.951576 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:56.952653 19884 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:56.953805 19831 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:56.954177 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2245) took 1383 ms. Trace:
I20260709 12:21:56.955006 20116 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:21:56.955368 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 440) took 2151 ms (client timeout 1722 ms). Trace:
I20260709 12:21:56.917270 20093 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:56.956130 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:56.957384 20117 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:56.909487 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:56.968321 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69412us
I20260709 12:21:56.968888 18136 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:56.960435 19936 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:21:56.971530 19852 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:56.973094 18806 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:56.976305 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.977113 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:56.977617 18099 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:56.933492 20126 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:56.982493 20088 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:56.983162 20052 raft_consensus.cc:1064] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:21:56.996017 18091 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:57.057206 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2337) took 1020 ms. Trace:
I20260709 12:21:56.991829 20056 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:57.040987 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.057020 18096 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:56.997259 20109 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:57.023052 20033 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.075232 18090 rpcz_store.cc:276] 0709 12:21:55.570525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.571407 (+   882us) service_pool.cc:224] Handling call
0709 12:21:55.571777 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.033961 (+462184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.953920 (+919959us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400052654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:56.954109 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1061117824}
I20260709 12:21:57.054486 17918 rpcz_store.cc:276] 0709 12:21:53.527085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.545636 (+ 18551us) service_pool.cc:224] Handling call
0709 12:21:56.946615 (+3400979us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:53.950675 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:53.951291 (+   616us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:53.951326 (+    35us) write_op.cc:435] Acquired schema lock
0709 12:21:53.951342 (+    16us) tablet.cc:662] Decoding operations
0709 12:21:53.951728 (+   386us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:53.951806 (+    78us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:53.951834 (+    28us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:53.952048 (+   214us) tablet.cc:701] Row locks acquired
0709 12:21:53.952061 (+    13us) write_op.cc:260] PREPARE: finished
0709 12:21:53.952235 (+   174us) write_op.cc:270] Start()
0709 12:21:53.952358 (+   123us) write_op.cc:276] Timestamp: P: 1783599713952172 usec, L: 0
0709 12:21:53.952375 (+    17us) op_driver.cc:348] REPLICATION: starting
0709 12:21:53.953094 (+   719us) log.cc:844] Serialized 538 byte log entry
0709 12:21:54.304840 (+351746us) op_driver.cc:464] REPLICATION: finished
0709 12:21:54.346727 (+ 41887us) write_op.cc:301] APPLY: starting
0709 12:21:54.346870 (+   143us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:54.347038 (+   168us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:54.347289 (+   251us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:54.347711 (+   422us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:54.347969 (+   258us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:54.348051 (+    82us) write_op.cc:312] APPLY: finished
0709 12:21:54.348426 (+   375us) log.cc:844] Serialized 23 byte log entry
0709 12:21:54.359763 (+ 11337us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:54.370459 (+ 10696us) write_op.cc:454] Released schema lock
0709 12:21:54.372122 (+  1663us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:54.391591 (+ 19469us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:54.392857 (+  1266us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1614057,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":41515,"apply.run_cpu_time_us":2820,"apply.run_wall_time_us":41463,"num_ops":1,"prepare.queue_time_us":246404,"prepare.run_cpu_time_us":2716,"prepare.run_wall_time_us":2715,"replication_time_us":352298,"thread_start_us":49239,"threads_started":1}]]}
I20260709 12:21:57.086506 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1499) took 1222 ms. Trace:
W20260709 12:21:57.078853 18815 rpcz_store.cc:269] 0709 12:21:54.804294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.804464 (+   170us) service_pool.cc:224] Handling call
0709 12:21:56.955164 (+2150700us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b5400494154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:56.955282 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":774442752}
I20260709 12:21:57.062458 20113 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:21:56.996922 20061 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 12:21:57.045691 20078 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.043733 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:57.070516 20077 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.071218 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.061362 20073 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.158587 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:57.160573 18022 rpcz_store.cc:276] 0709 12:21:56.036553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.586830 (+550277us) service_pool.cc:224] Handling call
0709 12:21:56.935226 (+348396us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592e9c 00007f6c13592a5b 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:57.057145 (+121919us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9134720}
W20260709 12:21:57.195670 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.161557 20055 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:21:57.195894 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2248) took 1615 ms (client timeout 1900 ms). Trace:
I20260709 12:21:57.085565 19884 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:57.143783 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 174933us
I20260709 12:21:57.123384 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:57.150420 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.035663 20070 rpcz_store.cc:276] 0709 12:21:53.511095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.530327 (+ 19232us) service_pool.cc:224] Handling call
0709 12:21:53.812491 (+282164us) ts_tablet_manager.cc:1151] Deleting tablet a29e00b535474906948e66659110db3b
0709 12:21:53.830086 (+ 17595us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:21:55.233881 (+1403795us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:55.712376 (+478495us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:56.936446 (+1224070us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":207308,"thread_start_us":73742,"threads_started":1}
I20260709 12:21:57.197291 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.200331 20095 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:57.221650 18020 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.161288 20051 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:21:57.032588 20141 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:57.200629 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.229128 18016 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:57.200879 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.200094 18171 rpcz_store.cc:276] 0709 12:21:55.864205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.876404 (+ 12199us) service_pool.cc:224] Handling call
0709 12:21:56.955954 (+1079550us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:57.086375 (+130421us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7990528}
W20260709 12:21:57.201330 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64300.2us
I20260709 12:21:57.136672 19852 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:57.230383 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:57.085879 20067 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.080372 20049 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.195201 20109 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:57.028203 20085 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.218204 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 512) took 1813 ms. Trace:
I20260709 12:21:57.066018 20020 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:57.225206 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:57.214570 18096 rpcz_store.cc:269] 0709 12:21:55.580947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.581361 (+   414us) service_pool.cc:224] Handling call
0709 12:21:56.946902 (+1365541us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:57.195818 (+248916us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6091392}
I20260709 12:21:57.229547 18016 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:57.211488 20077 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.212949 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.214437 20055 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:57.245301 20096 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:57.250495 18022 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:57.212731 20054 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:21:57.248375 20109 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.254390 19936 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 12:21:57.254716 20020 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.205904 20033 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.251066 18813 rpcz_store.cc:276] 0709 12:21:55.405081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.405242 (+   161us) service_pool.cc:224] Handling call
0709 12:21:56.412328 (+1007086us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.438670 (+ 26342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.215929 (+777259us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b54005f1c50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:57.218134 (+  2205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2260311296}
I20260709 12:21:57.216352 20073 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.270489 18016 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.285636 20020 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.240708 20142 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:57.302338 19852 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.302865 20142 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.298863 20155 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:57.303938 20033 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.304121 19852 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:57.304263 20155 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.305419 20142 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.306788 20127 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:57.307667 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.307921 20012 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.262063 20055 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:57.299300 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:57.310709 20157 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:21:57.283944 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67576.2us
I20260709 12:21:57.284441 20096 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:57.335057 20096 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.293140 18016 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:57.278111 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:57.294011 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.235406 20141 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.350665 20033 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:57.351300 20033 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:57.231832 20095 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.295212 20130 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:21:57.354470 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76492.5us
I20260709 12:21:57.300405 18094 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20260709 12:21:57.357259 18094 pending_rounds.cc:85] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:21:57.357690 18094 pending_rounds.cc:107] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20260709 12:21:57.358160 18094 raft_consensus.cc:2917] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260709 12:21:57.358758 18094 raft_consensus.cc:2889] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:57.287169 18020 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:57.302084 20038 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:57.359516 20132 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:57.360170 20038 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:57.360709 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.361265 20038 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.296922 20076 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:57.280025 18806 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:57.283565 20060 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:57.297328 19841 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.460688 18016 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:57.311340 20157 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.468788 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:21:57.514312 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 155403us
I20260709 12:21:57.515345 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2317) took 1486 ms. Trace:
I20260709 12:21:57.515879 18019 rpcz_store.cc:276] 0709 12:21:56.028986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.034212 (+  5226us) service_pool.cc:224] Handling call
0709 12:21:57.514453 (+1480241us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:21:57.514719 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.514959 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.514995 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.515044 (+    49us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:57.515084 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.515137 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.515264 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6583040}
I20260709 12:21:57.520255 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2238) took 1992 ms. Trace:
I20260709 12:21:57.436417 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.551695 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.553994 18016 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:57.520962 18098 rpcz_store.cc:276] 0709 12:21:55.527644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.527826 (+   182us) service_pool.cc:224] Handling call
0709 12:21:55.540191 (+ 12365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.970799 (+1430608us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:56.970835 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.970881 (+    46us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:56.970913 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.519798 (+548885us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54004123d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:57.519882 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.520129 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3289560832}
W20260709 12:21:57.580350 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.591272 18020 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:57.513511 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.593618 18020 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:57.610760 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.618445 18016 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:57.513356 20060 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.528404 20096 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:57.624724 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.606602 20141 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:57.616300 19926 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.011s	sys 0.014s
I20260709 12:21:57.643370 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:57.643808 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.644313 18020 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:57.644992 18019 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:57.646497 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.647255 18019 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:57.603200 18018 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:57.657460 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.658191 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.669031 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 184557us
I20260709 12:21:57.622556 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:21:57.653179 19926 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.011s	sys 0.014s
I20260709 12:21:57.570436 20132 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260709 12:21:57.640190 20038 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:57.692505 20012 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.697692 20012 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.693681 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.699222 18016 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:57.699716 20122 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:57.700436 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.642696 20096 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:57.702064 20096 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:57.703743 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.657341 20068 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.705994 20068 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.707060 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.707389 20033 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.653872 20060 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.707049 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.718493 18016 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:57.725980 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.727804 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:57.659502 20095 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.735605 20095 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.736228 20095 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.740299 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2330) took 1705 ms. Trace:
I20260709 12:21:57.741027 18015 rpcz_store.cc:276] 0709 12:21:56.034473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.753247 (+718774us) service_pool.cc:224] Handling call
0709 12:21:56.753512 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.768362 (+ 14850us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.739884 (+971522us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54001fca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:57.740179 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27791104}
W20260709 12:21:57.751713 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.752544 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.754447 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.764880 18015 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:57.766587 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.767632 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:57.762786 20157 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:57.769972 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:57.770702 18096 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:57.769515 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:57.771831 18098 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:57.790191 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:57.776201 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7346780939997188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
W20260709 12:21:57.792955 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6959780939996563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
I20260709 12:21:57.796890 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.797793 19884 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:57.805418 18815 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 2: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:21:57.814105 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:57.806527 20012 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.815383 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.815951 20128 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.816522 20128 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:57.816263 18815 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:57.819193 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.807763 20089 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:57.813540 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:57.814252 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:57.815579 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:21:57.828707 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.829669 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:57.809295 20085 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:57.811056 20151 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:57.825286 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:57.840009 18166 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:57.825502 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.826100 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:57.825212 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:57.834235 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.841434 18024 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:57.822121 20095 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.848268 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.850098 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.851233 18015 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:57.855337 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:57.845263 20079 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.079s	user 0.015s	sys 0.009s
I20260709 12:21:57.866827 18016 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:57.832274 20146 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:57.832804 20078 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.844723 20099 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:57.837167 20128 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:57.871763 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:21:57.842406 20098 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.992s	user 0.009s	sys 0.018s
I20260709 12:21:57.873104 18093 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:57.875386 20109 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.586s	user 0.007s	sys 0.007s
W20260709 12:21:57.879633 20097 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.884s	user 0.011s	sys 0.006s
W20260709 12:21:57.879741 20115 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.893s	user 0.011s	sys 0.005s
I20260709 12:21:57.860949 20085 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
W20260709 12:21:57.882100 20117 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.720s	user 0.011s	sys 0.011s
W20260709 12:21:57.882768 20055 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.573s	user 0.036s	sys 0.004s
I20260709 12:21:57.884735 20077 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:57.839728 20151 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:21:57.877247 20075 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.277s	user 0.008s	sys 0.008s
I20260709 12:21:57.862841 19841 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:57.863626 20076 raft_consensus.cc:2764] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260709 12:21:57.897637 20144 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.941s	user 0.012s	sys 0.006s
I20260709 12:21:57.935580 18092 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:57.840013 20089 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.894483 20079 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:57.914052 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:21:57.930603 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 105507us
I20260709 12:21:57.890564 20078 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.935469 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:57.942662 18024 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
W20260709 12:21:57.943435 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 225) took 2091 ms (client timeout 1722 ms). Trace:
W20260709 12:21:57.943881 18024 rpcz_store.cc:269] 0709 12:21:55.851953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.852114 (+   161us) service_pool.cc:224] Handling call
0709 12:21:57.833960 (+1981846us) spinlock_profiling.cc:243] Waited 943 us on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:57.943335 (+109375us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2169216}
I20260709 12:21:57.921610 20099 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:57.940079 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:57.952256 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:57.947281 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:57.953315 18024 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:57.951041 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:21:57.944964 19854 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.898s	user 0.015s	sys 0.019s
I20260709 12:21:57.937963 20162 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.918335 20109 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.893836 20072 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:57.952018 18810 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.885s	user 0.006s	sys 0.014s
I20260709 12:21:57.924914 20097 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.928040 20117 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.981263 20169 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:57.981153 20151 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.927595 20115 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.975682 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:57.942910 20144 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:57.965585 20093 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.805s	user 0.005s	sys 0.019s
W20260709 12:21:57.968977 20063 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.963s	user 0.012s	sys 0.006s
I20260709 12:21:57.983489 18096 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:57.980742 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:58.047516 18024 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:57.931156 20055 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:57.953295 20077 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:57.955886 20168 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:57.972254 20079 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:58.078361 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:58.079172 20097 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.079980 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.081353 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.082294 20099 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:58.086284 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:58.086685 20144 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:58.088233 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:58.092582 20055 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.062237 20093 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:58.089130 18024 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:58.067282 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:57.980110 18810 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:57.970695 19854 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:58.123603 20132 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:58.123095 18813 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:58.124151 20132 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:58.124585 20132 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:58.126083 19854 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.130131 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 523) took 1294 ms. Trace:
I20260709 12:21:58.130915 18810 rpcz_store.cc:276] 0709 12:21:56.835827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.839878 (+  4051us) service_pool.cc:224] Handling call
0709 12:21:58.129968 (+1290090us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.132825 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:57.979979 20162 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:58.133268 20160 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.435s	user 0.001s	sys 0.001s
I20260709 12:21:58.133808 20063 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.134451 20105 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:58.142590 18806 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:21:58.110016 18807 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:58.109560 19841 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
W20260709 12:21:58.148538 20165 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 12:21:58.071486 20077 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:21:58.109781 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:58.169292 18096 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:58.041935 20085 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
I20260709 12:21:58.100033 20060 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.555675 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:58.544356 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:58.563735 18024 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:21:58.528632 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.015s	sys 0.013s
W20260709 12:21:58.583670 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.020s	sys 0.013s
I20260709 12:21:58.284459 18815 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:58.285095 18015 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:21:58.302963 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.291319 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.446329 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 278) took 1890 ms (client timeout 1663 ms). Trace:
W20260709 12:21:58.239252 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 283878us
I20260709 12:21:58.564000 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:58.586983 18169 rpcz_store.cc:269] 0709 12:21:56.556024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.556183 (+   159us) service_pool.cc:224] Handling call
0709 12:21:58.446067 (+1889884us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:58.446223 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187608832}
I20260709 12:21:58.588002 18096 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:21:58.589047 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.589998 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:58.584436 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 621295us
I20260709 12:21:58.118026 20168 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:21:58.590266 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.591528 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.591837 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.595568 18169 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:58.595780 18170 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:21:58.596428 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.592120 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:58.560138 20104 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.004s	sys 0.015s
W20260709 12:21:58.471691 20068 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.765s	user 0.014s	sys 0.006s
I20260709 12:21:58.598548 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:58.558496 20096 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.856s	user 0.018s	sys 0.016s
I20260709 12:21:58.605369 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:58.447486 18018 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.710s	user 0.016s	sys 0.000s
I20260709 12:21:58.609944 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.612669 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:58.543386 20169 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.004s	sys 0.014s
I20260709 12:21:58.602530 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.565508 20109 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.602679 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:58.619508 18166 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:58.620862 20001 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:58.621104 18171 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:58.621516 20001 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:58.622036 20001 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:58.624681 20043 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:58.612547 20093 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:58.614900 20055 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:21:58.630432 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.337494 20033 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:58.382854 20098 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.603385 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.200577 19854 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.604640 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.448714 20116 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.644479 20116 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:58.645155 20001 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:58.522069 18093 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:58.537348 20054 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.607604 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:21:58.609014 20104 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.005s	sys 0.016s
I20260709 12:21:58.610248 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:58.341321 20115 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:58.200572 20117 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.610244 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:58.779652 18024 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:21:58.616822 20169 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.007s	sys 0.014s
I20260709 12:21:58.613541 18172 raft_consensus.cc:2368] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
W20260709 12:21:58.612128 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.475854 20163 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:58.594851 20060 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:58.729700 20079 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.630s	user 0.010s	sys 0.000s
I20260709 12:21:58.641853 18096 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:21:58.647975 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58150us
I20260709 12:21:58.202479 20063 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:58.613431 20096 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:58.783138 18810 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.569s	user 0.008s	sys 0.005s
I20260709 12:21:58.667241 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1493) took 2804 ms. Trace:
I20260709 12:21:58.612048 18018 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:58.669281 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.775900 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2274) took 1763 ms. Trace:
W20260709 12:21:58.695533 18063 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.722s (SENT)
I20260709 12:21:58.563819 20077 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:58.780303 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1884) took 1124 ms. Trace:
I20260709 12:21:58.776327 18165 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:58.770341 17988 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.037s (SENT)
I20260709 12:21:58.784695 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.785619 20177 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:58.647439 20001 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
W20260709 12:21:58.710351 16797 debug-util.cc:398] Leaking SignalData structure 0x7b080000f460 after lost signal to thread 17908
W20260709 12:21:58.738931 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.665360 20020 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.249s	user 0.017s	sys 0.006s
I20260709 12:21:58.630501 20055 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:58.790361 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 182889us
I20260709 12:21:58.792153 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:58.919590 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 127511us
I20260709 12:21:58.792276 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:58.711194 20125 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.648s	user 0.013s	sys 0.015s
I20260709 12:21:58.793705 20117 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:58.650198 20043 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:58.665557 20089 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.789562 20168 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:58.702927 20098 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:58.797246 20132 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.000s
W20260709 12:21:58.890815 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 195360us
I20260709 12:21:58.891081 18171 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:58.881675 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.892648 18098 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:21:58.920449 20020 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:58.880053 18167 rpcz_store.cc:276] 0709 12:21:55.862524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.403358 (+540834us) service_pool.cc:224] Handling call
0709 12:21:56.403800 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.421493 (+ 17693us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:56.421545 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.421613 (+    68us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:56.421661 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.666861 (+2245200us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:58.666939 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.667137 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39915264}
W20260709 12:21:58.899530 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 789835us
W20260709 12:21:58.991008 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.982369 20166 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.788514 20169 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:59.026366 20169 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.027066 20097 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:59.027590 20097 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:59.028600 20097 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.024729 20080 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.789747 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.032697 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.893333 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:21:59.034686 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 264415us
I20260709 12:21:59.035001 20155 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:58.921356 18096 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:59.037108 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 348) took 1141 ms. Trace:
I20260709 12:21:59.037513 18096 rpcz_store.cc:276] 0709 12:21:57.896498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.792041 (+895543us) service_pool.cc:224] Handling call
0709 12:21:59.037016 (+244975us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.038659 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:59.039284 18096 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:21:58.794293 20171 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:59.042258 20171 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.044474 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.045101 20171 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.652128 20033 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.047472 20033 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.783212 20054 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.047762 20162 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.048676 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:58.839223 20079 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:58.926808 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.050144 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:59.050634 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124796us
I20260709 12:21:59.051366 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.051445 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:59.052047 17919 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 289) took 5470 ms. Trace:
I20260709 12:21:59.052321 18170 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:59.053196 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:58.783741 19854 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.055902 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:58.669158 20038 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.898456 20177 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:58.648679 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2275) took 1624 ms. Trace:
I20260709 12:21:58.927390 20001 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.057334 20038 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.057982 20001 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.058566 20001 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:59.057699 18090 rpcz_store.cc:276] 0709 12:21:57.024470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.199034 (+174564us) service_pool.cc:224] Handling call
0709 12:21:57.199400 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.199808 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.199840 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.707751 (+507911us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:21:57.708404 (+   653us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.843718 (+135314us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:57.843771 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.843859 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.640910 (+797051us) raft_consensus.cc:1685] finished
0709 12:21:58.640933 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.648580 (+  7647us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":789,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":8065,"spinlock_wait_cycles":4602624,"thread_start_us":134821,"threads_started":1,"wal-append.queue_time_us":221675}
I20260709 12:21:59.060009 20127 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:59.060299 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.060544 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.061087 18020 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:59.061259 18090 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:58.931452 20117 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:21:58.931985 18813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 606) took 1718 ms (client timeout 2037 ms). Trace:
I20260709 12:21:58.933089 19910 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:58.933354 20063 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:58.934590 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 603) took 1722 ms (client timeout 1926 ms). Trace:
I20260709 12:21:58.935240 20096 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.926123 20116 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.936131 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:58.798599 20049 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.928000 20125 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:58.831707 20163 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:58.937316 20055 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.920083 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.878825 18810 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:21:58.915751 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2357) took 1649 ms (client timeout 2061 ms). Trace:
I20260709 12:21:58.970155 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1879) took 1335 ms. Trace:
W20260709 12:21:58.975790 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.061672 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1877) took 1427 ms. Trace:
I20260709 12:21:59.089076 18020 rpcz_store.cc:276] 0709 12:21:57.634086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.059974 (+1425888us) service_pool.cc:224] Handling call
0709 12:21:59.061603 (+  1629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.065053 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.090631 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:59.090785 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.056859 19854 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:59.091428 18807 rpcz_store.cc:269] 0709 12:21:57.211759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.273975 (+ 62216us) service_pool.cc:224] Handling call
0709 12:21:58.934356 (+1660381us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b5400590e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:58.934508 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1657395968}
I20260709 12:21:59.079028 20105 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.070996 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.093180 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:58.979499 18811 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:21:59.069581 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.094098 20144 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.094208 18814 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:59.094647 20020 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:59.094841 18814 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 448) took 2066 ms (client timeout 1722 ms). Trace:
I20260709 12:21:59.029338 20098 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.094750 20144 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:21:59.095443 18814 rpcz_store.cc:269] 0709 12:21:57.028555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.030185 (+  1630us) service_pool.cc:224] Handling call
0709 12:21:58.935912 (+1905727us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:59.094741 (+158829us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43737344}
I20260709 12:21:59.096154 18022 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
W20260709 12:21:59.096830 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 237) took 1861 ms (client timeout 2249 ms). Trace:
I20260709 12:21:59.096983 18809 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:59.089131 18097 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:59.097266 18022 rpcz_store.cc:269] 0709 12:21:57.235277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.009569 (+1774292us) service_pool.cc:224] Handling call
0709 12:21:59.096749 (+ 87180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.838220 20060 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.098464 18168 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.098634 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.100080 20060 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:58.990264 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 345) took 1136 ms. Trace:
I20260709 12:21:59.099706 20125 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:58.991406 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:58.890275 18024 rpcz_store.cc:276] 0709 12:21:57.656128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.595095 (+938967us) service_pool.cc:224] Handling call
0709 12:21:58.780229 (+185134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.978996 20089 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:58.980890 20072 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:58.991565 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:58.992966 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114390us
W20260709 12:21:58.982275 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 608) took 1605 ms (client timeout 1758 ms). Trace:
I20260709 12:21:58.994251 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:58.993403 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.000021 20126 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:21:59.022637 20102 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:59.023419 18099 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.227519 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 341) took 1383 ms. Trace:
I20260709 12:21:59.227923 18099 rpcz_store.cc:276] 0709 12:21:57.844258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.022803 (+1178545us) service_pool.cc:224] Handling call
0709 12:21:59.023209 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.227220 (+204011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.227452 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.229024 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.127967 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:59.231345 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 456) took 1152 ms. Trace:
I20260709 12:21:59.231666 18814 rpcz_store.cc:276] 0709 12:21:58.078964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.096701 (+1017737us) service_pool.cc:224] Handling call
0709 12:21:59.231277 (+134576us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.232851 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.233654 18814 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:21:59.236054 18171 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:59.214998 18096 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:59.243196 18172 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:59.057614 20177 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.029470 20166 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:21:59.164245 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.663s (SENT)
W20260709 12:21:59.260128 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95984.1us
I20260709 12:21:59.261474 20166 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.093258 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.262558 18016 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:59.263018 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1882) took 1627 ms. Trace:
I20260709 12:21:59.069020 20054 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.263401 18016 rpcz_store.cc:276] 0709 12:21:57.635750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.093002 (+1457252us) service_pool.cc:224] Handling call
0709 12:21:59.262949 (+169947us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.081252 20038 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:59.264647 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.264957 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.265373 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
W20260709 12:21:59.266881 20184 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.243s	user 0.004s	sys 0.000s
I20260709 12:21:59.099985 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.100783 18098 rpcz_store.cc:276] 0709 12:21:57.853713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.668965 (+815252us) service_pool.cc:224] Handling call
0709 12:21:58.990192 (+321227us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:59.267554 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.102088 18018 rpcz_store.cc:269] 0709 12:21:57.257163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.281744 (+ 24581us) service_pool.cc:224] Handling call
0709 12:21:58.906236 (+1624492us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.268217 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:58.879974 18094 rpcz_store.cc:276] 0709 12:21:57.012075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.194880 (+182805us) service_pool.cc:224] Handling call
0709 12:21:57.206707 (+ 11827us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.359698 (+152991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.775541 (+1415843us) spinlock_profiling.cc:243] Waited 93.1 ms on lock 0x7b540035edd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:58.775813 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":214129408}
I20260709 12:21:59.268991 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:21:58.926007 16797 kernel_stack_watchdog.cc:198] Thread 17908 stuck at /apache/dev/git/kudu/src/kudu/rpc/outbound_call.cc:307 for 386ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260709 12:21:59.067384 20067 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:59.269776 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 289) took 1456 ms. Trace:
I20260709 12:21:59.101024 18015 rpcz_store.cc:276] 0709 12:21:57.634688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.916313 (+281625us) service_pool.cc:224] Handling call
0709 12:21:58.969889 (+1053576us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:58.970054 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346751232}
I20260709 12:21:59.269513 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:59.270475 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.270774 19852 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:59.270836 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.271214 18018 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:59.271370 19852 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:59.270491 18166 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:59.272667 18016 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.272493 19852 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.273051 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2348) took 2233 ms. Trace:
I20260709 12:21:59.273391 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1869) took 1669 ms. Trace:
I20260709 12:21:59.069476 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2318) took 3040 ms. Trace:
I20260709 12:21:58.946061 20132 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.270236 18168 rpcz_store.cc:276] 0709 12:21:57.813354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.813498 (+   144us) service_pool.cc:224] Handling call
0709 12:21:57.813890 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.098942 (+1285052us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.269484 (+170542us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:59.269702 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2953681280}
I20260709 12:21:59.274919 18021 rpcz_store.cc:276] 0709 12:21:56.029353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.029477 (+   124us) service_pool.cc:224] Handling call
0709 12:21:56.042183 (+ 12706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.730559 (+688376us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.003686 (+2273127us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b540010a4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:59.069380 (+ 65694us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1278359296}
I20260709 12:21:59.276602 18021 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:58.975570 18165 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:21:59.277060 18168 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.090710 18023 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:58.976876 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.052479 17919 rpcz_store.cc:276] 0709 12:21:53.581922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.582152 (+   230us) service_pool.cc:224] Handling call
0709 12:21:59.051917 (+5469765us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:53.953582 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:53.966251 (+ 12669us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:53.966287 (+    36us) write_op.cc:435] Acquired schema lock
0709 12:21:53.966307 (+    20us) tablet.cc:662] Decoding operations
0709 12:21:53.966753 (+   446us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:53.966805 (+    52us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:53.966841 (+    36us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:53.967550 (+   709us) tablet.cc:701] Row locks acquired
0709 12:21:53.967581 (+    31us) write_op.cc:260] PREPARE: finished
0709 12:21:53.967915 (+   334us) write_op.cc:270] Start()
0709 12:21:53.970209 (+  2294us) write_op.cc:276] Timestamp: P: 1783599713967831 usec, L: 0
0709 12:21:53.970271 (+    62us) op_driver.cc:348] REPLICATION: starting
0709 12:21:53.984565 (+ 14294us) log.cc:844] Serialized 533 byte log entry
0709 12:21:54.354962 (+370397us) op_driver.cc:464] REPLICATION: finished
0709 12:21:55.209349 (+854387us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b60000af1c8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1a021161 00007f6c1a18df1c 00007f6c1a0ede62 00007f6c1a0ee489 00007f6c1a0f071a 00007f6c1a0f06a8 00007f6c1a0f0628 00007f6c1a0f05e3 00007f6c1a0ef7fc 00007f6c1c5222f6 00007f6c1c51caa8
0709 12:21:55.676110 (+466761us) write_op.cc:301] APPLY: starting
0709 12:21:55.676245 (+   135us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:55.676383 (+   138us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:55.676418 (+    35us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:55.720339 (+ 43921us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:55.720580 (+   241us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:55.720634 (+    54us) write_op.cc:312] APPLY: finished
0709 12:21:55.720896 (+   262us) log.cc:844] Serialized 23 byte log entry
0709 12:21:55.871808 (+150912us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:55.885590 (+ 13782us) write_op.cc:454] Released schema lock
0709 12:21:55.889087 (+  3497us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:55.952241 (+ 63154us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:55.986235 (+ 33994us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1037698,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":444777,"apply.run_cpu_time_us":17678,"apply.run_wall_time_us":244487,"num_ops":1,"prepare.queue_time_us":109486,"prepare.run_cpu_time_us":3847,"prepare.run_wall_time_us":44835,"replication_time_us":384538,"spinlock_wait_cycles":7677696,"thread_start_us":442169,"threads_started":2,"wal-append.queue_time_us":187270,"wal-append.run_cpu_time_us":1294,"wal-append.run_wall_time_us":1001288}]]}
I20260709 12:21:59.278304 18165 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.277244 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1880) took 1642 ms. Trace:
I20260709 12:21:59.098474 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.279016 18021 rpcz_store.cc:276] 0709 12:21:57.635047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.276102 (+1641055us) service_pool.cc:224] Handling call
0709 12:21:59.276413 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.276975 (+   562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.277183 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.279595 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1876) took 1646 ms. Trace:
I20260709 12:21:59.273474 18015 rpcz_store.cc:276] 0709 12:21:57.040066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.271985 (+2231919us) service_pool.cc:224] Handling call
0709 12:21:59.272467 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.272684 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.272719 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.272770 (+    51us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:59.272813 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.272874 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.272988 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31104}
I20260709 12:21:59.279968 18022 rpcz_store.cc:276] 0709 12:21:57.633681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.098188 (+1464507us) service_pool.cc:224] Handling call
0709 12:21:59.279524 (+181336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.083523 20097 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:59.243996 18813 rpcz_store.cc:269] 0709 12:21:57.213641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.298888 (+ 85247us) service_pool.cc:224] Handling call
0709 12:21:58.931750 (+1632862us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b54005429d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:58.931907 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1423097088}
I20260709 12:21:59.056813 18019 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
W20260709 12:21:59.286394 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 309663us
W20260709 12:21:59.088411 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1618) took 2233 ms (client timeout 2363 ms). Trace:
I20260709 12:21:58.980671 19841 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:59.240695 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.247584 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.249547 18020 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
W20260709 12:21:59.251212 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.251929 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 349) took 1331 ms. Trace:
I20260709 12:21:59.252803 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.069120 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:59.254136 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 447) took 2205 ms. Trace:
I20260709 12:21:59.254594 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.255167 20008 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.256721 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 455) took 1395 ms. Trace:
I20260709 12:21:59.223121 20073 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:59.257992 18807 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:21:59.204540 20096 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.102499 20056 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.258284 20150 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:59.358809 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.359548 18021 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
I20260709 12:21:59.359606 20056 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.360886 18090 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:59.362059 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:21:59.259119 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 267830us
I20260709 12:21:59.363420 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:59.365051 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:59.365322 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:59.366055 20150 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:59.367033 20181 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.344s	user 0.002s	sys 0.000s
I20260709 12:21:59.369913 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.092270 20117 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.273881 18016 rpcz_store.cc:276] 0709 12:21:57.603732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.272044 (+1668312us) service_pool.cc:224] Handling call
0709 12:21:59.272467 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.273138 (+   671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.273330 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95744}
I20260709 12:21:59.375344 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.375687 20151 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.376289 18016 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:21:59.377341 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:21:59.276978 20132 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.257192 20089 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.270910 18094 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.225075 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.366039 18090 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:59.383104 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.390632 18090 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:59.391726 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.392432 20089 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.392432 18090 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:59.394179 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:59.394820 18090 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:59.395709 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.399320 20132 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:59.399837 20132 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.400182 20132 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.403596 20183 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.404276 20183 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.404635 20183 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:59.076398 20084 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:59.257270 20001 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.296437 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.290432 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1881) took 1655 ms. Trace:
I20260709 12:21:59.286653 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2364) took 1465 ms. Trace:
I20260709 12:21:59.411088 18022 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:21:59.411293 18019 rpcz_store.cc:276] 0709 12:21:57.821317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.055540 (+1234223us) service_pool.cc:224] Handling call
0709 12:21:59.286577 (+231037us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:59.297333 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.412467 20001 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.413924 18166 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:59.415333 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.316435 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 602) took 2104 ms. Trace:
I20260709 12:21:59.323585 18017 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:59.323721 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:59.417639 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.410883 18023 rpcz_store.cc:276] 0709 12:21:57.635381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.007257 (+1371876us) service_pool.cc:224] Handling call
0709 12:21:59.290345 (+283088us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.418556 18168 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:59.418859 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.317356 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.419602 18096 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:59.329363 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.419953 18812 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:21:59.418040 18815 rpcz_store.cc:276] 0709 12:21:57.211521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.822620 (+611099us) service_pool.cc:224] Handling call
0709 12:21:58.111999 (+289379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.598379 (+486380us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.314637 (+716258us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54004b41d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:21:59.316062 (+  1425us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54004b41d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:59.316320 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036121216}
W20260709 12:21:59.420796 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91543.4us
I20260709 12:21:59.420881 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.421103 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.421298 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:59.421793 20130 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.422302 20130 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.422712 20130 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:59.323318 18024 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:21:59.242467 20020 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.426949 20020 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.427937 20183 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.273624 19852 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:59.334307 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45674.2us
I20260709 12:21:59.433002 20020 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:21:59.287686 18092 rpcz_store.cc:269] 0709 12:21:56.854844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.904277 (+ 49433us) service_pool.cc:224] Handling call
0709 12:21:59.088068 (+2183791us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400215054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:59.088305 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2369148160}
I20260709 12:21:59.434337 18017 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:59.434626 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:59.339437 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.436360 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.437058 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:21:59.339478 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.438542 18813 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:59.439563 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:59.440186 18813 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:59.444216 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:59.445811 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.446522 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:21:59.447655 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:59.448657 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.449213 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.449662 20073 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:59.450002 18172 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:21:59.450198 20073 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.450747 20073 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.451146 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.452375 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.452955 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:21:59.453514 20001 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.441530 20183 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.454190 20001 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.456851 20001 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:21:59.340484 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 357036us
W20260709 12:21:59.342049 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 184245us
I20260709 12:21:59.461304 18092 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:21:59.461753 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0096309089999522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2737s for a response
I20260709 12:21:59.227026 20102 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:59.458873 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:21:59.463584 20102 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.464157 20054 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:59.464672 20054 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:59.464727 20102 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:59.465126 20054 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:59.330708 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2263) took 2482 ms. Trace:
I20260709 12:21:59.242116 20180 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.465989 18097 rpcz_store.cc:276] 0709 12:21:56.848034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.921028 (+ 72994us) service_pool.cc:224] Handling call
0709 12:21:59.088668 (+2167640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.097646 (+  8978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.330369 (+232723us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b5400215050. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:59.330592 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4984010624}
I20260709 12:21:59.466214 20180 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.467458 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.468312 18097 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:59.469200 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:21:59.362182 20056 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.370764 20150 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.470423 18170 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:59.381704 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.281689 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.421660 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.474059 18097 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:59.474686 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53134.1us
I20260709 12:21:59.419576 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:59.323401 18814 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:59.476949 20055 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:59.477519 20055 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:59.474198 18095 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:59.478006 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.478266 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:21:59.478845 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:21:59.358615 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.479359 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.479436 20146 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:59.480127 18814 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:59.481176 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:59.478515 20055 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.469271 20180 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.482023 20056 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:59.482586 20056 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:21:59.483036 20056 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.483163 18094 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:59.483539 18814 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:21:59.484997 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.091261 20163 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.486959 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.284555 20154 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:59.487733 18811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 450) took 2436 ms (client timeout 2111 ms). Trace:
W20260709 12:21:59.488209 18811 rpcz_store.cc:269] 0709 12:21:57.051303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.158195 (+106892us) service_pool.cc:224] Handling call
0709 12:21:59.487528 (+2329333us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540009bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:59.487649 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3981538176}
I20260709 12:21:59.488741 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.489442 18015 raft_consensus.cc:2376] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
I20260709 12:21:59.482051 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.245071 19910 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:59.490716 18095 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:59.490722 18165 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:59.490722 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.490218 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.447651 18090 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:59.503651 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:21:59.523236 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.557592 18138 leader_election.cc:336] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.504s (SENT)
I20260709 12:21:59.485684 18814 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:21:59.558842 18138 leader_election.cc:336] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.585s (SENT)
I20260709 12:21:59.559432 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:21:59.406772 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:21:59.311590 20052 raft_consensus.cc:1068] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Unable to promote non-voter 910f8c8d787e4abbb092413c1f90ebf3: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:21:59.540439 18813 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:59.313278 20049 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.577142 20049 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.576486 20054 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.311865 20068 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:59.566759 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.580042 18813 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:21:59.583978 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 177277us
I20260709 12:21:59.584740 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.567411 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:57.321959 17908 catalog_manager.cc:5028] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): tablet a29e00b535474906948e66659110db3b (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:21:59.586719 18811 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:21:59.564544 18018 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:59.577632 20049 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:59.540180 20151 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
W20260709 12:21:59.663280 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.586790 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.323485 18810 rpcz_store.cc:276] 0709 12:21:57.861682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.132314 (+270632us) service_pool.cc:224] Handling call
0709 12:21:59.256647 (+1124333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.309211 20072 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.607151 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.697104 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.705942 18090 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:21:59.705801 18095 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:21:59.354463 20144 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.077977 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.416864 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:21:59.717350 18166 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:59.313833 20077 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.637524 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:21:59.726776 18811 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:59.727033 18813 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:59.727555 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 604) took 2515 ms. Trace:
I20260709 12:21:59.728185 18811 rpcz_store.cc:276] 0709 12:21:57.211995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.641932 (+2429937us) service_pool.cc:224] Handling call
0709 12:21:59.726524 (+ 84592us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.727221 (+   697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.727453 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.728652 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 607) took 2512 ms. Trace:
I20260709 12:21:59.340406 20008 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.591027 18097 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:21:59.637135 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:21:59.767992 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 130959us
W20260709 12:21:59.354054 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.693732 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.541532 20125 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:59.688738 20143 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.312107 19841 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:59.553970 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 454) took 1730 ms (client timeout 2002 ms). Trace:
I20260709 12:21:59.786381 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.787909 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:59.790191 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.792181 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:21:59.242950 18806 rpcz_store.cc:269] 0709 12:21:57.394133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.706712 (+312579us) service_pool.cc:224] Handling call
0709 12:21:58.942506 (+1235794us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:21:58.981733 (+ 39227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":835172096}
I20260709 12:21:59.818506 18811 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.323664 18093 rpcz_store.cc:276] 0709 12:21:57.920913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.876185 (+955272us) service_pool.cc:224] Handling call
0709 12:21:59.251850 (+375665us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.290042 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 243) took 1191 ms. Trace:
I20260709 12:21:59.821126 18020 rpcz_store.cc:276] 0709 12:21:58.098861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.090262 (+991401us) service_pool.cc:224] Handling call
0709 12:21:59.289960 (+199698us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.821497 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.721272 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:21:59.491528 20096 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.724314 18170 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: [1.4-1.4]
I20260709 12:21:59.609758 20051 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:59.698840 20076 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:59.725046 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2356) took 2492 ms. Trace:
W20260709 12:21:59.726161 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 139405us
I20260709 12:21:59.729054 18813 rpcz_store.cc:276] 0709 12:21:57.215974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.727731 (+2511757us) service_pool.cc:224] Handling call
0709 12:21:59.728065 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.728286 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.728317 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.728363 (+    46us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:59.728411 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.728463 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.728592 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.778555 20166 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:59.639940 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.732460 20102 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:59.738813 20154 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.564536 20080 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.162s
I20260709 12:21:59.840960 18018 rpcz_store.cc:276] 0709 12:21:57.225384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.272000 (+2046616us) service_pool.cc:224] Handling call
0709 12:21:59.325946 (+ 53946us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.612147 (+286201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.717215 (+105068us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28928}
I20260709 12:21:59.851085 20080 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:59.851512 20080 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:21:59.852461 20080 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:21:59.746910 18808 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:21:59.841518 20116 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:59.854318 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 445) took 2863 ms (client timeout 2255 ms). Trace:
I20260709 12:21:59.854383 20116 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:21:59.854835 18808 rpcz_store.cc:269] 0709 12:21:56.991238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.995080 (+  3842us) service_pool.cc:224] Handling call
0709 12:21:59.077671 (+2082591us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:21:59.854218 (+776547us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34462464}
I20260709 12:21:59.841925 20051 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.856554 20051 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.856778 20116 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.857106 20154 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:59.755173 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.756989 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 611) took 2219 ms. Trace:
I20260709 12:21:59.757994 18173 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:21:59.759087 20077 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:59.610087 20075 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:59.759949 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:21:59.601665 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 111521us
W20260709 12:21:59.883479 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.884233 18020 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:21:59.774745 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.780561 18807 rpcz_store.cc:269] 0709 12:21:57.823291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.092810 (+1269519us) service_pool.cc:224] Handling call
0709 12:21:59.542389 (+449579us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:59.637082 17908 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 2.31495e+06us
W20260709 12:21:59.745659 20105 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.652s	user 0.002s	sys 0.007s
I20260709 12:21:59.610307 20127 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.162s
I20260709 12:21:59.718284 20078 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:21:59.810096 18168 raft_consensus.cc:2376] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3. Re-sending same reply.
I20260709 12:21:59.906934 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.704243 20001 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260709 12:21:59.793715 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:21:59.822324 18095 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:21:59.611908 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.825006 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 609) took 2430 ms. Trace:
I20260709 12:21:59.827651 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.776790 20125 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:59.841457 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 120480us
I20260709 12:21:59.778178 20143 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:59.844748 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 204882us
I20260709 12:21:59.783954 20146 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:59.845654 20102 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.849792 20054 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:21:59.857999 18099 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:21:59.750208 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 174) took 4040 ms. Trace:
I20260709 12:21:59.932960 17921 rpcz_store.cc:276] 0709 12:21:55.710322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.710538 (+   216us) service_pool.cc:224] Handling call
0709 12:21:59.750101 (+4039563us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1992525,"reported_tablets":3}
I20260709 12:21:59.872979 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 526) took 1283 ms. Trace:
I20260709 12:21:59.856678 18808 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:59.892258 18020 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:59.893998 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.885735 20080 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:59.896919 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 407485us
I20260709 12:21:59.858448 19852 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:59.857776 20144 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.902067 20038 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.867589 20076 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:21:59.775462 19910 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.827839 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:21:59.968784 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:21:59.915457 18811 rpcz_store.cc:276] 0709 12:21:57.394700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.770304 (+2375604us) service_pool.cc:224] Handling call
0709 12:21:59.818193 (+ 47889us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.819487 (+  1294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.824907 (+  5420us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:59.926846 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.930470 20102 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:59.927788 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:21:59.938301 18806 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 4. Ops: []
I20260709 12:21:59.971653 18173 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:21:59.972486 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 413729us
I20260709 12:21:59.928215 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:21:59.925978 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.841146 20049 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.934572 18813 rpcz_store.cc:276] 0709 12:21:58.589234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.871606 (+1282372us) service_pool.cc:224] Handling call
0709 12:21:59.872090 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.872437 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:59.872470 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.872545 (+    75us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:59.872620 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.872704 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.872883 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.975086 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:21:59.935631 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108086us
I20260709 12:21:59.976837 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:21:59.941960 18094 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 1.
I20260709 12:21:59.813781 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:21:59.950825 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:21:59.950156 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 190275us
I20260709 12:21:59.954012 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:21:59.897843 20075 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:21:59.907931 20105 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:59.323606 18809 rpcz_store.cc:276] 0709 12:21:57.021978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.029710 (+  7732us) service_pool.cc:224] Handling call
0709 12:21:59.096413 (+2066703us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:21:59.096741 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.097558 (+   817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.226687 (+129129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10502400}
W20260709 12:21:59.916668 20128 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.067s	sys 0.088s
I20260709 12:21:59.964325 20054 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:59.929782 20143 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:59.923893 20125 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:59.967300 18165 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:59.949144 20130 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.526s	user 0.008s	sys 0.003s
I20260709 12:22:00.056764 18814 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:22:00.083844 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 458) took 1105 ms. Trace:
I20260709 12:22:00.057269 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:00.056013 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.058656 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:00.085356 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 131568us
I20260709 12:22:00.086427 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:00.007869 18138 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.763s (SENT)
I20260709 12:21:59.955446 20144 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.087881 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2288) took 1024 ms. Trace:
I20260709 12:21:59.978789 18166 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:59.945865 20020 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:59.960438 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:00.066869 18097 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:00.068049 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2287) took 1004 ms. Trace:
W20260709 12:22:00.146807 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.151016 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.096679 18094 rpcz_store.cc:276] 0709 12:21:59.063513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.086684 (+1023171us) service_pool.cc:224] Handling call
0709 12:22:00.087287 (+   603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.087508 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.087534 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.087580 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:00.087625 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.087692 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.087814 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.154771 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2290) took 1090 ms. Trace:
I20260709 12:22:00.161926 18099 rpcz_store.cc:276] 0709 12:21:59.063194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.067072 (+1003878us) service_pool.cc:224] Handling call
0709 12:22:00.067439 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.067684 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.067713 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.067759 (+    46us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:00.067819 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.067872 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.067988 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.967451 20038 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.099752 20067 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:00.165364 20067 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:00.166697 20038 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:00.166514 20067 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:00.167382 20038 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:00.167784 20038 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.168115 20038 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:00.075839 20068 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:00.170226 20068 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:21:59.978130 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 453) took 2281 ms. Trace:
W20260709 12:21:59.990768 20117 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.003s	sys 0.007s
W20260709 12:22:00.200852 20117 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.006s	sys 0.011s
I20260709 12:22:00.084730 18815 raft_consensus.cc:2368] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
I20260709 12:22:00.084259 18814 rpcz_store.cc:276] 0709 12:21:58.978622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.930320 (+951698us) service_pool.cc:224] Handling call
0709 12:21:59.975059 (+ 44739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.083373 (+108314us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.083766 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.203219 18814 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: [1.4-1.4]
I20260709 12:22:00.088881 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.206990 18173 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 4. Ops: []
I20260709 12:22:00.089421 18017 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:00.090171 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:00.209373 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 201558us
I20260709 12:22:00.081112 20163 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:21:59.877916 18810 rpcz_store.cc:276] 0709 12:21:57.538180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.755638 (+2217458us) service_pool.cc:224] Handling call
0709 12:21:59.756255 (+   617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.756529 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.756568 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.756618 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:59.756682 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.756747 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.756895 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.067535 18170 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:21:59.975898 18813 raft_consensus.cc:2376] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9. Re-sending same reply.
W20260709 12:22:00.075649 20128 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.067s	sys 0.089s
I20260709 12:22:00.097692 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 612) took 2454 ms. Trace:
I20260709 12:22:00.099648 20144 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:59.958595 20132 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.558s	user 0.005s	sys 0.006s
I20260709 12:22:00.155233 18094 rpcz_store.cc:276] 0709 12:21:59.064077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.153804 (+1089727us) service_pool.cc:224] Handling call
0709 12:22:00.154188 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.154398 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.154425 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.154470 (+    45us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:00.154517 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.154576 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.154704 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.183928 18811 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:59.980448 19852 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:00.068172 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:00.192397 18808 rpcz_store.cc:276] 0709 12:21:57.696307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.856010 (+2159703us) service_pool.cc:224] Handling call
0709 12:21:59.856439 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.945730 (+ 89291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.971539 (+ 25809us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.092769 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:00.084455 20166 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:00.084131 20054 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:00.305186 20054 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:00.306734 20190 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.243s	user 0.000s	sys 0.000s
I20260709 12:22:00.213430 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.213315 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:00.310756 18024 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:00.310009 20075 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.312161 18165 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:00.313972 20163 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.331598 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 617) took 1070 ms. Trace:
I20260709 12:22:00.332391 18809 rpcz_store.cc:276] 0709 12:21:59.261595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.209977 (+948382us) service_pool.cc:224] Handling call
0709 12:22:00.330581 (+120604us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.330928 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.330975 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.331053 (+    78us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:00.331162 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.331274 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.331489 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.335655 20049 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:00.296615 20132 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:00.366941 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.018500 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:00.239447 18138 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.900s (SENT)
W20260709 12:22:00.383714 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 144287us
I20260709 12:22:00.155572 19910 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:00.090039 20180 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:59.970737 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:00.256323 18098 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:00.441085 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2295) took 1182 ms. Trace:
I20260709 12:22:00.441743 18098 rpcz_store.cc:276] 0709 12:21:59.258754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.240622 (+981868us) service_pool.cc:224] Handling call
0709 12:22:00.251883 (+ 11261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.440595 (+188712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.440950 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.155999 20072 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:00.266531 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.270366 16797 debug-util.cc:398] Leaking SignalData structure 0x7b080000f640 after lost signal to thread 17908
I20260709 12:22:00.219408 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.161525 20020 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:00.262099 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 194033us
I20260709 12:22:00.386137 18018 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:00.294816 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 202180us
I20260709 12:22:00.212926 20116 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:00.296330 20127 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:00.302181 20166 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.302932 20055 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:00.303970 19852 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:00.494344 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:00.096171 20195 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:00.387164 18812 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:00.471633 20020 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:00.412623 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2299) took 1073 ms. Trace:
I20260709 12:22:00.317445 20163 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:00.247432 20117 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:00.433676 18169 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.036s	sys 0.048s
W20260709 12:22:00.436352 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.567842 18169 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.036s	sys 0.049s
I20260709 12:22:00.568715 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1947) took 1971 ms. Trace:
W20260709 12:22:00.439855 18139 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.039s	sys 0.052s
W20260709 12:22:00.357441 20150 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.887s	user 0.005s	sys 0.004s
W20260709 12:22:00.569641 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.005s	sys 0.007s
I20260709 12:22:00.209594 20038 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:00.569882 18139 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.039s	sys 0.052s
W20260709 12:22:00.570427 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.005s	sys 0.007s
I20260709 12:22:00.298856 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 459) took 1270 ms. Trace:
I20260709 12:22:00.571384 18815 rpcz_store.cc:276] 0709 12:21:58.989469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.421053 (+431584us) service_pool.cc:224] Handling call
0709 12:22:00.259698 (+838645us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54006153d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:00.259901 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1523874560}
I20260709 12:22:00.465997 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:00.573537 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107680us
I20260709 12:22:00.484668 20116 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:00.130725 20130 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:00.478601 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.362671 20096 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:00.448345 20051 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.591s	user 0.010s	sys 0.002s
I20260709 12:22:00.570333 20150 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:00.414952 20008 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:00.413549 20132 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:00.460966 20168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.882s	user 0.048s	sys 0.046s
W20260709 12:22:00.618367 20169 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s	user 0.029s	sys 0.064s
I20260709 12:22:00.498163 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:00.650039 18098 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:00.191700 18097 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260709 12:22:00.505765 18812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 613) took 1673 ms (client timeout 2211 ms). Trace:
W20260709 12:22:00.651250 18812 rpcz_store.cc:269] 0709 12:21:58.832956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.420850 (+587894us) service_pool.cc:224] Handling call
0709 12:22:00.505491 (+1084641us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54005f5854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:00.505677 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2219917184}
I20260709 12:22:00.506646 18813 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:00.251644 18806 rpcz_store.cc:276] 0709 12:21:57.643576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.858743 (+2215167us) service_pool.cc:224] Handling call
0709 12:21:59.898054 (+ 39311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.057871 (+159817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.091154 (+ 33283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:00.511875 18091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.191s	user 0.021s	sys 0.035s
W20260709 12:22:00.542963 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 614799us
W20260709 12:22:00.653079 18091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.332s	user 0.021s	sys 0.044s
I20260709 12:22:00.653373 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2293) took 1418 ms. Trace:
I20260709 12:22:00.554339 18094 rpcz_store.cc:276] 0709 12:21:59.338876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.258625 (+919749us) service_pool.cc:224] Handling call
0709 12:22:00.258914 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.369154 (+110240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.369205 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.369269 (+    64us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:00.369329 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.412181 (+ 42852us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400411ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:00.412267 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.412511 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252653952}
I20260709 12:22:00.443995 18093 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:00.252223 20076 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.566941 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:00.632503 20177 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.319s	user 0.012s	sys 0.005s
I20260709 12:22:00.574839 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:00.466456 18166 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
W20260709 12:22:00.713042 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.019s	sys 0.015s
W20260709 12:22:00.713426 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.019s	sys 0.015s
I20260709 12:22:00.569169 18169 rpcz_store.cc:276] 0709 12:21:58.597275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.617111 (+ 19836us) service_pool.cc:224] Handling call
0709 12:21:58.617501 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.617930 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.617962 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.618563 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.917369 (+298806us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:59.101018 (+183649us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b18003ca000. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:21:59.266160 (+165142us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:21:59.707063 (+440903us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b18003ca000. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf
0709 12:21:59.707244 (+   181us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.568325 (+861081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.568413 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.568464 (+    51us) raft_consensus.cc:1685] finished
0709 12:22:00.568478 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.568644 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":69173,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":155602,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"spinlock_wait_cycles":412460416,"thread_start_us":1177449,"threads_started":3,"wal-append.queue_time_us":366419}
W20260709 12:22:00.713726 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 787769us
I20260709 12:22:00.490531 20127 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.532141 18807 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:00.543403 20099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.522s	user 0.016s	sys 0.049s
I20260709 12:22:00.494974 20055 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:00.571312 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 594581us
W20260709 12:22:00.449056 20052 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.017s	sys 0.038s
I20260709 12:22:00.476233 20146 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
W20260709 12:22:00.754015 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.603417 18173 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:00.479544 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.502991 20195 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:00.627276 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.009s	sys 0.009s
W20260709 12:22:00.767711 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.010s	sys 0.009s
I20260709 12:22:00.768872 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:00.632247 18168 raft_consensus.cc:2368] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
I20260709 12:22:00.585551 19925 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:00.416584 20180 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:00.632001 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.001s	sys 0.019s
I20260709 12:22:00.635835 18090 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:00.653795 18097 rpcz_store.cc:276] 0709 12:21:59.235417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.190793 (+955376us) service_pool.cc:224] Handling call
0709 12:22:00.191309 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.653064 (+461755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.653310 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.681965 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 535) took 1053 ms. Trace:
I20260709 12:22:00.253007 19841 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:00.692106 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 211828us
W20260709 12:22:00.648059 20169 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.029s	sys 0.064s
W20260709 12:22:00.647603 20168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.069s	user 0.048s	sys 0.047s
W20260709 12:22:00.712122 17988 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.211s (SENT) [suppressed 50 similar messages]
I20260709 12:22:00.714491 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:00.626984 20001 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.009s	sys 0.010s
I20260709 12:22:00.744331 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2302) took 1386 ms. Trace:
I20260709 12:22:00.809648 18098 rpcz_store.cc:276] 0709 12:21:59.358209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.712563 (+1354354us) service_pool.cc:224] Handling call
0709 12:22:00.743405 (+ 30842us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.743739 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.743784 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.743852 (+    68us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:00.743929 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.744021 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.744221 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.751466 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1532) took 1248 ms. Trace:
I20260709 12:22:00.811237 18171 rpcz_store.cc:276] 0709 12:21:59.502815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.503329 (+   514us) service_pool.cc:224] Handling call
0709 12:22:00.734902 (+1231573us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400042ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:00.751344 (+ 16442us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2588783872}
I20260709 12:22:00.742551 20055 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:00.638617 20054 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.816691 18167 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:00.742754 20052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.986s	user 0.017s	sys 0.038s
W20260709 12:22:00.764529 20191 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 12:22:00.630401 20051 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.789588 20150 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:00.733704 20099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.744s	user 0.020s	sys 0.049s
W20260709 12:22:00.775686 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 290535us
I20260709 12:22:00.730515 20127 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:00.778798 18781 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.012s	sys 0.036s
W20260709 12:22:00.862193 18781 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.013s	sys 0.036s
I20260709 12:22:00.863727 20054 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:00.864539 20054 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:00.780731 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.001s	sys 0.019s
W20260709 12:22:00.865759 20194 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 12:22:00.866138 20212 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.866864 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:00.866809 20212 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:00.868046 18807 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:22:00.781631 18813 rpcz_store.cc:276] 0709 12:21:59.637673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.243546 (+605873us) service_pool.cc:224] Handling call
0709 12:22:00.506430 (+262884us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.652399 (+145969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.681718 (+ 29319us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54005f5850. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:00.681894 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":603495680}
I20260709 12:22:00.869242 18015 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:00.784425 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1636) took 1051 ms. Trace:
I20260709 12:22:00.869976 18097 rpcz_store.cc:276] 0709 12:21:59.732960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.783099 (+1050139us) service_pool.cc:224] Handling call
0709 12:22:00.783600 (+   501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.783905 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.783946 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.784012 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:00.784080 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.784159 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.784334 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:00.871176 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:00.789680 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:00.872738 18063 leader_election.cc:400] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
W20260709 12:22:00.872637 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 83047.3us
I20260709 12:22:00.874255 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:00.875010 18812 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:00.805529 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.808352 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93937.5us
I20260709 12:22:00.774523 19925 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.718360 20177 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:00.775951 20195 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:00.847751 20073 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.393s	user 0.016s	sys 0.000s
I20260709 12:22:00.797526 20180 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:00.772346 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:00.809378 20001 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.010s	sys 0.011s
I20260709 12:22:00.747490 20206 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20260709 12:22:00.857362 20051 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:00.857731 20198 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:00.858501 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:00.878692 20130 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:00.881091 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:00.924419 20169 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:00.873283 18063 leader_election.cc:403] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:22:00.891726 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1637) took 1158 ms. Trace:
I20260709 12:22:00.894876 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 346) took 3026 ms. Trace:
I20260709 12:22:00.978013 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2300) took 1627 ms. Trace:
W20260709 12:22:00.900943 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 326215us
I20260709 12:22:00.835114 18172 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:00.979295 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:00.865346 20054 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.721164 20116 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:00.868072 20202 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 12:22:00.877741 20125 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.725s	user 0.007s	sys 0.003s
W20260709 12:22:00.957549 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 222769us
I20260709 12:22:00.848026 20150 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.966781 20166 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.009184 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.010798 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:00.961336 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 249240us
I20260709 12:22:00.815308 20200 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.110s
I20260709 12:22:00.878440 20096 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:00.879501 20207 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.967144 18096 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:00.862195 20203 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:00.922657 19925 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:00.866197 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 904704us
W20260709 12:22:00.972971 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 91963.7us
I20260709 12:22:01.032095 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:00.958112 20073 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:00.974330 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 202117us
I20260709 12:22:00.978508 18092 rpcz_store.cc:276] 0709 12:21:59.350655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.259133 (+908478us) service_pool.cc:224] Handling call
0709 12:22:00.406402 (+147269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.406740 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:00.702122 (+295382us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.977512 (+275390us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.977613 (+   101us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:00.977681 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.977737 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.977929 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":432107,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":80,"thread_start_us":499839,"threads_started":2}
I20260709 12:22:00.978168 18090 rpcz_store.cc:276] 0709 12:21:57.868402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.706914 (+1838512us) service_pool.cc:224] Handling call
0709 12:21:59.736207 (+ 29293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.781255 (+1045048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.894480 (+113225us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54002963d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:00.894780 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2068436352}
I20260709 12:22:00.878053 19841 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260709 12:22:00.945351 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:00.878093 20117 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20260709 12:22:00.878911 20197 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20260709 12:22:00.978009 18098 rpcz_store.cc:276] 0709 12:21:59.733270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.841131 (+1107861us) service_pool.cc:224] Handling call
0709 12:22:00.890382 (+ 49251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.891121 (+   739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.891171 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.891240 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:00.891325 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.891413 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.891619 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.034152 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:00.979965 20051 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:00.879920 20099 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260709 12:22:00.878235 20132 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
W20260709 12:22:00.995266 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.981236 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:00.922327 20163 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 12:22:01.000798 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1891) took 1098 ms. Trace:
W20260709 12:22:00.976415 20143 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.876s	user 0.011s	sys 0.004s
W20260709 12:22:01.003963 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.013494 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:01.014307 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:01.014431 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:01.015115 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:01.016023 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.016956 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:01.017704 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:01.022058 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.023041 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1505) took 2238 ms. Trace:
I20260709 12:22:01.023263 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:01.024242 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1527) took 1610 ms. Trace:
I20260709 12:22:00.980880 20075 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:00.977367 20169 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:00.881640 18095 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.792s	user 0.009s	sys 0.002s
I20260709 12:22:01.122728 18016 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.122897 18095 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:01.033795 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:01.035136 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.033704 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:01.026974 20204 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:01.046718 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:00.997611 20038 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.051962 18024 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:01.019228 20150 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:00.998229 20105 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:22:01.027745 20212 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:01.033579 20096 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:01.005685 20205 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:01.086727 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:01.053252 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:01.273801 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 220741us
I20260709 12:22:01.274245 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1641) took 1539 ms. Trace:
I20260709 12:22:01.274838 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:01.274699 18090 rpcz_store.cc:276] 0709 12:21:59.734611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.273138 (+1538527us) service_pool.cc:224] Handling call
0709 12:22:01.273558 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.273848 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.273881 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.273934 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:01.273985 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.274046 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.274173 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.276440 18090 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:01.254489 20080 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.294s	user 0.007s	sys 0.008s
I20260709 12:22:01.276885 18090 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:01.043161 18812 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:22:01.071671 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 213251us
I20260709 12:22:01.278757 20132 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:01.279587 20132 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:01.109740 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:01.280331 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:01.280689 18018 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:22:01.113621 19841 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:01.281656 19841 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:01.119204 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.282454 20096 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.282626 19841 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.119274 18017 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:01.284354 19841 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.120003 18167 rpcz_store.cc:276] 0709 12:21:58.784904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.266039 (+481135us) service_pool.cc:224] Handling call
0709 12:21:59.266361 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.817472 (+1551111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.022710 (+205238us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54003ba1d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:01.022959 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3564534016}
W20260709 12:22:01.191160 20097 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.901s	user 0.014s	sys 0.003s
I20260709 12:22:01.120963 18173 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:01.286551 20077 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:01.287784 20210 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:22:01.288671 18097 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:01.276499 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:01.290759 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1638) took 1557 ms. Trace:
I20260709 12:22:01.291211 18097 rpcz_store.cc:276] 0709 12:21:59.733585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.871358 (+1137773us) service_pool.cc:224] Handling call
0709 12:22:00.871875 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.289112 (+417237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.290411 (+  1299us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:01.290670 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":957453568}
I20260709 12:22:01.108075 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1639) took 1374 ms. Trace:
I20260709 12:22:01.302616 18098 rpcz_store.cc:276] 0709 12:21:59.733933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.051764 (+1317831us) service_pool.cc:224] Handling call
0709 12:22:01.107274 (+ 55510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.107562 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.107601 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.107666 (+    65us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:01.107728 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.107806 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.107969 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.304771 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1643) took 1569 ms. Trace:
W20260709 12:22:01.122036 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.306317 20080 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.306874 20080 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.307175 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:01.308606 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:01.309495 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 531) took 1962 ms. Trace:
I20260709 12:22:00.997018 20206 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.028169 20130 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:01.027364 20203 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.019052 20125 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.311439 18090 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.041992 20001 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:01.045974 20207 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.082633 20225 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.329923 20203 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:01.121353 18172 rpcz_store.cc:276] 0709 12:21:59.413963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.453575 (+ 39612us) service_pool.cc:224] Handling call
0709 12:21:59.491805 (+ 38230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.978885 (+1487080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.023917 (+ 45032us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003b9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:01.024152 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3056417536}
W20260709 12:22:01.192744 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:01.228668 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 247561us
I20260709 12:22:01.056270 20221 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.181s
I20260709 12:22:01.053408 20116 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.126233 20209 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:01.126952 20220 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:01.127902 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:01.126004 20169 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.123458 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2305) took 1176 ms. Trace:
I20260709 12:22:01.098745 20117 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:01.098042 18806 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:01.128443 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2304) took 1639 ms. Trace:
I20260709 12:22:01.226707 20127 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:01.379244 20180 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:01.125234 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91962.7us
I20260709 12:22:01.126057 18166 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:01.231715 20038 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:01.124058 20072 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.082556 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 210067us
I20260709 12:22:01.128283 20055 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.123625 20051 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:01.233528 20073 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.237210 20146 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:22:01.237715 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1626) took 1996 ms (client timeout 2230 ms). Trace:
I20260709 12:22:01.238202 18099 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:01.238922 20177 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:01.239130 20102 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.167s	user 0.015s	sys 0.000s
I20260709 12:22:01.239681 20196 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:01.239975 20193 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.271s	user 0.001s	sys 0.000s
I20260709 12:22:01.241456 18016 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:01.192476 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1640) took 1458 ms. Trace:
I20260709 12:22:01.312520 20205 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.406687 18168 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:01.444726 18096 rpcz_store.cc:269] 0709 12:21:59.241047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.420635 (+179588us) service_pool.cc:224] Handling call
0709 12:22:01.237460 (+1816825us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400052154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:01.237644 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3553666432}
I20260709 12:22:01.445376 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:01.402516 18812 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEARNER]: Advancing to term 3
I20260709 12:22:01.445376 18094 rpcz_store.cc:276] 0709 12:21:59.489008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.723102 (+1234094us) service_pool.cc:224] Handling call
0709 12:22:00.723508 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.723960 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.723993 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.724549 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.929670 (+205121us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:00.929744 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.929894 (+   150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.128167 (+198273us) raft_consensus.cc:1685] finished
0709 12:22:01.128192 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.128364 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3779,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":32300,"thread_start_us":133493,"threads_started":1,"wal-append.queue_time_us":260543}
I20260709 12:22:01.452234 18092 rpcz_store.cc:276] 0709 12:21:59.734242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.108620 (+1374378us) service_pool.cc:224] Handling call
0709 12:22:01.191735 (+ 83115us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.191971 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.192002 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.192056 (+    54us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:01.192148 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.192232 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.192382 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.457125 20146 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:01.269999 20204 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:01.218557 20144 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.921s	user 0.010s	sys 0.005s
I20260709 12:22:01.120764 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:01.478334 18169 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.292173 20163 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:01.292524 20075 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:22:01.294381 18065 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.220s	user 0.012s	sys 0.008s
W20260709 12:22:01.296952 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.298081 20049 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:01.298730 20056 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.815s	user 0.013s	sys 0.005s
W20260709 12:22:01.509349 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.305243 18098 rpcz_store.cc:276] 0709 12:21:59.735220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.303795 (+1568575us) service_pool.cc:224] Handling call
0709 12:22:01.304190 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.304401 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.304427 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.304471 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:01.304515 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.304568 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.304703 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.312732 18064 leader_election.cc:400] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
I20260709 12:22:01.445746 20196 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:01.310052 18814 rpcz_store.cc:276] 0709 12:21:59.347248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.205481 (+858233us) service_pool.cc:224] Handling call
0709 12:22:00.209991 (+  4510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.712661 (+502670us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.712699 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.754252 (+ 41553us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.956293 (+202041us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:00.956348 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.956438 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.309190 (+352752us) raft_consensus.cc:1685] finished
0709 12:22:01.309225 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.309414 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12854,"raft.queue_time_us":732000,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":21080,"thread_start_us":689995,"threads_started":2,"wal-append.queue_time_us":511468}
I20260709 12:22:01.313995 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2306) took 1366 ms. Trace:
I20260709 12:22:01.315182 20080 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.303683 19925 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.533968 20075 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:01.193463 20054 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.538244 20054 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.537592 20075 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.540073 20075 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.541069 20054 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.449532 20205 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:01.534387 20056 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.546388 18093 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:01.549264 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1632) took 1943 ms. Trace:
I20260709 12:22:01.556807 18093 rpcz_store.cc:276] 0709 12:21:59.606392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.748362 (+1141970us) service_pool.cc:224] Handling call
0709 12:22:00.846472 (+ 98110us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.547132 (+700660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.548747 (+  1615us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54002cae54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:01.549108 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1596414208}
I20260709 12:22:01.445076 20117 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:01.571099 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:01.479893 20144 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.587468 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2276) took 4553 ms. Trace:
I20260709 12:22:01.587766 20204 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.418504 20220 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.457772 20146 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:01.322170 20084 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.837s	user 0.015s	sys 0.003s
I20260709 12:22:01.305415 20150 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.124110 18015 rpcz_store.cc:276] 0709 12:21:59.902773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.910878 (+  8105us) service_pool.cc:224] Handling call
0709 12:22:00.070662 (+159784us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:00.070950 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.869700 (+798750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.000368 (+130668us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:01.000691 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1846977152}
I20260709 12:22:01.625051 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.626359 18172 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:01.316895 20166 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.034314 20008 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.174s
I20260709 12:22:01.358855 20215 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:22:01.193792 20143 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.375115 18170 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:01.330497 20197 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 12:22:01.377931 18090 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:01.402397 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:01.406278 20154 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.502s	user 0.014s	sys 0.003s
W20260709 12:22:01.424786 19910 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.986s	user 0.009s	sys 0.005s
I20260709 12:22:01.461790 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:01.472867 20096 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.379582 20001 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.459882 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2292) took 2234 ms. Trace:
I20260709 12:22:01.658522 18099 rpcz_store.cc:276] 0709 12:21:59.225507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.163702 (+938195us) service_pool.cc:224] Handling call
0709 12:22:01.237982 (+1074280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.445357 (+207375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.459453 (+ 14096us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400052150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:01.459776 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2469206272}
I20260709 12:22:01.482059 18165 raft_consensus.cc:2368] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3.
I20260709 12:22:01.458410 20195 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.419040 20125 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:01.458133 20102 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.677213 20102 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:01.404661 20020 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.819s	user 0.014s	sys 0.000s
I20260709 12:22:01.678270 20020 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.656746 20154 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.553756 19925 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.683090 20154 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.318274 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:01.555209 20056 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.556107 20049 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:01.470665 20212 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:01.597756 20225 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
I20260709 12:22:01.599159 20051 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.660387 20096 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.619295 18169 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:01.317063 20232 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:01.631263 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:01.588976 20144 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.642328 18813 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.463292 20072 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:01.645933 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 234314us
I20260709 12:22:01.577031 20177 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.654032 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 192386us
I20260709 12:22:01.693013 20200 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.627235 20209 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.693904 20233 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:01.718921 20233 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.718914 20209 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.717680 20177 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.719727 20233 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:01.662583 18812 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:22:01.634202 20207 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.667973 20226 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.693277 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.644s (SENT)
I20260709 12:22:01.730937 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:01.660650 20001 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:01.734357 20144 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:01.735255 20177 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.742388 18807 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 12:22:01.685247 20127 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.692555 20196 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:01.459491 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:01.516361 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.535488 18064 leader_election.cc:403] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:01.121537 18020 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:22:01.471993 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:01.533708 18065 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.459s	user 0.016s	sys 0.012s
I20260709 12:22:01.486919 18023 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:01.588040 18091 rpcz_store.cc:276] 0709 12:21:57.034243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.199245 (+165002us) service_pool.cc:224] Handling call
0709 12:21:57.199482 (+   237us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.200928 (+  1446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.200984 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.944469 (+1743485us) spinlock_profiling.cc:243] Waited 5.31 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:21:59.319921 (+375452us) spinlock_profiling.cc:243] Waited 980 us on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:21:59.320750 (+   829us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.653575 (+1332825us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:00.653635 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.653708 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.587163 (+933455us) raft_consensus.cc:1685] finished
0709 12:22:01.587191 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.587381 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":111,"raft.queue_time_us":4690,"raft.run_cpu_time_us":36629,"raft.run_wall_time_us":1772412,"spinlock_wait_cycles":14471168,"thread_start_us":1332249,"threads_started":1,"wal-append.queue_time_us":1511332}
I20260709 12:22:01.406303 18095 rpcz_store.cc:276] 0709 12:21:59.947509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.956613 (+  9104us) service_pool.cc:224] Handling call
0709 12:22:01.123387 (+1166774us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.536010 18097 rpcz_store.cc:276] 0709 12:21:59.947932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.312856 (+1364924us) service_pool.cc:224] Handling call
0709 12:22:01.313294 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.313551 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.313586 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.313647 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:01.313702 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.313766 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.313925 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:01.612200 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2307) took 1664 ms. Trace:
I20260709 12:22:01.535259 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:01.640878 18164 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:01.651932 18168 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:01.654925 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 1960) took 1177 ms. Trace:
I20260709 12:22:01.685132 20049 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:01.787179 20180 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.267185 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1900) took 1252 ms. Trace:
I20260709 12:22:01.849392 20084 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:02.268083 18020 rpcz_store.cc:276] 0709 12:22:01.014067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.014181 (+   114us) service_pool.cc:224] Handling call
0709 12:22:02.266778 (+1252597us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:02.268429 20049 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:02.185529 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.272397 18170 rpcz_store.cc:276] 0709 12:22:00.477630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.479186 (+  1556us) service_pool.cc:224] Handling call
0709 12:22:01.654650 (+1175464us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b5400383954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:01.654834 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1404311424}
I20260709 12:22:01.776062 20117 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.132325 18092 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:01.728888 20146 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:02.116801 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.906733 20195 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.137230 18093 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:02.132732 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:02.297166 20120 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.797286 20226 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:02.297973 18090 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.008s
I20260709 12:22:02.298297 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2313) took 1248 ms. Trace:
W20260709 12:22:02.022993 20116 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.564s	user 0.013s	sys 0.000s
W20260709 12:22:02.012574 20169 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.592s	user 0.006s	sys 0.007s
W20260709 12:22:01.944299 20206 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.625s	user 0.009s	sys 0.003s
W20260709 12:22:02.136270 18094 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.001s	sys 0.000s
I20260709 12:22:02.209306 20020 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.268738 20084 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:02.020008 20203 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.621s	user 0.008s	sys 0.005s
I20260709 12:22:01.687935 20096 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:02.080392 20205 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.537s	user 0.013s	sys 0.000s
W20260709 12:22:01.870363 20240 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
W20260709 12:22:02.221691 20077 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.934s	user 0.016s	sys 0.024s
I20260709 12:22:02.298450 18096 rpcz_store.cc:276] 0709 12:21:59.948239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.447964 (+1499725us) service_pool.cc:224] Handling call
0709 12:22:01.448341 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.448593 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:01.576224 (+127631us) log.cc:844] Serialized 14 byte log entry
0709 12:22:01.611726 (+ 35502us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.611805 (+    79us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:01.611869 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.611951 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.612106 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":187257,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":269,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":110,"thread_start_us":130270,"threads_started":2,"wal-append.queue_time_us":158257}
I20260709 12:22:02.222043 20038 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:01.911262 20209 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.801237 20128 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:02.445401 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.804085 18813 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:02.210904 20051 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.481756 18098 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.485873 20038 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:02.480216 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1898) took 1466 ms (client timeout 1549 ms). Trace:
I20260709 12:22:02.486250 20180 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:22:02.498528 20127 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:02.504079 18094 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.003s	sys 0.015s
W20260709 12:22:02.189368 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 471232us
W20260709 12:22:02.360967 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 172937us
W20260709 12:22:02.480572 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 117499us
I20260709 12:22:02.480809 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1544) took 1294 ms. Trace:
I20260709 12:22:01.718180 20200 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.482733 18092 rpcz_store.cc:276] 0709 12:22:01.037196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.636784 (+599588us) service_pool.cc:224] Handling call
0709 12:22:02.131090 (+494306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.283719 (+152629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.285614 (+  1895us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:02.053431 20166 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:02.498395 18023 rpcz_store.cc:269] 0709 12:22:01.013402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.016719 (+  3317us) service_pool.cc:224] Handling call
0709 12:22:02.479682 (+1462963us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54003bb0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:02.479992 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":844326144}
I20260709 12:22:02.499689 20127 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:02.512231 20120 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.542229 19910 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:02.545109 20117 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.516022 20077 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:02.517630 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.05809e+06us
I20260709 12:22:02.547092 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:02.547638 20200 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.549113 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:02.506419 18168 rpcz_store.cc:276] 0709 12:22:01.008271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.124325 (+116054us) service_pool.cc:224] Handling call
0709 12:22:02.282540 (+1158215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:02.463943 20076 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:22:02.519670 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.722s (SENT)
W20260709 12:22:02.520740 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.20231e+06us
I20260709 12:22:02.520452 20205 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.211680 20072 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:01.907788 20220 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:01.743328 20143 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:02.389307 20130 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.097s	sys 0.241s
I20260709 12:22:02.537424 20196 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.550379 20127 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:01.744334 20055 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:02.554440 20180 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:02.558148 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:02.566243 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 625) took 1164 ms. Trace:
W20260709 12:22:02.558156 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.24726e+06us
I20260709 12:22:02.570465 20205 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:02.415401 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 288) took 4542 ms (client timeout 2249 ms). Trace:
W20260709 12:22:02.594875 18164 rpcz_store.cc:269] 0709 12:21:57.813024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.813965 (+   941us) service_pool.cc:224] Handling call
0709 12:22:02.273785 (+4459820us) spinlock_profiling.cc:243] Waited 3.8 s on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:02.355776 (+ 81991us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8745176960}
I20260709 12:22:02.299152 20150 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.298023 20054 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:02.574895 19910 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.586014 20128 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:02.359920 20075 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
I20260709 12:22:02.593003 18098 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:02.588784 20120 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.593179 18813 rpcz_store.cc:276] 0709 12:22:01.401403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.402065 (+   662us) service_pool.cc:224] Handling call
0709 12:22:02.565893 (+1163828us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:02.604995 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85398.4us
I20260709 12:22:02.601577 18095 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:02.604782 19910 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:02.574764 20051 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:02.610920 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61989.6us
W20260709 12:22:02.603928 20130 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.956s	user 0.101s	sys 0.245s
I20260709 12:22:02.617645 20209 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:22:02.614873 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 493132us
W20260709 12:22:02.614246 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.732s (SENT)
I20260709 12:22:02.616902 18090 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20260709 12:22:02.624555 20127 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.625243 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:02.627251 20120 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:02.621088 18090 pending_rounds.cc:85] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:02.632874 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2316) took 1137 ms. Trace:
I20260709 12:22:02.633289 18098 rpcz_store.cc:276] 0709 12:22:01.478963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.534791 (+ 55828us) service_pool.cc:224] Handling call
0709 12:22:02.611785 (+1076994us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:02.606979 20072 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.636401 20072 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.639299 20072 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:02.639988 20072 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:02.643222 20051 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.650089 20055 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:22:02.651165 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.800s (SENT)
I20260709 12:22:02.651962 20055 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:02.652453 20127 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.652695 20055 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:02.654726 20001 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.872s	user 0.009s	sys 0.013s
I20260709 12:22:02.655453 20146 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.655375 20001 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.656245 20055 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.658191 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:02.659775 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:02.660933 20209 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:02.661564 20209 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:02.661957 20204 raft_consensus.cc:2764] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 12:22:02.658381 20055 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.663066 20209 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.677964 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:02.678422 20020 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:02.678957 20020 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.679376 18164 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:02.679508 20096 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:02.680279 20096 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:02.679399 20020 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.684911 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:02.677886 20166 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:02.688796 20166 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.082733 20233 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:02.691294 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62318.2us
I20260709 12:22:02.685796 17921 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 3, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:22:02.691931 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.862s (SENT)
I20260709 12:22:02.630353 18090 pending_rounds.cc:107] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20260709 12:22:02.669675 20226 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.601224 20038 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.711426 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:02.709540 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:02.692878 19910 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:02.710824 18090 raft_consensus.cc:2917] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260709 12:22:02.711036 20120 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:22:02.717193 20220 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:02.706332 20206 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:02.717684 20203 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.719214 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:02.696992 20241 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:02.707173 20051 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.710470 17921 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
I20260709 12:22:02.727813 18023 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.731228 20241 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.735239 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:02.735966 20077 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:02.736178 20076 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:02.708215 20001 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:02.743052 18090 raft_consensus.cc:2889] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:02.748804 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 143167us
W20260709 12:22:02.739892 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.13712e+06us
I20260709 12:22:02.752535 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:02.754896 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.756309 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:02.745286 20241 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:02.760720 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:02.753796 17921 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 3 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:02.722818 20220 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.752511 20128 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.782778 20233 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:02.783711 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116113us
I20260709 12:22:02.761503 20169 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.673439 20084 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.785965 20038 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:02.681627 20096 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.808358 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:02.789846 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:02.810403 18091 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
W20260709 12:22:02.810887 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.814251 20117 raft_consensus.cc:2764] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:02.815883 20205 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.697636 20075 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
W20260709 12:22:02.793745 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58872us
I20260709 12:22:02.793399 20220 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.798148 20076 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:02.799772 18097 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:02.803606 20077 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:02.804265 18019 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:02.805362 20084 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:02.816630 20205 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.798842 18164 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.796983 18023 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:02.839869 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.749894 20049 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:02.850769 17989 leader_election.cc:400] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3.
I20260709 12:22:02.853992 17989 leader_election.cc:403] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:22:02.794760 19910 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:02.866067 20077 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.869580 20077 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:02.872038 20205 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:02.873183 20076 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:02.875229 20076 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.882709 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:02.883864 18020 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
W20260709 12:22:02.885988 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 213291us
W20260709 12:22:02.890497 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.969s (SENT)
I20260709 12:22:02.803128 20169 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.895211 20128 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.896988 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:02.898088 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.898182 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1902) took 1883 ms (client timeout 1558 ms). Trace:
W20260709 12:22:02.903053 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.891646 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:02.904924 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.905766 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:02.904556 18019 rpcz_store.cc:269] 0709 12:22:01.014712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.014856 (+   144us) service_pool.cc:224] Handling call
0709 12:22:02.897906 (+1883050us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:02.898089 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3829591936}
I20260709 12:22:02.799595 20203 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:02.926518 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1546) took 1078 ms. Trace:
I20260709 12:22:02.847693 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:02.914124 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.913556 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134183us
I20260709 12:22:02.848937 20117 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.928460 20221 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.927407 18098 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:02.942950 18095 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:02.948028 18164 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:02.950292 20020 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.951256 20117 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.931751 18166 rpcz_store.cc:276] 0709 12:22:01.623085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.623266 (+   181us) service_pool.cc:224] Handling call
0709 12:22:01.652978 (+ 29712us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:02.154979 (+502001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:02.155061 (+    82us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:02.203173 (+ 48112us) log.cc:844] Serialized 26 byte log entry
0709 12:22:02.489055 (+285882us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:02.489101 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.527894 (+ 38793us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.690391 (+162497us) raft_consensus.cc:1685] finished
0709 12:22:02.690445 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.701219 (+ 10774us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13416,"raft.queue_time_us":791456,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":457,"thread_start_us":285175,"threads_started":1,"wal-append.queue_time_us":400740}
W20260709 12:22:02.971818 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 124234us
I20260709 12:22:02.983336 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:02.986702 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 153974us
I20260709 12:22:02.987469 20245 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:02.988051 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:02.988438 20245 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:02.990485 20249 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.991360 20249 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:02.993392 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2317) took 1375 ms. Trace:
W20260709 12:22:02.994134 18778 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.127s (SENT)
I20260709 12:22:02.994788 18778 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:02.996492 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 99725.4us
W20260709 12:22:02.997058 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.128s (SENT)
I20260709 12:22:02.998637 20169 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:02.999534 20169 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:03.000242 20248 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:03.000844 20169 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:02.990672 20243 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:22:02.999114 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:03.010407 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.002549 20166 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:02.994990 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:02.961376 20206 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:03.021376 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60828.7us
I20260709 12:22:03.020417 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:03.050617 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.027657 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1642) took 3292 ms. Trace:
I20260709 12:22:03.028256 20075 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:03.028883 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2311) took 2148 ms. Trace:
I20260709 12:22:03.019699 19910 raft_consensus.cc:2764] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:03.031282 20220 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.031908 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.033089 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.034741 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.035491 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.020524 20054 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.034819 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.037943 20020 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.063614 18815 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:03.038791 20249 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.039657 18807 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:03.041132 20166 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:03.047498 20117 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:03.048223 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 69914.6us
W20260709 12:22:03.026960 20242 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:22:03.040686 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:03.051087 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54111.6us
W20260709 12:22:03.052486 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.051617 18090 rpcz_store.cc:276] 0709 12:21:59.734947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.275949 (+1541002us) service_pool.cc:224] Handling call
0709 12:22:01.276308 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.981797 (+1705489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.027552 (+ 45755us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:03.059690 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.060599 18095 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:03.070060 20220 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.070809 20226 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:03.081849 18813 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:03.071790 18806 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.071691 18812 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:03.038313 20051 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:03.073434 18814 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.002801 20150 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.993917 18099 rpcz_store.cc:276] 0709 12:22:01.534858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.165478 (+630620us) service_pool.cc:224] Handling call
0709 12:22:02.283592 (+118114us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.457362 (+173770us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:02.711052 (+253690us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.909963 (+198911us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:02.910051 (+    88us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:02.910151 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.910267 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.910477 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":350210,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":49,"raft.queue_time_us":55291,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":176,"thread_start_us":334122,"threads_started":2,"wal-append.queue_time_us":255232}
I20260709 12:22:03.079813 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.049219 20209 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:03.090685 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50176.1us
I20260709 12:22:03.091727 18064 leader_election.cc:400] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4.
I20260709 12:22:03.092620 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2308) took 3144 ms. Trace:
I20260709 12:22:03.093101 18094 rpcz_store.cc:276] 0709 12:21:59.948571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.452792 (+1504221us) service_pool.cc:224] Handling call
0709 12:22:01.453433 (+   641us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:01.474586 (+ 21153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.474642 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:01.475719 (+  1077us) log.cc:844] Serialized 26 byte log entry
0709 12:22:02.622942 (+1147223us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:02.670513 (+ 47571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.674162 (+  3649us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.829557 (+155395us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b1800383380. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:03.092151 (+262594us) raft_consensus.cc:1685] finished
0709 12:22:03.092178 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:03.092527 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19723,"raft.queue_time_us":20133,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":234,"spinlock_wait_cycles":38684416,"thread_start_us":888130,"threads_started":1,"wal-append.queue_time_us":983720}
I20260709 12:22:03.096231 20195 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:03.097280 20195 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.098302 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.099107 18019 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:03.101527 20077 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.102063 18099 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:22:03.102270 20077 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.102999 18099 raft_consensus.cc:606] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.103410 18099 raft_consensus.cc:614] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:03.104743 20077 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:03.106367 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.107311 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:22:03.091619 20249 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:03.114673 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.079156 20166 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:03.132006 20226 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:03.151265 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.151057 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:03.154714 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.156551 18092 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:22:03.112746 20130 raft_consensus.cc:1064] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:22:03.158095 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:03.114306 20195 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:03.166617 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.168527 18020 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:03.124020 20249 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:03.145239 20166 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:03.176956 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 187135us
I20260709 12:22:03.118808 20150 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:03.180707 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:03.160678 18098 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:03.125346 20051 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:03.110805 20245 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:03.185104 18023 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:03.184859 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.182602 18815 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
W20260709 12:22:03.187834 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.188036 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:03.059248 18093 rpcz_store.cc:276] 0709 12:22:00.880315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.560967 (+680652us) service_pool.cc:224] Handling call
0709 12:22:01.561635 (+   668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.491103 (+929468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.002357 (+511254us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b540032c154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:03.028774 (+ 26417us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320709376}
I20260709 12:22:03.193775 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:03.200305 18137 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.102s (SENT)
I20260709 12:22:03.176663 20249 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:03.189714 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.187s (SENT)
I20260709 12:22:03.206038 18090 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:03.201150 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:03.210677 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.211459 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.180791 20096 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:03.136238 20075 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.215976 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.216132 20096 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.231752 20096 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:03.232082 20051 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:03.234727 20257 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:03.234422 20226 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:03.241428 18780 connection.cc:256] RPC call timeout handler was delayed by -3.3618754509998325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
I20260709 12:22:03.220211 20120 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:03.242389 20120 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.242832 20120 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.244032 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.245014 18814 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:03.246439 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.247401 18806 raft_consensus.cc:2368] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5.
I20260709 12:22:03.248613 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.249655 18809 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.251412 20249 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.253854 20249 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.254204 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.198135 20076 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:03.202808 20206 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:03.263162 18092 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:03.277745 18019 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:03.278005 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.280953 18095 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:03.281749 18099 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.278972 18096 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:03.288580 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.300166 20054 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:03.300760 20054 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.301196 20054 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:03.302597 20096 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.303540 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.304343 18810 raft_consensus.cc:2376] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2. Re-sending same reply.
I20260709 12:22:03.327603 20241 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.319391 20049 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.330575 20049 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.334694 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.335969 18808 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6.
I20260709 12:22:03.305457 20206 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:03.345213 20206 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.280401 20203 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:03.321499 20127 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.668s	user 0.015s	sys 0.002s
W20260709 12:22:03.372547 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 141435us
W20260709 12:22:03.375011 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144803us
W20260709 12:22:03.377300 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 200782us
I20260709 12:22:03.377530 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.333604 20049 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:03.381352 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.381697 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.385416 18099 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:03.387467 18810 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:03.391036 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.388068 20038 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.567s	user 0.014s	sys 0.000s
W20260709 12:22:03.396585 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.397176 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.397192 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.401921 20067 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.402657 20067 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.403307 18812 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:03.404547 20221 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3.
I20260709 12:22:03.405213 20067 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:03.406716 18090 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:22:03.428438 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.422590 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.437613 18023 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:03.439268 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.390419 20076 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:03.419782 20096 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:03.446470 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 246231us
I20260709 12:22:03.446800 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:03.447430 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.447885 18812 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:03.448349 18137 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.516s (SENT)
I20260709 12:22:03.418840 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.452224 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.453264 18093 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.454124 20262 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.454408 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.454771 18808 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:03.455233 18093 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:03.455493 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.456278 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.457054 18093 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:03.419498 20075 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:03.458264 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.458742 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.459568 18096 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:03.460577 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.461402 18096 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:03.449054 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:03.461869 18806 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:03.464238 18166 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:03.465654 20169 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:03.465809 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.466334 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.454432 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.466866 18168 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:03.467226 18164 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.467326 18810 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:22:03.468477 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.470333 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.470528 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:03.470779 20268 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:22:03.471246 18096 raft_consensus.cc:2368] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5.
I20260709 12:22:03.416682 20241 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.478984 20150 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.479310 20120 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.482247 20241 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.482559 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.482923 20241 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:03.484082 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.483932 20130 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.484848 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.481995 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.485343 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.510281 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.511741 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.489593 18094 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:03.513224 18813 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:03.490921 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:03.517107 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 219544us
I20260709 12:22:03.517453 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.514762 18097 raft_consensus.cc:2368] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5.
I20260709 12:22:03.518658 18814 raft_consensus.cc:2376] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4. Re-sending same reply.
I20260709 12:22:03.485743 18815 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:03.523234 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.522742 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.500100 18098 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:03.535892 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.537272 18098 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.538976 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.540117 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.541040 18092 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:03.548848 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 108212us
I20260709 12:22:03.485378 20241 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:03.550583 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.549686 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.552099 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.552402 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.556442 18813 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.556607 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.557083 18091 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.524184 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.560513 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:03.561645 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:03.562320 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.559585 18095 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:03.558806 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.562846 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.563040 18099 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:03.563305 18097 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:03.566278 18094 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
W20260709 12:22:03.569931 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 175842us
I20260709 12:22:03.570776 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:03.571956 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.572270 18094 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:03.574312 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:03.576486 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.577267 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:03.578454 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.580338 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.580279 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.581192 18091 raft_consensus.cc:2376] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4. Re-sending same reply.
I20260709 12:22:03.581698 18098 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.584261 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.580617 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.585004 18813 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:03.585505 18093 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:03.558367 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.588505 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.565316 20150 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:22:03.570834 18810 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:03.585886 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.594017 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.594746 18170 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:03.574767 20120 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:03.579445 20075 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:03.581239 20195 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:03.585726 20262 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:03.602248 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.577198 20054 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.598206 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.609017 18815 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.672318 18168 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:03.673363 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89539.8us
I20260709 12:22:03.614974 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:03.676524 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61675.4us
I20260709 12:22:03.593258 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:03.581924 20096 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:03.601118 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.608464 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.677568 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:03.682844 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 89721.3us
I20260709 12:22:03.687270 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.688686 20273 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:22:03.664649 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.607435 20195 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:03.691473 20195 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:03.694905 20274 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:22:03.694659 20166 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.696296 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.696733 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.697345 18808 raft_consensus.cc:2368] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5.
I20260709 12:22:03.697715 18173 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.698516 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.699496 18808 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.700650 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.701593 18808 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:03.634505 20084 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.781s	user 0.012s	sys 0.005s
W20260709 12:22:03.695313 20265 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:22:03.705547 20226 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.697477 20166 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.613847 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.722967 18813 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:22:03.716828 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.715075 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:03.714766 18096 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:03.714833 18093 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:03.607518 20075 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:03.678664 20096 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:03.728567 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.725665 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.729271 18020 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:03.731300 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.732285 18813 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.682005 20262 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:03.734648 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.715982 20150 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:03.755786 20150 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:03.756529 20150 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:03.757670 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.758671 18171 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:03.760573 20051 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:03.754436 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:03.762504 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.762995 18099 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.763290 18023 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:03.763741 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.764637 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.765591 18015 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:03.765899 18017 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:03.766534 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.767338 18018 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:03.769243 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.769985 18016 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.770694 18807 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.766558 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.771770 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.772058 18019 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 1.
I20260709 12:22:03.772652 18807 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.772816 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:03.738165 18173 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: [1.4-1.4]
I20260709 12:22:03.737885 20120 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:22:03.736581 18811 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:03.734361 20096 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:03.745229 18815 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:03.716213 20067 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:03.754428 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.765327 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.737784 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.766666 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.601598 20249 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20260709 12:22:03.772135 18097 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:03.779306 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
W20260709 12:22:03.777720 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60881.5us
I20260709 12:22:03.830718 18168 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:03.866595 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 181913us
I20260709 12:22:03.778535 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.773533 18093 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:03.800096 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.839919 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 119799us
W20260709 12:22:03.785480 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.873028 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.817139 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:03.874634 18815 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:03.876108 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.836217 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:03.877218 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.877413 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.877940 18099 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:03.881986 20200 raft_consensus.cc:2764] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:22:03.882628 18808 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:03.882052 18168 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.883303 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.884104 18020 raft_consensus.cc:2368] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3.
I20260709 12:22:03.885010 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:03.882485 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.885061 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.885656 18171 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.888403 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:03.889532 20203 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:03.890153 20203 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:03.891088 18173 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:03.896862 18168 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:03.898211 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.898538 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.890041 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:03.796505 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.908246 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.875653 20166 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:03.911082 18168 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:03.911384 20001 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:03.911901 20001 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:03.912341 20001 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:03.878937 20067 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:03.914090 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:03.914294 20248 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:03.778017 20262 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.915099 18024 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:03.917109 18173 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:03.839167 20116 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:03.911150 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:03.918190 20116 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:03.918654 20116 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.919910 20054 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.921232 20195 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.922601 18165 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:03.928980 20273 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.929976 20273 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.930806 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.931581 18173 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:22:03.931748 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.920145 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.933226 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.914820 20248 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:03.936139 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.833861 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.733842 20075 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.934801 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:03.939648 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:03.940600 18137 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.896s (SENT)
I20260709 12:22:03.943307 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:03.943585 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:03.944042 18814 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.927743 20195 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:03.773656 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:03.941089 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.613384 20076 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.842197 20226 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:03.914325 20067 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:03.948509 18016 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:03.949435 18170 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:03.932566 20273 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:03.952041 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.954576 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:03.960124 18019 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:03.956851 18812 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:03.962219 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 269846us
I20260709 12:22:03.966153 18024 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:03.968955 20226 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:03.972216 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:03.973867 18137 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.149s (SENT)
I20260709 12:22:03.974678 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:03.954245 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:03.976924 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.978972 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.980041 18170 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 2: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:03.982386 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:03.989786 18021 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:22:03.990993 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:03.993907 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.994999 20008 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.025s	sys 0.053s
I20260709 12:22:03.891492 20203 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:03.997351 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:03.963810 20221 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:04.005858 20249 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.007709 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.956264 18164 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:04.009411 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89404.1us
I20260709 12:22:04.006744 18170 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:04.008121 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:04.020795 18173 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:04.022276 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111226us
I20260709 12:22:03.960595 18807 raft_consensus.cc:2376] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10. Re-sending same reply.
I20260709 12:22:03.995548 20051 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:04.000078 20054 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:04.000576 19910 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:04.030262 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 175) took 1384 ms. Trace:
I20260709 12:22:04.034410 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:03.995671 20273 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.033840 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:04.011341 20008 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.025s	sys 0.053s
W20260709 12:22:04.047859 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.058008 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:04.038335 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:04.061547 18167 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:04.065006 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:04.005721 20245 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:04.073839 20245 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:04.030357 20248 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.076102 18093 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:04.066154 18172 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:04.077699 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:04.067462 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:04.068383 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:04.027242 20130 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:04.084426 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:04.086023 20248 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:04.088688 18097 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.089994 18169 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:04.046309 20054 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.057763 20241 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:04.030988 20166 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:04.035005 20278 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:04.121254 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:04.108194 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:04.119516 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.123059 18169 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:04.109870 20054 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:04.149935 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:04.150283 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:04.157933 18093 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:04.152297 18092 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.603s	user 0.013s	sys 0.006s
I20260709 12:22:04.159006 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:04.159766 18092 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:04.163975 18019 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:04.170536 18173 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:04.171697 18091 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:04.166594 18168 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:04.176151 18098 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:22:04.178927 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 115500us
I20260709 12:22:04.142536 20120 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:04.148960 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:04.149480 20038 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:04.197275 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:04.192049 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4248570000004293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
I20260709 12:22:04.058796 17921 rpcz_store.cc:276] 0709 12:22:02.626753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.627069 (+   316us) service_pool.cc:224] Handling call
0709 12:22:04.010766 (+1383697us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:02.949636 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:02.950790 (+  1154us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:02.950872 (+    82us) write_op.cc:435] Acquired schema lock
0709 12:22:02.950932 (+    60us) tablet.cc:662] Decoding operations
0709 12:22:02.952071 (+  1139us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:02.952193 (+   122us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:02.952240 (+    47us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:22:02.952911 (+   671us) tablet.cc:701] Row locks acquired
0709 12:22:02.952985 (+    74us) write_op.cc:260] PREPARE: finished
0709 12:22:02.953263 (+   278us) write_op.cc:270] Start()
0709 12:22:02.953479 (+   216us) write_op.cc:276] Timestamp: P: 1783599722953165 usec, L: 0
0709 12:22:02.953512 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 12:22:02.969164 (+ 15652us) log.cc:844] Serialized 1355 byte log entry
0709 12:22:03.396934 (+427770us) op_driver.cc:464] REPLICATION: finished
0709 12:22:03.600629 (+203695us) write_op.cc:301] APPLY: starting
0709 12:22:03.615202 (+ 14573us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:03.630784 (+ 15582us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:03.690535 (+ 59751us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:03.691272 (+   737us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:03.691595 (+   323us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:22:03.691655 (+    60us) write_op.cc:312] APPLY: finished
0709 12:22:03.692015 (+   360us) log.cc:844] Serialized 35 byte log entry
0709 12:22:03.692256 (+   241us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:03.692372 (+   116us) write_op.cc:454] Released schema lock
0709 12:22:03.692418 (+    46us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:03.692996 (+   578us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:03.693028 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":180415,"apply.run_cpu_time_us":3487,"apply.run_wall_time_us":92388,"num_ops":3,"prepare.queue_time_us":146253,"prepare.run_cpu_time_us":42603,"prepare.run_wall_time_us":125634,"raft.queue_time_us":148326,"raft.run_cpu_time_us":25665,"raft.run_wall_time_us":184563,"replication_time_us":431517,"spinlock_wait_cycles":1920,"thread_start_us":339682,"threads_started":4,"wal-append.queue_time_us":195735}]]}
I20260709 12:22:04.149255 20241 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:04.198390 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3834570000003623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
I20260709 12:22:04.206524 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:04.206655 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3753570000003492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260709 12:22:04.208986 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2691570000001775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
W20260709 12:22:04.209380 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0855570000003354s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.188s for a response
W20260709 12:22:04.209774 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0675570000003063s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20260709 12:22:04.210170 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0603570000002946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
W20260709 12:22:04.210564 18064 connection.cc:256] RPC call timeout handler was delayed by -2.04505700000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1925s for a response
W20260709 12:22:04.210954 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0360570000002554s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1935s for a response
W20260709 12:22:04.211351 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9757570000001579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
W20260709 12:22:04.211757 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9091570000005049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
I20260709 12:22:04.175868 20128 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:04.224644 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2116570000002866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2851s for a response
I20260709 12:22:04.127187 20249 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.165727 20284 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:04.134953 20278 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.174546 20196 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.121s
I20260709 12:22:04.370683 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:22:04.231192 18169 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20260709 12:22:04.375803 20128 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.378590 20128 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.197009 20200 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.178s
W20260709 12:22:04.376085 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127438us
I20260709 12:22:04.349062 18806 raft_consensus.cc:2368] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4.
W20260709 12:22:04.344221 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 275968us
I20260709 12:22:04.382345 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:04.391494 20284 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.392266 20284 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:04.391839 17919 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 4, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:22:04.287696 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60708us
I20260709 12:22:04.388218 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:04.305433 17973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73568us
I20260709 12:22:04.231974 20038 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:04.406293 20038 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:04.408483 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.180024 20169 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:04.413110 20249 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:04.200095 20051 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:04.415161 20051 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.265916 18172 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:22:04.425329 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:04.426263 18813 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
W20260709 12:22:04.429092 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 258461us
W20260709 12:22:04.438090 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.402427 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:04.443718 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 278332us
W20260709 12:22:04.446924 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44659.9us
W20260709 12:22:04.449138 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.373674 20096 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.595s	user 0.011s	sys 0.002s
I20260709 12:22:04.460011 20196 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:04.460702 20196 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:04.460179 17919 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:04.461625 20196 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.462247 17919 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:04.464010 17919 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:04.468293 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:04.469663 20169 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:04.470916 20169 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.476492 18137 leader_election.cc:400] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5.
I20260709 12:22:04.478091 20055 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:04.479713 20055 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:04.481931 20054 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.484597 20054 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:04.450855 19910 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.485846 19910 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.488682 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:04.490262 20241 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.496742 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:04.469732 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:04.536049 18814 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.589s	user 0.012s	sys 0.008s
I20260709 12:22:04.536952 18814 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:04.543814 18171 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.656s	user 0.005s	sys 0.018s
I20260709 12:22:04.545344 18171 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:04.465652 17919 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:04.550197 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 632) took 1037 ms. Trace:
W20260709 12:22:04.558549 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 128301us
I20260709 12:22:04.461175 20221 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:22:04.561489 18814 rpcz_store.cc:276] 0709 12:22:03.512267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.543049 (+ 30782us) service_pool.cc:224] Handling call
0709 12:22:03.943130 (+400081us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:04.549992 (+606862us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3947776}
W20260709 12:22:04.564399 18095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.057s	sys 0.064s
I20260709 12:22:04.557289 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:04.574343 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.006s	sys 0.005s
W20260709 12:22:04.574774 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 179084us
I20260709 12:22:04.561851 17919 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:04.597791 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.006s	sys 0.013s
W20260709 12:22:04.601287 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 104782us
W20260709 12:22:04.632748 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 594533us
I20260709 12:22:04.633124 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:04.358608 20166 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.563315 20200 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:04.560042 18018 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.792s	user 0.019s	sys 0.003s
W20260709 12:22:04.575155 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 199129us
I20260709 12:22:04.557446 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:04.639041 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.641319 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 253015us
W20260709 12:22:04.635253 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58919.2us
I20260709 12:22:04.506145 20249 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.096s
I20260709 12:22:04.640125 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:04.579512 18095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.057s	sys 0.064s
I20260709 12:22:04.547310 20287 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:04.458541 20096 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.643989 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:04.544389 20203 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:04.645231 18097 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.536s	user 0.013s	sys 0.004s
I20260709 12:22:04.648641 18016 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:04.644593 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:04.650538 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:04.563690 20245 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:04.579166 20288 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.651456 18097 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:22:04.601970 20293 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:04.655921 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98893.8us
I20260709 12:22:04.644986 20226 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.640760 18018 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:04.650290 20292 raft_consensus.cc:2764] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 12:22:04.668977 18138 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.658s (SENT)
I20260709 12:22:04.659130 17919 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:04.660859 20054 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:04.694532 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50632.1us
I20260709 12:22:04.654695 20130 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:04.669170 20245 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:04.648988 20051 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
W20260709 12:22:04.671909 20116 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.753s	user 0.018s	sys 0.006s
I20260709 12:22:04.665222 20293 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.673031 20200 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:04.696226 18810 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.064s	user 0.016s	sys 0.000s
W20260709 12:22:04.746320 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 69380us
W20260709 12:22:04.759020 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.773252 20294 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.097s
W20260709 12:22:04.774772 20298 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:22:04.795384 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:04.691782 20302 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:04.775660 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113268us
I20260709 12:22:04.775795 17919 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:04.776706 20291 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:04.694398 20076 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:04.779454 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:04.814329 18171 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:04.781955 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:04.785270 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:04.815312 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:04.696674 20286 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:04.796577 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:04.774976 20306 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:22:04.741348 20130 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:04.812270 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.757768 20245 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:04.814476 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:04.786640 18814 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:04.838236 18167 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:04.819727 18138 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.732s (SENT)
I20260709 12:22:04.839071 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:04.678601 20226 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:04.843313 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.821346 18165 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:04.811120 17919 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 3, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 3 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:04.813782 18169 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:04.815342 18168 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:04.831948 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:04.669095 20288 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.768302 20293 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:04.847394 17919 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 9, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 9 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:04.861138 17919 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:04.778301 18810 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:04.866003 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:04.867899 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 182348us
I20260709 12:22:04.869536 20296 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.134s
I20260709 12:22:04.918038 20311 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:04.918416 20296 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:04.918892 20096 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:04.919894 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 474) took 1451 ms. Trace:
I20260709 12:22:04.920527 20226 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:04.920323 18810 rpcz_store.cc:276] 0709 12:22:03.468512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.508358 (+ 39846us) service_pool.cc:224] Handling call
0709 12:22:04.919801 (+1411443us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11008}
W20260709 12:22:04.921330 20248 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.801s	user 0.013s	sys 0.000s
W20260709 12:22:04.870409 20283 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
I20260709 12:22:04.924625 20297 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:04.925098 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:04.758407 20055 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:22:04.926213 20311 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:04.926515 20303 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260709 12:22:04.886117 18173 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:04.926887 20311 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:04.928081 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.929499 20297 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:04.930243 20297 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:04.931543 20310 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:04.932195 20310 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:04.934118 20226 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:04.934810 20301 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
I20260709 12:22:04.934727 20226 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.843609 20245 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:04.936546 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 260871us
I20260709 12:22:04.886441 20200 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.937659 18778 leader_election.cc:400] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5.
I20260709 12:22:04.938165 18778 leader_election.cc:403] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:22:04.889485 20278 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:04.820689 20076 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:04.832285 20150 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.075s	user 0.012s	sys 0.004s
I20260709 12:22:04.835719 20195 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.777580 20116 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.841305 20130 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:04.916158 20001 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.003s	user 0.012s	sys 0.013s
I20260709 12:22:04.945318 18168 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:04.931571 20297 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:04.975240 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 247265us
I20260709 12:22:04.977929 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:04.933439 20310 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.986385 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:04.988289 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:04.991954 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:04.992878 18171 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:04.940719 20076 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.943847 20084 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:04.948925 20248 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:05.015707 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.016320 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.016635 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.948104 20054 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:04.855465 20286 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.992628 20278 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:05.032740 18093 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:22:05.033851 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123475us
W20260709 12:22:05.014218 20067 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.060s	user 0.019s	sys 0.008s
I20260709 12:22:05.038802 20067 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:05.040046 20128 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.002s
I20260709 12:22:05.039474 20054 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.040531 20128 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.012230 20084 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:05.042424 20226 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:05.043321 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:05.042872 20084 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.044246 18096 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:05.020249 20317 raft_consensus.cc:2764] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:05.047113 20286 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:05.002956 20304 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5.
W20260709 12:22:05.017207 20273 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.950s	user 0.014s	sys 0.005s
I20260709 12:22:05.049566 20273 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:05.050446 20278 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:05.051406 18172 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.052078 20278 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:04.947983 20150 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.044248 20084 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:05.056363 20150 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:05.043641 19910 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.557s	user 0.015s	sys 0.000s
I20260709 12:22:04.972952 20001 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:04.950527 20195 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.057710 20196 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:05.072682 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.059360 20117 raft_consensus.cc:993] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:22:05.058530 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:05.057909 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:05.072801 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.083690 18092 raft_consensus.cc:2368] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3.
I20260709 12:22:05.083915 20304 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:05.073134 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:05.064682 20128 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:05.066062 20317 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:22:05.076311 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:05.086803 20317 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:05.086995 18093 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:05.087385 20317 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:05.088666 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.088419 20317 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.089470 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:05.090024 20248 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:05.074052 20055 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.091141 18098 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:05.082674 20117 raft_consensus.cc:1081] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:22:05.093164 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.084786 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.090829 20055 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.069643 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:05.099503 18171 raft_consensus.cc:2376] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4. Re-sending same reply.
I20260709 12:22:05.072260 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:05.085708 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:05.082028 18090 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:05.083722 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:05.129036 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:05.115334 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:05.041579 20067 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:05.140707 18166 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:05.090036 20314 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:05.145774 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.146046 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.151221 18090 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:05.163349 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 343687us
I20260709 12:22:05.165352 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:05.131444 20292 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:05.141945 20241 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:05.150132 20314 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:05.143890 20304 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:05.166225 18092 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:05.174250 18138 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.740s (SENT)
I20260709 12:22:05.176326 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:05.183323 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.170737 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87097.8us
I20260709 12:22:05.203527 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 556) took 1514 ms. Trace:
W20260709 12:22:05.175734 20305 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.320s	user 0.000s	sys 0.004s
I20260709 12:22:05.091223 20248 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.205592 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:05.214766 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.217854 20195 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:05.218984 20116 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:05.223018 20001 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:05.225364 20311 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:05.225042 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:05.282415 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 117734us
I20260709 12:22:05.143323 20128 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 12:22:05.183833 20055 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:05.186908 20318 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.230870 20288 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:05.180207 20292 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.200502 20241 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:05.130887 19910 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:05.234193 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:05.319018 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 533833us
W20260709 12:22:05.235149 20075 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.271s	user 0.009s	sys 0.005s
W20260709 12:22:05.235566 20284 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.800s	user 0.013s	sys 0.000s
I20260709 12:22:05.238066 18168 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:05.239526 20310 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:05.240430 18807 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:05.241219 20314 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.246250 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:05.247128 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:05.254703 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.259649 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.278074 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:05.283366 18138 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.779s (SENT)
W20260709 12:22:05.290279 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93745.1us
W20260709 12:22:05.291559 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116237us
I20260709 12:22:05.294148 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.256126 20221 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 12:22:05.322301 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:05.323123 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:05.325681 18811 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.015s	sys 0.009s
I20260709 12:22:05.328891 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:05.331678 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.329787 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:05.325285 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.001s	sys 0.020s
I20260709 12:22:05.299044 20304 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:05.298503 20150 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.503746 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:05.504644 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:05.467542 17988 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.795s (SENT)
I20260709 12:22:05.611317 18015 raft_consensus.cc:2368] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5.
W20260709 12:22:05.612010 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.950s	user 0.008s	sys 0.025s
I20260709 12:22:05.456038 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:05.433728 18090 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:05.494231 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:05.612089 18091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.008s	sys 0.018s
W20260709 12:22:05.546180 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.611119 18811 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.015s	sys 0.013s
I20260709 12:22:05.498477 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:05.612023 18096 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:05.610600 18016 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:05.315647 20292 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:05.609766 18098 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:05.433941 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:05.610085 18813 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:05.333927 20291 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:05.302083 20318 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.628931 18024 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
I20260709 12:22:05.337103 20116 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.342877 20075 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.641040 18022 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4.
I20260709 12:22:05.601426 19910 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:05.886685 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.999284 18093 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:22:05.682811 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:05.995136 18091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.008s	sys 0.034s
I20260709 12:22:05.682808 18092 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 3
W20260709 12:22:05.614149 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.610983 20054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.034s	sys 0.033s
I20260709 12:22:05.683027 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:05.683795 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44836.2us
W20260709 12:22:05.617875 20196 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.531s	user 0.006s	sys 0.006s
I20260709 12:22:06.037854 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:06.041185 18019 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4.
W20260709 12:22:05.761619 20293 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.842s	user 0.004s	sys 0.006s
W20260709 12:22:05.621919 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 292219us
W20260709 12:22:05.726899 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.994371 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 380933us
I20260709 12:22:05.641791 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:06.061352 18016 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:06.062466 18090 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:05.563304 20245 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.613s	user 0.008s	sys 0.006s
W20260709 12:22:05.984587 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.35253e+06us
I20260709 12:22:06.002799 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:05.828918 20130 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.826s	user 0.015s	sys 0.000s
W20260709 12:22:05.640348 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 172869us
W20260709 12:22:05.991348 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 707970us
I20260709 12:22:05.616490 18020 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:22:06.028981 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.068523 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:06.070219 18021 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:05.629340 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 559) took 1939 ms. Trace:
I20260709 12:22:05.613948 20304 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:06.010186 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 360707us
I20260709 12:22:06.031353 18023 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:05.677444 20302 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.039464 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.851974 20096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.020s	sys 0.015s
I20260709 12:22:05.616397 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:05.746522 20287 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.098292 18138 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.828s (SENT)
I20260709 12:22:06.047518 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2369) took 1021 ms. Trace:
I20260709 12:22:05.632138 20249 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.099045 18094 rpcz_store.cc:276] 0709 12:22:05.026575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.026980 (+   405us) service_pool.cc:224] Handling call
0709 12:22:05.027462 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.137504 (+110042us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.137558 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.145419 (+  7861us) log.cc:844] Serialized 26 byte log entry
0709 12:22:05.432873 (+287454us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:05.434219 (+  1346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.434321 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.047246 (+612925us) raft_consensus.cc:1685] finished
0709 12:22:06.047271 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.047425 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71636,"raft.queue_time_us":91606,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":11878,"thread_start_us":157474,"threads_started":1,"wal-append.queue_time_us":571592}
I20260709 12:22:05.805013 20297 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:05.988942 20314 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:05.664062 20150 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:05.745162 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 561) took 1344 ms. Trace:
I20260709 12:22:05.689991 18806 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:22:06.054832 18099 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:06.056790 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.057057 18096 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:06.145311 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1574) took 1076 ms. Trace:
I20260709 12:22:06.057788 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:05.690873 20325 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:06.063906 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.064652 18812 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:06.065434 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:06.146049 18173 rpcz_store.cc:276] 0709 12:22:05.068830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.074303 (+  5473us) service_pool.cc:224] Handling call
0709 12:22:05.074825 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.075322 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.075355 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.075923 (+   568us) log.cc:844] Serialized 26 byte log entry
0709 12:22:05.434873 (+358950us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:05.434937 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.435033 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.145035 (+710002us) raft_consensus.cc:1685] finished
0709 12:22:06.145062 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.145234 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1303,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":177,"thread_start_us":356401,"threads_started":1,"wal-append.queue_time_us":984983}
W20260709 12:22:05.462893 20308 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:22:06.104768 20321 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:06.061297 20116 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.345083 20284 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.080317 20294 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.080840 20075 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.081305 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:05.684055 20317 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.373s
I20260709 12:22:06.085404 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:06.093794 18092 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:06.175796 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 90502us
W20260709 12:22:06.097220 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.098958 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:05.607214 20241 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:05.344369 20319 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.101135 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:06.189499 18168 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:06.143644 20291 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.106443 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.106653 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:06.197041 18094 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:06.198417 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:06.199237 18094 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:06.199431 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.074685 18811 rpcz_store.cc:276] 0709 12:22:03.689640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.886711 (+197071us) service_pool.cc:224] Handling call
0709 12:22:03.887032 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:03.887276 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:04.136886 (+249610us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.348035 (+211149us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b1800150a00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:22:05.628770 (+1280735us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:05.628830 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:05.628903 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.629055 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.629251 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52160,"prepare.queue_time_us":285275,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":473505,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":144,"spinlock_wait_cycles":6211968,"thread_start_us":1474677,"threads_started":3,"wal-append.queue_time_us":479672,"wal-append.run_cpu_time_us":2895,"wal-append.run_wall_time_us":1205281}
I20260709 12:22:06.104275 20287 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.135712 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:06.135927 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:06.092442 20245 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.101889 20296 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.094348 20304 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.102041 20055 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.137212 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68768us
I20260709 12:22:06.097683 20130 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.139058 20150 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.139686 18807 rpcz_store.cc:276] 0709 12:22:04.400623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.443748 (+ 43125us) service_pool.cc:224] Handling call
0709 12:22:05.208930 (+765182us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.350607 (+141677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.744838 (+394231us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b5400781750. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:05.745061 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1671821696}
W20260709 12:22:06.098418 20096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.025s	sys 0.019s
I20260709 12:22:06.067391 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:06.058481 20293 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.150740 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:06.150539 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:06.078567 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:05.264503 18808 rpcz_store.cc:276] 0709 12:22:03.688717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.883475 (+194758us) service_pool.cc:224] Handling call
0709 12:22:03.883894 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:03.884219 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:04.076505 (+192286us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.096999 (+1020494us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:05.153962 (+ 56963us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:05.154057 (+    95us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:05.154150 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.203194 (+ 49044us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.203408 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70505,"prepare.queue_time_us":203464,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":72,"raft.queue_time_us":348360,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"spinlock_wait_cycles":23569280,"thread_start_us":1125356,"threads_started":3,"wal-append.queue_time_us":549024}
I20260709 12:22:06.159415 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.164302 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:06.168073 20284 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:06.101644 20302 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.171607 18016 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:05.495440 20076 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.192260 18169 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:06.201859 18173 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:06.152716 20321 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:06.107556 20249 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.154103 20325 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:06.501946 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.372507 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2353) took 2313 ms. Trace:
I20260709 12:22:06.217362 18094 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:06.721992 18094 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:06.721786 18099 rpcz_store.cc:276] 0709 12:22:04.058825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.058981 (+   156us) service_pool.cc:224] Handling call
0709 12:22:04.059416 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.145213 (+2085797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.372109 (+226896us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:06.372373 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4588253952}
I20260709 12:22:06.254730 18095 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:06.409083 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 264) took 1077 ms. Trace:
I20260709 12:22:06.724354 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1667) took 1466 ms. Trace:
W20260709 12:22:06.224444 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.492511 18168 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:06.553649 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.725239 18093 rpcz_store.cc:276] 0709 12:22:05.258074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.833785 (+575711us) service_pool.cc:224] Handling call
0709 12:22:06.724286 (+890501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.353899 18090 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:06.720630 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 636233us
W20260709 12:22:06.709357 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.003s	sys 0.008s
W20260709 12:22:06.727715 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:06.727844 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.004s	sys 0.009s
W20260709 12:22:06.642637 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:06.728344 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 540078us
W20260709 12:22:06.646763 20295 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.003s	sys 0.011s
W20260709 12:22:06.729251 20295 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.004s	sys 0.013s
I20260709 12:22:06.723197 18096 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:06.729885 20295 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.730002 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1658) took 1593 ms. Trace:
I20260709 12:22:06.724488 18023 rpcz_store.cc:276] 0709 12:22:05.331546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.341151 (+  9605us) service_pool.cc:224] Handling call
0709 12:22:06.408812 (+1067661us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:06.408990 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":959119616}
I20260709 12:22:06.730495 18096 rpcz_store.cc:276] 0709 12:22:05.136984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.150220 (+1013236us) service_pool.cc:224] Handling call
0709 12:22:06.729903 (+579683us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.730468 20295 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.731418 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.732199 18096 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:06.732685 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.732879 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1676) took 1235 ms. Trace:
W20260709 12:22:06.386812 17988 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 3.163s (SENT) [suppressed 31 similar messages]
I20260709 12:22:06.733454 18023 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:06.726948 18138 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.841s (SENT)
W20260709 12:22:06.734944 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2404143340004339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20260709 12:22:06.735340 18138 connection.cc:256] RPC call timeout handler was delayed by -1.1938148540002658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20260709 12:22:06.735005 20354 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.735829 18138 connection.cc:256] RPC call timeout handler was delayed by -1.1704148540002279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22690000000000002s for a response
W20260709 12:22:06.736200 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0695333580001716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
I20260709 12:22:06.186393 20196 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:06.449417 18015 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:06.733299 18096 rpcz_store.cc:276] 0709 12:22:05.497340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.731572 (+1234232us) service_pool.cc:224] Handling call
0709 12:22:06.732015 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.732629 (+   614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.732820 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.726941 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1672) took 1231 ms. Trace:
I20260709 12:22:06.181787 20297 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:06.723162 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:06.214859 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:06.679342 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.001s	sys 0.010s
I20260709 12:22:06.724083 18815 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:06.406935 20341 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.738498 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 351805us
I20260709 12:22:06.740378 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2396) took 1135 ms. Trace:
W20260709 12:22:06.779459 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 714331us
W20260709 12:22:06.739455 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2393) took 1135 ms (client timeout 1502 ms). Trace:
W20260709 12:22:06.739720 18138 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.055s (SENT)
I20260709 12:22:06.742105 18093 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:06.787622 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:06.788442 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1674) took 1291 ms. Trace:
I20260709 12:22:06.788902 18093 rpcz_store.cc:276] 0709 12:22:05.496610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.741497 (+1244887us) service_pool.cc:224] Handling call
0709 12:22:06.741893 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.788062 (+ 46169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.788347 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.780949 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2383) took 1276 ms. Trace:
I20260709 12:22:06.784875 18019 rpcz_store.cc:276] 0709 12:22:05.605154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.146853 (+541699us) service_pool.cc:224] Handling call
0709 12:22:06.740280 (+593427us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:06.781404 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58355.9us
I20260709 12:22:06.779785 18813 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:06.791945 18023 rpcz_store.cc:276] 0709 12:22:05.504833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.732429 (+1227596us) service_pool.cc:224] Handling call
0709 12:22:06.780839 (+ 48410us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.739871 18090 rpcz_store.cc:276] 0709 12:22:05.495660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.292860 (+797200us) service_pool.cc:224] Handling call
0709 12:22:06.353649 (+ 60789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.726676 (+373027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.726873 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:06.786141 18015 rpcz_store.cc:269] 0709 12:22:05.604024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.663969 (+ 59945us) service_pool.cc:224] Handling call
0709 12:22:06.739376 (+1075407us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:06.795750 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 728401us
I20260709 12:22:06.796356 18096 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:06.796967 18015 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:06.746266 18094 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:06.334779 20331 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
W20260709 12:22:06.252678 20340 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:22:06.810324 18094 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:06.811878 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 12:22:06.370371 20310 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.025s	user 0.015s	sys 0.000s
W20260709 12:22:06.271143 20328 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:06.740067 20200 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.802256 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1677) took 1304 ms. Trace:
I20260709 12:22:06.824455 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:06.805256 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2386) took 1292 ms. Trace:
I20260709 12:22:06.805619 18024 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:06.195853 18092 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:22:06.825646 18015 rpcz_store.cc:276] 0709 12:22:05.515267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.796059 (+1280792us) service_pool.cc:224] Handling call
0709 12:22:06.796571 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.797760 (+  1189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.798247 (+   487us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.826401 18092 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:06.826994 18092 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:06.817909 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:06.786177 20248 raft_consensus.cc:2764] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:22:06.208801 20128 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.810406 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:06.290053 20316 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.314735 20245 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:06.740130 20321 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:06.315994 20336 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.268s	user 0.004s	sys 0.000s
I20260709 12:22:06.290179 20273 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.408229 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 261) took 1087 ms. Trace:
I20260709 12:22:06.844686 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:06.312287 20287 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.786935 20291 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.740166 20169 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:06.626974 20314 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:06.822912 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2385) took 1307 ms. Trace:
I20260709 12:22:06.824436 18096 rpcz_store.cc:276] 0709 12:22:05.497695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.739593 (+1241898us) service_pool.cc:224] Handling call
0709 12:22:06.796140 (+ 56547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.796973 (+   833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.802159 (+  5186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.803901 18098 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:06.854277 18023 rpcz_store.cc:276] 0709 12:22:05.505544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.793566 (+1288022us) service_pool.cc:224] Handling call
0709 12:22:06.811632 (+ 18066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.812418 (+   786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.812758 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.813263 20196 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:06.832144 20200 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.209161 20296 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.863322 20196 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.778200 20055 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.180763 20294 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.866106 20055 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.809302 20297 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.867296 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:06.867707 20055 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:06.867777 20297 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.833331 20295 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.196074 20166 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.627s
I20260709 12:22:06.818605 20284 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:06.871515 20284 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.872014 20284 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:06.847666 20316 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.874967 18813 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:06.876709 20316 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.831700 20310 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.838653 20128 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.852177 20273 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.823904 20349 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.838045 20292 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.797s	user 0.008s	sys 0.008s
I20260709 12:22:06.799839 20341 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.857539 20248 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:06.858371 20245 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:06.859278 20314 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.859964 20293 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:06.861655 20130 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.803048 20354 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.910633 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2365) took 1899 ms. Trace:
I20260709 12:22:06.862026 18810 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:22:06.862141 20287 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:06.863178 20291 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.848767 18015 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:06.913483 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2387) took 1315 ms. Trace:
I20260709 12:22:06.913944 18015 rpcz_store.cc:276] 0709 12:22:05.601275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.827124 (+1225849us) service_pool.cc:224] Handling call
0709 12:22:06.848476 (+ 21352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.913067 (+ 64591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.913382 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:06.839468 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:06.879346 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.847711 18018 rpcz_store.cc:276] 0709 12:22:05.320929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.332305 (+ 11376us) service_pool.cc:224] Handling call
0709 12:22:05.612378 (+280073us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.683453 (+ 71075us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.683507 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.684140 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.172349 (+488209us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:06.178138 (+  5789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.193510 (+ 15372us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.402988 (+209478us) raft_consensus.cc:1685] finished
0709 12:22:06.407822 (+  4834us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.408096 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"thread_start_us":460309,"threads_started":1,"wal-append.queue_time_us":580124}
I20260709 12:22:06.918710 20287 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.920027 20296 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:06.904527 20245 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.911207 18091 rpcz_store.cc:276] 0709 12:22:05.010914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.012746 (+  1832us) service_pool.cc:224] Handling call
0709 12:22:05.026826 (+ 14080us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.030224 (+  3398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.030277 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.074592 (+ 44315us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b4c004c60c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14983a20 00007f6c14994a29 00007f6c1495a644 00007f6c14981f2d 00007f6c14981b93 00007f6c166b218b 00007f6c166b2cb5 00007f6c166b4004 00007f6c166b3729 00007f6c166b8d96 00007f6c166adadb
0709 12:22:05.075863 (+  1271us) log.cc:844] Serialized 556 byte log entry
0709 12:22:06.012252 (+936389us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:06.012313 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.012415 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.906735 (+894320us) raft_consensus.cc:1685] finished
0709 12:22:06.906760 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.910511 (+  3751us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95840640,"thread_start_us":935936,"threads_started":1,"wal-append.queue_time_us":1084072}
I20260709 12:22:06.895351 20347 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260709 12:22:06.849977 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.893991 18815 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:06.202085 20319 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.890156 18092 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.916499 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1668) took 1653 ms. Trace:
I20260709 12:22:06.890318 18094 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:22:06.889870 18023 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:06.916282 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.901328 20292 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.896765 20310 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.900213 20349 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:06.972441 18015 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260709 12:22:07.206866 20346 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:22:07.212096 20302 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:07.149567 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 363386us
I20260709 12:22:07.213591 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2388) took 1611 ms. Trace:
I20260709 12:22:07.214013 18015 rpcz_store.cc:276] 0709 12:22:05.601860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.920525 (+1318665us) service_pool.cc:224] Handling call
0709 12:22:06.972064 (+ 51539us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.213192 (+241128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.213477 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:07.215675 18019 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3. Re-sending same reply.
W20260709 12:22:07.205622 20350 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 12:22:07.217451 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.218315 20296 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.218861 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2384) took 1713 ms. Trace:
I20260709 12:22:07.219614 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:07.219457 18019 rpcz_store.cc:276] 0709 12:22:05.505190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.791114 (+1285924us) service_pool.cc:224] Handling call
0709 12:22:07.218584 (+427470us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:07.218765 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880215424}
I20260709 12:22:07.221040 18094 pending_rounds.cc:85] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 1
I20260709 12:22:07.221122 20169 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:07.221455 18094 pending_rounds.cc:107] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:22:07.221726 20169 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.023722 18024 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:07.202412 20355 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.155176 17988 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.537s (SENT)
W20260709 12:22:07.146111 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.976915 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 564) took 1651 ms. Trace:
I20260709 12:22:06.924667 20293 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.008780 20294 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:06.979897 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 155540us
W20260709 12:22:07.515543 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 300714us
W20260709 12:22:07.210548 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 392760us
W20260709 12:22:07.134549 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:07.517179 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.517980 18094 raft_consensus.cc:2889] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:07.518198 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:07.222890 20169 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:07.061905 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 565) took 1736 ms. Trace:
W20260709 12:22:07.521476 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 366379us
I20260709 12:22:07.003680 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2371) took 1933 ms. Trace:
W20260709 12:22:07.224684 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:07.504541 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 764535us
I20260709 12:22:07.460004 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:07.389925 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1582) took 1317 ms. Trace:
I20260709 12:22:06.951457 19910 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.521155 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 482) took 1369 ms (client timeout 1665 ms). Trace:
I20260709 12:22:07.526886 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:07.520452 18098 rpcz_store.cc:276] 0709 12:22:05.263347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.999394 (+736047us) service_pool.cc:224] Handling call
0709 12:22:06.913760 (+914366us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b540025e254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:06.916426 (+  2666us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1672608512}
I20260709 12:22:07.218904 20296 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:07.222059 20359 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.348s
W20260709 12:22:07.528652 17988 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.592s (SENT)
W20260709 12:22:07.535799 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 712633us
I20260709 12:22:07.916889 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:07.597769 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.530109 20293 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:06.966617 20297 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:07.050952 20338 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.998s	user 0.005s	sys 0.013s
W20260709 12:22:07.929903 18815 rpcz_store.cc:269] 0709 12:22:06.151337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.151573 (+   236us) service_pool.cc:224] Handling call
0709 12:22:07.521069 (+1369496us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:07.516496 18810 rpcz_store.cc:276] 0709 12:22:05.325557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.332425 (+  6868us) service_pool.cc:224] Handling call
0709 12:22:05.686265 (+353840us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:06.829425 (+1143160us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592f8c 00007f6c13592ef6 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:06.837422 (+  7997us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.920470 (+ 83048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.976798 (+ 56328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":114635392}
I20260709 12:22:07.503134 18023 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:06.967402 20245 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:07.553121 20241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.006s	sys 0.010s
I20260709 12:22:07.531809 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1670) took 2215 ms. Trace:
I20260709 12:22:07.052541 20291 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.946543 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1585) took 1612 ms. Trace:
I20260709 12:22:07.549568 20348 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:07.944258 20360 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.389001 20304 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.586s	user 0.012s	sys 0.000s
I20260709 12:22:07.948326 20360 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:07.948680 20304 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:07.941519 20317 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:07.951737 20304 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:07.514256 20319 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.570068 20318 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:07.171957 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2389) took 1520 ms. Trace:
I20260709 12:22:07.050747 20341 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.061434 20273 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:07.056473 20128 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:07.054602 20076 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:07.604139 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:07.528267 18814 rpcz_store.cc:276] 0709 12:22:05.325872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.336424 (+ 10552us) service_pool.cc:224] Handling call
0709 12:22:05.683917 (+347493us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:06.015605 (+331688us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.015931 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.015972 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.052306 (+ 36334us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.547470 (+495164us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:06.547547 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.547675 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.011724 (+464049us) raft_consensus.cc:1685] finished
0709 12:22:07.011756 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.061769 (+ 50013us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2599808,"thread_start_us":494551,"threads_started":1,"wal-append.queue_time_us":825690}
I20260709 12:22:07.529016 18095 rpcz_store.cc:276] 0709 12:22:05.070271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.089744 (+ 19473us) service_pool.cc:224] Handling call
0709 12:22:05.104920 (+ 15176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.781136 (+1676216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.003241 (+222105us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400215054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:07.003567 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2643520768}
W20260709 12:22:07.929962 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.932016 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:07.916574 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.964128 18019 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 3. Ops: []
I20260709 12:22:07.941550 19910 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:07.965087 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2395) took 2360 ms. Trace:
I20260709 12:22:07.934038 18092 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:07.965504 18019 rpcz_store.cc:276] 0709 12:22:05.604808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.221015 (+1616207us) service_pool.cc:224] Handling call
0709 12:22:07.963705 (+742690us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:07.963926 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.964783 (+   857us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.965020 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":692879616}
I20260709 12:22:07.937268 20379 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.967818 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:07.939198 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 410559us
I20260709 12:22:07.967581 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:06.997145 20352 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:22:07.974066 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:45040 (request call id 2376) took 2820 ms. Trace:
I20260709 12:22:07.941921 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2409) took 1088 ms. Trace:
W20260709 12:22:07.647163 20321 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.768s	user 0.012s	sys 0.001s
I20260709 12:22:07.944310 18020 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 4. Ops: []
W20260709 12:22:07.942322 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.054219 20314 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.763312 20287 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.945502 18094 rpcz_store.cc:276] 0709 12:22:05.316683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.216672 (+899989us) service_pool.cc:224] Handling call
0709 12:22:06.217172 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.531384 (+1314212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.531718 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:07.009845 20325 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.947400 20348 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.605513 20310 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:07.388163 20200 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:07.520710 20249 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:07.925612 20293 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.007825 20354 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.541929 20116 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:07.604278 20358 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:07.919772 18171 rpcz_store.cc:276] 0709 12:22:06.072255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.072416 (+   161us) service_pool.cc:224] Handling call
0709 12:22:06.073033 (+   617us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.086284 (+ 13251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.086315 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.102591 (+ 16276us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.458856 (+356265us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:06.458924 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.459051 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.206156 (+747105us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:07.212936 (+  6780us) raft_consensus.cc:1685] finished
0709 12:22:07.212961 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.389771 (+176810us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12591,"raft.queue_time_us":19443,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":5482,"spinlock_wait_cycles":48065280,"thread_start_us":321312,"threads_started":1,"wal-append.queue_time_us":777872}
W20260709 12:22:07.006502 20278 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.944s	user 0.007s	sys 0.025s
I20260709 12:22:07.968896 18098 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:22:07.969499 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2390) took 2366 ms (client timeout 2074 ms). Trace:
W20260709 12:22:08.018738 18024 rpcz_store.cc:269] 0709 12:22:05.602818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.150463 (+547645us) service_pool.cc:224] Handling call
0709 12:22:07.969292 (+1818829us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b540024e354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:07.969420 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191095552}
I20260709 12:22:08.020185 20341 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:07.971534 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:07.968912 18021 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:07.692860 20362 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:07.976231 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2378) took 1798 ms. Trace:
I20260709 12:22:08.022277 18099 rpcz_store.cc:276] 0709 12:22:06.177892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.798742 (+620850us) service_pool.cc:224] Handling call
0709 12:22:06.916408 (+117666us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.916681 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:07.224790 (+308109us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.930620 (+705830us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:07.930690 (+    70us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:07.930749 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.930831 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.975933 (+ 45102us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b540025e250. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:07.976134 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":629148,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":96,"raft.queue_time_us":379081,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":10575,"spinlock_wait_cycles":269419904,"thread_start_us":598037,"threads_started":2,"wal-append.queue_time_us":697903}
W20260709 12:22:07.952920 20241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.089s	user 0.023s	sys 0.015s
I20260709 12:22:07.974563 18092 rpcz_store.cc:276] 0709 12:22:05.146712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.176391 (+1029679us) service_pool.cc:224] Handling call
0709 12:22:07.966642 (+1790251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:07.968621 18022 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:22:07.980247 18023 rpcz_store.cc:276] 0709 12:22:06.853488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.866651 (+ 13163us) service_pool.cc:224] Handling call
0709 12:22:07.941847 (+1075196us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:07.979422 17988 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.744s (SENT)
I20260709 12:22:08.036193 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:08.036509 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.037261 18023 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:08.038268 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:07.985798 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.987286 18095 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:07.988080 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:07.990021 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 380986us
I20260709 12:22:08.039326 18019 raft_consensus.cc:2376] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3. Re-sending same reply.
I20260709 12:22:07.990517 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1673) took 2494 ms. Trace:
I20260709 12:22:07.992053 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:08.041275 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:08.035573 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2400) took 2130 ms (client timeout 2243 ms). Trace:
I20260709 12:22:07.991616 18017 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:07.996682 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:07.968007 20291 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:07.999181 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2394) took 2394 ms. Trace:
I20260709 12:22:08.044582 18020 rpcz_store.cc:276] 0709 12:22:05.604429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.118871 (+1514442us) service_pool.cc:224] Handling call
0709 12:22:07.944081 (+825210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.983626 (+ 39545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.998938 (+ 15312us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b540004c4d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:07.999104 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":964141184}
I20260709 12:22:07.999445 18090 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:08.002035 18094 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:08.046126 18094 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:08.041532 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.046960 18094 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:07.979734 20297 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.047760 18093 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:08.048033 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:08.004071 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:08.021971 20278 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.968s	user 0.026s	sys 0.049s
I20260709 12:22:07.998137 20348 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:07.947108 18164 rpcz_store.cc:276] 0709 12:22:06.333985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.334191 (+   206us) service_pool.cc:224] Handling call
0709 12:22:06.334675 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.350914 (+ 16239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.350955 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.839012 (+488057us) log.cc:844] Serialized 26 byte log entry
0709 12:22:07.222228 (+383216us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:07.222283 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.222372 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.917332 (+694960us) raft_consensus.cc:1685] finished
0709 12:22:07.929296 (+ 11964us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.944438 (+ 15142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":470,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":365789,"threads_started":1,"wal-append.queue_time_us":845419}
I20260709 12:22:08.014703 20380 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:07.961550 18018 rpcz_store.cc:276] 0709 12:22:05.602384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.919054 (+1316670us) service_pool.cc:224] Handling call
0709 12:22:06.919500 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.924115 (+  4615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.945748 (+ 21633us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:06.968858 (+ 23110us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:06.973907 (+  5049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.012152 (+ 38245us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.123087 (+110935us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":395,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":141}
I20260709 12:22:08.069011 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.040585 20358 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.070745 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.071416 20364 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:08.071931 20364 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:08.072261 20358 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:08.072703 18018 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:08.072836 20364 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:07.990562 20321 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:08.006778 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 480006us
I20260709 12:22:08.020115 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:07.970711 18016 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:08.023770 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.024466 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2392) took 2420 ms. Trace:
W20260709 12:22:08.015404 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.034215 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.967468 20347 consensus_queue.cc:1243] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:08.041217 18168 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:08.041198 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:07.435720 20355 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:07.985199 20318 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 12:22:08.047051 18090 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:08.040032 18097 rpcz_store.cc:276] 0709 12:22:05.496222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.317400 (+821178us) service_pool.cc:224] Handling call
0709 12:22:06.333691 (+ 16291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.367261 (+ 33570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.367299 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.786854 (+419555us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.978751 (+191897us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:07.904487 (+925736us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.990140 (+ 85653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.990230 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.990274 (+    44us) raft_consensus.cc:1685] finished
0709 12:22:07.990289 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.990449 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103700,"prepare.queue_time_us":620288,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":85377,"raft.run_cpu_time_us":514,"raft.run_wall_time_us":22618,"thread_start_us":513483,"threads_started":2,"wal-append.queue_time_us":1068426}
W20260709 12:22:08.005259 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73335.1us
W20260709 12:22:08.042232 18022 rpcz_store.cc:269] 0709 12:22:05.905850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.001246 (+1095396us) service_pool.cc:224] Handling call
0709 12:22:07.967655 (+966409us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:08.035490 (+ 67835us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":692621568}
I20260709 12:22:07.988984 20319 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:22:08.052925 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:08.053452 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2407) took 1864 ms (client timeout 1899 ms). Trace:
I20260709 12:22:08.055756 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:08.056803 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 87259.9us
I20260709 12:22:08.061865 18094 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.061730 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:08.062258 18019 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:08.058470 20297 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.063872 20380 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.046778 20362 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:08.066179 20296 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:08.067615 20284 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.195s	user 0.016s	sys 0.000s
I20260709 12:22:08.058929 20304 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:08.073493 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2403) took 1973 ms. Trace:
I20260709 12:22:08.027716 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.035852 20292 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:07.982967 20379 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:08.074513 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.473567 18024 raft_consensus.cc:2376] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
I20260709 12:22:08.075870 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:08.476544 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2391) took 2873 ms. Trace:
W20260709 12:22:08.224650 18778 connection.cc:256] RPC call timeout handler was delayed by -3.0663788560000285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20260709 12:22:08.485988 18778 connection.cc:256] RPC call timeout handler was delayed by -3.0501788560000023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
W20260709 12:22:08.486706 18778 connection.cc:256] RPC call timeout handler was delayed by -2.8791788560001805s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21640000000000004s for a response
W20260709 12:22:08.487233 18778 connection.cc:256] RPC call timeout handler was delayed by -2.8053788560000612s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
I20260709 12:22:08.487543 18092 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:08.474606 18021 rpcz_store.cc:276] 0709 12:22:05.603606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.052939 (+1449333us) service_pool.cc:224] Handling call
0709 12:22:07.968438 (+915499us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:07.968664 (+   226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.022098 (+ 53434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.024373 (+  2275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":717533440}
I20260709 12:22:08.410210 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.489387 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:22:08.104318 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.075567 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.480803 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2397) took 2875 ms. Trace:
W20260709 12:22:08.492092 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 411533us
W20260709 12:22:08.482002 18017 rpcz_store.cc:269] 0709 12:22:06.189309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.929348 (+740039us) service_pool.cc:224] Handling call
0709 12:22:07.931835 (+1002487us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592e9c 00007f6c13592a5b 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:08.053263 (+121428us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:08.053380 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66608128}
W20260709 12:22:08.495299 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 419551us
I20260709 12:22:08.495012 18019 rpcz_store.cc:276] 0709 12:22:05.605559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.051203 (+2445644us) service_pool.cc:224] Handling call
0709 12:22:08.061993 (+ 10790us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.480336 (+418343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.480702 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:08.484655 18016 rpcz_store.cc:276] 0709 12:22:05.603235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.000432 (+1397197us) service_pool.cc:224] Handling call
0709 12:22:07.970303 (+969871us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:07.970509 (+   206us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.476219 (+505710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.476464 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693385600}
I20260709 12:22:08.497980 20353 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:08.492322 20116 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.498529 18807 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:08.527330 20116 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.499065 20366 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:08.528679 20366 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:08.492506 20221 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.529804 20366 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.530706 18098 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:08.530483 20221 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.499476 20365 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.503266 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:08.475737 18099 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:08.532876 18098 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:08.533962 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:08.534698 18099 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:08.504765 20360 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:08.541962 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.505913 18024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:22:08.508175 18017 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:22:08.510015 20375 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.497s	user 0.000s	sys 0.001s
I20260709 12:22:08.509094 20362 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:08.510011 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 457148us
I20260709 12:22:08.558248 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:08.510785 18021 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:08.494580 20284 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.474913 18018 rpcz_store.cc:276] 0709 12:22:06.100213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.071887 (+1971674us) service_pool.cc:224] Handling call
0709 12:22:08.072290 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.073189 (+   899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.073429 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:08.222334 20245 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.514143 18022 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:08.576819 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2410) took 1673 ms. Trace:
I20260709 12:22:08.514983 19910 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:08.509707 18094 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 11. (term mismatch)
I20260709 12:22:08.579291 18094 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 10
I20260709 12:22:08.579684 18094 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 9.11
I20260709 12:22:08.580029 18094 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:08.060238 20341 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:08.057125 20314 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:08.057464 20291 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.521649 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.522732 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.592320 18814 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:08.554996 20296 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.219606 20249 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.508129 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2405) took 2363 ms. Trace:
I20260709 12:22:08.504765 20297 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:08.472584 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.537129 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:08.542867 18098 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:22:08.602571 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.606061 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.543614 18024 raft_consensus.cc:606] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:08.545953 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:08.607211 18016 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:08.555394 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.556263 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2406) took 2398 ms. Trace:
I20260709 12:22:08.609017 18017 rpcz_store.cc:276] 0709 12:22:06.144747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.507085 (+2362338us) service_pool.cc:224] Handling call
0709 12:22:08.507527 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.542937 (+ 35410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.543188 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16256}
I20260709 12:22:08.436625 20364 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:08.559211 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:08.559950 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2408) took 1745 ms. Trace:
I20260709 12:22:08.620680 18021 rpcz_store.cc:276] 0709 12:22:06.814540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.510098 (+1695558us) service_pool.cc:224] Handling call
0709 12:22:08.510518 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.559583 (+ 49065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.559873 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:08.570413 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:08.518217 20380 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:08.577494 18022 rpcz_store.cc:276] 0709 12:22:06.906394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.513324 (+1606930us) service_pool.cc:224] Handling call
0709 12:22:08.513743 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.576356 (+ 62613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.576715 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:08.581796 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.582711 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.585667 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.586611 20116 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:08.587037 20341 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.588482 20349 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:08.571903 20278 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.589094 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:08.589903 20221 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:08.590298 20284 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:08.539479 20384 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:08.591248 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2399) took 2767 ms. Trace:
W20260709 12:22:08.593518 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.525799 20355 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:08.594587 18813 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.122s	sys 0.108s
W20260709 12:22:08.680289 18813 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.123s	sys 0.109s
I20260709 12:22:08.572261 20370 raft_consensus.cc:2764] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:08.603446 18099 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:08.217723 20293 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.602768 18015 rpcz_store.cc:276] 0709 12:22:06.144314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.507037 (+2362723us) service_pool.cc:224] Handling call
0709 12:22:08.507506 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.507754 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.507780 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:08.507825 (+    45us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:08.507858 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.507920 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.508056 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32256}
I20260709 12:22:08.623014 18093 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:08.642635 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.698984 18099 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:08.627141 18095 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:08.629463 18019 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:08.629443 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.590610 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:08.651923 20116 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:08.650297 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.718066 18815 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:08.649621 18021 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:08.619601 20296 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.654438 18016 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:08.598495 20385 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:08.600806 20200 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.649161 18810 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:08.668586 18023 rpcz_store.cc:276] 0709 12:22:05.824474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.054935 (+2230461us) service_pool.cc:224] Handling call
0709 12:22:08.055374 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.062171 (+  6797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.062201 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:08.062888 (+   687us) log.cc:844] Serialized 26 byte log entry
0709 12:22:08.475563 (+412675us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:08.475676 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.475928 (+   252us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.591022 (+115094us) raft_consensus.cc:1685] finished
0709 12:22:08.591042 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.591184 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2339,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":180382,"threads_started":1,"wal-append.queue_time_us":516185}
I20260709 12:22:08.671088 18022 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:08.742974 18095 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:08.716120 18809 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:08.721239 18808 raft_consensus.cc:2376] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4. Re-sending same reply.
I20260709 12:22:08.750967 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.740119 20385 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.754274 20385 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:08.729031 20116 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.681968 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:08.703325 20293 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.756857 20116 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:08.757776 20321 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:08.678073 17987 connection.cc:256] RPC call timeout handler was delayed by -1.7808045550000315s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
I20260709 12:22:08.688151 18097 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:22:08.765220 17987 connection.cc:256] RPC call timeout handler was delayed by -1.7691045550000126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1537s for a response
I20260709 12:22:08.692677 18093 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:08.626281 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.539317 20365 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.707461 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2411) took 1761 ms. Trace:
I20260709 12:22:08.785271 18019 rpcz_store.cc:276] 0709 12:22:06.945884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.593103 (+1647219us) service_pool.cc:224] Handling call
0709 12:22:08.629174 (+ 36071us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.707068 (+ 77894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.707377 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:08.745041 18806 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:22:08.663473 20349 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:08.667485 18807 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:08.720427 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.652434 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.681581 20310 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.658437 20278 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.638151 20360 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260709 12:22:08.625888 20249 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.684818 20372 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:08.731709 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.685629 20361 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:08.728138 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:08.696417 20384 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:08.737299 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.739533 18016 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:08.645444 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.747120 20200 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.628444 20314 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.683326 18092 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:08.769793 17987 connection.cc:256] RPC call timeout handler was delayed by -1.720504554999934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
W20260709 12:22:08.853160 17987 connection.cc:256] RPC call timeout handler was delayed by -1.554904555000121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260709 12:22:08.714995 18099 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:08.854007 18099 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:08.853763 17987 connection.cc:256] RPC call timeout handler was delayed by -1.5369045550000919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
W20260709 12:22:08.856561 17987 connection.cc:256] RPC call timeout handler was delayed by -1.5099045550000483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20260709 12:22:08.857014 17987 connection.cc:256] RPC call timeout handler was delayed by -1.4289045549999173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20260709 12:22:08.857395 17987 connection.cc:256] RPC call timeout handler was delayed by -1.2012045550000039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
W20260709 12:22:08.857864 17987 connection.cc:256] RPC call timeout handler was delayed by -1.169704554999953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
W20260709 12:22:08.858254 17987 connection.cc:256] RPC call timeout handler was delayed by -1.1328045549998933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22440000000000004s for a response
W20260709 12:22:08.858652 17987 connection.cc:256] RPC call timeout handler was delayed by -1.0950045549998322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22860000000000003s for a response
W20260709 12:22:08.859050 17987 connection.cc:256] RPC call timeout handler was delayed by -1.023904555000172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23650000000000004s for a response
W20260709 12:22:08.860065 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 289784us
I20260709 12:22:08.817068 20381 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:08.787380 18093 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.829074 18021 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:08.829948 18814 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:08.820441 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 199025us
I20260709 12:22:08.868033 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.831425 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:08.834733 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.754277 20348 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.689s	user 0.008s	sys 0.003s
W20260709 12:22:08.825248 20392 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:22:08.877707 18093 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:08.846259 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 118305us
I20260709 12:22:08.867667 18099 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.863301 17987 leader_election.cc:400] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3.
W20260709 12:22:08.864632 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 233906us
I20260709 12:22:08.860780 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:08.882206 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:08.882478 18095 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:08.846151 20245 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:08.855146 20321 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:08.882362 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:08.883051 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.870818 18090 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:08.898990 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:08.869199 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 199538us
I20260709 12:22:08.870435 18812 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:22:08.894891 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 127736us
I20260709 12:22:08.902562 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:08.914060 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.830890 20349 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.927088 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.931340 18099 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:08.932225 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69163.3us
W20260709 12:22:08.932294 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51006.5us
I20260709 12:22:08.933238 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:08.934887 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.935091 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:08.936435 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:08.939630 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.868772 20381 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:08.941655 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:08.947172 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65090.2us
I20260709 12:22:08.851007 20310 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.954413 18015 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:22:08.958917 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:08.893342 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.965684 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:08.965755 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:08.896041 20293 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:08.971873 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70442.3us
I20260709 12:22:08.972417 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:08.975066 18015 raft_consensus.cc:606] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:08.975566 18015 raft_consensus.cc:614] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:08.926643 20347 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:08.977732 18814 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:08.870043 20314 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:08.926232 20291 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:08.964845 18093 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:22:08.981138 18093 pending_rounds.cc:85] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:22:08.986541 18093 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:22:08.941926 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:08.985178 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52045.3us
I20260709 12:22:08.948122 20245 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.926818 20297 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:22:08.937682 20321 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:08.987154 18093 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:08.877715 20348 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.990888 18814 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:08.859614 20372 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.860440 20361 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:08.982227 20396 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.973935 20241 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:08.995486 18093 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:22:08.980078 20347 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.979770 20366 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:22:08.993688 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:08.998148 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:08.966725 20130 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:22:09.002297 18093 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:08.987848 20400 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:09.004909 20321 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:08.996977 20390 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.267s	user 0.000s	sys 0.002s
I20260709 12:22:08.976778 19910 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:22:08.992221 20384 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.006517 20314 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:09.150851 20398 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.243s
I20260709 12:22:09.311887 20412 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:09.151508 20278 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:09.319252 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 396387us
I20260709 12:22:09.068959 20406 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:09.322353 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.974s (SENT)
I20260709 12:22:09.085203 20405 raft_consensus.cc:2764] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:09.314690 20200 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:09.091454 20310 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:09.031104 20365 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:09.324007 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.246s (SENT)
W20260709 12:22:09.031462 20169 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.474s	user 0.012s	sys 0.005s
I20260709 12:22:09.325299 20372 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:09.325840 20348 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:09.325412 20169 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:09.006893 20381 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.328140 20169 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:09.328325 20381 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:09.197216 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 315092us
I20260709 12:22:09.023334 18091 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:09.313359 20400 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:08.998116 20245 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:09.310127 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 287259us
I20260709 12:22:09.321027 18814 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:09.208746 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.318934 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 344398us
I20260709 12:22:09.330726 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:09.004953 20394 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 12:22:09.061841 20393 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:22:09.341941 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:09.347437 20400 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:09.348938 20400 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.349200 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:09.349371 20385 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.006s
I20260709 12:22:09.197574 20249 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:09.220635 20383 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.484s	user 0.004s	sys 0.001s
I20260709 12:22:09.350057 19910 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:09.349936 20385 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.351264 19910 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:09.354303 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2401) took 3331 ms. Trace:
W20260709 12:22:09.354579 20402 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:09.088482 20349 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:09.355273 20385 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:09.157310 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:09.036743 20397 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.313493 20347 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:09.351938 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:09.353474 18808 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:09.368862 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.310983 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:09.371721 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:09.372040 20379 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:09.372586 18815 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:09.356226 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 486) took 1836 ms. Trace:
I20260709 12:22:09.376266 18813 rpcz_store.cc:276] 0709 12:22:07.519958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.520179 (+   221us) service_pool.cc:224] Handling call
0709 12:22:07.544828 (+ 24649us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.874983 (+330155us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:07.875037 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:07.946274 (+ 71237us) log.cc:844] Serialized 26 byte log entry
0709 12:22:08.680945 (+734671us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:08.681019 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.681159 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.219576 (+538417us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:09.355909 (+136333us) raft_consensus.cc:1685] finished
0709 12:22:09.355933 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.356155 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26874,"raft.queue_time_us":391642,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":146,"spinlock_wait_cycles":7423872,"thread_start_us":1036650,"threads_started":2,"wal-append.queue_time_us":750729}
W20260709 12:22:09.357563 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 363956us
I20260709 12:22:09.378753 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:09.379572 18018 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.380249 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:09.320429 20412 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:09.381785 20412 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.383616 20412 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:09.387748 20361 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:09.388736 20341 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:09.389292 20341 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:09.389712 20341 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:09.389570 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:09.390322 18016 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:09.394480 18024 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:09.397354 18064 leader_election.cc:400] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3.
I20260709 12:22:09.397941 18064 leader_election.cc:403] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:09.359251 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:09.354770 18020 rpcz_store.cc:276] 0709 12:22:06.022533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.058207 (+2035674us) service_pool.cc:224] Handling call
0709 12:22:08.229679 (+171472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.411772 (+182093us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.411822 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:08.479517 (+ 67695us) log.cc:844] Serialized 26 byte log entry
0709 12:22:08.769999 (+290482us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:08.770063 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.770181 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.354003 (+583822us) raft_consensus.cc:1685] finished
0709 12:22:09.354040 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.354232 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":243267,"threads_started":1,"wal-append.queue_time_us":318247}
I20260709 12:22:09.360730 20399 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:09.412410 20399 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:09.412864 20399 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:09.413785 20399 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:09.361275 18810 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.611s	user 0.008s	sys 0.004s
I20260709 12:22:09.363904 20348 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:09.024215 20396 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:09.365110 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.365507 20293 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.400413 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:09.401412 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:09.402619 20398 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:22:09.402822 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72200.9us
I20260709 12:22:09.000856 20296 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:09.406306 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47188us
I20260709 12:22:09.347505 20245 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:09.410537 18016 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:09.498329 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:09.426347 20399 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.438434 18024 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:22:09.439153 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87340.5us
I20260709 12:22:09.457358 18815 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:09.459136 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2395) took 1437 ms (client timeout 1914 ms). Trace:
I20260709 12:22:09.441045 20349 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:22:09.425076 18810 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:09.484254 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:09.489923 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.445626 20364 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.803s	user 0.015s	sys 0.000s
I20260709 12:22:09.430573 20407 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3.
I20260709 12:22:09.432868 20348 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:09.467665 20398 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:09.463811 20361 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260709 12:22:09.469319 20293 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.427927 20130 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:09.502775 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.590312 18015 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:09.517307 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:09.517554 18091 rpcz_store.cc:269] 0709 12:22:08.021336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.050169 (+ 28833us) service_pool.cc:224] Handling call
0709 12:22:09.435027 (+1384858us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:09.459042 (+ 24015us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1214440704}
I20260709 12:22:09.518462 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:09.598668 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80374us
I20260709 12:22:09.602397 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:09.521037 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:09.533315 17989 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.545s (SENT)
W20260709 12:22:09.605929 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72651.5us
W20260709 12:22:09.534790 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50727.9us
I20260709 12:22:09.592625 18815 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:09.547912 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:09.616819 18813 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:09.436393 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:09.609349 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.684s (SENT)
I20260709 12:22:09.629977 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:09.587826 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.003s	sys 0.006s
I20260709 12:22:09.519630 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:09.631156 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.004s	sys 0.006s
W20260709 12:22:09.632695 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.712s (SENT)
I20260709 12:22:09.561445 20397 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:09.633740 20310 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:09.634740 20304 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:09.634986 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 115521us
W20260709 12:22:09.631541 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 645140us
I20260709 12:22:09.623813 18815 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:09.636732 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:09.502174 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.529747 20360 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.011s	sys 0.022s
I20260709 12:22:09.641402 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:09.642247 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 205924us
W20260709 12:22:09.642760 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 97853.9us
I20260709 12:22:09.548216 18809 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:09.647454 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:09.542444 20348 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:09.658231 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49859us
I20260709 12:22:09.670747 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:09.659679 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.521036 20379 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:09.676132 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:09.677531 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:09.640336 20249 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:09.586426 18807 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.745s	user 0.014s	sys 0.000s
I20260709 12:22:09.545733 20398 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:09.679184 20278 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:09.545707 20422 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:09.681427 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 177836us
I20260709 12:22:09.628820 19910 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:09.738991 19910 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:09.740168 19910 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.669575 20310 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:09.742054 20395 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:22:09.656332 20360 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.012s	sys 0.022s
I20260709 12:22:09.638983 20362 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:09.591660 20130 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.696743 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.685199 20358 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:09.700554 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:22:09.700881 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.700783 20249 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:09.701790 18810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 488) took 1656 ms (client timeout 1503 ms). Trace:
I20260709 12:22:09.703145 18811 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:09.703361 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:09.704738 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.705592 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:09.638039 20361 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 12:22:09.678280 20200 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:22:09.840548 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:09.667634 20408 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:22:09.656334 20424 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:09.710007 18806 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:09.667862 20400 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 12:22:09.712715 18022 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20260709 12:22:09.748031 20169 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:09.715065 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.754554 20407 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.705605 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.719972 18812 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:22:09.686180 18807 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:09.681942 20418 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.166s
W20260709 12:22:09.743407 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.733335 20422 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.873085 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.873525 18814 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:22:09.753075 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.754213 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.818837 18168 raft_consensus.cc:606] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:09.819433 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:09.762158 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:09.821956 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57738us
I20260709 12:22:09.876151 18021 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:09.876652 18168 raft_consensus.cc:614] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:09.847699 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.824213 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.833702 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.719879 20278 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.835947 18019 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:09.888361 18019 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:09.838754 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.889703 18016 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:09.713078 20379 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:09.842626 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:09.907950 18167 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:22:09.908272 18165 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.844115 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.910811 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.909569 18023 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:09.915079 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:09.848716 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 110531us
I20260709 12:22:09.916744 18170 raft_consensus.cc:2410] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:09.917212 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:09.713390 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44288.6us
I20260709 12:22:09.919464 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:09.851935 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 567) took 1269 ms. Trace:
I20260709 12:22:09.920478 18811 rpcz_store.cc:276] 0709 12:22:08.582904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.597720 (+ 14816us) service_pool.cc:224] Handling call
0709 12:22:09.702901 (+1105181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.830054 (+127153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.851595 (+ 21541us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54005f64d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:09.851836 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2427859712}
I20260709 12:22:09.822696 20358 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.813477 20362 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:09.814483 20130 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:09.731325 18809 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:22:09.862375 18812 pending_rounds.cc:85] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 5
I20260709 12:22:09.929564 18812 pending_rounds.cc:107] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:22:09.929944 18812 raft_consensus.cc:2889] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:09.753075 20372 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:09.830754 20397 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.829011 20364 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:09.864305 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.831274 20396 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:09.949295 18020 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:09.871412 18173 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:09.886035 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.880089 20407 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:09.861218 20169 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.753039 20341 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:09.923372 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.886611 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 570) took 1302 ms. Trace:
I20260709 12:22:09.925803 20358 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:09.931895 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.937633 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.939220 18024 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:09.972121 18023 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:09.945200 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:09.946233 20349 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:09.942483 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.907378 20278 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.928651 18019 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:09.906484 20365 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.957444 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.855283 20415 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.105s
W20260709 12:22:09.829154 18810 rpcz_store.cc:269] 0709 12:22:08.045549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.051373 (+  5824us) service_pool.cc:224] Handling call
0709 12:22:09.701547 (+1650174us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:09.701699 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":331489024}
I20260709 12:22:09.891067 20422 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:09.963013 20169 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.963578 20396 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:09.973465 18020 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:09.932060 20362 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:09.974239 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:10.030189 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77017.2us
I20260709 12:22:09.926851 20245 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:10.041121 18090 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:10.041381 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.042188 18099 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:10.042527 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.043639 18093 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:22:09.990358 18064 connection.cc:256] RPC call timeout handler was delayed by -4.7356496659999721s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260709 12:22:10.045094 18064 connection.cc:256] RPC call timeout handler was delayed by -4.6348496659998091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
W20260709 12:22:10.045612 18064 connection.cc:256] RPC call timeout handler was delayed by -4.5061496660000557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
W20260709 12:22:10.046038 18064 connection.cc:256] RPC call timeout handler was delayed by -4.500749666000047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
W20260709 12:22:10.046444 18064 connection.cc:256] RPC call timeout handler was delayed by -4.42244966599992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20260709 12:22:10.046831 18064 connection.cc:256] RPC call timeout handler was delayed by -4.3936496659998738s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20260709 12:22:10.047227 18064 connection.cc:256] RPC call timeout handler was delayed by -4.3036496659997283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260709 12:22:10.047616 18064 connection.cc:256] RPC call timeout handler was delayed by -4.2352496660000725s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20260709 12:22:10.047992 18064 connection.cc:256] RPC call timeout handler was delayed by -4.2217496660000506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
W20260709 12:22:10.048391 18064 connection.cc:256] RPC call timeout handler was delayed by -4.2217496660000506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
W20260709 12:22:10.048771 18064 connection.cc:256] RPC call timeout handler was delayed by -4.1641496659999575s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
W20260709 12:22:10.049189 18064 connection.cc:256] RPC call timeout handler was delayed by -4.1416496659999211s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
I20260709 12:22:09.986696 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 572) took 1402 ms. Trace:
I20260709 12:22:09.993782 18170 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:09.959822 18173 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:09.988888 20349 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:09.995234 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:09.947389 20296 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:09.938750 20364 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:09.973385 18166 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:10.079468 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 134861us
I20260709 12:22:10.095743 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.096647 18166 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:10.097028 18166 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:10.019013 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.020364 18811 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:09.989289 18167 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:09.975534 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:09.950914 20372 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.051769 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:09.989006 20318 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:10.050081 18812 rpcz_store.cc:276] 0709 12:22:08.584494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.989383 (+404889us) service_pool.cc:224] Handling call
0709 12:22:08.989790 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.930430 (+940640us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.986383 (+ 55953us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b54005e01d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:09.986606 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1658055040}
I20260709 12:22:10.058327 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:09.951460 20397 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:10.049530 18064 connection.cc:256] RPC call timeout handler was delayed by -4.1389496659999168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20260709 12:22:10.248740 18064 connection.cc:256] RPC call timeout handler was delayed by -4.074149665999812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2242s for a response
W20260709 12:22:10.249559 18064 connection.cc:256] RPC call timeout handler was delayed by -3.5206496659998265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2857s for a response
W20260709 12:22:10.249967 18064 connection.cc:256] RPC call timeout handler was delayed by -3.2983496659999219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3104s for a response
I20260709 12:22:10.251403 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:10.063711 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.962393 18807 rpcz_store.cc:276] 0709 12:22:08.583915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.588757 (+  4842us) service_pool.cc:224] Handling call
0709 12:22:09.874098 (+1285341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:10.063377 20245 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.092921 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.023321 20293 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:10.035881 20421 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:22:10.092864 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:10.114979 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.065855 17920 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.030184 17919 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 290) took 6352 ms. Trace:
W20260709 12:22:10.185184 18138 connection.cc:256] RPC call timeout handler was delayed by -2.5077923349999764s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:22:10.266479 18138 connection.cc:256] RPC call timeout handler was delayed by -2.0631923349997123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
I20260709 12:22:10.260483 18091 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:22:10.237392 20304 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.066713 20349 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:10.063691 20396 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:10.018297 20365 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.204373 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:10.223381 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 248777us
I20260709 12:22:10.231900 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.279228 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.035218 20321 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:10.290221 18099 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:10.290737 20245 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.290537 20321 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.292801 20321 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.292711 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:10.234246 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 258799us
I20260709 12:22:10.224069 18166 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.295821 18166 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:10.298084 20318 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:10.298528 20396 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.299465 20372 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:10.304669 20348 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.617s	user 0.011s	sys 0.001s
I20260709 12:22:10.184880 20296 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:10.306957 20296 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.254834 18098 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:10.311349 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.311893 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.312553 18099 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:10.313299 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:10.255697 20416 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.420s	user 0.000s	sys 0.002s
I20260709 12:22:10.314417 20407 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:10.255967 20437 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:10.258843 20364 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.315793 20437 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:10.316627 20364 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:10.234356 17918 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.319303 17918 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 4, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 4 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:10.320220 18018 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.940s	user 0.013s	sys 0.001s
I20260709 12:22:10.320986 18018 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:10.256352 20426 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260709 12:22:10.321698 20292 raft_consensus.cc:2764] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:10.091813 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.314208 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:10.322661 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 265) took 1803 ms (client timeout 2059 ms). Trace:
I20260709 12:22:10.323319 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.323427 18099 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:10.324759 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.325366 18170 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:10.268266 18138 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.500s (SENT)
I20260709 12:22:10.270886 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:10.326426 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58214.5us
I20260709 12:22:10.261861 17920 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:22:10.328212 20398 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.003s
I20260709 12:22:10.328647 18165 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:10.329959 20348 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:10.323202 18018 rpcz_store.cc:269] 0709 12:22:08.519642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.585878 (+ 66236us) service_pool.cc:224] Handling call
0709 12:22:09.378564 (+792686us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592e9c 00007f6c13592a5b 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:10.322579 (+944015us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10725120}
I20260709 12:22:10.062606 20430 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:10.283088 20304 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.286432 20341 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.341845 18098 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:10.288803 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.328702 17920 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.289610 20365 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:10.005288 20249 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.278931 18167 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5. Re-sending same reply.
I20260709 12:22:10.233927 20432 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.342566 18090 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:10.347926 20432 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.349237 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:10.349745 20347 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.917s	user 0.014s	sys 0.000s
I20260709 12:22:10.349951 18090 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:10.350361 20432 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.350298 20347 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.352022 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.067251 20379 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:10.341614 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.354890 18097 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:10.278031 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:10.336244 18094 raft_consensus.cc:2376] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4. Re-sending same reply.
I20260709 12:22:10.356173 18093 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:10.340927 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:10.367715 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89672.3us
I20260709 12:22:10.336882 18166 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:10.370353 18168 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:10.371901 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.339726 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.321408 17918 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.347263 20304 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.375700 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:10.376641 18098 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5. Re-sending same reply.
I20260709 12:22:10.373466 18092 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:22:10.329097 20398 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.352818 20347 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.380165 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.375418 17918 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.385033 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:10.387127 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.387784 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.402129 18099 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
W20260709 12:22:10.493742 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 156039us
W20260709 12:22:10.489023 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.006s	sys 0.020s
W20260709 12:22:10.492897 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.544531 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:10.571707 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:10.451233 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.548759 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:10.578538 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.010s	sys 0.022s
I20260709 12:22:10.578707 18091 raft_consensus.cc:2376] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3. Re-sending same reply.
I20260709 12:22:10.411408 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:10.266320 17919 rpcz_store.cc:276] 0709 12:22:03.665444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.665771 (+   327us) service_pool.cc:224] Handling call
0709 12:22:10.017467 (+6351696us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:05.331787 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:05.757339 (+425552us) spinlock_profiling.cc:243] Waited 8.37 ms on lock 0x7b5400040ca8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1ca739e1 00007f6c1ca7264c 00007f6c1a101d5d 00007f6c1a0ed0a6 00007f6c1a0ecd58 00007f6c1a0f19f1 00007f6c1a0f19a9 00007f6c1a0f1939 00007f6c1a0f1901 00007f6c1a0f0b2d 0000000000388634
0709 12:22:05.757463 (+   124us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:05.757525 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:22:05.757551 (+    26us) tablet.cc:662] Decoding operations
0709 12:22:06.175952 (+418401us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:06.188280 (+ 12328us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:06.191410 (+  3130us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:22:06.351242 (+159832us) tablet.cc:701] Row locks acquired
0709 12:22:06.351265 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:22:06.351462 (+   197us) write_op.cc:270] Start()
0709 12:22:06.351751 (+   289us) write_op.cc:276] Timestamp: P: 1783599726351403 usec, L: 0
0709 12:22:06.351770 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 12:22:06.352677 (+   907us) log.cc:844] Serialized 4370 byte log entry
0709 12:22:08.694485 (+2341808us) op_driver.cc:464] REPLICATION: finished
0709 12:22:08.938122 (+243637us) write_op.cc:301] APPLY: starting
0709 12:22:08.967539 (+ 29417us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:09.007566 (+ 40027us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:09.022266 (+ 14700us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:09.356714 (+334448us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:09.357013 (+   299us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 12:22:09.357068 (+    55us) write_op.cc:312] APPLY: finished
0709 12:22:09.360084 (+  3016us) log.cc:844] Serialized 83 byte log entry
0709 12:22:09.638671 (+278587us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:09.744501 (+105830us) write_op.cc:454] Released schema lock
0709 12:22:09.753341 (+  8840us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:09.864331 (+110990us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:09.871255 (+  6924us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":242720,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":197805,"apply.run_cpu_time_us":17093,"apply.run_wall_time_us":919782,"num_ops":11,"prepare.queue_time_us":353459,"prepare.run_cpu_time_us":23430,"prepare.run_wall_time_us":1441769,"raft.queue_time_us":608814,"raft.run_cpu_time_us":23035,"raft.run_wall_time_us":234044,"replication_time_us":2303781,"spinlock_wait_cycles":19239808,"thread_start_us":1119156,"threads_started":5,"wal-append.queue_time_us":760964,"wal-append.run_cpu_time_us":60956,"wal-append.run_wall_time_us":2381950}]]}
I20260709 12:22:10.579541 18098 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:10.343060 20430 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:10.424197 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:10.405702 18094 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 12:22:10.524461 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 138909us
I20260709 12:22:10.501434 18090 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:10.411409 20424 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:22:10.610148 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.611606 18090 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:10.413573 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:10.614909 18093 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:10.577306 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:10.579603 18092 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:22:10.411336 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.606154 18095 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:10.558180 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 263366us
I20260709 12:22:10.591028 18170 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:10.598757 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:10.579046 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 816996us
I20260709 12:22:10.622902 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:10.623008 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.581619 18167 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:10.624217 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:10.624439 18173 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:10.624969 18815 raft_consensus.cc:2376] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5. Re-sending same reply.
I20260709 12:22:10.607378 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:10.423722 20412 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.040s	user 0.012s	sys 0.004s
W20260709 12:22:10.618134 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 126521us
I20260709 12:22:10.628129 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:10.423334 20399 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.895s	user 0.012s	sys 0.003s
I20260709 12:22:10.629420 18815 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:10.443456 20278 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.469237 18164 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:10.425009 20397 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:10.605091 20296 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:10.634646 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.662765 18170 raft_consensus.cc:2376] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3. Re-sending same reply.
W20260709 12:22:10.636205 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.668319 18164 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:10.641099 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.644081 18169 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:10.645548 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.670667 18173 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:10.645351 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:10.549674 20415 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.433545 20292 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:10.646749 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.597308 20432 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:10.648113 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70938.3us
I20260709 12:22:10.677181 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.678031 18165 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:10.649118 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:10.678964 18815 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:10.631645 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:10.554957 20360 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 12:22:10.450579 20348 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:10.605754 20430 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:10.689244 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.691594 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:10.695567 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60630.9us
I20260709 12:22:10.699076 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.655335 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:10.652117 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.430173 20130 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.653162 20249 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:10.654151 20364 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:10.627228 20408 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.760201 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.761179 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.674674 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:10.466787 20358 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.765148 20408 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.766203 18171 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:10.767660 20278 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:10.768784 20397 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:10.671218 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:10.770479 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 99336.4us
I20260709 12:22:10.771936 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:10.673570 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:10.774309 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 100802us
I20260709 12:22:10.661176 20296 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.647693 20442 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:22:10.674162 20292 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:10.783545 20321 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:10.784044 20442 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:10.784241 20292 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.785493 20292 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:10.674386 20304 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:10.734686 18166 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:10.737093 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.791976 18137 connection.cc:256] RPC call timeout handler was delayed by -1.6963414249999005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18660000000000002s for a response
W20260709 12:22:10.727926 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 110090us
I20260709 12:22:10.794883 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.795686 18170 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:10.740374 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:10.742805 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.796759 18165 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:10.785434 20442 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:10.762691 20398 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:10.764580 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89998.7us
I20260709 12:22:10.779737 18172 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:10.777663 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:10.782912 20348 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:10.810359 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:10.782449 20353 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:10.785943 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 319) took 1101 ms. Trace:
W20260709 12:22:10.786446 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 184776us
W20260709 12:22:10.787045 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54816.7us
I20260709 12:22:10.689955 20432 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.747941 18173 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:10.791601 20397 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:10.681985 20443 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.118s
I20260709 12:22:10.679720 20440 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260709 12:22:10.795346 20438 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:10.801658 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:10.804817 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.806277 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.859716 18167 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:10.808531 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69003.2us
I20260709 12:22:10.786253 20296 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:10.767525 20358 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:10.839922 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.868829 18173 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:10.635020 20169 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.003s
I20260709 12:22:10.812017 20449 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:10.767637 20408 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:10.838471 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:10.817152 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:10.812779 18164 rpcz_store.cc:276] 0709 12:22:09.684292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.685499 (+  1207us) service_pool.cc:224] Handling call
0709 12:22:10.785708 (+1100209us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:10.785871 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1442034048}
I20260709 12:22:10.851765 20397 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.846112 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:10.850661 20379 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.688072 20415 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:10.760012 20364 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.853828 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.856791 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 324) took 1023 ms. Trace:
I20260709 12:22:10.793545 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:10.908643 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 115187us
I20260709 12:22:10.800798 20442 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:10.863206 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61659.6us
I20260709 12:22:10.910898 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:10.870146 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.912875 18165 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:10.842016 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.913918 18166 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:10.839747 20304 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:10.829150 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.891088 18173 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:10.893635 18815 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:10.893481 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76691.1us
I20260709 12:22:10.744849 20349 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.846385 20348 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:10.903131 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d440327a63754b889f7edd91e1373942"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:10.923143 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:10.930143 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.931156 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.930213 18167 raft_consensus.cc:606] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:10.932181 18165 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:10.933429 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.933717 18812 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:10.899241 20360 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:10.907444 18169 rpcz_store.cc:276] 0709 12:22:09.833013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.833313 (+   300us) service_pool.cc:224] Handling call
0709 12:22:10.855466 (+1022153us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b5400042ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:10.855718 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1756665600}
I20260709 12:22:10.874890 20249 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:10.879279 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:10.856763 20200 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:22:10.896368 20400 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:10.949798 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:22:10.896715 19910 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:10.972847 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:10.893354 20449 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:10.939258 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.945148 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:10.944545 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:10.945631 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:10.974951 20449 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.976207 20449 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:10.970448 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:10.946636 20360 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:10.979890 20360 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:10.896076 20399 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:10.981002 20399 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:10.984041 20422 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.949s	user 0.013s	sys 0.000s
I20260709 12:22:10.979342 20432 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:10.984913 20455 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:22:10.984162 20442 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.879784 20169 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:10.986086 18167 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:10.979019 20249 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:10.907879 20379 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:10.989070 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:10.987136 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:10.934942 20441 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:10.929852 20364 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:10.936444 20408 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:10.991806 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 153724us
I20260709 12:22:10.925961 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:10.950644 20304 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:10.996837 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.986153 20400 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:11.026695 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95778.3us
I20260709 12:22:10.971115 20398 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:22:11.028116 18165 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:11.025534 18168 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:11.044914 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:10.901371 20397 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:11.061143 18811 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:11.048170 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 122378us
W20260709 12:22:11.048879 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.072090 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:11.072484 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:10.995719 20422 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.072957 18019 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:11.030462 20441 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.073410 18023 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:11.079203 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 217852us
I20260709 12:22:11.081833 18016 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:11.082659 20349 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:11.022367 20348 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.082964 20304 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.031589 20130 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:11.084036 20169 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:11.081717 20432 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.084705 20400 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:11.088629 20379 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:11.053541 20358 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:11.053257 20442 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.091316 20399 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:11.094058 20413 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:11.094745 20456 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:22:11.095350 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:11.059238 20396 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.000s	sys 0.012s
I20260709 12:22:11.098244 20310 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:11.027513 20453 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260709 12:22:11.101778 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 323) took 1278 ms. Trace:
I20260709 12:22:11.166742 18172 rpcz_store.cc:276] 0709 12:22:09.823596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.823797 (+   201us) service_pool.cc:224] Handling call
0709 12:22:11.101496 (+1277699us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54001b4c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:11.101689 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069930368}
I20260709 12:22:11.102563 18164 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:11.103286 20415 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:11.103569 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 320) took 1407 ms (client timeout 1809 ms). Trace:
W20260709 12:22:11.171923 18171 rpcz_store.cc:269] 0709 12:22:09.696222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.696421 (+   199us) service_pool.cc:224] Handling call
0709 12:22:11.103329 (+1406908us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b5400383454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:11.103491 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2174827776}
I20260709 12:22:11.176478 18171 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:22:11.060482 20408 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:11.112826 18020 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:11.120731 18015 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:11.179423 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 262778us
I20260709 12:22:11.181478 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:11.166085 20396 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.000s	sys 0.013s
I20260709 12:22:11.128449 18017 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:11.119114 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.122835 18167 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:11.119033 18023 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:11.073894 20406 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260709 12:22:11.108230 20441 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:11.157714 20432 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:11.161446 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.163244 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.164197 18022 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
W20260709 12:22:11.164300 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45983.5us
I20260709 12:22:11.135205 20443 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.196077 18171 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:11.162493 20413 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:11.166493 20310 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:11.160583 20461 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:11.147527 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 102723us
W20260709 12:22:11.419430 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.418596 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.186s (SENT) [suppressed 38 similar messages]
W20260709 12:22:11.218077 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 137511us
I20260709 12:22:11.422681 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:11.330632 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.419327 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.425056 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.436580 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:11.454905 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 267843us
I20260709 12:22:11.455942 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:11.456471 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:11.460356 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:11.202459 20471 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:22:11.206671 20458 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
W20260709 12:22:11.204248 20362 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.141s	user 0.011s	sys 0.003s
I20260709 12:22:11.202082 20130 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.205054 19910 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:11.487424 18811 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:11.196872 20469 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:11.487958 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:11.519495 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:11.488143 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:11.456799 20430 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.715s	user 0.011s	sys 0.004s
W20260709 12:22:11.483815 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.956s (SENT)
I20260709 12:22:11.203904 20422 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:11.434952 20396 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.499209 20449 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:11.492164 20310 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.297276 20379 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:11.480304 20443 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.463488 20453 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:11.437340 20461 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:11.515026 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59199.1us
I20260709 12:22:11.528860 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:11.232965 20364 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.313030 20360 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.512682 20432 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:11.538321 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:11.513337 19910 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:11.489410 20468 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:11.556960 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 1988) took 1499 ms. Trace:
I20260709 12:22:11.529755 20437 raft_consensus.cc:2764] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:11.562243 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:11.481657 20472 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260709 12:22:11.552402 20442 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:11.494253 20292 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:22:11.206131 20408 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:11.582304 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.584398 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100611us
I20260709 12:22:11.576678 20461 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:11.586596 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48397.2us
I20260709 12:22:11.618837 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:11.558188 20296 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.664s	user 0.013s	sys 0.000s
I20260709 12:22:11.575256 20310 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:11.592283 18164 rpcz_store.cc:276] 0709 12:22:10.057327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.901138 (+843811us) service_pool.cc:224] Handling call
0709 12:22:11.102335 (+201197us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.168727 (+ 66392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.556529 (+387802us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54001b4c50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:11.556838 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461785856}
I20260709 12:22:11.607230 20349 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:11.519730 20130 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.608587 18170 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:11.609517 18165 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:11.590945 20438 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:11.535892 20430 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.515288 20362 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.597299 20449 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:11.624858 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:11.560510 20379 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.552307 20396 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.612974 20432 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:11.614461 20468 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:11.644164 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:11.646931 20396 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:11.647567 20440 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:11.649037 18166 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:22:11.658471 18166 pending_rounds.cc:85] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 4
I20260709 12:22:11.658905 18166 pending_rounds.cc:107] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:22:11.659250 18166 raft_consensus.cc:2889] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:11.636490 20442 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:11.664237 18813 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:11.628122 20296 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.645344 20487 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:11.644320 20465 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:11.673611 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:11.645160 20292 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:22:11.663915 20440 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:11.675313 20484 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
W20260709 12:22:11.672736 20397 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.543s	user 0.008s	sys 0.005s
I20260709 12:22:11.668929 20348 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:11.696251 18169 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:11.697772 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72997.3us
I20260709 12:22:11.634539 19910 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.671409 20442 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:11.675037 20487 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:11.702128 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:11.703055 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.164s (SENT)
I20260709 12:22:11.684680 20449 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:11.715727 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.718375 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49799us
I20260709 12:22:11.718904 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:11.717478 20449 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.717950 20364 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:11.704540 20397 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.723680 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:11.721551 20472 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:11.729122 20397 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:11.714718 20443 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:11.722337 19910 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:11.722924 20437 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:11.811085 20492 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:22:11.723560 20468 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:11.816917 20480 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:11.731020 20360 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:11.841635 20491 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:22:11.843880 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 580) took 1036 ms. Trace:
W20260709 12:22:11.844842 20495 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.161s	user 0.003s	sys 0.000s
W20260709 12:22:11.845988 20488 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:22:11.726228 20453 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 12:22:11.752763 20443 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:11.865648 18098 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:11.870930 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.871796 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:11.872531 18099 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:11.876786 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.879745 18167 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:11.880693 18164 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:11.885665 18098 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:11.888985 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:11.899358 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.900899 18167 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:11.918066 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:11.918347 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:11.919152 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:11.921183 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1608) took 1588 ms. Trace:
I20260709 12:22:11.921694 18172 rpcz_store.cc:276] 0709 12:22:10.332323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.167790 (+835467us) service_pool.cc:224] Handling call
0709 12:22:11.168234 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.193480 (+ 25246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.193534 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.194209 (+   675us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.663045 (+468836us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:11.663103 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.663202 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.843854 (+180652us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:11.891905 (+ 48051us) raft_consensus.cc:1685] finished
0709 12:22:11.891947 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.921096 (+ 29149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24720,"raft.queue_time_us":36377,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"spinlock_wait_cycles":5176704,"thread_start_us":350269,"threads_started":1,"wal-append.queue_time_us":391291}
I20260709 12:22:11.922123 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:11.919083 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:11.928781 18099 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:11.870042 20360 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.954973 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:11.956923 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.882067 20364 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.961983 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.871042 20362 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:11.961246 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:11.889437 20443 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.966393 18099 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:11.907019 20469 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:11.968488 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.917150 20379 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:11.971663 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:11.983281 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:11.973309 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:11.983455 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:11.984264 18016 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:11.984694 18020 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:11.985782 18016 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:11.983371 18023 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:11.974561 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:11.975099 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:11.988193 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.857985 20432 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.101s
I20260709 12:22:11.927790 20460 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:11.832690 20440 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.982214 18099 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
W20260709 12:22:11.964006 20485 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.318s	user 0.001s	sys 0.001s
I20260709 12:22:11.990792 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:11.988265 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:11.993245 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:11.821688 20487 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:11.993650 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1602) took 1663 ms (client timeout 1598 ms). Trace:
I20260709 12:22:11.994686 18164 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:11.996124 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.964911 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:12.040027 18091 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:12.041481 18168 rpcz_store.cc:269] 0709 12:22:10.330510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.371500 (+ 40990us) service_pool.cc:224] Handling call
0709 12:22:11.993381 (+1621881us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400382f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:11.993560 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1500808576}
I20260709 12:22:11.908396 20304 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 12:22:12.036345 18167 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:11.994400 20489 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.295s	user 0.002s	sys 0.000s
I20260709 12:22:11.871671 20442 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:12.048564 18806 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.069003 20449 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:12.075389 20498 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:22:12.076933 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1603) took 1746 ms. Trace:
I20260709 12:22:11.962568 20360 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.052874 18097 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:12.052187 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 65950.1us
I20260709 12:22:12.056546 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:12.087252 18099 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:12.059026 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:11.968918 20364 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:11.994134 20349 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:12.011188 20397 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:12.077529 18169 rpcz_store.cc:276] 0709 12:22:10.330853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.946635 (+615782us) service_pool.cc:224] Handling call
0709 12:22:10.946995 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.701063 (+754068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.076535 (+375472us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b54001b5b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:12.076837 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1722320256}
I20260709 12:22:11.979091 20443 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:11.987766 20469 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:12.108662 20481 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.211s	user 0.002s	sys 0.000s
I20260709 12:22:11.844549 18810 rpcz_store.cc:276] 0709 12:22:10.807420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.812293 (+  4873us) service_pool.cc:224] Handling call
0709 12:22:10.947354 (+135061us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.011068 (+ 63714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.018542 (+  7474us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.491885 (+473343us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b4c002c62c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14983a20 00007f6c14994a29 00007f6c1495a644 00007f6c14981f2d 00007f6c14981b93 00007f6c166b218b 00007f6c166b2cb5 00007f6c166b4004 00007f6c166b3729 00007f6c166b8d96 00007f6c166adadb
0709 12:22:11.522597 (+ 30712us) log.cc:844] Serialized 556 byte log entry
0709 12:22:11.636878 (+114281us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:11.642740 (+  5862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.652309 (+  9569us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.843349 (+191040us) raft_consensus.cc:1685] finished
0709 12:22:11.843392 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.843768 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38481792,"thread_start_us":82470,"threads_started":1,"wal-append.queue_time_us":96042}
I20260709 12:22:11.954119 20413 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.138190 18099 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:12.020052 20460 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.147261 20460 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.118638 20349 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:12.152311 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 428713us
I20260709 12:22:12.148583 18024 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 2: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:12.118279 20130 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.154934 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:12.085784 20494 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20260709 12:22:12.134143 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 582) took 1322 ms. Trace:
I20260709 12:22:12.165064 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1597) took 2060 ms. Trace:
I20260709 12:22:12.165555 18170 rpcz_store.cc:276] 0709 12:22:10.105149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.907957 (+802808us) service_pool.cc:224] Handling call
0709 12:22:10.955026 (+ 47069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.628402 (+673376us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.164759 (+536357us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:12.164990 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1502443392}
I20260709 12:22:12.178012 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.181598 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:12.188009 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1607) took 1856 ms. Trace:
I20260709 12:22:12.190008 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:12.123287 20397 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.190806 18022 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:12.190600 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:12.188627 18165 rpcz_store.cc:276] 0709 12:22:10.332039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.055858 (+723819us) service_pool.cc:224] Handling call
0709 12:22:11.056291 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.628408 (+572117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.168401 (+539993us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:12.187906 (+ 19505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1271805312}
I20260709 12:22:12.210574 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:12.201231 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:12.152187 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 283) took 1080 ms. Trace:
I20260709 12:22:12.173518 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:12.212653 18780 connection.cc:256] RPC call timeout handler was delayed by -1.1251477980003983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27320000000000005s for a response
I20260709 12:22:12.229079 18017 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:12.212865 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.230583 18023 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
W20260709 12:22:12.216221 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63128us
I20260709 12:22:12.189385 20432 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.215348 20396 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:12.243428 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:12.159372 20502 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.083s
I20260709 12:22:12.248970 18019 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.264781 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:12.268800 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:12.269038 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58595.4us
I20260709 12:22:12.269667 18097 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.272145 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:12.212206 20348 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:22:12.275178 20310 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.645s	user 0.015s	sys 0.001s
I20260709 12:22:12.247859 20432 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.274633 20501 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:12.271268 20130 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:22:12.284358 20505 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:12.274399 20424 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
W20260709 12:22:12.327248 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98003.5us
I20260709 12:22:12.300932 20396 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:12.332474 18099 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:12.336258 18019 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:12.314457 20310 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.347657 18099 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:12.349251 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:12.354442 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:12.326588 20364 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.345414 18097 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:12.361366 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 128377us
I20260709 12:22:12.290493 20442 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:22:12.333148 20396 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:12.337852 20443 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.351472 20360 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:12.369714 20364 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:12.360306 20384 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:12.388206 18022 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:22:12.352100 20353 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:22:12.175110 18809 rpcz_store.cc:276] 0709 12:22:10.808194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.812517 (+  4323us) service_pool.cc:224] Handling call
0709 12:22:10.902288 (+ 89771us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.972129 (+ 69841us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.977740 (+  5611us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.978578 (+   838us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.273397 (+294819us) spinlock_profiling.cc:243] Waited 7.76 ms on lock 0x7b18002e0540. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:22:11.925719 (+652322us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:11.956775 (+ 31056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.963102 (+  6327us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.077112 (+114010us) raft_consensus.cc:1685] finished
0709 12:22:12.077142 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.130833 (+ 53691us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3865,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":153,"spinlock_wait_cycles":17845504,"thread_start_us":477231,"threads_started":1,"wal-append.queue_time_us":1013929}
I20260709 12:22:12.346576 20304 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:22:12.346753 20438 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
W20260709 12:22:12.396380 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.397226 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.397431 20490 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.713s	user 0.000s	sys 0.006s
I20260709 12:22:12.397980 20397 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:12.400718 19910 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.642s	user 0.012s	sys 0.001s
I20260709 12:22:12.388463 20364 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:12.387878 18806 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:12.408699 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:12.374086 20396 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.352710 20362 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:22:12.433030 20304 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:12.433486 20304 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:12.434486 20304 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.435890 20304 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.442862 18170 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:12.417934 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:12.424618 20379 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:12.444959 20379 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:12.445437 20379 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:12.428560 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 163884us
I20260709 12:22:12.353996 20310 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.375038 20469 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.405663 20398 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.427254 20364 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.464921 20278 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:12.430819 20384 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:12.475172 20465 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:12.478881 20432 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.480842 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 576) took 1677 ms. Trace:
W20260709 12:22:12.509604 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 155281us
W20260709 12:22:12.501966 20461 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.879s	user 0.011s	sys 0.006s
I20260709 12:22:12.499428 20465 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.482153 20398 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:12.526870 20441 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.039s	user 0.016s	sys 0.000s
I20260709 12:22:12.521763 20461 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:12.532239 20461 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:12.534335 20472 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.806s	user 0.012s	sys 0.004s
I20260709 12:22:12.534984 20472 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:12.246430 18018 rpcz_store.cc:276] 0709 12:22:11.069542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.074019 (+  4477us) service_pool.cc:224] Handling call
0709 12:22:11.085682 (+ 11663us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.157868 (+ 72186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.160198 (+  2330us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.243553 (+ 83355us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:22:11.244315 (+   762us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.512266 (+267951us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b18003908e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:11.565261 (+ 52995us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:11.582491 (+ 17230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.596154 (+ 13663us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.143403 (+547249us) raft_consensus.cc:1685] finished
0709 12:22:12.143435 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.149381 (+  5946us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22818,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":8148,"spinlock_wait_cycles":12475520,"thread_start_us":294655,"threads_started":1,"wal-append.queue_time_us":366069}
I20260709 12:22:12.398324 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.579123 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
W20260709 12:22:12.587183 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 314892us
I20260709 12:22:12.590205 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:12.596953 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 343) took 1677 ms. Trace:
I20260709 12:22:12.597872 18171 rpcz_store.cc:276] 0709 12:22:10.918821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.435989 (+517168us) service_pool.cc:224] Handling call
0709 12:22:11.479576 (+ 43587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.479900 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.479927 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.489167 (+  9240us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.648252 (+159085us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:11.929964 (+281712us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.930305 (+   341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.930394 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.396450 (+466056us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:12.595805 (+199355us) raft_consensus.cc:1685] finished
0709 12:22:12.596068 (+   263us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.596527 (+   459us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12762,"prepare.queue_time_us":58663,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2960,"raft.queue_time_us":105,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":8945,"spinlock_wait_cycles":12845312,"thread_start_us":385881,"threads_started":2,"wal-append.queue_time_us":170886}
I20260709 12:22:12.603475 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:12.607153 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:12.611151 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:12.614349 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:12.509518 18807 rpcz_store.cc:276] 0709 12:22:10.803208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.811075 (+  7867us) service_pool.cc:224] Handling call
0709 12:22:12.480544 (+1669469us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:12.480750 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1479140224}
I20260709 12:22:12.619707 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:12.621677 18806 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:12.622923 20304 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:12.623695 20304 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:12.624176 20304 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.634697 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:12.636006 18807 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:12.639122 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 497) took 2781 ms. Trace:
I20260709 12:22:12.639667 18813 rpcz_store.cc:276] 0709 12:22:09.857850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.858603 (+   753us) service_pool.cc:224] Handling call
0709 12:22:09.875773 (+ 17170us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.664812 (+1789039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.638432 (+973620us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54006f2cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:12.638833 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4107234688}
I20260709 12:22:12.644043 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:12.646989 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 342) took 1728 ms. Trace:
I20260709 12:22:12.647784 18166 rpcz_store.cc:276] 0709 12:22:10.918547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.233000 (+314453us) service_pool.cc:224] Handling call
0709 12:22:11.233472 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.665444 (+431972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.633573 (+968129us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54003b9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:12.646724 (+ 13151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954752896}
I20260709 12:22:12.648725 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:12.652725 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.656230 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.656944 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.662523 20353 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:12.524474 20444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.015s	sys 0.002s
W20260709 12:22:12.678762 20444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.016s	sys 0.002s
W20260709 12:22:12.684043 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.654608 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:12.691548 20397 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:12.693500 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.702484 20505 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:12.702809 20453 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:12.703109 20442 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.703825 20440 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:12.703661 20518 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:12.704348 20442 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.703145 20505 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:12.706281 20460 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.706593 20505 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.706502 20461 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:12.510946 20349 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.707381 20460 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.710243 20296 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:12.710387 20472 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:12.710887 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:12.712302 18021 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:12.712359 20502 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:12.714839 20502 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:12.714454 20453 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.716610 20502 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.706691 20438 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260709 12:22:12.719658 20398 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:12.727836 20502 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.699514 20450 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.728907 20450 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.714141 20444 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:12.730201 20444 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:12.722901 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:12.721611 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.724490 18810 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEARNER]: Advancing to term 5
I20260709 12:22:12.732640 18022 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:12.725834 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:12.726060 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.734630 20460 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.734964 18018 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:12.726585 20453 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.731667 20430 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:12.698015 20364 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.701534 20468 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:22:12.742305 20487 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.744374 20364 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:12.528445 20441 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.755535 20413 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.758191 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:12.758622 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:12.760002 20503 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:12.663419 20384 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.766975 20441 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.742194 20461 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:12.768707 20461 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:12.769217 20461 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:12.761164 20503 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:12.770056 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115731us
I20260709 12:22:12.770478 20461 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.772821 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:12.772565 20503 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.774019 20360 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:12.794360 20310 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.796471 18807 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:12.822177 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.823334 18024 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:12.823640 20396 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.825544 18017 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:12.827315 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.828558 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.830657 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.825385 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:12.818487 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157503us
I20260709 12:22:12.835388 20461 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:12.835783 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:12.842676 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 396436us
I20260709 12:22:12.828717 18810 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:12.849138 20396 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:12.854254 20349 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.855757 20535 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:12.856393 20535 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.856791 20535 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.810426 20438 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:12.858122 20438 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:12.858542 20438 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.860221 20537 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:12.819406 20360 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:12.861384 20537 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.863366 20360 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.865378 20360 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.808825 20450 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.832765 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:12.802577 20353 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:12.892740 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:12.895502 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.887363 18015 raft_consensus.cc:2376] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3. Re-sending same reply.
I20260709 12:22:12.917896 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.929718 20533 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.931110 20533 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:12.938915 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 332068us
W20260709 12:22:12.941730 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 386351us
I20260709 12:22:12.943102 20533 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:12.944120 18018 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 4
I20260709 12:22:12.944576 18018 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:22:12.944950 18018 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:12.950192 18023 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:12.950590 20353 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.951963 18806 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:12.953214 18021 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.954015 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.955971 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:12.957059 18019 raft_consensus.cc:2368] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4.
W20260709 12:22:12.958057 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106499us
I20260709 12:22:12.959230 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:12.960651 18171 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:12.943102 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:12.968420 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:12.970582 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:12.961407 18138 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.043s (SENT)
I20260709 12:22:12.972402 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:12.973945 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:12.974291 18138 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.049s (SENT)
I20260709 12:22:12.974393 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:12.974961 18170 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:12.976656 20441 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:12.977334 20441 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:12.979616 20465 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.982286 20396 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:12.992677 20396 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:12.993146 20396 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:12.994932 20379 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.549s	user 0.009s	sys 0.012s
I20260709 12:22:12.994757 20396 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.995575 17918 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 275) took 7534 ms. Trace:
I20260709 12:22:12.937191 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:12.996506 20396 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:12.997380 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108609us
I20260709 12:22:12.984853 20441 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:12.988133 20532 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:12.988408 20424 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:13.014020 20296 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:13.014281 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76335.7us
I20260709 12:22:12.989212 20517 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:12.774755 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:12.990869 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:12.731765 20444 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:12.986372 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:12.987552 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:13.024590 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.995630 20379 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.024461 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:13.012403 17920 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 176) took 7522 ms. Trace:
I20260709 12:22:13.025521 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.032581 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.035509 18019 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:12.908885 20505 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.014103 20424 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:13.014993 20296 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.061128 20503 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.062224 20503 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.067233 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:13.067770 18172 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:13.069285 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.070314 18024 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:13.070894 18015 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:13.072046 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.072638 20278 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:13.072052 20465 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:13.072930 18024 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:13.078043 18016 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:13.032100 20532 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:13.079830 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54221.5us
I20260709 12:22:13.086592 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:13.091719 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60162.8us
I20260709 12:22:12.921514 20537 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.057617 20424 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:13.059271 20444 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:13.149389 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.113334 20362 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:13.113945 20442 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:13.119442 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.119170 17921 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 3, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.022351 20130 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:22:13.121582 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:13.122089 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.122237 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.090829 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.077301 20503 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.122872 18806 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:22:13.124342 20304 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.127288 20379 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:13.127714 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.544s (SENT)
I20260709 12:22:13.128263 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:13.130537 20505 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.191612 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.131059 18093 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:13.134488 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.135773 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:13.138113 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71101.8us
I20260709 12:22:13.138379 20360 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:13.140002 20278 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:22:13.141335 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.033241 17920 rpcz_store.cc:276] 0709 12:22:05.490361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.494598 (+  4237us) service_pool.cc:224] Handling call
0709 12:22:13.012062 (+7517464us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:10.790266 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:10.949337 (+159071us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:10.953429 (+  4092us) write_op.cc:435] Acquired schema lock
0709 12:22:10.969198 (+ 15769us) tablet.cc:662] Decoding operations
0709 12:22:11.110284 (+141086us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:11.114335 (+  4051us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:11.122124 (+  7789us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:22:11.190695 (+ 68571us) tablet.cc:701] Row locks acquired
0709 12:22:11.194342 (+  3647us) write_op.cc:260] PREPARE: finished
0709 12:22:11.229894 (+ 35552us) write_op.cc:270] Start()
0709 12:22:11.230055 (+   161us) write_op.cc:276] Timestamp: P: 1783599731229803 usec, L: 0
0709 12:22:11.230083 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:22:11.230997 (+   914us) log.cc:844] Serialized 1293 byte log entry
0709 12:22:11.834653 (+603656us) op_driver.cc:464] REPLICATION: finished
0709 12:22:12.106617 (+271964us) write_op.cc:301] APPLY: starting
0709 12:22:12.106762 (+   145us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:12.106919 (+   157us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:12.115409 (+  8490us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:12.116036 (+   627us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:12.116280 (+   244us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:22:12.116332 (+    52us) write_op.cc:312] APPLY: finished
0709 12:22:12.116601 (+   269us) log.cc:844] Serialized 35 byte log entry
0709 12:22:12.116832 (+   231us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:12.116938 (+   106us) write_op.cc:454] Released schema lock
0709 12:22:12.116988 (+    50us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:12.117488 (+   500us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:12.117520 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":129803,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":267436,"apply.run_cpu_time_us":2850,"apply.run_wall_time_us":10887,"num_ops":3,"prepare.queue_time_us":316429,"prepare.run_cpu_time_us":29568,"prepare.run_wall_time_us":660866,"raft.queue_time_us":510941,"raft.run_cpu_time_us":49954,"raft.run_wall_time_us":526875,"replication_time_us":512604,"thread_start_us":753367,"threads_started":4,"wal-append.queue_time_us":215598}]]}
I20260709 12:22:13.195567 18168 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:13.145367 20473 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:13.196650 20398 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:22:13.197870 20473 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:13.198362 20473 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:13.198208 17921 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 4, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.199616 20473 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:13.155328 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60834.3us
I20260709 12:22:13.169092 18090 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:13.122902 20494 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.172s
I20260709 12:22:13.161262 20533 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260709 12:22:13.170652 18170 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:13.176934 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.183735 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.185611 18024 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:13.196324 18020 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:13.155831 20384 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
I20260709 12:22:13.208976 18099 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:13.235060 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2013) took 1232 ms. Trace:
I20260709 12:22:13.202157 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:13.202397 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.205216 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:12.996155 17918 rpcz_store.cc:276] 0709 12:22:05.461291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.490723 (+ 29432us) service_pool.cc:224] Handling call
0709 12:22:12.995251 (+7504528us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:11.447233 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:11.447973 (+   740us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:11.451996 (+  4023us) write_op.cc:435] Acquired schema lock
0709 12:22:11.452024 (+    28us) tablet.cc:662] Decoding operations
0709 12:22:11.491320 (+ 39296us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:11.491381 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:11.491416 (+    35us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:22:11.491900 (+   484us) tablet.cc:701] Row locks acquired
0709 12:22:11.491919 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:22:11.492111 (+   192us) write_op.cc:270] Start()
0709 12:22:11.492258 (+   147us) write_op.cc:276] Timestamp: P: 1783599731492046 usec, L: 0
0709 12:22:11.492283 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:22:11.499467 (+  7184us) log.cc:844] Serialized 1689 byte log entry
0709 12:22:12.061869 (+562402us) op_driver.cc:464] REPLICATION: finished
0709 12:22:12.136161 (+ 74292us) write_op.cc:301] APPLY: starting
0709 12:22:12.145066 (+  8905us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:12.153282 (+  8216us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:12.153427 (+   145us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:12.269981 (+116554us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:12.293031 (+ 23050us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 12:22:12.296666 (+  3635us) write_op.cc:312] APPLY: finished
0709 12:22:12.308494 (+ 11828us) log.cc:844] Serialized 41 byte log entry
0709 12:22:12.324524 (+ 16030us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:12.329851 (+  5327us) write_op.cc:454] Released schema lock
0709 12:22:12.332061 (+  2210us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:12.373633 (+ 41572us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:12.376686 (+  3053us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":294653,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":60562,"apply.run_cpu_time_us":4922,"apply.run_wall_time_us":225817,"num_ops":4,"prepare.queue_time_us":814200,"prepare.run_cpu_time_us":6198,"prepare.run_wall_time_us":52686,"replication_time_us":569360,"thread_start_us":180007,"threads_started":1}]]}
I20260709 12:22:13.203750 18095 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:13.206208 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.250866 18021 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:13.202368 18091 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:13.289453 18168 rpcz_store.cc:276] 0709 12:22:12.011584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.043160 (+ 31576us) service_pool.cc:224] Handling call
0709 12:22:13.234614 (+1191454us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000438d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:13.234895 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2481259264}
I20260709 12:22:13.212905 18017 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:13.201927 20472 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.172000 20517 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.136s
I20260709 12:22:13.308137 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.309190 18094 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:22:13.309597 20408 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.007s	sys 0.033s
W20260709 12:22:13.310246 20408 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.008s	sys 0.033s
I20260709 12:22:13.201619 17921 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:13.313975 18807 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:13.315476 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.317029 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:13.318109 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.318323 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.318441 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.319243 18023 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:13.319479 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.320358 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.321254 18093 pending_rounds.cc:85] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 2
I20260709 12:22:13.321655 20503 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:13.140008 20532 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.322801 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.323439 18092 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:22:13.323699 18017 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:13.324769 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.325328 18092 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:13.327692 18173 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:13.304558 20413 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 12:22:13.330291 20503 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:13.332840 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:13.202755 20278 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:13.292508 20438 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:13.322897 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 276845us
I20260709 12:22:13.321211 18023 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:13.321671 18093 pending_rounds.cc:107] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20260709 12:22:13.337579 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.313179 17921 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.319221 18095 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:13.212630 20506 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.201705 20473 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.220065 20296 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:13.207857 20292 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.202943 20360 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.222512 20468 raft_consensus.cc:2764] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:22:13.208209 20304 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:13.367014 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.207855 20362 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:13.368516 18019 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:13.367296 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.223673 20449 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.125s
I20260709 12:22:13.375370 18015 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:13.406855 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 262191us
I20260709 12:22:13.372776 18093 raft_consensus.cc:2889] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:13.380560 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:13.377471 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:13.378656 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.010s	sys 0.063s
I20260709 12:22:13.379637 20494 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:22:13.382333 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 367341us
W20260709 12:22:13.383174 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50468.8us
I20260709 12:22:13.392148 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.399716 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:13.404947 18813 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:13.417196 17921 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:13.409696 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.011s	sys 0.064s
I20260709 12:22:13.402323 20360 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:13.406203 20379 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:22:13.441547 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61104.4us
I20260709 12:22:13.321174 20535 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.443320 18091 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:13.448483 18021 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:13.461879 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.462507 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.463577 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:13.466912 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 339375us
I20260709 12:22:13.440136 20408 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:13.476397 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49962.2us
I20260709 12:22:13.483338 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:13.419716 20472 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:13.491983 18778 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.556s (SENT)
I20260709 12:22:13.492686 18778 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:13.493618 20408 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:13.494068 20468 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:13.494724 20468 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:13.495335 20468 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:13.495029 20408 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.417533 20506 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.509514 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.480829 20494 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.446949 20292 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.450310 20362 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.525457 20464 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:13.515786 18165 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:13.487023 20461 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:13.511543 20379 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:13.525892 20462 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:13.551203 20501 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:13.544034 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:13.551937 19910 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:13.559871 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56812.4us
W20260709 12:22:13.571986 20551 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:13.573841 20461 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.575003 18168 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:13.576802 20461 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.577057 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.583048 18807 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:13.584803 20296 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:13.585556 20296 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:13.587163 20518 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:13.586979 20296 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.587805 20518 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:13.602136 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 171873us
I20260709 12:22:13.553171 20462 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:13.557443 20501 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:13.630894 18170 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:13.630631 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63804.5us
I20260709 12:22:13.518412 20543 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 12:22:13.506860 20278 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.565146 20464 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:13.628894 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:13.639190 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1605) took 3301 ms. Trace:
I20260709 12:22:13.639827 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:13.647142 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 160067us
W20260709 12:22:13.647578 20547 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
W20260709 12:22:13.648077 18778 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.583s (SENT)
I20260709 12:22:13.642666 18173 rpcz_store.cc:276] 0709 12:22:10.331443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.976230 (+644787us) service_pool.cc:224] Handling call
0709 12:22:10.976640 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:13.371260 (+2394620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.625306 (+254046us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54003025d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:13.632921 (+  7615us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5405914880}
I20260709 12:22:13.654575 20292 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.654753 20462 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.656327 20462 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:13.657209 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.622510 20438 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.659159 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.660317 20278 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:13.659425 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 1947) took 1676 ms (client timeout 1991 ms). Trace:
I20260709 12:22:13.567329 20494 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:13.662987 18020 rpcz_store.cc:269] 0709 12:22:11.985248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.985526 (+   278us) service_pool.cc:224] Handling call
0709 12:22:13.659119 (+1673593us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400347bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:13.659328 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2531202816}
I20260709 12:22:13.665128 20494 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:13.666167 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.658823 20501 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.668782 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:13.661800 20464 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:13.667258 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110469us
W20260709 12:22:13.649540 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.616s (SENT)
I20260709 12:22:13.676540 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.667414 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:13.589186 20518 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.676975 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:13.676697 20396 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.680s	user 0.019s	sys 0.000s
I20260709 12:22:13.625525 20487 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 12:22:13.668020 20430 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:13.707319 18095 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:13.684567 20437 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:13.692539 18016 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:13.716190 20555 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:13.717022 20545 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
W20260709 12:22:13.718369 20548 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:22:13.719656 20503 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:13.639544 20505 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:22:13.720522 20503 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.721578 20505 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:13.711064 20517 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 12:22:13.722391 20505 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:13.722797 20505 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:13.723034 20362 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.723840 20505 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:13.725114 20503 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.725380 20505 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:13.726763 20518 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:13.731547 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.714138 20396 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:22:13.734634 20506 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:13.721608 18165 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:13.734649 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.741036 18168 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:13.749868 20555 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:13.750316 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100845us
I20260709 12:22:13.751423 20555 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:13.700120 20501 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:13.683547 20379 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:22:13.707363 20535 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:13.770514 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:13.776721 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.780566 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:13.793392 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84544us
I20260709 12:22:13.795704 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.777982 20503 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:13.802574 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:13.794354 20424 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.601s	user 0.009s	sys 0.005s
W20260709 12:22:13.795876 20444 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.634s	user 0.014s	sys 0.001s
I20260709 12:22:13.785746 20437 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:13.816496 18168 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:22:13.819115 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82781.9us
I20260709 12:22:13.823172 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:13.797026 20506 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:13.807046 20292 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:13.834339 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.806613 20503 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:13.841864 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:13.840817 20292 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:13.866518 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55278.7us
I20260709 12:22:13.842844 20430 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:13.853293 20437 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:13.822347 20424 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:13.882807 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 250627us
W20260709 12:22:13.888837 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.874348 20292 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:13.883785 20506 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:13.917455 18169 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:13.936872 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60042.2us
W20260709 12:22:13.938524 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 187364us
W20260709 12:22:13.939550 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.787s (SENT)
I20260709 12:22:13.940209 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:13.963186 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:13.963353 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:13.964339 18168 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:13.967803 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:13.968616 18168 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:13.971405 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.980857 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:13.986860 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:13.998250 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 129486us
I20260709 12:22:14.001048 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:14.002526 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:14.004463 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:14.006776 18023 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:14.010183 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:14.011691 18815 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:14.013216 18015 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:14.018373 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 384576us
I20260709 12:22:14.013213 20472 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.026954 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:14.006939 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:13.981839 20543 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.037842 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:14.048895 18095 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:14.013088 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:14.048038 18813 raft_consensus.cc:2376] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:14.054764 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:14.056759 18094 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:14.054790 18811 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:14.074460 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:14.075604 18165 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:14.056595 18807 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:14.089087 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:14.089381 18015 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:14.044584 20384 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:22:14.092480 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.093987 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89654us
I20260709 12:22:14.097369 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:14.100826 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:14.105187 18168 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:14.107769 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:14.077111 18018 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:14.074807 20487 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.066570 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:14.119858 18164 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:22:14.120376 18023 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:13.924960 20503 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.122350 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:14.124089 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:14.131637 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 192169us
I20260709 12:22:14.132230 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:14.137425 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.129164 18164 pending_rounds.cc:85] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 5
I20260709 12:22:14.139292 18164 pending_rounds.cc:107] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:22:14.139703 18164 raft_consensus.cc:2889] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:14.106437 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:14.164760 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57110.1us
W20260709 12:22:14.167131 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.809s (SENT)
I20260709 12:22:14.167814 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:14.131304 20558 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:22:14.075273 20543 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:14.137089 20468 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.641s	user 0.018s	sys 0.000s
I20260709 12:22:14.162671 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:14.158155 20562 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:22:14.179507 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:14.185762 20557 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.607s	user 0.004s	sys 0.001s
W20260709 12:22:14.189543 20564 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:22:14.190719 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121088us
I20260709 12:22:14.198748 20424 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:14.220427 18094 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:14.222719 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1707) took 1096 ms. Trace:
I20260709 12:22:14.223349 18098 rpcz_store.cc:276] 0709 12:22:13.148149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.162132 (+ 13983us) service_pool.cc:224] Handling call
0709 12:22:13.230243 (+ 68111us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.231066 (+   823us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:13.231129 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.231981 (+   852us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.447842 (+215861us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:13.447911 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.448022 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.184711 (+736689us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:14.222310 (+ 37599us) raft_consensus.cc:1685] finished
0709 12:22:14.222343 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.222607 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7129,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":189,"spinlock_wait_cycles":38907264,"thread_start_us":215249,"threads_started":1,"wal-append.queue_time_us":345371}
I20260709 12:22:14.228606 20505 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:14.214768 20379 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:14.203419 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 120334us
I20260709 12:22:14.123204 20506 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.219010 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:14.207676 18099 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:14.135301 20523 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.242167 20442 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:14.241390 20494 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:14.241911 20384 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:14.135797 20570 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.096s
W20260709 12:22:14.216189 20473 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.832s	user 0.015s	sys 0.001s
W20260709 12:22:14.245297 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78916.5us
I20260709 12:22:14.144958 19910 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.241564 20379 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:14.103214 20437 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:14.252383 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 182926us
I20260709 12:22:14.254375 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:14.272830 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 105753us
I20260709 12:22:14.259865 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:14.274901 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.280668 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:14.213744 20468 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.303138 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:14.304715 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:14.275882 20384 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:14.320078 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.320776 20535 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:14.325640 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.929s (SENT)
I20260709 12:22:14.327308 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:14.299978 20494 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:22:14.321638 20278 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.661s	user 0.015s	sys 0.011s
I20260709 12:22:14.320734 20473 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:14.334020 20494 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.357955 18138 leader_election.cc:400] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4.
I20260709 12:22:14.370204 18138 leader_election.cc:403] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:22:14.358685 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2442) took 1510 ms. Trace:
I20260709 12:22:14.339078 19910 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.362567 20468 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.360956 20384 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.364323 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:14.365123 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 110848us
W20260709 12:22:14.365904 18024 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.540s	user 0.007s	sys 0.003s
I20260709 12:22:14.371102 18019 rpcz_store.cc:276] 0709 12:22:12.847971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.414829 (+566858us) service_pool.cc:224] Handling call
0709 12:22:13.415273 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.462896 (+ 47623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:13.462954 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.479621 (+ 16667us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.839032 (+359411us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:14.233133 (+394101us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b18000825a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf
0709 12:22:14.233329 (+   196us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.239459 (+  6130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.239649 (+   190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.342901 (+103252us) raft_consensus.cc:1685] finished
0709 12:22:14.358429 (+ 15528us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.358604 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8841,"prepare.queue_time_us":170366,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":70,"raft.queue_time_us":5825,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"spinlock_wait_cycles":92159232,"thread_start_us":370586,"threads_started":2,"wal-append.queue_time_us":434394}
I20260709 12:22:14.377866 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:14.387773 18024 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:14.384004 20518 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.657s	user 0.017s	sys 0.002s
I20260709 12:22:14.345546 20543 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.376080 20494 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:14.378945 20278 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4.
I20260709 12:22:14.382243 20535 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:14.340938 20505 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
W20260709 12:22:14.411173 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127578us
I20260709 12:22:14.414397 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:14.393929 20468 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.386927 20473 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:14.429286 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 115055us
I20260709 12:22:14.423501 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:14.416808 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:14.398711 20379 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:14.439779 20580 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
W20260709 12:22:14.449672 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2453) took 1505 ms (client timeout 1901 ms). Trace:
W20260709 12:22:14.450589 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 147541us
W20260709 12:22:14.461261 18024 rpcz_store.cc:269] 0709 12:22:12.944476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.302473 (+357997us) service_pool.cc:224] Handling call
0709 12:22:14.449382 (+1146909us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:14.449574 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":959266176}
W20260709 12:22:14.462186 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.461943 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:14.421160 20569 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:14.466087 18098 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:14.468254 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:14.411381 20518 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:14.423032 19910 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:14.420176 20535 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:14.485093 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.486873 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:14.453761 20296 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:14.468791 20555 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:14.493268 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:14.479002 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:14.419759 20462 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.062s
I20260709 12:22:14.471712 20543 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.480819 20472 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:14.479017 19910 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:14.507330 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89290us
I20260709 12:22:14.509312 18092 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:22:14.445745 20468 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.514806 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:14.517488 20468 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:14.519307 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:14.474784 20506 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:22:14.488179 20588 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:14.505988 20360 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.027s	user 0.015s	sys 0.008s
I20260709 12:22:14.496937 20296 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.489049 20523 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:22:14.508173 20364 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:14.522904 19910 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:14.548840 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 214532us
I20260709 12:22:14.525203 20398 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:22:14.552973 18097 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:14.535706 20296 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:14.544819 20555 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:22:14.552414 20169 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:14.555809 20535 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:14.578984 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74593.8us
I20260709 12:22:14.544895 20360 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:14.586016 20584 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:22:14.586499 20360 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:14.532115 20473 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.591100 20462 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:22:14.591902 20501 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.821s	user 0.014s	sys 0.003s
I20260709 12:22:14.593470 18172 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:14.594599 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.527858 20518 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:14.621096 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 160119us
W20260709 12:22:14.623139 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 108474us
W20260709 12:22:14.626497 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67159.4us
W20260709 12:22:14.630146 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 314798us
I20260709 12:22:14.620476 20462 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:14.634403 20292 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.656s	user 0.012s	sys 0.003s
W20260709 12:22:14.653635 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.059s (SENT)
I20260709 12:22:14.654282 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:14.661339 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:14.662623 18021 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:14.622428 20588 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:22:14.648329 20462 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:14.667781 20437 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:14.671798 20605 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.649781 20472 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:14.676808 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51104.6us
I20260709 12:22:14.677174 18090 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:14.681447 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:14.682581 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:14.695807 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:14.761312 18090 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:14.762807 18090 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.764343 18090 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:14.766296 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:14.774994 18099 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:14.791517 18093 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:14.706149 20605 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.769237 20494 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:14.792526 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 160605us
I20260709 12:22:14.673965 20603 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.696866 20505 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:14.809442 20589 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260709 12:22:14.809901 20579 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.321s	user 0.000s	sys 0.002s
I20260709 12:22:14.704350 20278 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:14.654259 20292 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.810904 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:14.778314 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:14.768353 20572 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:22:14.813385 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:14.865640 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52385.7us
I20260709 12:22:14.830343 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:14.829264 20505 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:14.866235 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 220286us
I20260709 12:22:14.867137 18024 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:14.880896 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:14.881889 18810 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:14.882853 20360 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:14.883425 20360 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:14.883836 20360 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:14.884953 20472 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.887991 20605 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.889107 20613 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:14.890676 20597 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:14.892287 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 192384us
I20260709 12:22:14.864271 20501 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.841686 20278 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:14.865038 20384 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:14.900813 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:14.905200 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 139296us
I20260709 12:22:14.868098 20505 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:14.915140 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:14.883823 20593 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260709 12:22:14.826915 20437 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.891394 20597 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:14.890707 20608 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.934325 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:14.906049 20595 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:14.941314 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2447) took 1019 ms. Trace:
I20260709 12:22:14.941973 18091 rpcz_store.cc:276] 0709 12:22:13.922307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.922466 (+   159us) service_pool.cc:224] Handling call
0709 12:22:13.981142 (+ 58676us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:14.003460 (+ 22318us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.003505 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:14.037734 (+ 34229us) log.cc:844] Serialized 26 byte log entry
0709 12:22:14.256487 (+218753us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b18002a3a60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:22:14.466494 (+210007us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:14.466523 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.466576 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.935453 (+468877us) raft_consensus.cc:1685] finished
0709 12:22:14.935486 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.941204 (+  5718us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32702,"raft.queue_time_us":48579,"raft.run_cpu_time_us":1895,"raft.run_wall_time_us":26127,"spinlock_wait_cycles":7158528,"thread_start_us":209807,"threads_started":1,"wal-append.queue_time_us":529772}
I20260709 12:22:14.951778 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:14.857012 20603 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.960152 20595 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:14.952629 18813 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:14.950332 20608 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.961084 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:14.961609 20595 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.936699 20617 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.976388 20616 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:14.977538 20616 consensus_queue.cc:1243] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:14.973654 18815 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:14.981954 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:14.919528 20501 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:14.954061 20169 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:14.956754 20518 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.922511 20296 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:14.999097 20598 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:22:15.007704 20292 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.981194 20617 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:15.015226 20617 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:14.993505 20618 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:15.018621 20618 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:15.019129 18810 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:15.020227 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:15.019805 20618 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.029145 20501 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:15.031836 20604 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:15.032255 19910 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.037202 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.038245 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.039280 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.039676 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.040343 18094 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:15.040746 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.041399 18097 raft_consensus.cc:2376] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4. Re-sending same reply.
I20260709 12:22:15.049662 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:15.050810 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:15.051759 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.994117 20169 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:15.055389 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:15.038270 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56429.4us
I20260709 12:22:15.060053 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:14.978483 20278 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:15.073154 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:15.065831 20169 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:15.071943 18098 raft_consensus.cc:2376] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3. Re-sending same reply.
I20260709 12:22:15.056468 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:15.079156 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.058993 20494 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:15.083299 18095 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 2: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:15.087167 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:15.094918 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.099913 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.100183 18091 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:15.102007 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.096046 18097 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:15.081032 20437 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:15.105283 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.009s	sys 0.048s
I20260709 12:22:15.089859 20278 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:15.089648 20494 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:15.111424 18098 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:15.092270 20618 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:15.087291 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:15.115777 20296 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:15.101560 20605 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.106973 18094 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:15.090266 20597 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:15.118026 18806 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:22:15.106973 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.010s	sys 0.048s
I20260709 12:22:15.119602 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.130919 18095 raft_consensus.cc:2376] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3. Re-sending same reply.
I20260709 12:22:15.114638 20613 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.132181 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.120609 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.133554 18097 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:15.134589 18091 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:15.136039 18091 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:15.127295 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:15.129596 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.129770 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.132251 20613 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:15.138346 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 593696us
I20260709 12:22:15.110963 20437 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.147841 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.155591 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.160140 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.163200 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:15.164064 18168 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:15.168574 20398 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:15.182336 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.183718 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:15.128643 20462 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.142537 20605 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.433274 18024 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:15.236279 18021 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:15.197459 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1641) took 1577 ms. Trace:
I20260709 12:22:15.249181 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:15.191192 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:15.229391 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.304242 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:15.216396 18168 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:15.489693 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 381486us
W20260709 12:22:15.368884 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 416497us
I20260709 12:22:15.489143 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.498122 18020 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:22:15.502992 18091 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:15.186754 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.190698 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.257241 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:15.434330 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:15.504029 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.496832 18172 rpcz_store.cc:276] 0709 12:22:13.619744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.619967 (+   223us) service_pool.cc:224] Handling call
0709 12:22:13.631025 (+ 11058us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.627234 (+996209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.197091 (+569857us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x7b54003011d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:15.197366 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2212782848}
I20260709 12:22:15.518091 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:15.283581 20563 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.220708 20398 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.289036 20503 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:15.496551 20472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.014s	sys 0.011s
W20260709 12:22:15.525312 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 276283us
I20260709 12:22:15.517556 18017 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:22:15.186689 20468 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:15.527570 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:15.513646 18015 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:15.169742 20437 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:15.495888 20592 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.305248 20506 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:15.572999 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46562.7us
I20260709 12:22:15.504559 20623 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:15.583216 18023 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:15.575448 20472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.015s	sys 0.011s
I20260709 12:22:15.568113 18173 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:15.575537 18016 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 12:22:15.584971 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66980.1us
I20260709 12:22:15.595181 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:15.353636 20518 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:15.567442 20505 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.005s
I20260709 12:22:15.604048 20505 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:15.518922 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.009s	sys 0.016s
W20260709 12:22:15.606070 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.010s	sys 0.016s
I20260709 12:22:15.610807 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.611984 18091 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:15.598814 18024 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:22:15.588907 20595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.029s	sys 0.012s
I20260709 12:22:15.617691 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:15.619029 18015 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:15.586890 20575 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:15.527921 20533 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20260709 12:22:15.621543 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.632370 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.637719 18169 raft_consensus.cc:2368] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5.
I20260709 12:22:15.596539 20424 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:15.615903 20595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.031s	sys 0.012s
I20260709 12:22:15.592679 20506 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:15.652993 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.653524 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:15.591157 20468 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:15.654246 18024 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
W20260709 12:22:15.644495 18094 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.502s	user 0.014s	sys 0.006s
I20260709 12:22:15.578331 20398 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:15.657433 18168 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:22:15.660697 18021 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
W20260709 12:22:15.642858 20621 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:22:15.576608 20563 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.644719 20505 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:15.652022 20618 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.535s	user 0.014s	sys 0.000s
I20260709 12:22:15.655403 18094 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:15.691030 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.692901 20379 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:15.704164 20398 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:15.691468 20487 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.597312 20592 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.629878 20518 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.707708 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 514) took 1479 ms. Trace:
I20260709 12:22:15.719636 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:15.744268 18021 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
W20260709 12:22:15.720770 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.672468 20613 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:15.739933 18094 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:15.731364 20473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.062s	sys 0.110s
I20260709 12:22:15.745263 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:15.708653 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87423.1us
I20260709 12:22:15.655557 20424 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.743976 18811 rpcz_store.cc:276] 0709 12:22:14.228815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.229387 (+   572us) service_pool.cc:224] Handling call
0709 12:22:14.229900 (+   513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.234063 (+  4163us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:14.648467 (+414404us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.670773 (+ 22306us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.868569 (+197796us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:14.868636 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:14.868703 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.707309 (+838606us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b54004e5f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:15.707375 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.707555 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144673,"prepare.queue_time_us":108221,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":338835,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":8772,"spinlock_wait_cycles":8685824,"thread_start_us":512583,"threads_started":3}
I20260709 12:22:15.758275 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:15.759979 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:15.760902 20424 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:15.762418 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.755012 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:15.752369 18097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.002s	sys 0.022s
W20260709 12:22:15.766703 18097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.003s	sys 0.022s
I20260709 12:22:15.767385 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:15.768527 18024 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:15.757462 20473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.063s	sys 0.110s
I20260709 12:22:15.780776 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.781715 18094 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.769421 20364 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:15.796696 18172 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:15.769488 20169 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.692s	user 0.017s	sys 0.000s
I20260709 12:22:15.798461 18021 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:15.761744 20424 raft_consensus.cc:2764] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260709 12:22:15.811007 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.811492 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:15.743439 20379 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.818859 18021 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:15.822042 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:15.751596 20487 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:15.807284 20364 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:22:15.841089 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 265001us
I20260709 12:22:15.705775 20506 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:15.825075 18778 connection.cc:256] RPC call timeout handler was delayed by -1.8148539819999314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
W20260709 12:22:15.819422 20616 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.017s	sys 0.017s
W20260709 12:22:15.849486 18778 connection.cc:256] RPC call timeout handler was delayed by -1.8142670130000624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
I20260709 12:22:15.863273 18091 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:15.810329 20462 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:15.866719 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67642.6us
I20260709 12:22:15.813756 20608 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:15.884521 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.043s (SENT)
I20260709 12:22:15.879766 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:15.849941 20364 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:15.894418 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:15.867805 20616 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.018s	sys 0.017s
I20260709 12:22:15.893132 18021 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:15.923236 18094 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:15.922398 18091 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:15.925509 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:15.813428 20169 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:15.930550 18023 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: [3.3-4.4]
I20260709 12:22:15.931875 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:15.881395 18015 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:15.931911 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.924885 20462 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.892287 20506 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:15.945871 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:15.944375 18094 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:15.946488 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58412us
I20260709 12:22:15.948094 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:15.938782 20398 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:22:15.952693 20442 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:15.953505 20442 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:15.954227 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74165.4us
W20260709 12:22:15.954914 18778 connection.cc:256] RPC call timeout handler was delayed by -1.729836205999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 12:22:15.955248 18778 connection.cc:256] RPC call timeout handler was delayed by -1.6526188049997472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2848s for a response
W20260709 12:22:15.955534 18778 connection.cc:256] RPC call timeout handler was delayed by -1.642517432999739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3435s for a response
I20260709 12:22:15.956146 20442 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:15.956449 18778 connection.cc:256] RPC call timeout handler was delayed by -1.4980539819998739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20260709 12:22:15.957271 18024 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:15.939162 20618 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:15.967259 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:15.922423 20437 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:15.967415 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:15.965744 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.973071 18091 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:15.920902 20464 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.026s	sys 0.040s
I20260709 12:22:15.978101 18170 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:15.944661 20468 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:15.989306 20487 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:15.993335 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:15.994212 18093 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:15.995105 20379 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:15.957038 20442 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:16.000519 20398 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:16.001140 20398 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:16.001837 20296 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:16.002219 20398 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:16.003185 20360 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.119s	user 0.011s	sys 0.007s
I20260709 12:22:16.003784 20398 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:16.003693 20360 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.006235 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:16.007403 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.007740 20296 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:16.008764 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.009429 18172 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:16.010000 20603 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:16.010593 20637 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:22:16.010914 20296 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:16.011905 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260709 12:22:16.011819 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:16.012660 18813 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:22:15.974737 20278 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.860s	user 0.016s	sys 0.001s
I20260709 12:22:16.014058 20563 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:16.015579 20628 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:22:16.015693 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.016068 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:16.017483 18092 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:16.018318 18166 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:15.977120 20597 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.942772 20462 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.039211 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 308) took 1173 ms. Trace:
I20260709 12:22:16.039860 18019 rpcz_store.cc:276] 0709 12:22:14.843753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.843948 (+   195us) service_pool.cc:224] Handling call
0709 12:22:14.871815 (+ 27867us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:14.905432 (+ 33617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.905470 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:14.914457 (+  8987us) log.cc:844] Serialized 26 byte log entry
0709 12:22:15.488505 (+574048us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:15.488572 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.488666 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.017088 (+528422us) raft_consensus.cc:1685] finished
0709 12:22:16.017116 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.017267 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13802,"raft.queue_time_us":43539,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":18187,"thread_start_us":433671,"threads_started":1,"wal-append.queue_time_us":839366}
I20260709 12:22:16.041966 18021 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:15.971979 20533 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:16.019735 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 495001us
I20260709 12:22:16.022012 18170 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:16.035221 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.012931 20296 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.043318 18171 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:15.981519 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.017098 20592 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:16.057086 18164 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:16.062045 18814 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:16.063230 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:15.980314 20464 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.026s	sys 0.040s
I20260709 12:22:16.006378 20360 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:16.070264 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.071441 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.044023 20169 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:16.072456 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:16.074225 18094 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:16.074249 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.013s	sys 0.004s
W20260709 12:22:16.077396 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.013s	sys 0.004s
W20260709 12:22:16.079985 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.080585 18064 connection.cc:256] RPC call timeout handler was delayed by -2.781477272000302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3014s for a response
I20260709 12:22:16.062567 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:16.088587 18093 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:16.095746 18167 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:16.096540 18809 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:16.071013 20533 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:16.098407 20533 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.055550 20597 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:16.103941 20364 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:16.104750 20597 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:16.106492 18167 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:16.107216 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.111111 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.112201 18099 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 1.
I20260709 12:22:16.114254 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.115175 18092 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:16.118218 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:16.115672 20364 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:16.113632 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:16.120556 18099 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:16.121399 20462 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:16.121801 18091 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:16.126035 18064 connection.cc:256] RPC call timeout handler was delayed by -2.6275772720000532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3185s for a response
W20260709 12:22:16.126902 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9322353640000074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17090000000000002s for a response
W20260709 12:22:16.127285 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8602353640003457s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1789s for a response
W20260709 12:22:16.127612 18064 connection.cc:256] RPC call timeout handler was delayed by -1.783735364000222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20260709 12:22:16.127952 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7495353640001667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260709 12:22:16.128286 18064 connection.cc:256] RPC call timeout handler was delayed by -1.724335364000126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
W20260709 12:22:16.128624 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6784353640000518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
I20260709 12:22:16.129433 20592 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:16.130555 20592 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.130997 20563 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.132296 20563 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:16.134289 20468 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:16.135118 20468 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.136613 20563 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:16.129004 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6775353640000503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19920000000000002s for a response
W20260709 12:22:16.155288 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5794353640003465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
W20260709 12:22:16.158025 18064 connection.cc:256] RPC call timeout handler was delayed by -1.50743536400023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
W20260709 12:22:16.159803 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4516353640001398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20260709 12:22:16.166973 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.168807 18173 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:16.168123 18170 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:16.069423 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 311) took 1169 ms. Trace:
I20260709 12:22:16.175374 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:16.177233 20473 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:16.178081 20473 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:16.179479 20473 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.182806 20364 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:22:16.172981 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 496221us
I20260709 12:22:16.195972 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:16.199891 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.254456 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.255502 18092 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:16.202994 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:16.256999 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.257730 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:16.209643 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:16.259325 20503 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.681s	user 0.012s	sys 0.001s
I20260709 12:22:16.260032 18095 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:16.212761 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:16.218065 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.234248 20604 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.260427 20620 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.262444 20604 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:16.216235 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:16.225564 20360 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:16.266119 20360 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:16.263616 20620 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.267968 20503 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:16.235574 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.236657 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.268846 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.232548 18094 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:16.238238 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.240139 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.211931 20618 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:22:16.258270 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.293920 18165 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:16.294451 18099 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:16.260615 20396 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:16.309994 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114160us
I20260709 12:22:16.311466 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:16.224027 20592 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:16.188647 20535 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.000s	sys 0.023s
I20260709 12:22:16.272508 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.331692 18091 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:16.272877 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:16.278126 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68588us
W20260709 12:22:16.264616 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48499.2us
I20260709 12:22:16.270028 18093 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:16.270498 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.335536 18092 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:16.264182 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72665us
I20260709 12:22:16.273782 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:16.187863 20588 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.413s	user 0.035s	sys 0.006s
I20260709 12:22:16.267421 20360 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.318612 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:16.351868 18815 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 2.
I20260709 12:22:16.353157 18024 raft_consensus.cc:2376] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
I20260709 12:22:16.329284 18099 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:16.242990 20575 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:16.355579 18094 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:16.339649 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:16.357650 18017 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:22:16.273097 20620 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:16.264817 20384 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:16.239050 20468 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:16.364774 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:16.337812 20588 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.570s	user 0.035s	sys 0.007s
I20260709 12:22:16.368927 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:16.369679 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:16.362268 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:16.373898 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:16.326769 20517 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:16.405224 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:16.406006 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 439268us
I20260709 12:22:16.326546 20472 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:16.381055 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.407378 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.271617 20605 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:16.269737 19910 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:16.336997 20535 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.000s	sys 0.024s
I20260709 12:22:16.410807 18023 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:16.352994 20592 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:16.412947 18024 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:16.371294 20384 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:16.435711 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:16.426023 20643 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:22:16.401105 20437 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:22:16.327067 20396 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:16.327109 20364 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:16.354779 20575 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:16.405464 20647 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:16.265650 20604 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:16.453117 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:16.439337 18015 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:16.442936 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 485356us
W20260709 12:22:16.439319 20570 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.016s	sys 0.051s
I20260709 12:22:16.434837 20487 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260709 12:22:16.360661 20618 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.406003 20517 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.321708 20462 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:16.461508 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:16.434928 20620 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:16.463927 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:16.455044 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:16.461432 20570 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.016s	sys 0.051s
W20260709 12:22:16.459390 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.632s (SENT) [suppressed 22 similar messages]
I20260709 12:22:16.441035 20384 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:16.445258 18021 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:16.548978 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.576234 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:16.449131 20506 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.450687 20364 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.452888 20647 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.449144 20648 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.430524 20472 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:16.448058 20592 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:16.818470 18168 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.012s	sys 0.023s
I20260709 12:22:17.060963 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:17.068318 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 604463us
W20260709 12:22:16.891350 18166 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.008s	sys 0.033s
I20260709 12:22:16.814388 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:17.069495 18166 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.012s	sys 0.037s
I20260709 12:22:17.069975 18015 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:17.059502 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:16.855518 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.067597 18168 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.922s	user 0.018s	sys 0.031s
W20260709 12:22:17.062597 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.004s	sys 0.006s
W20260709 12:22:17.071933 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.004s	sys 0.006s
W20260709 12:22:16.682714 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.075485 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 616079us
I20260709 12:22:16.174284 18018 rpcz_store.cc:276] 0709 12:22:14.899934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.904912 (+  4978us) service_pool.cc:224] Handling call
0709 12:22:14.940514 (+ 35602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.958104 (+ 17590us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:15.118884 (+160780us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b18002a34c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:15.566485 (+447601us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.972617 (+406132us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.974788 (+  2171us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:15.981592 (+  6804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.037946 (+ 56354us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.065714 (+ 27768us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4045,"prepare.queue_time_us":58525,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":19177,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":49316,"spinlock_wait_cycles":27913344,"thread_start_us":839336,"threads_started":2,"wal-append.queue_time_us":627759}
W20260709 12:22:16.879200 18167 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.004s	sys 0.035s
W20260709 12:22:17.102684 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.733s (SENT)
I20260709 12:22:17.083372 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:17.104274 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:17.105751 18167 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.010s	sys 0.040s
I20260709 12:22:17.072893 18171 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:22:16.887259 20473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.010s	sys 0.024s
W20260709 12:22:17.108248 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.011s	sys 0.012s
I20260709 12:22:17.109310 18164 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:17.115821 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.012s	sys 0.012s
W20260709 12:22:16.854816 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.021s	sys 0.016s
W20260709 12:22:16.947502 20296 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.023s	sys 0.087s
I20260709 12:22:17.075835 20364 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:17.117918 20645 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.447s	user 0.000s	sys 0.001s
I20260709 12:22:17.087404 20648 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:16.721040 20505 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:16.996759 20623 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.004s	sys 0.021s
I20260709 12:22:16.956125 20643 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.594576 20563 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:16.574587 20533 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.142187 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.001s	sys 0.014s
I20260709 12:22:17.115494 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:17.142724 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.001s	sys 0.014s
I20260709 12:22:17.074108 20620 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.128196 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:17.083067 20437 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:17.102816 19910 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:17.154467 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:17.128685 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 723300us
I20260709 12:22:17.126902 18021 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:16.879738 20613 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:17.068447 20517 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:16.763964 20615 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:17.115764 20473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.011s	sys 0.025s
W20260709 12:22:17.005167 20442 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49522 (request call id 83) took 23436 ms (client timeout 29999 ms). Trace:
I20260709 12:22:17.082435 20647 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:17.121189 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.026s	sys 0.023s
W20260709 12:22:17.144364 20623 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.005s	sys 0.022s
W20260709 12:22:17.134750 20296 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.028s	sys 0.088s
I20260709 12:22:16.764524 20604 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.371s
W20260709 12:22:17.167022 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64390.9us
I20260709 12:22:17.172024 18023 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
W20260709 12:22:17.174991 20642 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.964s	user 0.009s	sys 0.024s
W20260709 12:22:17.150956 20398 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.147s	user 0.014s	sys 0.003s
I20260709 12:22:17.177975 20651 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.120690 20462 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:16.626798 20618 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.134460 20605 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:17.202057 20605 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:17.202641 20605 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.203007 20605 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:17.082747 20506 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:17.207998 20506 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:17.208460 20364 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.163920 20437 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.211755 20437 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:17.165026 20620 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:17.151710 20643 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.216461 20643 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:17.164451 20533 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.217973 20651 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.219254 20533 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:17.211592 18096 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:22:17.181134 20615 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.165616 20563 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:17.172334 20442 rpcz_store.cc:269] 0709 12:21:53.501190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.502215 (+  1025us) service_pool.cc:224] Handling call
0709 12:21:53.555098 (+ 52883us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b54002ec5a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c16634520 00007f6c166bdb42 00007f6c166bc800 00007f6c166b59fa 00007f6c1b67d4c9 00007f6c14bf9c54 00007f6c14bf9be2 00007f6c14bf9b11 00007f6c14bf9a8c 00007f6c14bf8d42
0709 12:21:53.600319 (+ 45221us) log.cc:844] Serialized 494 byte log entry
0709 12:21:53.974928 (+374609us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b180033ff40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c16637b26 00007f6c1663149a 00007f6c16630714
0709 12:21:54.007390 (+ 32462us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b54002ec5a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c16637b26 00007f6c1663149a 00007f6c16630714 00007f6c16610eaa 00007f6c16617d53 00007f6c16617cd9 00007f6c16617c69 00007f6c16617c31 00007f6c16616dfd 0000000000388634
0709 12:21:55.227886 (+1220496us) spinlock_profiling.cc:243] Waited 7.12 ms on lock 0x7b54002ec5a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c16630142 00007f6c16610eaa 00007f6c16617d53 00007f6c16617cd9 00007f6c16617c69 00007f6c16617c31 00007f6c16616dfd 0000000000388634 0000000000388479 00007f6c135ca475
0709 12:22:16.891490 (+21663604us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60050,"raft.queue_time_us":16160,"raft.run_cpu_time_us":33449,"raft.run_wall_time_us":1982725,"spinlock_wait_cycles":81892480,"thread_start_us":40128,"threads_started":1,"wal-append.queue_time_us":85129,"wal-append.run_cpu_time_us":4356,"wal-append.run_wall_time_us":1273517}
I20260709 12:22:17.223532 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:17.228479 18020 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.006s	sys 0.029s
I20260709 12:22:16.892488 20575 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:17.229034 18020 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.007s	sys 0.029s
I20260709 12:22:17.180493 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:17.193437 19910 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:17.439465 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:17.164867 20613 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:17.441066 18024 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:17.446344 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.834s (SENT)
I20260709 12:22:17.453169 18023 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:17.474160 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2476) took 1046 ms. Trace:
I20260709 12:22:17.474716 18024 rpcz_store.cc:276] 0709 12:22:16.427279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.435826 (+  8547us) service_pool.cc:224] Handling call
0709 12:22:17.474081 (+1038255us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:17.475739 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.015s	sys 0.040s
W20260709 12:22:17.477417 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.015s	sys 0.040s
I20260709 12:22:17.477744 18015 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:17.480463 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2481) took 1026 ms. Trace:
I20260709 12:22:17.480866 18015 rpcz_store.cc:276] 0709 12:22:16.453172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.489136 (+ 35964us) service_pool.cc:224] Handling call
0709 12:22:17.479007 (+989871us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:17.486269 20473 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.486785 20396 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:17.487340 20563 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:17.486979 20473 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:17.487504 20396 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:17.488219 20651 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:17.489096 18096 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:17.488736 20613 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:17.489921 20618 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:17.490407 20613 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:17.491019 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 195175us
W20260709 12:22:17.491885 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.186s (SENT)
I20260709 12:22:17.492313 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:17.494755 20618 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:17.493765 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.848s (SENT)
I20260709 12:22:17.498703 20648 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:17.480733 20296 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:17.500190 20296 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:17.500736 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:17.501740 18813 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:17.504688 20618 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:17.505513 20618 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:17.507705 20644 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.081s	user 0.005s	sys 0.008s
I20260709 12:22:17.507383 20462 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:17.508680 20618 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:17.513087 18096 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:17.514012 20379 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:17.514499 20379 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:17.515199 18018 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:22:17.515604 20379 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.516480 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:17.517133 20379 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:17.517366 18096 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:17.520324 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:17.522782 18015 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:17.523626 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2477) took 1096 ms. Trace:
I20260709 12:22:17.527526 18023 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:22:17.535763 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:17.538956 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 1953) took 2162 ms. Trace:
W20260709 12:22:17.540491 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.542120 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2474) took 1115 ms. Trace:
I20260709 12:22:17.524060 18015 rpcz_store.cc:276] 0709 12:22:16.427624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.521896 (+1094272us) service_pool.cc:224] Handling call
0709 12:22:17.522334 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.523272 (+   938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.523545 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:17.544515 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2467) took 2390 ms. Trace:
I20260709 12:22:17.539465 18016 rpcz_store.cc:276] 0709 12:22:15.370810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.371267 (+   457us) service_pool.cc:224] Handling call
0709 12:22:15.414788 (+ 43521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:15.598304 (+183516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.532577 (+1934273us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:17.532870 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":259719936}
W20260709 12:22:17.546543 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 432) took 2406 ms (client timeout 1689 ms). Trace:
W20260709 12:22:17.555454 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.556950 20396 raft_consensus.cc:2764] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:17.542762 18023 rpcz_store.cc:276] 0709 12:22:16.426640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.484737 (+1058097us) service_pool.cc:224] Handling call
0709 12:22:17.527181 (+ 42444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.541621 (+ 14440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.542003 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:17.501274 20296 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:17.577363 20296 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:17.515623 18018 pending_rounds.cc:85] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 1
I20260709 12:22:17.577951 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 600) took 1376 ms. Trace:
I20260709 12:22:17.578811 18092 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:17.483177 20604 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:17.578100 18018 pending_rounds.cc:107] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 3.2
W20260709 12:22:17.582155 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.548306 18098 rpcz_store.cc:276] 0709 12:22:15.144899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.145180 (+   281us) service_pool.cc:224] Handling call
0709 12:22:15.189700 (+ 44520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:15.192554 (+  2854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:15.192591 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:15.193267 (+   676us) log.cc:844] Serialized 26 byte log entry
0709 12:22:16.078606 (+885339us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:16.078676 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.080252 (+  1576us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.506958 (+1426706us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:17.527368 (+ 20410us) raft_consensus.cc:1685] finished
0709 12:22:17.527393 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.535092 (+  7699us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3723,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":26502,"spinlock_wait_cycles":9830784,"thread_start_us":589015,"threads_started":1,"wal-append.queue_time_us":1157913}
W20260709 12:22:17.556715 18095 rpcz_store.cc:269] 0709 12:22:15.120771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.131840 (+ 11069us) service_pool.cc:224] Handling call
0709 12:22:17.526943 (+2395103us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:17.527116 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2591966592}
W20260709 12:22:17.588701 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.012s	sys 0.012s
W20260709 12:22:17.544116 20666 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 12:22:17.594244 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.021s	sys 0.020s
W20260709 12:22:17.594735 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.021s	sys 0.021s
W20260709 12:22:17.596005 18017 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.013s	sys 0.034s
I20260709 12:22:17.535815 19910 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:17.592165 18018 raft_consensus.cc:2889] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:17.596719 18017 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.013s	sys 0.034s
W20260709 12:22:17.613997 18166 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.010s	sys 0.027s
W20260709 12:22:17.616746 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.613423 20675 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:17.601018 20506 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.635506 20675 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:17.604295 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 488856us
I20260709 12:22:17.636480 20506 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:17.637657 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:17.620596 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 486342us
W20260709 12:22:17.616662 18166 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.010s	sys 0.028s
I20260709 12:22:17.643005 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:17.601238 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2473) took 1178 ms. Trace:
I20260709 12:22:17.645298 18022 rpcz_store.cc:276] 0709 12:22:16.422558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.441768 (+ 19210us) service_pool.cc:224] Handling call
0709 12:22:16.454436 (+ 12668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.486652 (+ 32216us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:17.044126 (+557474us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.600810 (+556684us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.600876 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:17.600945 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.601024 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.601173 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15437,"prepare.queue_time_us":746605,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":25171,"raft.queue_time_us":15571,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":5037,"thread_start_us":938525,"threads_started":2}
I20260709 12:22:17.578505 18809 rpcz_store.cc:276] 0709 12:22:16.201772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.211743 (+  9971us) service_pool.cc:224] Handling call
0709 12:22:16.242983 (+ 31240us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.243417 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.243471 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:16.244375 (+   904us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.190080 (+945705us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:17.201933 (+ 11853us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.202107 (+   174us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.577669 (+375562us) raft_consensus.cc:1685] finished
0709 12:22:17.577703 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.577872 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":935178,"threads_started":1,"wal-append.queue_time_us":1312263}
I20260709 12:22:17.674113 20472 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:17.679023 20675 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:17.592903 20604 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.682646 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:17.690006 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2471) took 1330 ms. Trace:
I20260709 12:22:17.683727 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 365) took 1554 ms. Trace:
I20260709 12:22:17.654353 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:17.720412 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:17.657316 20648 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:17.755721 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 532316us
W20260709 12:22:17.756894 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 102673us
I20260709 12:22:17.757227 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:17.715627 18166 rpcz_store.cc:276] 0709 12:22:16.129881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.130117 (+   236us) service_pool.cc:224] Handling call
0709 12:22:16.131231 (+  1114us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.131532 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:17.070120 (+938588us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.683039 (+612919us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.683143 (+   104us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:17.683247 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.683376 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.683613 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42027,"prepare.queue_time_us":996786,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":11318,"thread_start_us":1508422,"threads_started":2,"wal-append.queue_time_us":627309}
I20260709 12:22:17.715031 18017 rpcz_store.cc:276] 0709 12:22:16.359263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.361102 (+  1839us) service_pool.cc:224] Handling call
0709 12:22:16.399071 (+ 37969us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.426733 (+ 27662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.426770 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:16.427384 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.072421 (+645037us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:17.615877 (+543456us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.616220 (+   343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.616317 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.689562 (+ 73245us) raft_consensus.cc:1685] finished
0709 12:22:17.689600 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.689893 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53531,"prepare.queue_time_us":421534,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":30073,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":24852,"thread_start_us":1187454,"threads_started":2,"wal-append.queue_time_us":764779}
W20260709 12:22:17.715400 20570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.000s	sys 0.039s
I20260709 12:22:17.693966 20618 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.753738 18171 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.022s	sys 0.010s
W20260709 12:22:17.809619 18171 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.022s	sys 0.010s
I20260709 12:22:17.761502 20364 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:17.695183 20623 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.125s
I20260709 12:22:17.761840 20648 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.751950 20472 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:17.754686 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2478) took 1323 ms. Trace:
W20260709 12:22:17.810434 20570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.001s	sys 0.044s
W20260709 12:22:17.832239 20629 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.608s	user 0.019s	sys 0.032s
W20260709 12:22:17.865468 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 127985us
W20260709 12:22:17.869014 20629 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.657s	user 0.019s	sys 0.032s
W20260709 12:22:17.880930 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.882939 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 366) took 1738 ms. Trace:
I20260709 12:22:17.895110 20677 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:17.896207 20677 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.859390 20648 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:17.886633 18168 rpcz_store.cc:276] 0709 12:22:16.144348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.144538 (+   190us) service_pool.cc:224] Handling call
0709 12:22:16.145046 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.145379 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:17.072033 (+926654us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.882455 (+810422us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.882541 (+    86us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:17.882604 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.882694 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.882863 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1006014,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":88516,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":55115,"thread_start_us":1735547,"threads_started":3,"wal-append.queue_time_us":625246}
I20260709 12:22:17.694505 20672 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.194s
W20260709 12:22:17.911783 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.754946 19910 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.933638 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:17.833002 20618 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.839395 20689 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20260709 12:22:17.915102 20472 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:17.913784 20681 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:17.926644 20676 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.941236 20667 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.959292 20605 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.756s	user 0.015s	sys 0.000s
I20260709 12:22:17.860512 18018 rpcz_store.cc:276] 0709 12:22:16.427982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.514288 (+1086306us) service_pool.cc:224] Handling call
0709 12:22:17.514668 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.660785 (+146117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.751279 (+ 90494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:17.915738 20592 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:18.000427 20671 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:17.962347 20468 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:18.006138 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128772us
I20260709 12:22:18.012290 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:18.013783 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 255776us
I20260709 12:22:18.015148 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:18.015473 20592 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:18.021327 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:18.030102 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:18.032262 20689 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:18.041966 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 284818us
W20260709 12:22:18.041203 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122992us
I20260709 12:22:18.043227 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:18.045333 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2468) took 1726 ms. Trace:
I20260709 12:22:18.046083 18020 rpcz_store.cc:276] 0709 12:22:16.318913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.319051 (+   138us) service_pool.cc:224] Handling call
0709 12:22:16.319389 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.319686 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:17.439521 (+1119835us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.044818 (+605297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.044895 (+    77us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:18.044980 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.045077 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.045245 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10727,"prepare.queue_time_us":851132,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"thread_start_us":1356688,"threads_started":3}
I20260709 12:22:17.992558 20614 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:18.052536 20614 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.053082 20604 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:18.061175 20629 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:18.001653 20667 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.054914 20681 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:17.999419 20676 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:17.999094 20570 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:18.061790 20680 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.002s
W20260709 12:22:18.069051 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.068316 20523 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:18.128928 20523 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:18.130184 20523 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.131999 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1667) took 1060 ms. Trace:
I20260709 12:22:18.126739 20667 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.127111 18090 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:18.135545 18090 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:18.054168 20690 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:18.063807 20687 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:18.063275 20692 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.083809 18099 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:18.132107 20523 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:18.123528 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:18.076902 20570 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:18.134999 18171 rpcz_store.cc:276] 0709 12:22:17.071134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.072040 (+   906us) service_pool.cc:224] Handling call
0709 12:22:17.072475 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.072758 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:17.227935 (+155177us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.879872 (+651937us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.131201 (+251329us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b1800329c80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:22:18.131484 (+   283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.131556 (+    72us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:18.131629 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.131725 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.131909 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64471,"prepare.queue_time_us":39034,"raft.queue_time_us":323067,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":1325,"spinlock_wait_cycles":100337792,"thread_start_us":581910,"threads_started":1,"wal-append.queue_time_us":674105}
I20260709 12:22:18.136838 18090 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.141466 20692 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.144459 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2472) took 1745 ms. Trace:
I20260709 12:22:18.160260 18019 rpcz_store.cc:276] 0709 12:22:16.398988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.431285 (+ 32297us) service_pool.cc:224] Handling call
0709 12:22:16.449444 (+ 18159us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.578668 (+129224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.578727 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.214911 (+636184us) spinlock_profiling.cc:243] Waited 6.22 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:22:17.215423 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.694449 (+479026us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:17.759840 (+ 65391us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.952767 (+192927us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.952925 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.144045 (+191120us) raft_consensus.cc:1685] finished
0709 12:22:18.144165 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.144370 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":100386,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":74,"raft.queue_time_us":540784,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":55238,"spinlock_wait_cycles":14297984,"thread_start_us":670918,"threads_started":2,"wal-append.queue_time_us":698978}
I20260709 12:22:18.144327 20690 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.008948 20468 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:22:18.180152 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.146281 18095 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.128026 18022 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:22:18.062839 20686 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:22:18.188205 18090 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:18.193697 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:18.103868 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54734.8us
W20260709 12:22:18.164371 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134391us
I20260709 12:22:18.198715 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:18.124857 20613 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.612s	user 0.013s	sys 0.001s
W20260709 12:22:18.064985 20679 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
W20260709 12:22:18.146957 20505 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.995s	user 0.008s	sys 0.007s
I20260709 12:22:18.199971 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:18.002678 20605 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.073226 20603 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:18.076588 20676 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:18.220192 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:18.206995 20620 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.992s	user 0.015s	sys 0.000s
I20260709 12:22:18.074636 20681 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:18.196359 20468 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.228353 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:18.234974 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 160175us
I20260709 12:22:18.221341 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.242187 18091 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:18.253212 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:18.249707 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.257020 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.259171 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1665) took 1809 ms. Trace:
I20260709 12:22:18.242506 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:18.260667 18171 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:18.264300 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.251416 18020 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:18.224061 20505 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:18.267083 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.270830 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:18.268381 18019 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:18.271322 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72697.1us
I20260709 12:22:18.272153 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:18.270697 18091 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:18.275964 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.280467 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:18.259852 18169 rpcz_store.cc:276] 0709 12:22:16.450494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.450645 (+   151us) service_pool.cc:224] Handling call
0709 12:22:17.110224 (+659579us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.110594 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.110635 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.129253 (+ 18618us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.609033 (+479780us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:17.609087 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.609166 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.258815 (+649649us) raft_consensus.cc:1685] finished
0709 12:22:18.258851 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.259062 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":435073,"threads_started":1,"wal-append.queue_time_us":652462}
I20260709 12:22:18.283720 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.230432 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.286334 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:18.287868 20614 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.288558 20614 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:18.288882 20702 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:18.241083 20613 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:18.061877 20472 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:18.286764 20384 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.179s	user 0.009s	sys 0.005s
I20260709 12:22:18.248119 20689 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:18.300225 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:18.302644 18015 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:18.304805 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet aef65d70106f4f458174493864b4ba3d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:18.306211 20695 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:18.306972 20695 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.307788 20692 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.308892 20692 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.311338 20690 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.312556 20690 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:18.313086 20690 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:18.315196 20605 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.316187 20605 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:18.316955 20605 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.318149 20696 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:22:18.320034 20689 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:18.279448 20437 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.067s	user 0.017s	sys 0.000s
W20260709 12:22:18.321383 20379 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.804s	user 0.013s	sys 0.005s
W20260709 12:22:18.293223 20647 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.092s	user 0.014s	sys 0.000s
I20260709 12:22:18.274214 20570 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.322732 20647 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.323582 20570 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.325405 20647 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.318257 20681 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.295943 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:18.270510 20603 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.293793 18168 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:22:18.327857 18017 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:18.329136 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:18.298830 18020 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:18.350091 17988 connection.cc:256] RPC call timeout handler was delayed by -1.6548015470002611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20260709 12:22:18.351233 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:18.330576 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.315969 20592 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:18.332446 18091 pending_rounds.cc:85] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:18.357980 18091 pending_rounds.cc:107] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20260709 12:22:18.340039 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:18.340942 18090 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
W20260709 12:22:18.342921 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.344950 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.301537 20703 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:18.347450 20618 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.347235 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:18.366320 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.022s	sys 0.091s
I20260709 12:22:18.348989 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.287870 20604 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 12:22:18.352079 18020 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:22:18.335454 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59121.7us
W20260709 12:22:18.315299 20683 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20260709 12:22:18.345587 18095 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:18.309530 20695 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.361987 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:18.363552 18091 raft_consensus.cc:2917] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20260709 12:22:18.377744 18091 raft_consensus.cc:2889] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:18.378434 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 469974us
I20260709 12:22:18.296581 20613 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.350137 18019 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.368937 18099 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:18.370573 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:18.372206 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.022s	sys 0.091s
I20260709 12:22:18.249213 20620 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:18.381278 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 562955us
I20260709 12:22:18.348209 20676 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:18.321696 20437 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.362922 20647 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:18.289490 20702 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.365783 20675 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.381429 20296 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.804s	user 0.012s	sys 0.003s
I20260709 12:22:18.389020 18091 pending_rounds.cc:107] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260709 12:22:18.309628 20692 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.384485 20690 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.392287 20676 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.392853 20715 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:18.393205 20684 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.360896 18090 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 2
I20260709 12:22:18.360823 20603 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.332943 20384 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.370168 20623 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.399065 18091 raft_consensus.cc:2889] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:18.386876 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.421742 18090 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:22:18.433349 18090 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:18.425522 20620 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.393240 20618 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:22:18.386350 20613 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:18.424166 20690 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:18.414115 20675 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.464747 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:18.474298 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:18.436882 20296 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:18.397650 20647 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.492161 20712 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:18.492916 20712 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.493377 20712 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:18.510689 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.421622 18778 connection.cc:256] RPC call timeout handler was delayed by -1.9291351100000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 12:22:18.426187 20676 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:18.421923 20379 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.544183 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.428380 20682 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:18.388774 20703 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:18.388012 20695 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:18.402030 20517 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.220s	user 0.012s	sys 0.004s
I20260709 12:22:18.436738 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2475) took 2327 ms. Trace:
I20260709 12:22:18.493757 20603 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:18.552202 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2020) took 2736 ms (client timeout 1896 ms). Trace:
W20260709 12:22:18.552886 18173 rpcz_store.cc:269] 0709 12:22:15.809424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.809613 (+   189us) service_pool.cc:224] Handling call
0709 12:22:18.543371 (+2733758us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:18.546136 (+  2765us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":389149440}
I20260709 12:22:18.543946 20705 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:22:18.550765 18097 rpcz_store.cc:276] 0709 12:22:16.109200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.116423 (+  7223us) service_pool.cc:224] Handling call
0709 12:22:16.116945 (+   522us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.117295 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:16.448965 (+331670us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.435993 (+1987028us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18004289e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:22:18.436247 (+   254us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.436309 (+    62us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:18.436385 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.436487 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.436657 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":820789,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":76,"raft.queue_time_us":2558,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":2748,"spinlock_wait_cycles":25643776,"thread_start_us":828918,"threads_started":2,"wal-append.queue_time_us":766405,"wal-append.run_cpu_time_us":1321,"wal-append.run_wall_time_us":1001484}
I20260709 12:22:18.544312 20379 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:18.524591 18778 connection.cc:256] RPC call timeout handler was delayed by -1.8139351099998748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
I20260709 12:22:18.429400 20701 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:18.555300 20705 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:22:18.558627 20709 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:18.558683 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.562786 18813 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:22:18.556882 18778 connection.cc:256] RPC call timeout handler was delayed by -1.810335109999869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 12:22:18.565925 18778 connection.cc:256] RPC call timeout handler was delayed by -1.543935109999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260709 12:22:18.566849 20706 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:22:18.439414 20684 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.562095 20379 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:18.572582 18778 connection.cc:256] RPC call timeout handler was delayed by -1.4233351100001528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
I20260709 12:22:18.571552 20684 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:18.575047 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.574884 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:18.575207 20278 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:18.575670 18814 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.576311 20437 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.469877 20623 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.576227 20278 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.578285 20605 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:18.578898 20278 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:18.578907 20623 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.562757 20517 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.537554 20296 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 12:22:18.571765 20695 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.586434 20675 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.538834 20620 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:22:18.559209 20709 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.588747 20709 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.556455 20518 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:18.593590 18167 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.987s	user 0.023s	sys 0.032s
I20260709 12:22:18.588124 20695 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:18.562264 20708 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:22:18.554280 20703 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:18.451228 20715 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:18.599498 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.032s	sys 0.036s
W20260709 12:22:18.594059 18167 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.987s	user 0.023s	sys 0.032s
W20260709 12:22:18.581164 20648 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.002s
I20260709 12:22:18.611021 18093 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:18.572916 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:18.610183 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.033s	sys 0.036s
I20260709 12:22:18.555449 20618 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:18.585546 20517 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.612391 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 367) took 2465 ms. Trace:
I20260709 12:22:18.613242 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.614396 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:18.615492 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 571782us
I20260709 12:22:18.571607 20384 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.619819 18811 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:18.612109 20667 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:18.626837 20667 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.627480 18807 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:18.628324 20692 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:18.617208 20618 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.634352 20618 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.617074 18167 rpcz_store.cc:276] 0709 12:22:16.146631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.146821 (+   190us) service_pool.cc:224] Handling call
0709 12:22:16.166503 (+ 19682us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.166808 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:17.106469 (+939661us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.610258 (+1503789us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.610332 (+    74us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.610392 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.612055 (+  1663us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.612285 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8283,"prepare.queue_time_us":1007867,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":112586,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":4003,"thread_start_us":2415000,"threads_started":3,"wal-append.queue_time_us":646913}
I20260709 12:22:18.590341 20468 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:18.638439 20703 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:18.590741 20702 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:18.639959 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.625764 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:18.639753 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.625861 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:18.959316 18810 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:18.959574 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.960865 18812 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:18.961339 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.962054 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:18.600752 20575 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:18.963235 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.963958 18098 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:18.965951 18098 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:18.966588 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:18.890918 18811 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:18.968534 18811 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:18.858384 20689 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.708822 19910 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.015s	sys 0.006s
W20260709 12:22:18.689980 20462 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.011s	sys 0.010s
I20260709 12:22:18.851890 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:18.689702 18063 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.890s	user 0.017s	sys 0.015s
W20260709 12:22:18.998644 18063 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.199s	user 0.022s	sys 0.021s
W20260709 12:22:18.999101 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.2011e+06us
W20260709 12:22:18.709353 18024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.004s	sys 0.028s
I20260709 12:22:18.962257 18813 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:19.002167 18024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.370s	user 0.006s	sys 0.036s
I20260709 12:22:18.969558 18807 pending_rounds.cc:85] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 3
W20260709 12:22:18.986977 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.004354 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.004891 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.687727 18814 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:18.996551 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.617930 20648 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:19.149883 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.012s	sys 0.052s
I20260709 12:22:19.387902 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:19.029727 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:19.193451 18091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.002s	sys 0.011s
W20260709 12:22:19.390026 18091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.003s	sys 0.011s
W20260709 12:22:19.195116 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.222890 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 287314us
W20260709 12:22:19.388514 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.021s	user 0.018s	sys 0.070s
W20260709 12:22:19.182716 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.009s	sys 0.004s
I20260709 12:22:19.116887 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:19.394249 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.012s	user 0.009s	sys 0.006s
W20260709 12:22:19.394842 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.01297e+06us
I20260709 12:22:19.085575 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:19.397050 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.394979 18099 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:22:19.193356 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.006s	sys 0.007s
W20260709 12:22:19.398610 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.006s	sys 0.009s
I20260709 12:22:18.941807 20652 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.994575 20533 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.998118 19910 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.016s	sys 0.014s
W20260709 12:22:19.393309 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.661s (SENT)
I20260709 12:22:19.458277 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:19.150640 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.073s	sys 0.083s
W20260709 12:22:19.461426 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.076s	sys 0.105s
I20260709 12:22:18.972263 20647 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:19.457466 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.461194 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:19.402992 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.998027 20462 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.100s	user 0.014s	sys 0.017s
I20260709 12:22:19.396651 18813 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:19.462129 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.10024e+06us
W20260709 12:22:19.221252 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.014508 18807 pending_rounds.cc:107] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
W20260709 12:22:19.480537 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87390.7us
W20260709 12:22:19.012041 20364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.006s	sys 0.020s
I20260709 12:22:19.016400 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:19.470329 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.478677 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 85115us
I20260709 12:22:19.483260 18093 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:19.484409 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:19.181015 20472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.010s	sys 0.015s
I20260709 12:22:19.459120 20702 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:19.333468 20676 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.013s	sys 0.013s
W20260709 12:22:19.487346 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.544374 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.494535 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:19.545514 18096 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.495107 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:19.399811 20296 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.399560 20716 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.491562 20364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.201s	user 0.006s	sys 0.024s
I20260709 12:22:19.003810 20468 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:19.399250 20620 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.185839 18808 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:19.493850 20570 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.570068 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:19.574185 18099 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:19.576030 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:19.357427 20715 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:19.554670 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53030.8us
I20260709 12:22:19.555768 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:19.557226 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72934.1us
I20260709 12:22:19.580794 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:19.458451 20517 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:19.193275 20671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.056s	user 0.010s	sys 0.023s
W20260709 12:22:19.539338 20472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.021s	sys 0.022s
W20260709 12:22:19.538916 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59848us
W20260709 12:22:19.559504 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.479382 20710 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.561286 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:19.586823 20711 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.094s	user 0.059s	sys 0.067s
W20260709 12:22:19.602826 20711 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.143s	user 0.060s	sys 0.067s
I20260709 12:22:19.486459 20690 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:19.604732 20613 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.168s	user 0.012s	sys 0.003s
I20260709 12:22:19.565228 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:19.547133 20676 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.017s	sys 0.014s
I20260709 12:22:19.486904 18807 raft_consensus.cc:2917] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
W20260709 12:22:19.551890 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.498663 20605 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.479187 20533 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.577107 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.589751 18093 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:22:19.593487 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.613099 18168 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:19.608644 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:19.015367 20689 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:19.599673 20699 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.338s	user 0.002s	sys 0.013s
W20260709 12:22:19.589442 20671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.452s	user 0.013s	sys 0.023s
I20260709 12:22:19.590306 20570 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:19.593425 20517 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:19.569743 20716 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:19.627286 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.354s	user 0.008s	sys 0.018s
W20260709 12:22:19.642387 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.369s	user 0.008s	sys 0.018s
I20260709 12:22:19.642652 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:19.643085 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.37092e+06us
I20260709 12:22:19.643383 18168 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.548015 20702 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:19.558205 20709 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.960s	user 0.017s	sys 0.000s
I20260709 12:22:19.623034 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:19.629680 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:19.650902 18097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.010s	sys 0.028s
I20260709 12:22:19.651767 18169 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:19.574075 19910 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:19.486517 20652 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.644021 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:19.594347 20647 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:19.609391 20718 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:19.643867 20716 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.660940 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2480) took 3232 ms. Trace:
I20260709 12:22:19.641870 20711 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:22:19.622244 20605 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:19.661398 18016 rpcz_store.cc:276] 0709 12:22:16.430045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.550602 (+1120557us) service_pool.cc:224] Handling call
0709 12:22:17.593934 (+ 43332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.594299 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.594332 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.601639 (+  7307us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.941325 (+339686us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:17.965158 (+ 23833us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.371624 (+406466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.371745 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.588755 (+1217010us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:19.660595 (+ 71840us) raft_consensus.cc:1685] finished
0709 12:22:19.660713 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.660869 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26369,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":4876,"raft.queue_time_us":107505,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":73,"spinlock_wait_cycles":10630400,"thread_start_us":703964,"threads_started":2,"wal-append.queue_time_us":527623}
I20260709 12:22:19.661419 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:19.664314 20605 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:19.664577 18021 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:19.653236 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:19.667888 18093 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:19.619359 20462 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:19.634972 20719 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:22:19.573632 20620 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.582345 20296 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.656103 18171 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:19.643792 20689 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:19.630712 20710 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:19.654814 20709 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.658675 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:19.688476 20702 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.689131 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 695) took 1037 ms. Trace:
I20260709 12:22:19.689534 18809 rpcz_store.cc:276] 0709 12:22:18.652291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.658293 (+1006002us) service_pool.cc:224] Handling call
0709 12:22:19.689048 (+ 30755us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:19.690831 18168 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:19.659149 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.659258 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:19.666054 18099 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:19.641321 18807 raft_consensus.cc:2889] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:19.651788 18097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.010s	sys 0.028s
I20260709 12:22:19.677711 18096 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:22:19.679425 20712 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.185s	user 0.013s	sys 0.004s
I20260709 12:22:19.696394 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:19.697289 18165 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:19.699476 18096 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:19.605396 20613 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.700842 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:19.697548 18093 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:19.701671 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.700122 18021 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:19.708287 18168 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:19.673996 20462 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:19.644757 20472 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.047s
W20260709 12:22:19.715279 18095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.019s	sys 0.026s
W20260709 12:22:19.716007 18095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.020s	sys 0.026s
I20260709 12:22:19.719455 18171 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:19.722260 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2486) took 2690 ms (client timeout 1737 ms). Trace:
W20260709 12:22:19.722797 18021 rpcz_store.cc:269] 0709 12:22:17.017942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.169460 (+151518us) service_pool.cc:224] Handling call
0709 12:22:19.661177 (+2491717us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:19.707833 (+ 46656us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3231488}
I20260709 12:22:19.684928 20689 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.730259 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:19.732002 18096 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:19.726404 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:19.733157 18093 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:22:19.672333 20647 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:19.745687 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 86547.9us
W20260709 12:22:19.747207 18098 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.017s	sys 0.029s
I20260709 12:22:19.722996 20718 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.723321 20462 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:19.747967 18098 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.017s	sys 0.029s
I20260709 12:22:19.770629 18169 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:19.739315 20648 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:19.743305 20681 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.014s	sys 0.023s
I20260709 12:22:19.709110 20712 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.779894 18064 leader_election.cc:400] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5.
I20260709 12:22:19.785406 18064 leader_election.cc:403] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:19.779573 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:19.785935 18093 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:19.786726 18169 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:19.783146 18811 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.025s	sys 0.058s
I20260709 12:22:19.789755 18096 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:19.735406 20689 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.788288 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:19.778810 20709 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.794804 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.749908 20718 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:19.788772 18811 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.025s	sys 0.059s
W20260709 12:22:19.784969 20681 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.015s	sys 0.024s
I20260709 12:22:19.799779 18169 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:19.806507 18093 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:19.814647 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:19.817589 18198 heartbeater.cc:646] Failed to heartbeat to 127.16.102.62:42293 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.102.62:42293 timed out after 15.000s (SENT)
I20260709 12:22:19.819307 20615 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.819885 20613 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.823619 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2484) took 2941 ms. Trace:
I20260709 12:22:19.824124 20689 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.824934 20689 raft_consensus.cc:2764] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 12:22:19.828507 20730 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:22:19.831395 20620 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.832924 20296 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:19.833690 20296 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:19.833957 20517 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:19.834137 20296 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.835158 20296 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:19.838495 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1748) took 1621 ms. Trace:
I20260709 12:22:19.839195 18091 rpcz_store.cc:276] 0709 12:22:18.216558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.331647 (+115089us) service_pool.cc:224] Handling call
0709 12:22:18.332164 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.464914 (+132750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.464972 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.465714 (+   742us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.390804 (+925090us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:19.390875 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.390995 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.838077 (+447082us) raft_consensus.cc:1685] finished
0709 12:22:19.838115 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.838383 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":877729,"threads_started":1,"wal-append.queue_time_us":1239591}
I20260709 12:22:19.841953 20613 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.842768 20613 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.845279 20613 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.846575 20652 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:19.847244 20667 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:19.794641 18094 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.034s	user 0.007s	sys 0.010s
W20260709 12:22:19.851660 18094 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.007s	sys 0.010s
I20260709 12:22:19.852583 18167 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:19.777576 20727 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 12:22:19.841742 20570 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:19.824220 18023 rpcz_store.cc:276] 0709 12:22:16.882014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.584032 (+702018us) service_pool.cc:224] Handling call
0709 12:22:17.606478 (+ 22446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.638098 (+ 31620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.638137 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.638721 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.873347 (+234626us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:17.900636 (+ 27289us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.049160 (+148524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.823058 (+1773898us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b54003476d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:19.823134 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.823200 (+    66us) raft_consensus.cc:1685] finished
0709 12:22:19.823348 (+   148us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.823524 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11930,"prepare.queue_time_us":945,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":72,"raft.queue_time_us":176272,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":15467,"spinlock_wait_cycles":71511808,"thread_start_us":349172,"threads_started":2,"wal-append.queue_time_us":484738,"wal-append.run_cpu_time_us":3011,"wal-append.run_wall_time_us":1172875}
W20260709 12:22:19.873287 20618 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.238s	user 0.010s	sys 0.004s
I20260709 12:22:19.874409 20296 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.876880 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.891927 18167 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:19.890339 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1746) took 1674 ms. Trace:
I20260709 12:22:19.893083 18094 rpcz_store.cc:276] 0709 12:22:18.215708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.215876 (+   168us) service_pool.cc:224] Handling call
0709 12:22:18.220464 (+  4588us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.220873 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:18.760271 (+539398us) log.cc:844] Serialized 14 byte log entry
0709 12:22:19.855102 (+1094831us) log.cc:844] Serialized 14 byte log entry
0709 12:22:19.855361 (+   259us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:19.855424 (+    63us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:19.887154 (+ 31730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.887302 (+   148us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.890245 (+  2943us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2283,"prepare.queue_time_us":1362514,"prepare.run_cpu_time_us":169,"prepare.run_wall_time_us":15422,"thread_start_us":1554391,"threads_started":2}
I20260709 12:22:19.858237 20689 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5.
I20260709 12:22:19.879587 20727 consensus_queue.cc:1243] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:19.912969 20689 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:19.913743 20689 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:19.910197 20473 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:19.927546 18810 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.967s	user 0.014s	sys 0.001s
I20260709 12:22:19.928453 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.930039 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:19.930900 18170 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:19.938848 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 458877us
I20260709 12:22:19.958940 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:19.961462 18167 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:19.898779 20618 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:19.969066 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.014839 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:20.014839 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.622s (SENT)
I20260709 12:22:20.019301 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:20.018571 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 240418us
W20260709 12:22:20.025638 18778 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.632s (SENT)
I20260709 12:22:20.032331 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:20.032933 18170 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:20.033705 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.942193 18810 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:20.066964 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:20.065284 18024 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.944s	user 0.005s	sys 0.056s
I20260709 12:22:20.064947 18815 raft_consensus.cc:2368] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5.
I20260709 12:22:20.001744 20681 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:20.070220 18024 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.006s	sys 0.057s
I20260709 12:22:19.999872 20608 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.072283 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:20.074900 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 470915us
I20260709 12:22:20.077188 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2475) took 3650 ms. Trace:
W20260709 12:22:20.081184 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.689s (SENT)
I20260709 12:22:20.028702 20690 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:20.084861 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2496) took 1121 ms. Trace:
I20260709 12:22:20.077845 18024 rpcz_store.cc:276] 0709 12:22:16.426990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.510325 (+1083335us) service_pool.cc:224] Handling call
0709 12:22:17.510749 (+   424us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:17.579460 (+ 68711us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.579511 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:17.597206 (+ 17695us) log.cc:844] Serialized 52 byte log entry
0709 12:22:19.002818 (+1405612us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:20.071475 (+1068657us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.073623 (+  2148us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.076662 (+  3039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.076800 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.076864 (+    64us) raft_consensus.cc:1685] finished
0709 12:22:20.076885 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.077098 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35246,"raft.queue_time_us":73093,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":17594,"thread_start_us":2320730,"threads_started":2,"wal-append.queue_time_us":1884078}
I20260709 12:22:20.022756 20647 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.088426 18097 rpcz_store.cc:276] 0709 12:22:18.963164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.965926 (+  2762us) service_pool.cc:224] Handling call
0709 12:22:18.966290 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:18.966751 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.966786 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:18.967376 (+   590us) log.cc:844] Serialized 52 byte log entry
0709 12:22:19.697127 (+729751us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:19.710842 (+ 13715us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.084097 (+373255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.084210 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.084262 (+    52us) raft_consensus.cc:1685] finished
0709 12:22:20.084275 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.084780 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13702,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":60,"raft.queue_time_us":116418,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":24485,"thread_start_us":1085635,"threads_started":2,"wal-append.queue_time_us":838505}
I20260709 12:22:20.034919 20710 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:20.091049 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 476597us
I20260709 12:22:20.094246 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.030943 20677 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.106478 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:20.107033 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2502) took 1141 ms. Trace:
I20260709 12:22:20.083133 20487 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.118132 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:20.087730 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 691) took 1460 ms. Trace:
I20260709 12:22:20.101533 20712 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.025385 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:20.118948 18166 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:20.113767 18812 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.003s	sys 0.018s
I20260709 12:22:20.121349 18098 rpcz_store.cc:276] 0709 12:22:18.965162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.966751 (+  1589us) service_pool.cc:224] Handling call
0709 12:22:18.967031 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.967226 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:19.748590 (+781364us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.100155 (+351565us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.100244 (+    89us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:20.100319 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.102358 (+  2039us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.102611 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":642942,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":1132040,"threads_started":2}
W20260709 12:22:20.137583 18812 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.003s	sys 0.018s
W20260709 12:22:20.132015 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 106738us
I20260709 12:22:20.136701 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.128203 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:20.084394 20681 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.141153 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:20.103614 20728 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:20.112723 20652 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:20.144923 20575 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.181s	user 0.016s	sys 0.000s
I20260709 12:22:20.088583 20608 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.133771 20473 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:20.130491 20487 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.164705 18173 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:20.147958 20468 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.150110 20747 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:20.113090 20613 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:20.183650 20360 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:20.183483 20613 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.109977 20647 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.186406 20647 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:20.187523 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 690) took 1575 ms (client timeout 1762 ms). Trace:
W20260709 12:22:20.188027 18815 rpcz_store.cc:269] 0709 12:22:18.612676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.612851 (+   175us) service_pool.cc:224] Handling call
0709 12:22:20.187255 (+1574404us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54004b46d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:20.187443 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3254348672}
W20260709 12:22:20.155701 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.103722 20741 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 12:22:20.153123 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:20.191234 20741 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:20.191844 20741 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:20.192831 20741 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.193306 20517 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:20.194053 20517 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:20.194623 20517 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:20.121795 20615 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:20.195398 20615 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.161121 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:20.199795 20711 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:20.203492 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.205058 18098 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:20.174325 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:20.157347 20398 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:20.161933 20473 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.207701 20473 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:20.168845 20712 raft_consensus.cc:2764] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:20.165536 20169 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.211382 20169 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.176882 20757 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:20.213378 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.213645 20757 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:20.178584 20742 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:22:20.214313 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.215183 18171 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:20.215907 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2506) took 1173 ms. Trace:
I20260709 12:22:20.216135 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.214334 20757 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:20.177698 20468 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.216312 18092 rpcz_store.cc:276] 0709 12:22:19.042586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.042860 (+   274us) service_pool.cc:224] Handling call
0709 12:22:19.351322 (+308462us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:19.358600 (+  7278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.358633 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:19.472045 (+113412us) log.cc:844] Serialized 52 byte log entry
0709 12:22:19.888836 (+416791us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:19.888900 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.889007 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.215654 (+326647us) raft_consensus.cc:1685] finished
0709 12:22:20.215675 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.215834 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6523,"raft.queue_time_us":268,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":78775,"thread_start_us":350435,"threads_started":1,"wal-append.queue_time_us":479376}
I20260709 12:22:20.219347 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.220214 20468 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:20.179375 20739 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
W20260709 12:22:20.180183 20743 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:22:20.221724 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:20.222815 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:20.180888 20740 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 12:22:20.223800 18024 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:20.223914 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.225355 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:20.227435 18814 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:20.225696 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.222267 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.216372 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.229581 18023 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:20.230064 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.005s	sys 0.017s
I20260709 12:22:20.228397 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:20.230402 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.232741 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.236155 20613 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.237129 20506 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.222666 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 703) took 1326 ms. Trace:
I20260709 12:22:20.238080 18811 rpcz_store.cc:276] 0709 12:22:18.896403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.971522 (+ 75119us) service_pool.cc:224] Handling call
0709 12:22:18.971856 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.003685 (+ 31829us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.003728 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.014024 (+ 10296us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.806296 (+792272us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:20.109889 (+303593us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.160583 (+ 50694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.161186 (+   603us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.222404 (+ 61218us) raft_consensus.cc:1685] finished
0709 12:22:20.222426 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.222567 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":352130,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":42585,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":14635,"thread_start_us":1091405,"threads_started":3,"wal-append.queue_time_us":899325}
I20260709 12:22:20.111107 20570 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:22:20.241117 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:20.241952 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.243013 18166 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:20.230417 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.005s	sys 0.017s
W20260709 12:22:20.246035 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 637438us
I20260709 12:22:20.127602 18810 rpcz_store.cc:276] 0709 12:22:18.613093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.614397 (+  1304us) service_pool.cc:224] Handling call
0709 12:22:20.073134 (+1458737us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:20.133865 20727 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20260709 12:22:20.247984 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.225749 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.248879 20681 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.250268 20618 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.250459 18097 pending_rounds.cc:85] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 5
I20260709 12:22:20.140885 20756 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.119s
I20260709 12:22:20.253505 20608 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.253914 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:20.257570 18169 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:20.258176 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.259234 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.163945 20689 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.256330 20398 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:20.326761 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.327059 20398 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.328501 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.329280 18023 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:20.329972 20690 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:20.271678 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 203034us
I20260709 12:22:20.275120 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.331998 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.290505 20718 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.332999 18021 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:20.334018 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.334173 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.334753 18021 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:20.335088 20462 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.559s	user 0.012s	sys 0.000s
I20260709 12:22:20.335614 20462 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.265695 18024 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:20.261962 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.338263 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.253576 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:20.339919 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.340160 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.341334 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.288009 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.288769 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.288246 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:20.292129 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.298542 18172 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:20.298813 18164 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:22:20.299568 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.257401 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:20.338107 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84298.6us
I20260709 12:22:20.341668 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.367524 18097 pending_rounds.cc:107] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:22:20.348281 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.368310 18097 raft_consensus.cc:2889] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.368629 18814 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:20.366672 18022 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
W20260709 12:22:20.373361 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.374346 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.375922 18169 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:20.257320 20719 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:22:20.304934 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.303529 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75580.6us
I20260709 12:22:20.338340 18017 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:20.289451 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.390612 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.367664 18166 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:20.376011 18021 raft_consensus.cc:2368] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:20.392884 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.395921 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:20.396759 18169 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:20.333019 17989 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.528s (SENT)
I20260709 12:22:20.166595 20296 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:20.170820 20652 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:20.383723 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 84233.8us
I20260709 12:22:20.401589 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.402417 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:20.285338 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:20.292007 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 698) took 1576 ms. Trace:
I20260709 12:22:20.405601 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:20.399025 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66029.6us
I20260709 12:22:20.409399 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.343243 18806 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:20.414441 18168 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:20.318882 20604 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:20.406590 18017 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:20.405982 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.342933 20728 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:20.422290 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.433261 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.434068 18811 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
W20260709 12:22:20.434455 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 149229us
I20260709 12:22:20.435348 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:20.439231 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.440263 18171 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:20.439306 18015 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:20.427027 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:20.428244 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.445240 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:20.446437 18811 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:20.395397 20711 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:20.456845 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.456845 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.458261 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:20.458946 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51024.4us
I20260709 12:22:20.419086 20647 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.460623 20570 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.461169 18090 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:20.461315 20570 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.463964 20570 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.464574 20570 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.465499 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.388926 20473 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.466228 20689 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:20.443583 20709 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:20.467257 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:20.467648 20709 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:20.418675 20681 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:20.475314 18164 raft_consensus.cc:2368] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:20.453060 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:20.466245 18094 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:20.471295 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.471570 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.476665 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:20.477614 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.476679 18814 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:20.472280 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:20.466977 20689 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:20.474601 20473 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:20.488994 18166 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 5.
I20260709 12:22:20.429770 20641 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.465293 20570 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.491092 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:20.484230 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:20.492441 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.491065 20296 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:20.494668 20296 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:20.495428 20652 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.499085 20641 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.480736 20709 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.415522 18807 rpcz_store.cc:276] 0709 12:22:18.694932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.968996 (+274064us) service_pool.cc:224] Handling call
0709 12:22:18.969342 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.783527 (+814185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.787085 (+  3558us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.822131 (+ 35046us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.036372 (+214241us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:20.065033 (+ 28661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.074533 (+  9500us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.240557 (+166024us) raft_consensus.cc:1685] finished
0709 12:22:20.245188 (+  4631us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.271322 (+ 26134us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":205180,"threads_started":1,"wal-append.queue_time_us":294980}
I20260709 12:22:20.449640 20364 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:20.491392 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.458343 20575 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:20.468402 20696 raft_consensus.cc:2764] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:22:20.492518 20681 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:20.532737 20681 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.481037 20667 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:20.490190 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:20.536248 18091 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.537645 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.538553 18099 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:20.487205 20711 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:20.518141 18168 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:20.518514 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.458083 20360 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 12:22:20.481328 20746 raft_consensus.cc:2764] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:22:20.506095 20462 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.531307 20487 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.526688 20689 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.577066 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:20.549011 20735 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:20.566895 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.597005 18811 raft_consensus.cc:2368] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5.
W20260709 12:22:20.567670 20763 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:22:20.570281 20570 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:20.570676 20757 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.579129 18091 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:22:20.555904 18169 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:20.624738 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.625761 18091 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:20.626986 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.591012 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:20.602972 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.630287 18094 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:20.673290 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.676033 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:20.676779 18811 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:22:20.679234 18090 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:20.679394 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 186526us
I20260709 12:22:20.680544 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.603621 20667 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:20.682844 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:20.692031 18094 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:20.681232 18099 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:20.695708 18091 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:20.689213 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:20.691066 20763 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:22:20.720795 17988 leader_election.cc:400] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5.
I20260709 12:22:20.730320 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:20.731189 17988 leader_election.cc:403] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:20.724972 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:20.728968 18099 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:22:20.733907 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.734331 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:20.735448 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 297378us
I20260709 12:22:20.738430 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:20.707847 20517 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.002s
I20260709 12:22:20.741047 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.742563 18169 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:20.745014 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.746557 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.747356 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:20.719388 20487 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:20.708940 20728 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.764927 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1670) took 1308 ms. Trace:
W20260709 12:22:20.767325 18064 connection.cc:256] RPC call timeout handler was delayed by -3.6659163879999141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20260709 12:22:20.767815 18064 connection.cc:256] RPC call timeout handler was delayed by -3.6335163879998618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1537s for a response
W20260709 12:22:20.768138 18064 connection.cc:256] RPC call timeout handler was delayed by -3.5453163879997192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
W20260709 12:22:20.768437 18064 connection.cc:256] RPC call timeout handler was delayed by -3.519216387999677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20260709 12:22:20.768733 18064 connection.cc:256] RPC call timeout handler was delayed by -3.4967163879996406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20260709 12:22:20.769024 18064 connection.cc:256] RPC call timeout handler was delayed by -3.3248163879998174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.188s for a response
W20260709 12:22:20.769301 18064 connection.cc:256] RPC call timeout handler was delayed by -3.3230163879998145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
W20260709 12:22:20.769635 18064 connection.cc:256] RPC call timeout handler was delayed by -3.3104163879997941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1896s for a response
W20260709 12:22:20.769923 18064 connection.cc:256] RPC call timeout handler was delayed by -3.2564163879997068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20260709 12:22:20.770186 18064 connection.cc:256] RPC call timeout handler was delayed by -3.241116387999682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
W20260709 12:22:20.761083 20424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.000s	sys 0.024s
I20260709 12:22:20.743577 20608 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:20.755848 20613 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:20.771763 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:20.773420 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:20.773977 18064 connection.cc:256] RPC call timeout handler was delayed by -3.2393163879996791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
I20260709 12:22:20.697708 20462 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.049s
W20260709 12:22:20.774353 18064 connection.cc:256] RPC call timeout handler was delayed by -3.2321163879996675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
I20260709 12:22:20.765463 18165 rpcz_store.cc:276] 0709 12:22:19.456878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.457086 (+   208us) service_pool.cc:224] Handling call
0709 12:22:20.759800 (+1302714us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54000320d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:20.764834 (+  5034us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516729600}
W20260709 12:22:20.793651 18064 connection.cc:256] RPC call timeout handler was delayed by -3.0485163879998254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20260709 12:22:20.793895 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46078.9us
W20260709 12:22:20.795681 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50737.4us
I20260709 12:22:20.796725 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.798085 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:20.805759 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:20.746786 20651 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 5
W20260709 12:22:20.770864 20424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.000s	sys 0.024s
I20260709 12:22:20.765650 20487 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.756726 20748 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:20.825460 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:20.798329 20735 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:20.773322 20728 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:20.823103 20487 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:20.801148 20641 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:20.772094 20613 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.828610 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2512) took 1273 ms. Trace:
I20260709 12:22:20.746165 20473 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260709 12:22:20.746354 20709 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
W20260709 12:22:20.856248 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114217us
I20260709 12:22:20.810405 20747 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:20.690676 20711 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.775053 20741 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.857482 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:20.858491 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52863.1us
W20260709 12:22:20.859746 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.756145 20517 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.855345 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:20.931279 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1678) took 1275 ms. Trace:
I20260709 12:22:20.810199 20681 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:20.939349 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 81986.8us
I20260709 12:22:20.930613 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:20.917728 20778 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:20.861162 20613 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:20.940997 20641 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:20.955570 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100343us
I20260709 12:22:20.825325 20748 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.959288 20748 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:20.940599 20741 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.823956 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:20.970278 18166 raft_consensus.cc:606] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:20.969985 20641 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:20.970762 18166 raft_consensus.cc:614] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:20.857632 18096 rpcz_store.cc:276] 0709 12:22:19.554970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.804447 (+249477us) service_pool.cc:224] Handling call
0709 12:22:19.804855 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.073515 (+268660us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:20.099363 (+ 25848us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.308198 (+208835us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:20.309035 (+   837us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.309272 (+   237us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:20.309321 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.309432 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.828206 (+518774us) raft_consensus.cc:1685] finished
0709 12:22:20.828252 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.828487 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12334,"prepare.queue_time_us":29855,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":3268,"raft.queue_time_us":83341,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":33319,"thread_start_us":423911,"threads_started":2,"wal-append.queue_time_us":327736}
I20260709 12:22:20.968663 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:20.815893 20667 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:20.975322 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:20.836968 20750 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:20.914847 20648 raft_consensus.cc:1064] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:22:20.978319 20468 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:20.944943 20533 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:20.980177 20741 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:20.980324 20781 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:22:20.988077 20768 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.180s	user 0.001s	sys 0.001s
I20260709 12:22:20.989404 20757 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:20.991039 18171 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:20.919376 20759 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260709 12:22:20.859169 20652 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:21.059301 20652 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:20.929950 20744 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:20.918593 20767 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:22:21.004087 20727 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:22:21.003747 20750 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:20.908885 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:21.075525 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 166746us
I20260709 12:22:21.031775 20620 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:22:21.014497 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:20.923437 20765 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.068s	user 0.002s	sys 0.000s
W20260709 12:22:21.023664 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.867326 20772 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:21.086483 20772 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.086958 20772 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:21.034299 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65312.3us
I20260709 12:22:21.037303 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:21.090792 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53604.4us
I20260709 12:22:21.091651 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:21.031370 20648 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.012233 20517 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:21.092886 20773 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:21.093323 20773 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.093693 20773 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:21.077692 20750 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:21.095331 20757 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:21.095822 20757 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.096217 20757 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:21.018708 20755 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.429s	user 0.000s	sys 0.003s
I20260709 12:22:21.026352 20604 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:21.052691 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:21.098109 20728 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:20.880745 20747 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:21.099305 20774 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:21.099838 20774 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.100203 20774 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:21.100262 20747 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.004278 20462 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:21.102105 20462 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:21.102416 20764 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.338s	user 0.001s	sys 0.001s
I20260709 12:22:21.102566 20462 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:21.103144 18806 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.651s	user 0.013s	sys 0.002s
I20260709 12:22:21.103574 18806 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:20.977370 20681 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.050655 20751 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.104784 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1760) took 1488 ms. Trace:
I20260709 12:22:21.105402 20751 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.105365 18093 rpcz_store.cc:276] 0709 12:22:19.616926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.824013 (+207087us) service_pool.cc:224] Handling call
0709 12:22:19.824385 (+   372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:20.150739 (+326354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.150772 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:20.151439 (+   667us) log.cc:844] Serialized 52 byte log entry
0709 12:22:20.538597 (+387158us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:20.617318 (+ 78721us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.617666 (+   348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.617767 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.012384 (+394617us) spinlock_profiling.cc:243] Waited 5.51 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:21.104360 (+ 91976us) raft_consensus.cc:1685] finished
0709 12:22:21.104496 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.104702 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":64006,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":79859,"raft.run_cpu_time_us":7710,"raft.run_wall_time_us":68811,"spinlock_wait_cycles":12666112,"thread_start_us":707175,"threads_started":2,"wal-append.queue_time_us":544443}
I20260709 12:22:21.108709 20751 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:21.113937 20651 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5.
I20260709 12:22:21.119215 20169 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:21.123144 20777 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:22:21.104830 20681 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:21.126953 20766 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.143s	user 0.001s	sys 0.001s
I20260709 12:22:21.089495 20487 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.132449 20487 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:21.135676 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45379.1us
I20260709 12:22:20.939625 18173 rpcz_store.cc:276] 0709 12:22:19.655758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.655926 (+   168us) service_pool.cc:224] Handling call
0709 12:22:20.930976 (+1275050us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b5400032ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:20.931188 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080054528}
I20260709 12:22:21.061420 20652 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:21.139336 20506 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:21.076727 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:21.142340 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65685.9us
I20260709 12:22:21.142594 20608 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.143204 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:21.009714 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:21.143292 20608 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.144754 20741 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:21.144825 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 135279us
I20260709 12:22:21.145355 20741 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:21.081835 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67418.8us
I20260709 12:22:21.079835 20716 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:21.084488 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.066781 20744 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.079177 20614 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:21.145757 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.151360 20614 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.151849 20614 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.152936 20744 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:21.148391 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:21.156323 18093 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:21.157600 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:21.153592 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.084587 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.168069 18813 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:21.170862 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:21.205835 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43978.2us
I20260709 12:22:21.168639 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:21.220728 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:21.225970 18094 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:21.195194 18016 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
W20260709 12:22:21.228348 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 85238.3us
W20260709 12:22:21.231178 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62653.2us
I20260709 12:22:21.147881 20741 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:21.212913 20689 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:21.204993 20169 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:21.150252 20608 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:21.186635 20754 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:21.167730 20468 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:21.146070 20762 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:21.186453 20744 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:21.277143 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.262609 20689 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.148742 20716 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.231336 20751 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 12:22:21.265805 20506 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:21.254739 20667 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:21.532172 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 447646us
I20260709 12:22:21.671936 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:21.484074 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.296229 20608 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.359s
W20260709 12:22:21.392575 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 219190us
I20260709 12:22:21.673887 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:21.532567 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 301647us
W20260709 12:22:21.466361 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.560956 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 335733us
W20260709 12:22:21.393591 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.533s (SENT)
W20260709 12:22:21.705889 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 312340us
W20260709 12:22:21.606451 20788 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20260709 12:22:21.733322 20718 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:21.714066 18809 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.009s	sys 0.009s
I20260709 12:22:21.666339 20689 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:21.734184 18809 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.010s	sys 0.009s
I20260709 12:22:21.298870 20762 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.735862 20762 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:21.314870 20783 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:21.296262 20169 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:21.770138 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58315.3us
I20260709 12:22:21.771315 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.772837 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.273898 20754 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.271608 20747 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.463s
W20260709 12:22:21.649030 20613 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.689s	user 0.011s	sys 0.003s
W20260709 12:22:21.637526 20795 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:22:21.392269 18807 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.769s	user 0.010s	sys 0.005s
W20260709 12:22:21.766304 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 94559.6us
W20260709 12:22:21.767457 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.665s (SENT)
I20260709 12:22:21.789321 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:21.777374 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:21.657610 20744 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:21.593312 20681 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.787721 20608 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:21.784718 20773 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.691s	user 0.010s	sys 0.003s
I20260709 12:22:21.800596 20608 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.800961 20773 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.802963 20608 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:21.803239 20773 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:21.803910 20773 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.804440 20773 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:21.805627 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.779573 20748 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.819s	user 0.004s	sys 0.012s
I20260709 12:22:21.807104 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:21.806957 20748 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:21.807528 20756 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:21.807868 18810 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:21.808104 20756 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:21.808632 20570 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:21.809101 20570 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.779701 20741 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:21.782538 20651 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:21.820506 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:21.829360 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:21.842734 18091 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:21.844305 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:21.847606 18091 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:21.814749 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:21.778599 20506 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:21.781970 20754 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:21.809441 20748 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:21.796115 20744 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:21.809505 20570 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:21.841349 20652 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.678s	user 0.009s	sys 0.001s
W20260709 12:22:21.859624 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.008s	sys 0.012s
W20260709 12:22:21.877437 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.009s	sys 0.013s
W20260709 12:22:21.877964 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 644534us
W20260709 12:22:21.859871 18092 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.016s	sys 0.004s
I20260709 12:22:21.872152 18090 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.5-9.5]
W20260709 12:22:21.879499 18092 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.017s	sys 0.006s
I20260709 12:22:21.880662 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:21.872490 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.650s (SENT)
W20260709 12:22:21.882692 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.659s (SENT)
W20260709 12:22:21.811491 20772 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.724s	user 0.012s	sys 0.001s
I20260709 12:22:21.883265 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:21.883584 20772 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.792796 18807 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:21.886229 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 715) took 1369 ms. Trace:
I20260709 12:22:21.886665 18807 rpcz_store.cc:276] 0709 12:22:20.516712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.523965 (+  7253us) service_pool.cc:224] Handling call
0709 12:22:21.886149 (+1362184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.788341 20613 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.893999 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:21.810318 20798 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:21.895395 18091 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:21.811492 20800 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:21.894884 18090 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:21.901732 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87068.6us
W20260709 12:22:21.903797 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.961s (SENT)
W20260709 12:22:21.875550 20648 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.030s	sys 0.047s
I20260709 12:22:21.819195 20169 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:21.915140 17921 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 279) took 17083 ms (client timeout 15000 ms). Trace:
I20260709 12:22:21.876879 20741 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:21.917680 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:21.933279 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.931006 18810 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:21.921293 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:21.964512 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46919.5us
I20260709 12:22:21.969280 18097 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:21.971334 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:21.972101 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:21.936460 20709 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:21.916834 20801 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:21.973541 20798 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:21.973842 20709 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:21.916687 20697 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:21.975227 20709 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.982625 20697 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.983162 20681 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:21.918648 20789 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:21.985914 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 716) took 1463 ms. Trace:
I20260709 12:22:21.986367 18815 rpcz_store.cc:276] 0709 12:22:20.522166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.693064 (+170898us) service_pool.cc:224] Handling call
0709 12:22:20.693441 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.693726 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.693763 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:20.694315 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.069399 (+375084us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:21.069448 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.069532 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.985602 (+916070us) raft_consensus.cc:1685] finished
0709 12:22:21.985631 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.985804 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":305620,"threads_started":1,"wal-append.queue_time_us":1018011}
I20260709 12:22:21.870939 20506 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.910315 20751 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:21.809121 20756 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:21.965059 20614 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.813s	user 0.009s	sys 0.005s
W20260709 12:22:22.005018 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.095s (SENT)
I20260709 12:22:22.005772 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:21.993423 20798 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:22.008315 20798 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:21.983163 20697 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:22.010259 20741 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.010986 20615 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.011823 20689 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:21.973982 20801 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:22.013590 18099 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:21.927567 20620 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.008s	sys 0.018s
W20260709 12:22:22.014777 20620 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.008s	sys 0.018s
W20260709 12:22:21.912029 20648 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.030s	sys 0.047s
I20260709 12:22:22.015283 20613 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:21.909862 20772 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.013602 20801 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:22.016109 20614 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:21.977048 20774 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.876s	user 0.011s	sys 0.004s
I20260709 12:22:21.899932 20652 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:21.977800 20462 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.875s	user 0.015s	sys 0.001s
I20260709 12:22:22.001336 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:22.032227 18093 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.861s	user 0.003s	sys 0.027s
I20260709 12:22:21.937397 20800 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:21.993767 20718 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:21.975987 20651 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:22.018824 20614 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:21.991446 20750 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.897s	user 0.008s	sys 0.006s
W20260709 12:22:21.927603 17921 rpcz_store.cc:269] 0709 12:22:04.832029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.832465 (+   436us) service_pool.cc:224] Handling call
0709 12:22:21.914990 (+17082525us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:14.237844 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:14.449397 (+211553us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:14.464872 (+ 15475us) write_op.cc:435] Acquired schema lock
0709 12:22:14.464908 (+    36us) tablet.cc:662] Decoding operations
0709 12:22:14.526769 (+ 61861us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:14.526884 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:14.526915 (+    31us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:22:14.777390 (+250475us) tablet.cc:701] Row locks acquired
0709 12:22:14.839698 (+ 62308us) write_op.cc:260] PREPARE: finished
0709 12:22:14.878319 (+ 38621us) write_op.cc:270] Start()
0709 12:22:14.891021 (+ 12702us) write_op.cc:276] Timestamp: P: 1783599734868580 usec, L: 0
0709 12:22:14.891055 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 12:22:14.904190 (+ 13135us) log.cc:844] Serialized 2064 byte log entry
0709 12:22:16.454349 (+1550159us) op_driver.cc:464] REPLICATION: finished
0709 12:22:17.525519 (+1071170us) write_op.cc:301] APPLY: starting
0709 12:22:17.525643 (+   124us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:17.525921 (+   278us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:17.587163 (+ 61242us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:17.924296 (+337133us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:17.924804 (+   508us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0709 12:22:17.924882 (+    78us) write_op.cc:312] APPLY: finished
0709 12:22:17.925429 (+   547us) log.cc:844] Serialized 47 byte log entry
0709 12:22:18.188935 (+263506us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:18.210912 (+ 21977us) write_op.cc:454] Released schema lock
0709 12:22:18.229319 (+ 18407us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:18.309978 (+ 80659us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:18.310015 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11524,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":1034757,"apply.run_cpu_time_us":14242,"apply.run_wall_time_us":749837,"num_ops":5,"prepare.queue_time_us":482115,"prepare.run_cpu_time_us":40722,"prepare.run_wall_time_us":941321,"raft.queue_time_us":707654,"raft.run_cpu_time_us":17247,"raft.run_wall_time_us":820994,"replication_time_us":1539386,"thread_start_us":1445660,"threads_started":5,"wal-append.queue_time_us":999684,"wal-append.run_cpu_time_us":29581,"wal-append.run_wall_time_us":2990314}]]}
I20260709 12:22:21.918919 20398 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 12:22:22.002054 20681 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:22.006248 20773 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.028024 18812 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:21.912897 20799 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:22.034101 18093 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.003s	sys 0.027s
W20260709 12:22:21.991365 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 214150us
I20260709 12:22:22.068161 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:22.058537 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:22.044152 20757 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.947s	user 0.015s	sys 0.000s
I20260709 12:22:22.027482 20506 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:22.073561 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.077370 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:22.055971 20613 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:21.997061 20751 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.092608 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:22.101306 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:22.094337 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:22.044741 20716 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.104823 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:22.056565 20667 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:22.056031 20462 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.029289 20774 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.121107 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:22.095736 20681 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:22.096093 20773 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:22.064085 20750 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.075150 20757 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.114287 20651 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:22.120185 20620 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:22.135488 20772 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:22.235121 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114134us
I20260709 12:22:22.248349 18064 leader_election.cc:400] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:22.248862 18064 leader_election.cc:403] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:22.119983 20806 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:22.181488 20613 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:22.073519 20799 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.100893 20506 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:22.285091 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:22.290344 18095 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:22.306317 17989 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.929s (SENT) [suppressed 22 similar messages]
I20260709 12:22:22.244853 20620 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:22.272593 20806 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:22.302114 20773 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
W20260709 12:22:22.370465 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 466258us
I20260709 12:22:22.301971 20681 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:22:22.341182 20726 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.000s	sys 0.040s
I20260709 12:22:22.321545 20689 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.346438 20620 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:22.346954 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 609) took 3755 ms (client timeout 1989 ms). Trace:
I20260709 12:22:22.344100 20806 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:22.346939 20814 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:22.391719 20726 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.000s	sys 0.042s
I20260709 12:22:22.284489 20800 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.369251 20810 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:22:22.403072 20652 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.412336 20773 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:22.432026 18097 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:22.366925 20667 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.393620 20681 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.439538 20773 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.463773 20773 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:22.405299 20613 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.466817 20751 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.469439 20462 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.486678 18813 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260709 12:22:22.485918 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 480901us
W20260709 12:22:22.396900 18806 rpcz_store.cc:269] 0709 12:22:18.564740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.566665 (+  1925us) service_pool.cc:224] Handling call
0709 12:22:20.174083 (+1607418us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:22.292417 (+2118334us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400781754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:22.319550 (+ 27133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":326359168}
I20260709 12:22:22.471873 20774 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:22.499867 20802 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.268s	user 0.000s	sys 0.002s
W20260709 12:22:22.507968 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 3.435s (SENT)
W20260709 12:22:22.521126 20812 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:22:22.526539 20816 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:22:22.502580 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 722) took 1363 ms. Trace:
I20260709 12:22:22.527746 18809 rpcz_store.cc:276] 0709 12:22:21.138747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.146199 (+  7452us) service_pool.cc:224] Handling call
0709 12:22:21.146669 (+   470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.146935 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:21.734690 (+587755us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.416651 (+681961us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.417907 (+  1256us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:22.435154 (+ 17247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.483163 (+ 48009us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.499392 (+ 16229us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":751075,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"thread_start_us":1226614,"threads_started":3,"wal-append.queue_time_us":622724}
I20260709 12:22:22.512389 20757 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:22.533763 20424 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:22.538897 20424 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:22.541260 20746 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:22.510728 20807 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.277s	user 0.002s	sys 0.000s
I20260709 12:22:22.536917 20728 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:22.545410 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 726) took 1313 ms. Trace:
I20260709 12:22:22.546005 18813 rpcz_store.cc:276] 0709 12:22:21.231533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.231728 (+   195us) service_pool.cc:224] Handling call
0709 12:22:22.486230 (+1254502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.487170 (+   940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.545131 (+ 57961us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400494150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:22.545317 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2884190976}
I20260709 12:22:22.541530 20424 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:22.565616 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.545050 20697 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.004s
W20260709 12:22:22.548008 20762 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.812s	user 0.010s	sys 0.005s
I20260709 12:22:22.516491 20750 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:22.548990 20746 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:22.504267 20801 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.582711 20774 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:22.740845 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 365374us
I20260709 12:22:22.724066 20728 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:22.752432 18137 connection.cc:256] RPC call timeout handler was delayed by -1.18122733600012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260709 12:22:22.752880 20728 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.756188 20728 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.757278 20604 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:22.689392 20819 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:22.753408 18137 connection.cc:256] RPC call timeout handler was delayed by -1.015627336000307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20260709 12:22:22.658089 20814 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:22.797468 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:22.802824 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:22.805982 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:22.833302 18169 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:22.832445 20824 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:22:22.830827 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:22.832098 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2523) took 2455 ms. Trace:
I20260709 12:22:22.836143 18171 raft_consensus.cc:2376] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5. Re-sending same reply.
W20260709 12:22:22.634711 20817 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:22:22.840910 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:22.764995 20762 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.837515 18098 rpcz_store.cc:276] 0709 12:22:20.377176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.377401 (+   225us) service_pool.cc:224] Handling call
0709 12:22:20.377882 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.432713 (+ 54831us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:20.456104 (+ 23391us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.708236 (+252132us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.708588 (+   352us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.708660 (+    72us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:20.708748 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.125806 (+417058us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b1800394240. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:22:21.596379 (+470573us) spinlock_profiling.cc:243] Waited 9.51 ms on lock 0x7b1800394240. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:22.831785 (+1235406us) spinlock_profiling.cc:243] Waited 54.5 ms on lock 0x7b54002977d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:22.831859 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.832022 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9845,"prepare.queue_time_us":36267,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":114,"raft.queue_time_us":34738,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":11329,"spinlock_wait_cycles":176645632,"thread_start_us":251042,"threads_started":1}
I20260709 12:22:22.845677 18168 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:22.846561 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.000s	sys 0.039s
I20260709 12:22:22.714409 20799 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:22.862555 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:22.864044 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.000s	sys 0.040s
W20260709 12:22:22.867597 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 876140us
I20260709 12:22:22.742799 20681 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:22.878500 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 727) took 1088 ms. Trace:
I20260709 12:22:22.873041 18171 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:22.628330 20697 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.738168 20774 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:22.793066 20604 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.742408 20825 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.150s
I20260709 12:22:22.846319 20814 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.815779 20746 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:22.948279 18167 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:22:23.434362 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.460665 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1786) took 2305 ms. Trace:
W20260709 12:22:23.458521 18139 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.175s	user 0.013s	sys 0.028s
W20260709 12:22:23.427139 17989 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.119s	user 0.015s	sys 0.006s
W20260709 12:22:23.463410 18139 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.190s	user 0.013s	sys 0.029s
I20260709 12:22:23.461540 18092 rpcz_store.cc:276] 0709 12:22:21.154871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.156319 (+  1448us) service_pool.cc:224] Handling call
0709 12:22:21.156672 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.900787 (+744115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.900899 (+   112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.976610 (+ 75711us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.356545 (+379935us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:22.356609 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.356723 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:23.459567 (+1102844us) raft_consensus.cc:1685] finished
0709 12:22:23.459659 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.460304 (+   645us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58447,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":19696,"thread_start_us":1017590,"threads_started":2,"wal-append.queue_time_us":522062}
W20260709 12:22:23.463554 17989 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.156s	user 0.021s	sys 0.011s
I20260709 12:22:23.063153 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:23.466118 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.15975e+06us
I20260709 12:22:22.898099 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:22.928360 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:23.470260 18138 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.000s	sys 0.007s
W20260709 12:22:23.470970 18138 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.001s	sys 0.008s
W20260709 12:22:23.471993 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.000s	sys 0.010s
W20260709 12:22:23.472565 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.000s	sys 0.010s
W20260709 12:22:23.473106 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 728321us
W20260709 12:22:23.097565 18778 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.017s	sys 0.022s
W20260709 12:22:23.475391 18778 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.017s	sys 0.023s
I20260709 12:22:23.474915 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:23.476095 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 958926us
I20260709 12:22:23.470774 18168 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 8
W20260709 12:22:23.471884 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 725519us
W20260709 12:22:23.357290 20773 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.893s	user 0.016s	sys 0.001s
W20260709 12:22:23.505143 18097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.010s	sys 0.009s
W20260709 12:22:23.157904 20830 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.322s	user 0.000s	sys 0.001s
W20260709 12:22:23.507751 18097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.011s	sys 0.010s
W20260709 12:22:23.297772 20754 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.419s	user 0.018s	sys 0.000s
W20260709 12:22:23.514861 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.166s	user 0.010s	sys 0.009s
W20260709 12:22:22.875598 20614 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.005s	sys 0.014s
W20260709 12:22:23.519243 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.227s	user 0.011s	sys 0.009s
W20260709 12:22:22.868584 20744 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.974s	user 0.015s	sys 0.003s
W20260709 12:22:23.231320 20711 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.009s	sys 0.012s
W20260709 12:22:23.050788 20718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.951s	user 0.012s	sys 0.025s
I20260709 12:22:22.868577 20750 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:23.145437 20798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.013s	sys 0.027s
W20260709 12:22:23.164705 20831 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
W20260709 12:22:22.941042 20817 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
W20260709 12:22:23.158432 20833 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.444s	user 0.000s	sys 0.001s
W20260709 12:22:23.362874 20806 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.950s	user 0.012s	sys 0.004s
I20260709 12:22:23.154677 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 732) took 1322 ms. Trace:
I20260709 12:22:23.533664 20841 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:23.538784 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:22.897918 18814 rpcz_store.cc:276] 0709 12:22:21.790148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.790323 (+   175us) service_pool.cc:224] Handling call
0709 12:22:21.790812 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.791499 (+   687us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:22.121354 (+329855us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.878023 (+756669us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.878102 (+    79us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:22.878162 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.878248 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.878408 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":174846,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"raft.queue_time_us":297225,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11846,"thread_start_us":1085398,"threads_started":3,"wal-append.queue_time_us":672583}
I20260709 12:22:23.528136 20651 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:23.302119 20762 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:23.548506 20841 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:23.524004 20681 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.545428 20651 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:23.554286 20798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s	user 0.014s	sys 0.041s
I20260709 12:22:23.560997 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:23.240787 20801 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.556723 20841 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:23.546941 20750 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:23.567618 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 630661us
I20260709 12:22:23.576993 20398 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.576699 20744 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.577283 20846 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:23.578150 20846 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:23.577937 20398 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:23.575562 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101091us
W20260709 12:22:23.580775 20718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.449s	user 0.019s	sys 0.038s
W20260709 12:22:23.586162 20711 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.016s	sys 0.019s
W20260709 12:22:23.577018 20614 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.352s	user 0.021s	sys 0.042s
W20260709 12:22:23.581610 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.29649e+06us
I20260709 12:22:23.590981 20718 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:22.939291 20829 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.644s
I20260709 12:22:23.594394 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1788) took 2432 ms. Trace:
I20260709 12:22:23.594200 20718 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.594712 20846 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.595458 18808 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:23.604529 20754 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.605271 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.606666 18091 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:23.607797 18173 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:23.608067 20754 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:22.860144 20620 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.618606 20697 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:23.606607 20774 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.619024 20681 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.643913 20750 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.648519 20814 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:23.649296 20604 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:23.652061 20398 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.658167 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:23.659930 18098 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:23.660171 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55933.6us
I20260709 12:22:23.661033 20744 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:23.662734 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:23.664510 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:23.596935 18093 rpcz_store.cc:276] 0709 12:22:21.161366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.161714 (+   348us) service_pool.cc:224] Handling call
0709 12:22:21.162151 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:22.065599 (+903448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.065637 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:22.076129 (+ 10492us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.367537 (+291408us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:22.367598 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.367709 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:23.592563 (+1224854us) raft_consensus.cc:1685] finished
0709 12:22:23.592620 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.594139 (+  1519us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8958,"raft.queue_time_us":82518,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":188,"thread_start_us":1124329,"threads_started":2,"wal-append.queue_time_us":497800}
I20260709 12:22:23.666222 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:23.667888 18095 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:23.678083 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:23.682988 20750 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:23.686224 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49951.3us
W20260709 12:22:23.687300 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.012s (SENT)
I20260709 12:22:23.687891 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:23.690595 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:23.692762 20762 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:23.694113 20762 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:23.694517 20854 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:23.695394 20854 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:23.698166 20858 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.192s	user 0.000s	sys 0.002s
I20260709 12:22:23.698585 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:23.699236 18812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 725) took 2529 ms (client timeout 2939 ms). Trace:
I20260709 12:22:23.683137 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2542) took 1755 ms. Trace:
I20260709 12:22:23.582336 20806 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:23.699774 18812 rpcz_store.cc:269] 0709 12:22:21.169499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.169938 (+   439us) service_pool.cc:224] Handling call
0709 12:22:23.698711 (+2528773us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54006f2054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:23.699029 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1823571712}
I20260709 12:22:23.597522 20773 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.720918 20847 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:23.713754 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80955.2us
I20260709 12:22:23.714313 20814 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:23.702266 18097 rpcz_store.cc:276] 0709 12:22:21.928203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.482337 (+554134us) service_pool.cc:224] Handling call
0709 12:22:22.482730 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.482972 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:22.863261 (+380289us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.636530 (+773269us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.642041 (+  5511us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.642185 (+   144us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:23.642297 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.642442 (+   145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.683018 (+ 40576us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":757685,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":119,"raft.queue_time_us":20411,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":34455,"thread_start_us":1015921,"threads_started":2,"wal-append.queue_time_us":641877}
I20260709 12:22:23.736593 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:23.736630 18098 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:23.738698 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74456.7us
I20260709 12:22:23.719704 20798 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:23.758653 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:23.759194 20398 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:23.759905 18166 raft_consensus.cc:606] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:23.760401 18166 raft_consensus.cc:614] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:23.760160 20398 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.763192 20398 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.763676 20854 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:23.720942 20604 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:22:23.775744 20861 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:23.776719 20861 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.779765 20861 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:23.582338 18807 rpcz_store.cc:276] 0709 12:22:21.821855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.889386 (+ 67531us) service_pool.cc:224] Handling call
0709 12:22:21.889734 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.905284 (+ 15550us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:22.389787 (+484503us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.864501 (+474714us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.923127 (+ 58626us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:22.930002 (+  6875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.040420 (+110418us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.143893 (+103473us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":60190,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":320436,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":14540,"thread_start_us":915225,"threads_started":2}
I20260709 12:22:23.783344 18093 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:23.784870 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:23.786120 20798 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:23.787799 20798 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:23.788270 20798 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:23.789726 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:23.790042 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.790185 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.791790 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:23.791769 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.794723 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:23.795655 20651 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.797039 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.801497 20604 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:23.798523 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2541) took 1882 ms. Trace:
I20260709 12:22:23.802220 20604 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:23.800120 18136 connection.cc:256] RPC call timeout handler was delayed by -1.829253019999669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
I20260709 12:22:23.805959 20862 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.806767 20862 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.812539 20641 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.813556 20641 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:23.814590 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78832.9us
I20260709 12:22:23.815768 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:23.818192 20801 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:23.815872 20829 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.816777 20641 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:23.803851 20604 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.834070 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:23.835187 18094 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:23.834652 20829 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:23.837894 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.838048 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 139702us
I20260709 12:22:23.839998 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:23.841048 18017 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:23.842032 20849 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:22:23.842478 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.844400 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:23.847303 20811 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:23.848021 20811 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:23.848198 20806 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:23.843497 18172 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:23.849963 20620 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.849473 20811 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.851090 20811 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.852159 20681 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.852547 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:23.853260 18168 raft_consensus.cc:606] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:23.853928 20773 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:23.854154 18097 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:23.856515 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:23.856833 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.679s (SENT)
I20260709 12:22:23.849298 20798 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.857780 18098 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:23.857430 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:23.847124 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:23.869585 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:23.860635 18169 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:23.886096 18173 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:22:23.858275 20843 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.599s	user 0.000s	sys 0.003s
I20260709 12:22:23.873263 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:23.837690 20829 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:23.893020 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:23.898890 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 333) took 3000 ms (client timeout 2171 ms). Trace:
I20260709 12:22:23.904042 20804 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:23.905052 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:22:23.906704 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65033.8us
I20260709 12:22:23.907037 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:23.965989 18169 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:23.907486 20854 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:23.908388 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51631.3us
W20260709 12:22:23.880875 20570 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.927s	user 0.025s	sys 0.000s
I20260709 12:22:23.910136 20810 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:23.910339 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.911118 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:23.912426 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:23.924074 20862 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:23.926662 18062 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45040 dropped due to backpressure: service queue is full with 50 items
W20260709 12:22:23.927654 20852 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:22:23.928782 20726 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:23.968889 18173 raft_consensus.cc:606] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:23.930743 20841 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:23.932083 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 545) took 2110 ms. Trace:
I20260709 12:22:23.935101 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:23.936342 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:23.941431 20604 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.962589 18172 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:23.947259 18017 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:23.909373 20798 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:23.946334 18016 rpcz_store.cc:269] 0709 12:22:20.897973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.903147 (+  5174us) service_pool.cc:224] Handling call
0709 12:22:23.898267 (+2995120us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:23.898547 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323045632}
W20260709 12:22:23.973177 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.972415 20647 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:23.977866 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:23.963972 20773 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.979846 18173 raft_consensus.cc:614] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:23.980975 18093 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:23.983603 18094 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:24.020725 18812 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:24.010978 18019 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.003355 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a07f730a06794e2cb530edca57871763"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:24.012579 18166 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:23.972437 20854 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:23.975850 20726 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:23.982504 20772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.582s	user 0.004s	sys 0.018s
I20260709 12:22:24.011629 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.035492 18807 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.974561 20570 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.994614 18020 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:24.044184 20772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.749s	user 0.006s	sys 0.019s
W20260709 12:22:24.058506 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80936us
I20260709 12:22:24.062019 18022 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:22:24.063055 18811 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:24.065224 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.066200 18017 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.066682 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:24.066756 20726 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:24.067386 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.067512 18023 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:24.068410 18016 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:24.070402 18090 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:24.070549 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:24.071478 18018 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:24.071718 20570 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:24.046727 20810 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:23.802466 18099 rpcz_store.cc:276] 0709 12:22:21.916138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.020732 (+104594us) service_pool.cc:224] Handling call
0709 12:22:22.062531 (+ 41799us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.062880 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:22.063424 (+   544us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.706231 (+1642807us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b1800393d60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:22:23.771694 (+ 65463us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.797969 (+ 26275us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:23.798088 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.798175 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.798331 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":195,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":2455,"raft.queue_time_us":15908,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":25511,"spinlock_wait_cycles":6190976,"thread_start_us":23021,"threads_started":1,"wal-append.queue_time_us":1719944}
I20260709 12:22:24.075165 18024 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:22:24.075412 20810 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:24.076718 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:24.077543 20841 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:24.078475 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.079458 18015 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.081152 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.082376 18021 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:23.979558 18808 rpcz_store.cc:276] 0709 12:22:21.821384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.856647 (+ 35263us) service_pool.cc:224] Handling call
0709 12:22:21.934523 (+ 77876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.624757 (+1690234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.931644 (+306887us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54006153d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:23.931962 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3774649600}
I20260709 12:22:24.085100 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.089048 20746 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.075210 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2538) took 2166 ms. Trace:
W20260709 12:22:24.092599 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 274731us
W20260709 12:22:24.095048 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47236.1us
I20260709 12:22:23.995303 20773 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:24.091352 18091 rpcz_store.cc:276] 0709 12:22:21.909126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.909601 (+   475us) service_pool.cc:224] Handling call
0709 12:22:21.909978 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.607379 (+1697401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.074843 (+467464us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b540035e8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:24.075131 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3900962560}
I20260709 12:22:24.101706 18023 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.114552 18023 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:24.112320 18020 pending_rounds.cc:85] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 5
I20260709 12:22:24.059896 20772 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:24.119725 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.120481 18017 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:24.118172 18020 pending_rounds.cc:107] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:22:23.973239 18064 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45040 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 461)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1806)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1807)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2545)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2546)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2549)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 453)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 465)
I20260709 12:22:24.130034 18020 raft_consensus.cc:2889] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:24.071480 20854 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:24.139662 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.140764 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.141390 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.152413 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:24.159103 18017 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:24.109509 20570 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:22:24.137189 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.170665 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2536) took 2279 ms. Trace:
I20260709 12:22:24.171193 18090 rpcz_store.cc:276] 0709 12:22:21.891036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.907180 (+ 16144us) service_pool.cc:224] Handling call
0709 12:22:21.932860 (+ 25680us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.071094 (+2138234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.170223 (+ 99129us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54000523d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:24.170565 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4914865024}
I20260709 12:22:24.184379 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.185397 18172 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:24.181849 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.194887 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.196228 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.198271 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.199709 18169 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:24.208026 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.208353 20772 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:24.208905 18091 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5. Re-sending same reply.
I20260709 12:22:24.209527 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.210062 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.210652 18015 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:24.211024 18091 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:24.211789 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:24.212059 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.212497 18015 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:24.212816 18091 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:24.213172 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.213637 20773 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:24.216457 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.218321 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.448s (SENT)
I20260709 12:22:24.218999 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:24.220100 20615 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:24.224278 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.226159 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.227948 18815 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:24.232510 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.233299 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.234737 18173 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:24.235560 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.237039 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.237892 20773 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:24.233373 18169 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:24.239425 20773 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:24.239470 18169 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:24.239856 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.239900 20773 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:24.241343 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.241972 20773 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:24.243480 18099 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:24.244000 20773 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:24.245004 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.246989 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.266661 18018 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.264847 18812 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:24.270663 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.274130 18809 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:24.276345 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.278705 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:23.973869 18062 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45040 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 458)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 445)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 450)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 447)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 462)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 463)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 464)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 467)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 471)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 449)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 460)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 461)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 456)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 468)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 469)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 454)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 470)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 444)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2543)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 438)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 441)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 443)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1806)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1807)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2545)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2546)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2549)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 453)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 457)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 465)
I20260709 12:22:24.270728 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.283519 18015 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.286206 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.288869 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.289685 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.290556 18169 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:24.290828 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.292891 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.293665 18169 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:22:24.294143 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.295099 18093 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.298919 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.299254 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.299949 18814 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:24.301158 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.248699 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.250833 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:24.286265 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68699.3us
I20260709 12:22:24.301162 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.302260 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.192044 18808 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:24.315400 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:24.315479 18017 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:24.320941 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:24.324752 18810 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.280212 18806 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:22:24.371642 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.372630 18812 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.372731 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:24.328106 18090 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:24.325356 18173 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:24.370074 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.374506 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:24.373569 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49727.1us
W20260709 12:22:24.322288 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2527) took 2494 ms (client timeout 1555 ms). Trace:
W20260709 12:22:24.396822 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.401441 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.402452 18021 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:24.395676 18096 rpcz_store.cc:269] 0709 12:22:21.819005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.820182 (+  1177us) service_pool.cc:224] Handling call
0709 12:22:24.302421 (+2482239us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54004123d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:24.313572 (+ 11151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1300318592}
I20260709 12:22:24.416407 20806 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:24.417090 20806 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:24.418370 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.419147 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.419852 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49731.6us
I20260709 12:22:24.421322 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.421758 20748 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:24.422204 18811 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:24.422650 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.423265 20748 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:24.423727 20748 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:24.424055 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.424397 18172 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:24.420114 18094 raft_consensus.cc:2368] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5.
I20260709 12:22:24.424829 18092 raft_consensus.cc:2376] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4. Re-sending same reply.
I20260709 12:22:24.487989 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.484290 18093 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:24.543471 18018 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:24.613566 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.426103 18098 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:24.614701 18021 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:24.606724 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.500739 18019 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:24.571022 18091 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:24.425534 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.620855 18015 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:24.631269 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.635511 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.423738 18807 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:22:24.400302 20570 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:24.610108 20726 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.656926 18813 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:24.648372 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:24.649977 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.650574 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.657981 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.658226 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.648340 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.643065 18023 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:24.489073 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:24.654263 20800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.341s	user 0.025s	sys 0.031s
I20260709 12:22:24.655076 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.659152 18094 raft_consensus.cc:2376] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4. Re-sending same reply.
I20260709 12:22:24.661362 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.661204 18810 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:24.666318 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:24.667157 18807 raft_consensus.cc:2376] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5. Re-sending same reply.
I20260709 12:22:24.668368 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.668742 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.424993 20748 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:24.671011 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.672084 20810 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:22:24.674145 18093 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:24.674500 20810 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:24.675011 20810 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:24.675326 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.676199 20810 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:24.676842 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.677472 20442 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.639266 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:24.487967 20806 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:24.798477 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.666350 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:24.699399 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.678166 18093 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 12:22:24.710055 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.665408 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.692891 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:24.682027 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 261149us
I20260709 12:22:24.759423 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.699016 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.738740 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.738735 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.744791 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.823002 18810 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:24.724167 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.815861 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.829154 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.722591 20877 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.439s	user 0.003s	sys 0.000s
I20260709 12:22:24.700096 18092 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.787663 18098 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:24.724073 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.657732 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.848973 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:24.824229 20746 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.706s	user 0.013s	sys 0.003s
I20260709 12:22:24.849938 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.857101 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.857621 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2548) took 1057 ms. Trace:
I20260709 12:22:24.840767 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.852320 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.852285 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.858111 18091 rpcz_store.cc:276] 0709 12:22:23.800178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.663683 (+863505us) service_pool.cc:224] Handling call
0709 12:22:24.857531 (+193848us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.853147 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1804) took 1058 ms. Trace:
I20260709 12:22:24.853374 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.687034 20810 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.669682 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.866604 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:24.847643 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.868414 18021 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:24.856696 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.870298 18099 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:24.872017 18093 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:24.856925 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.664682 20800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.471s	user 0.026s	sys 0.032s
I20260709 12:22:24.859414 18098 rpcz_store.cc:276] 0709 12:22:23.790086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.308499 (+518413us) service_pool.cc:224] Handling call
0709 12:22:24.847928 (+539429us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:24.865448 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.895936 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.853753 20772 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.015s	sys 0.024s
W20260709 12:22:24.867210 18137 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.989s (SENT)
I20260709 12:22:24.786053 20854 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:24.673066 20842 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.182s
I20260709 12:22:24.901101 18813 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:24.901738 18815 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:24.905748 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.902521 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.907289 18018 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5. Re-sending same reply.
W20260709 12:22:24.898250 18166 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.003s	sys 0.041s
W20260709 12:22:24.909073 18166 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.004s	sys 0.041s
I20260709 12:22:24.910683 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.913353 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.918112 18094 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:24.916222 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 448) took 1064 ms. Trace:
I20260709 12:22:24.918744 18098 rpcz_store.cc:276] 0709 12:22:23.851438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.886054 (+1034616us) service_pool.cc:224] Handling call
0709 12:22:24.916148 (+ 30094us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.930127 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 451) took 1077 ms. Trace:
I20260709 12:22:24.918864 18096 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:24.897992 20772 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.015s	sys 0.024s
I20260709 12:22:24.934962 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:24.930923 18094 rpcz_store.cc:276] 0709 12:22:23.852482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.834922 (+982440us) service_pool.cc:224] Handling call
0709 12:22:24.929999 (+ 95077us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28032}
I20260709 12:22:24.939324 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:24.941462 18064 connection.cc:256] RPC call timeout handler was delayed by -2.6552967779998653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260709 12:22:24.941694 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:24.947487 18095 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:24.949267 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.949394 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:24.954381 18096 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:22:24.954747 18098 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:22:24.958201 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 444) took 1172 ms. Trace:
I20260709 12:22:24.909562 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:24.961258 18064 connection.cc:256] RPC call timeout handler was delayed by -2.5940967779997663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
I20260709 12:22:24.834471 20442 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:24.962677 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2543) took 1170 ms. Trace:
I20260709 12:22:24.963372 18094 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:24.963208 18093 rpcz_store.cc:276] 0709 12:22:23.782691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.914321 (+1131630us) service_pool.cc:224] Handling call
0709 12:22:24.943797 (+ 29476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.952951 (+  9154us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.952988 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:24.953047 (+    59us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:24.953095 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.953175 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.953339 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.959055 18095 rpcz_store.cc:276] 0709 12:22:23.795081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.910480 (+1115399us) service_pool.cc:224] Handling call
0709 12:22:24.958071 (+ 47591us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.965986 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 441) took 1170 ms. Trace:
I20260709 12:22:24.966046 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:24.966620 18096 rpcz_store.cc:276] 0709 12:22:23.784747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.932678 (+1147931us) service_pool.cc:224] Handling call
0709 12:22:24.953840 (+ 21162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.955125 (+  1285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.955448 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.967819 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 438) took 1172 ms. Trace:
I20260709 12:22:24.968544 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 443) took 1179 ms. Trace:
I20260709 12:22:24.969166 18094 rpcz_store.cc:276] 0709 12:22:23.785514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.940602 (+1155088us) service_pool.cc:224] Handling call
0709 12:22:24.960700 (+ 20098us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.964002 (+  3302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.964385 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.968305 18098 rpcz_store.cc:276] 0709 12:22:23.783543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.920001 (+1136458us) service_pool.cc:224] Handling call
0709 12:22:24.950836 (+ 30835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.955573 (+  4737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.955819 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.971524 18811 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
W20260709 12:22:24.962419 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4905967780000537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260709 12:22:24.972553 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4824967780000406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20260709 12:22:24.973027 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3609967779998442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
W20260709 12:22:24.973522 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3501967779998267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20260709 12:22:24.973973 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2988967779997438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260709 12:22:24.974282 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2907967779997307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19530000000000003s for a response
W20260709 12:22:24.974568 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1044967779998842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20260709 12:22:24.974854 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1035967779998828s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
W20260709 12:22:24.975157 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0936967779998668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21720000000000003s for a response
W20260709 12:22:24.975445 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0567967779998071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
W20260709 12:22:24.975849 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0459967779997896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
W20260709 12:22:24.976275 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5851967779999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2737s for a response
I20260709 12:22:24.978269 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.978437 18093 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:24.979159 18165 raft_consensus.cc:2368] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:24.977281 18164 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:24.984916 18170 raft_consensus.cc:606] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:24.985409 18170 raft_consensus.cc:614] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:24.986868 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1807) took 1188 ms. Trace:
I20260709 12:22:24.987360 18093 rpcz_store.cc:276] 0709 12:22:23.791202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.965859 (+1174657us) service_pool.cc:224] Handling call
0709 12:22:24.977850 (+ 11991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.979348 (+  1498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.979669 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:24.969847 20875 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.516s	user 0.006s	sys 0.006s
I20260709 12:22:24.989487 18096 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:24.990388 18095 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:24.915288 20854 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:24.993891 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2546) took 1194 ms. Trace:
I20260709 12:22:24.995296 18096 rpcz_store.cc:276] 0709 12:22:23.799444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.967486 (+1168042us) service_pool.cc:224] Handling call
0709 12:22:24.984242 (+ 16756us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.992182 (+  7940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.993774 (+  1592us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.992532 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2545) took 1193 ms. Trace:
I20260709 12:22:24.997008 18095 rpcz_store.cc:276] 0709 12:22:23.799070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.966454 (+1167384us) service_pool.cc:224] Handling call
0709 12:22:24.982706 (+ 16252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.992232 (+  9526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.992466 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.000614 18091 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:25.001497 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1806) took 1210 ms. Trace:
I20260709 12:22:25.001904 18091 rpcz_store.cc:276] 0709 12:22:23.790857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.954601 (+1163744us) service_pool.cc:224] Handling call
0709 12:22:24.980496 (+ 25895us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.001175 (+ 20679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.001429 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.005055 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.006491 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:24.986828 20570 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:25.007759 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2549) took 1207 ms. Trace:
I20260709 12:22:25.009261 18098 rpcz_store.cc:276] 0709 12:22:23.800513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.970708 (+1170195us) service_pool.cc:224] Handling call
0709 12:22:25.001369 (+ 30661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.007317 (+  5948us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.007351 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.007417 (+    66us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:25.007454 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.007517 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.007678 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.008610 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:24.979696 18809 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:25.012676 18811 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:25.005903 18021 raft_consensus.cc:2376] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5. Re-sending same reply.
I20260709 12:22:25.014748 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2547) took 1215 ms. Trace:
I20260709 12:22:25.016906 18094 rpcz_store.cc:276] 0709 12:22:23.799809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.970602 (+1170793us) service_pool.cc:224] Handling call
0709 12:22:24.993788 (+ 23186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.014408 (+ 20620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.014432 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.014475 (+    43us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:25.014509 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.014562 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.014689 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.023080 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.024030 18015 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:25.025214 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.030432 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:25.003746 20884 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.266s	user 0.000s	sys 0.002s
I20260709 12:22:25.034030 18098 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:25.034785 18095 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:25.035226 18015 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:25.035943 18172 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:22:25.040956 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:25.036536 20880 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.161s	user 0.000s	sys 0.002s
I20260709 12:22:24.998157 20772 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:25.044127 18094 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:25.044127 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 457) took 1127 ms. Trace:
I20260709 12:22:25.044783 18095 rpcz_store.cc:276] 0709 12:22:23.917227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.997954 (+1080727us) service_pool.cc:224] Handling call
0709 12:22:25.012409 (+ 14455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.043764 (+ 31355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.044053 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:25.037457 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 188144us
I20260709 12:22:25.039109 18813 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:25.042034 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 465) took 1122 ms. Trace:
I20260709 12:22:25.043789 18093 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:25.046860 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 453) took 1193 ms. Trace:
I20260709 12:22:25.047227 18093 rpcz_store.cc:276] 0709 12:22:23.853154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.996604 (+1143450us) service_pool.cc:224] Handling call
0709 12:22:25.004638 (+  8034us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.046525 (+ 41887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.046795 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.045928 18813 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:25.049238 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.006836 18806 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:25.050239 18017 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:25.050544 18098 rpcz_store.cc:276] 0709 12:22:23.919977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.010114 (+1090137us) service_pool.cc:224] Handling call
0709 12:22:25.033737 (+ 23623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.041669 (+  7932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.041956 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.054338 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:25.055889 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.056849 18165 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:24.981769 20613 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.141s
I20260709 12:22:25.056722 18021 raft_consensus.cc:2376] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3. Re-sending same reply.
I20260709 12:22:25.060824 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:25.061164 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.056988 18093 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:25.061937 18023 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:25.049701 18813 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.060056 20442 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:25.062963 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:25.068066 18090 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:25.072294 18093 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:25.074812 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.074971 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.074558 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.075806 18023 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5. Re-sending same reply.
I20260709 12:22:25.075802 18019 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:22:25.077926 18813 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.081079 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2551) took 1277 ms. Trace:
I20260709 12:22:25.081974 18096 rpcz_store.cc:276] 0709 12:22:23.801331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.002012 (+1200681us) service_pool.cc:224] Handling call
0709 12:22:25.035669 (+ 33657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.071757 (+ 36088us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.071789 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.078545 (+  6756us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:25.078606 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.078679 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.078819 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.083237 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:25.083220 17987 connection.cc:256] RPC call timeout handler was delayed by -1.9913220759999604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
W20260709 12:22:25.085264 17987 connection.cc:256] RPC call timeout handler was delayed by -1.8942597989998831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260709 12:22:25.085551 17987 connection.cc:256] RPC call timeout handler was delayed by -1.7988597990001836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20260709 12:22:25.085848 17987 connection.cc:256] RPC call timeout handler was delayed by -1.7763597990001472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
W20260709 12:22:25.086197 17987 connection.cc:256] RPC call timeout handler was delayed by -1.6035597989998678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
W20260709 12:22:25.086457 17987 connection.cc:256] RPC call timeout handler was delayed by -1.53695979899976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
W20260709 12:22:25.086709 17987 connection.cc:256] RPC call timeout handler was delayed by -1.51535979900018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
W20260709 12:22:25.086949 17987 connection.cc:256] RPC call timeout handler was delayed by -1.4856597990001319s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20260709 12:22:25.087199 17987 connection.cc:256] RPC call timeout handler was delayed by -1.3344597989998874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
W20260709 12:22:25.087446 17987 connection.cc:256] RPC call timeout handler was delayed by -1.2012597990001268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2384s for a response
I20260709 12:22:25.088848 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:25.052073 20888 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:25.097092 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:25.098044 18172 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:22:25.099443 18172 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:25.100996 18172 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:25.101521 18172 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:25.102773 18172 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.103981 18172 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.108274 18091 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:25.113440 18090 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:22:25.113787 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.114718 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.115579 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 458785us
I20260709 12:22:25.074457 20772 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:25.117105 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:25.118923 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.122017 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2514) took 1050 ms. Trace:
I20260709 12:22:25.125177 20604 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:22:25.127986 20811 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:25.128590 20811 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:25.051476 20889 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.078s
I20260709 12:22:25.130966 18023 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:25.130118 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1715) took 1323 ms. Trace:
I20260709 12:22:25.127902 20604 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:22:25.134431 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 136035us
I20260709 12:22:25.137522 18173 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:25.161715 18019 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:25.132316 20829 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:25.133796 20800 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:25.129652 20811 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.132675 18171 rpcz_store.cc:276] 0709 12:22:23.741934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.084869 (+342935us) service_pool.cc:224] Handling call
0709 12:22:24.085390 (+   521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.101288 (+ 15898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.101324 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:24.113496 (+ 12172us) log.cc:844] Serialized 26 byte log entry
0709 12:22:24.187724 (+ 74228us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:24.214878 (+ 27154us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.215646 (+   768us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.215764 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.425496 (+209732us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:25.048629 (+623133us) raft_consensus.cc:1685] finished
0709 12:22:25.064075 (+ 15446us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.064260 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28919,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":121,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":178,"spinlock_wait_cycles":77333376,"thread_start_us":68442,"threads_started":1,"wal-append.queue_time_us":169617}
I20260709 12:22:25.178287 18165 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:25.122548 18016 rpcz_store.cc:276] 0709 12:22:24.068025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.069154 (+  1129us) service_pool.cc:224] Handling call
0709 12:22:24.069625 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.086463 (+ 16838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.086505 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:24.087161 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 12:22:24.128497 (+ 41336us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:24.322240 (+193743us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.323742 (+  1502us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.323838 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.774534 (+450696us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:25.110352 (+335818us) raft_consensus.cc:1685] finished
0709 12:22:25.118448 (+  8096us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.118660 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10699,"prepare.queue_time_us":803015,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":3589,"raft.queue_time_us":6848,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":19843,"spinlock_wait_cycles":44591488,"thread_start_us":196539,"threads_started":2,"wal-append.queue_time_us":99352}
W20260709 12:22:25.186888 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.189225 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:25.190547 18173 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:25.174434 20829 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.121138 20570 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.161023 20842 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:22:25.181597 20800 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:25.206708 18172 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:25.208415 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2504) took 1160 ms. Trace:
I20260709 12:22:25.205761 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2520) took 1114 ms. Trace:
W20260709 12:22:25.217521 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 324066us
I20260709 12:22:25.203581 20829 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.214175 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2515) took 1145 ms. Trace:
I20260709 12:22:25.217955 18165 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:25.215174 18022 rpcz_store.cc:276] 0709 12:22:24.096627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.143144 (+ 46517us) service_pool.cc:224] Handling call
0709 12:22:24.172252 (+ 29108us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:24.230577 (+ 58325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.230605 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:24.231366 (+   761us) log.cc:844] Serialized 52 byte log entry
0709 12:22:24.497863 (+266497us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:24.916887 (+419024us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.204995 (+288108us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.205385 (+   390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.205473 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.205518 (+    45us) raft_consensus.cc:1685] finished
0709 12:22:25.205531 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.205689 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":167999,"prepare.queue_time_us":510524,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":86159,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":6060,"thread_start_us":792100,"threads_started":3,"wal-append.queue_time_us":574327}
I20260709 12:22:25.208114 20800 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.245298 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:25.252636 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76234.8us
I20260709 12:22:25.208869 18020 rpcz_store.cc:276] 0709 12:22:24.048357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.057811 (+  9454us) service_pool.cc:224] Handling call
0709 12:22:24.104949 (+ 47138us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.177065 (+ 72116us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.177119 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:24.192321 (+ 15202us) log.cc:844] Serialized 26 byte log entry
0709 12:22:24.497023 (+304702us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:24.961680 (+464657us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.204131 (+242451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.207499 (+  3368us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.208134 (+   635us) raft_consensus.cc:1685] finished
0709 12:22:25.208156 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.208319 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141345,"prepare.queue_time_us":500790,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":1590,"raft.queue_time_us":108155,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":103,"thread_start_us":800068,"threads_started":3,"wal-append.queue_time_us":401289}
I20260709 12:22:25.227941 18024 rpcz_store.cc:276] 0709 12:22:24.068414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.089450 (+ 21036us) service_pool.cc:224] Handling call
0709 12:22:24.111835 (+ 22385us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:24.155143 (+ 43308us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.155171 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:24.171225 (+ 16054us) log.cc:844] Serialized 52 byte log entry
0709 12:22:24.413227 (+242002us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:24.913770 (+500543us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.946014 (+ 32244us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.210188 (+264174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.210286 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.210331 (+    45us) raft_consensus.cc:1685] finished
0709 12:22:25.210345 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.214077 (+  3732us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22909,"prepare.queue_time_us":522819,"prepare.run_wall_time_us":49,"raft.queue_time_us":140267,"raft.run_cpu_time_us":679,"raft.run_wall_time_us":53951,"spinlock_wait_cycles":4490112,"thread_start_us":748116,"threads_started":3,"wal-append.queue_time_us":489599}
I20260709 12:22:25.266243 20169 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:25.266847 20169 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:25.233078 20604 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:22:25.267916 20169 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.269521 20169 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.227891 20854 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:25.281582 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49711.5us
I20260709 12:22:25.281205 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:25.286837 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.270913 20811 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:25.301860 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2550) took 1501 ms (client timeout 1631 ms). Trace:
W20260709 12:22:25.302459 18097 rpcz_store.cc:269] 0709 12:22:23.800922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.856079 (+ 55157us) service_pool.cc:224] Handling call
0709 12:22:25.301549 (+1445470us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540032b9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.301734 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2579078912}
W20260709 12:22:25.293974 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74895.3us
W20260709 12:22:25.294266 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.295985 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.299188 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71595.2us
I20260709 12:22:25.308760 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.279736 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:25.319802 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 144418us
W20260709 12:22:25.321898 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.321485 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:25.323875 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:25.219791 20772 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.327643 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.332083 18024 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:25.333137 18024 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:22:25.333630 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.292533 20894 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.281102 20892 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:25.336647 18780 connection.cc:256] RPC call timeout handler was delayed by -2.71936011400021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
W20260709 12:22:25.345623 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66088.7us
W20260709 12:22:25.348636 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65749.8us
W20260709 12:22:25.345742 18780 connection.cc:256] RPC call timeout handler was delayed by -2.2450601140003528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
W20260709 12:22:25.351051 18780 connection.cc:256] RPC call timeout handler was delayed by -2.2450601140003528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
I20260709 12:22:25.349715 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:25.359064 18020 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:25.362552 18022 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:25.364164 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.365312 20897 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:25.366482 20842 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:25.367046 18809 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:25.403632 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122530us
I20260709 12:22:25.337961 20854 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.412936 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.413482 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.414119 18024 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:25.411475 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75758.5us
I20260709 12:22:25.417752 18096 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:25.421037 18090 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:22:25.415426 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:25.425446 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.381282 20900 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:25.430737 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51393.7us
W20260709 12:22:25.440083 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 116304us
I20260709 12:22:25.409965 20901 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:25.442186 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:25.447971 20886 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.440s	user 0.003s	sys 0.000s
I20260709 12:22:25.412225 20772 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:25.452052 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.662s (SENT)
W20260709 12:22:25.454141 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 104541us
I20260709 12:22:25.376238 20894 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.457038 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:25.451735 20902 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:25.455564 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.367676 20892 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.433748 20903 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:25.453644 20901 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:25.469574 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.419617 20854 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.478788 18812 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:22:25.478358 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.479592 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 622) took 1230 ms. Trace:
I20260709 12:22:25.480293 20893 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:25.481880 20846 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:25.482934 20846 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:25.483304 20846 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.483671 20846 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:25.485031 20854 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:25.487073 20726 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.816s	user 0.013s	sys 0.002s
I20260709 12:22:25.488883 20748 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:25.490339 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 475) took 1263 ms (client timeout 1512 ms). Trace:
W20260709 12:22:25.490841 18099 rpcz_store.cc:269] 0709 12:22:24.227232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.270218 (+ 42986us) service_pool.cc:224] Handling call
0709 12:22:25.490104 (+1219886us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b5400410fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.490262 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934919936}
I20260709 12:22:25.494356 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.494657 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.495289 18022 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:25.496349 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.493654 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.479933 18812 rpcz_store.cc:276] 0709 12:22:24.249255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.867808 (+618553us) service_pool.cc:224] Handling call
0709 12:22:25.478263 (+610455us) spinlock_profiling.cc:243] Waited 979 us on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:25.478496 (+   233us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:25.479280 (+   784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.479526 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2250624}
I20260709 12:22:25.463128 20772 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.497179 18022 raft_consensus.cc:2368] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:25.508961 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.511379 18167 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:25.497958 20907 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:25.513013 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.514714 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.515221 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2047) took 1282 ms. Trace:
I20260709 12:22:25.513069 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:25.516888 18023 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:25.517508 18016 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.515622 18015 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:25.519806 18813 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:22:25.520761 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71043.9us
I20260709 12:22:25.516846 20896 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:25.515669 18166 rpcz_store.cc:276] 0709 12:22:24.232487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.266224 (+ 33737us) service_pool.cc:224] Handling call
0709 12:22:24.299512 (+ 33288us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:24.300097 (+   585us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.300132 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:24.300848 (+   716us) log.cc:844] Serialized 52 byte log entry
0709 12:22:24.909576 (+608728us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:24.937368 (+ 27792us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.937970 (+   602us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.938257 (+   287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.938397 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.447078 (+508681us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:25.481558 (+ 34480us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b180044d220. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:25.514960 (+ 33402us) raft_consensus.cc:1685] finished
0709 12:22:25.514983 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.515124 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":82254,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":3589,"raft.queue_time_us":2486,"raft.run_cpu_time_us":457,"raft.run_wall_time_us":87820,"spinlock_wait_cycles":92860544,"thread_start_us":608147,"threads_started":1,"wal-append.queue_time_us":698324}
W20260709 12:22:25.502242 20810 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.609s	user 0.012s	sys 0.003s
W20260709 12:22:25.523640 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44135.5us
W20260709 12:22:25.526899 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57407.8us
I20260709 12:22:25.531325 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:25.532529 20898 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:25.534461 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1703) took 2736 ms (client timeout 2123 ms). Trace:
I20260709 12:22:25.535272 20892 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:25.474992 20901 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.536636 20894 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:25.537035 20896 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:25.559267 20896 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.560794 20896 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:25.566562 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.471573 20902 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:25.542528 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:25.544847 18164 rpcz_store.cc:269] 0709 12:22:22.797718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.815832 (+ 18114us) service_pool.cc:224] Handling call
0709 12:22:25.534124 (+2718292us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.534342 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4716355840}
I20260709 12:22:25.547324 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.547183 18017 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:25.550597 20772 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:25.552044 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:25.577960 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:25.578787 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:25.538751 20909 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:25.581986 20902 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.583009 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 28) took 1580 ms. Trace:
I20260709 12:22:25.583434 18170 rpcz_store.cc:276] 0709 12:22:24.002392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.003164 (+   772us) service_pool.cc:224] Handling call
0709 12:22:25.582760 (+1579596us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.582905 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2198557952}
I20260709 12:22:25.585901 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:25.587903 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:25.588831 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.569766 18168 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:25.575992 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:25.590448 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 130028us
W20260709 12:22:25.590766 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43564.5us
I20260709 12:22:25.590476 18016 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:25.592952 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51668.8us
I20260709 12:22:25.594494 20647 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:25.606976 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.608654 18018 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:22:25.609264 18018 pending_rounds.cc:85] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 2
I20260709 12:22:25.609580 18018 pending_rounds.cc:107] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:22:25.609935 18018 raft_consensus.cc:2889] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:25.613323 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.993s (SENT)
I20260709 12:22:25.614096 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:25.611479 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:25.596532 20772 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:25.596370 20613 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:25.613207 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:22:25.626504 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.625768 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:25.627854 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:25.621598 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.605674 20647 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.643358 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:25.643462 20647 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.652477 18806 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:25.663407 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:25.694296 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76504.1us
W20260709 12:22:25.698601 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153220us
I20260709 12:22:25.623458 20772 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.631311 20570 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.631809 20893 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:25.702770 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.630482 20748 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:25.707587 20889 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:25.603459 20899 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 12:22:25.718652 20772 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:25.735533 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 129571us
I20260709 12:22:25.736147 18812 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:25.741668 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:25.730010 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 273050us
I20260709 12:22:25.733038 20748 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:25.757689 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:25.759022 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56425.7us
I20260709 12:22:25.761938 20912 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:25.762755 20912 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.738199 20889 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.767748 20889 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:25.773489 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.775645 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55431.1us
W20260709 12:22:25.777011 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 314545us
I20260709 12:22:25.735088 20842 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:25.736240 20908 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:25.726974 20570 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.730228 20829 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.784516 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.757279 20748 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.753893 20894 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:25.792850 20613 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:25.793777 20894 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.794940 20894 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.796254 20846 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.778349 20773 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:25.798471 20773 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.782871 20912 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:25.801934 20773 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:25.793170 20806 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.804071 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:25.818271 18812 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:25.807942 20570 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:25.808326 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 454) took 1954 ms. Trace:
I20260709 12:22:25.822404 18092 rpcz_store.cc:276] 0709 12:22:23.853505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.853647 (+1000142us) service_pool.cc:224] Handling call
0709 12:22:25.808111 (+954464us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b54002977d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.808263 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1498805632}
I20260709 12:22:25.808898 18095 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:25.824127 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.809360 20746 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:25.802704 20800 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.521s	user 0.012s	sys 0.004s
I20260709 12:22:25.811303 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:25.813649 20901 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.814414 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2057) took 1090 ms. Trace:
W20260709 12:22:25.816222 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.816462 20829 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:25.826643 18167 rpcz_store.cc:276] 0709 12:22:24.723752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.723911 (+   159us) service_pool.cc:224] Handling call
0709 12:22:25.814201 (+1090290us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400383954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.814319 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527329536}
I20260709 12:22:25.826733 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:25.827862 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.831043 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47266.6us
I20260709 12:22:25.828785 20746 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:25.832463 20726 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.832093 20613 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:25.832432 20800 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.832662 20918 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:22:25.834527 20726 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:25.850200 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.851665 20726 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:25.834733 20908 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.853477 20908 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:25.854720 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.834826 20746 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.859369 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2563) took 1085 ms. Trace:
I20260709 12:22:25.859504 20746 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.859941 18095 rpcz_store.cc:276] 0709 12:22:24.774432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.045661 (+271229us) service_pool.cc:224] Handling call
0709 12:22:25.808717 (+763056us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.828948 (+ 20231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.859054 (+ 30106us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54002977d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:25.859258 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1706268544}
W20260709 12:22:25.835296 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51622.5us
W20260709 12:22:25.862562 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78062.2us
I20260709 12:22:25.838335 18064 leader_election.cc:400] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 5.
I20260709 12:22:25.863745 18064 leader_election.cc:403] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:22:25.842154 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89882.4us
I20260709 12:22:25.864971 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.865949 18064 leader_election.cc:400] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:25.842271 20570 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.846475 20842 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:25.866986 20570 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:25.867471 20570 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.868865 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:25.870060 20888 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 5.
I20260709 12:22:25.870872 20888 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:25.871450 20888 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:25.872531 20888 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.833767 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:25.877669 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43981.6us
I20260709 12:22:25.878266 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:25.887248 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:25.882550 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:25.907867 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44492.8us
I20260709 12:22:25.908972 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.843204 20846 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:25.836838 20881 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.833s	user 0.005s	sys 0.000s
I20260709 12:22:25.914832 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:25.839129 20800 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:25.915853 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47114.4us
I20260709 12:22:25.913709 20889 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:25.914378 20726 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:22:25.923836 20772 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.847374 20894 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.899369 20896 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:25.838543 20773 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:25.928241 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:25.912601 20918 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:25.933144 20613 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:25.946233 20800 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:25.931005 20899 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:25.935140 20889 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.935863 20898 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:25.935217 20908 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:25.955497 20910 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:25.956573 20908 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.936755 20829 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:25.936940 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:25.937477 17921 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 4, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:25.939347 20901 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:25.917464 20916 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:22:25.942262 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:25.943674 20914 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 12:22:25.944860 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.952943 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:25.965747 20726 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:22:25.958169 20748 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.978062 20905 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:25.977809 20889 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:25.978910 20169 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:25.979847 20748 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:25.952495 20899 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.983217 20899 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.983282 20912 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.931104 20896 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:25.964274 17921 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:25.952301 20613 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:25.962029 20846 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:22:25.962723 20829 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:26.001065 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:25.963243 20842 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:25.999838 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47020.4us
I20260709 12:22:26.002117 18812 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:26.006084 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:25.986217 20918 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:26.005412 17921 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.010221 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:25.996304 20896 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.957715 20908 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.012603 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:26.016990 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74915.9us
I20260709 12:22:26.017469 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:26.021487 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:26.019754 17921 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.025321 20829 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.026741 20613 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.027535 20829 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.007743 20918 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.034507 20918 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:25.996176 20442 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:22:26.003428 20647 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:26.057576 18811 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:22:26.058598 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46101us
I20260709 12:22:26.034991 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:26.048172 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:26.019730 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:26.060526 18092 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:22:26.060621 18095 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.053290 17921 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:26.010809 20772 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:26.035401 20917 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:22:26.014448 20901 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.017743 20748 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
W20260709 12:22:26.071007 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62272.7us
I20260709 12:22:26.074398 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:26.069967 17921 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.082643 18137 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:22:26.083184 18137 leader_election.cc:403] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:26.086439 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:26.087819 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:26.088083 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:26.089023 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:26.008975 20811 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:22:26.090117 20726 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:26.090058 20772 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:26.091095 20726 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:26.091336 20772 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:26.092286 20811 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:26.092804 20811 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.093205 20811 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.068934 20898 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:26.093781 20746 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:26.094167 20773 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:26.094641 20905 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:26.094741 20773 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:26.015205 20604 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:22:26.095211 20905 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:26.095379 20773 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.095737 20604 consensus_queue.cc:1243] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:26.095839 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:26.096853 18092 raft_consensus.cc:2410] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:26.096350 20905 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.042949 20718 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:26.098059 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:26.073128 20910 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:26.100112 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.094410 20898 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.100047 20910 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.072935 20810 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:22:26.101516 20897 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:26.094429 20746 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.101998 20846 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:26.102479 20746 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:26.102547 20846 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:26.103714 20846 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.096374 20604 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:26.102180 20888 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:22:26.104714 20604 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:26.105293 20846 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.105912 20604 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.094223 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:26.094964 18017 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:26.111423 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:26.099853 20905 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:26.099619 17921 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 4 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.130260 17921 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.057878 20894 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:26.141073 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51523.7us
I20260709 12:22:26.142323 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:26.132735 17921 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.145265 18095 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:26.145437 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:26.149358 18096 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20260709 12:22:26.149385 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66949.3us
I20260709 12:22:26.150343 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:26.149930 18096 pending_rounds.cc:85] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 1
I20260709 12:22:26.153402 18096 pending_rounds.cc:107] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:22:26.153715 18096 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:26.150774 17921 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 4 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.168802 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:26.070788 20842 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:26.041568 20570 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.174242 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:26.171088 17921 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.175552 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:26.181581 17921 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.184795 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:26.185096 17921 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.191908 18092 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:22:26.191296 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:26.193670 18096 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:26.196688 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78501us
I20260709 12:22:26.169541 20894 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.199887 18097 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:26.203313 18021 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:26.178651 20169 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:22:26.187248 20604 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:26.226086 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.186429 20810 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:26.229283 17921 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.252776 17921 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 5, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.173995 20898 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:26.207232 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:26.092348 20726 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.260936 18023 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:26.262079 18021 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:26.259125 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 108389us
I20260709 12:22:26.259423 20933 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:26.264062 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:26.264137 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:26.260459 17921 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.243760 20718 raft_consensus.cc:2764] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260709 12:22:26.270432 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.272519 18020 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:26.275895 20169 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:26.276429 20169 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:26.276844 20169 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:26.277647 20570 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:26.274513 20748 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:22:26.238998 20892 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:22:26.280795 20892 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:26.281211 20892 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:26.282423 20892 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.283769 18064 leader_election.cc:400] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:26.205343 20842 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:22:26.284185 18064 leader_election.cc:403] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:22:26.280028 20748 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:26.285449 20748 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:26.286479 20748 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.287426 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:26.287714 20806 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:26.288041 20748 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:26.288324 20918 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.288823 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:26.292138 20912 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:26.302351 18021 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:26.304913 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:26.306103 20806 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:26.306864 20910 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:26.307782 20806 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:26.309520 20806 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:26.310746 20937 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:26.324213 20806 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:26.312270 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:26.271034 20889 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:22:26.319650 18020 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:26.310669 20901 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:26.371683 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.327345 18167 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:26.328605 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:26.330986 18015 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:26.373026 18096 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:26.204190 20894 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:26.366498 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54331.5us
I20260709 12:22:26.380069 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:26.366668 20937 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:26.388671 18164 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:26.388881 18171 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:26.269760 17921 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 2, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 2 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.331768 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:26.374750 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:26.374868 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:26.376791 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:26.378799 18165 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:26.394591 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62989.6us
I20260709 12:22:26.327004 20933 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:26.411142 17921 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 4 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.361526 20892 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:26.315155 20842 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:22:26.389137 20570 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:26.416436 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.413475 17921 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 5, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:22:26.422147 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 253456us
I20260709 12:22:26.430462 18165 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:26.431742 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 256292us
I20260709 12:22:26.410019 20910 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:26.441303 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.449900 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.429489 17921 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.464887 17921 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:26.417181 20908 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:26.369326 20718 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 12:22:26.467391 17921 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 5, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:22:26.511835 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 98260.3us
W20260709 12:22:26.515367 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88249.1us
I20260709 12:22:26.516781 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:26.441167 20570 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:26.525460 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 155771us
W20260709 12:22:26.525698 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 145766us
I20260709 12:22:26.413128 20854 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 12:22:26.508328 17921 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:26.530686 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.550945 18164 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:22:26.551859 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.428108 20773 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.507648 20908 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:26.531278 20918 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:26.586143 18813 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:26.568703 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:26.571273 20898 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:26.574913 18809 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:26.577212 20570 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.579560 20746 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.586843 18813 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:26.559428 20937 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:26.625591 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.640434 18164 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:26.621547 18170 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
W20260709 12:22:26.638689 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.510159 20806 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.646801 20940 raft_consensus.cc:2764] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:22:26.647411 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:26.648157 18167 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:26.650444 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 122198us
I20260709 12:22:26.652987 18171 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:26.656355 18813 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.657660 18813 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:26.667174 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.668123 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:26.721908 18099 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:26.745702 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:26.746263 18167 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
W20260709 12:22:26.749619 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.749632 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.756240 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:26.753160 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.560115 20933 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.765334 18094 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:26.792632 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.771667 18813 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:26.577294 20908 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.775748 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:26.845232 20169 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.004s
I20260709 12:22:26.720011 20947 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:26.761274 20894 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:26.841794 20772 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.750s	user 0.012s	sys 0.004s
I20260709 12:22:26.829502 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:26.739144 20854 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:26.867762 18806 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:26.930174 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.901925 20896 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.881s	user 0.008s	sys 0.008s
I20260709 12:22:26.899330 20773 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:26.950423 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:26.974565 20938 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
W20260709 12:22:26.977430 20952 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:22:26.984957 18169 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:22:26.930344 20945 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.000041 18167 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:26.940621 20772 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:26.935453 20948 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:26.934836 20854 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:26.996585 20899 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.013s	user 0.018s	sys 0.000s
W20260709 12:22:27.034818 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 366757us
I20260709 12:22:26.939431 20965 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.942852 18806 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:26.925724 20169 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:27.008827 20829 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.966s	user 0.010s	sys 0.005s
I20260709 12:22:27.052038 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:27.028501 20939 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.463s	user 0.002s	sys 0.000s
I20260709 12:22:27.051548 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:27.056005 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:27.074532 20854 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:26.936594 20959 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.119s
I20260709 12:22:27.076066 20854 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.055135 20899 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.081351 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:26.964527 20896 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:26.896538 20937 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.053158 20916 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.088101 18018 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:27.087610 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.116657 18015 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:27.084053 20966 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.085106 20971 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:27.118404 20966 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.119478 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:27.121269 20966 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:27.056577 20746 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.122351 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:27.122896 18172 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:22:26.932906 20947 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.130939 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:27.085625 20938 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:22:27.130384 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:27.139971 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58747.9us
W20260709 12:22:27.141711 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.153594 20748 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.865s	user 0.011s	sys 0.006s
I20260709 12:22:27.122375 20896 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:27.143642 20946 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260709 12:22:27.070082 20948 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.089087 20957 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:27.152230 20604 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.908s	user 0.004s	sys 0.008s
I20260709 12:22:27.019321 20945 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.208851 20917 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.082983 20888 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:27.130486 20916 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:27.198100 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 112115us
I20260709 12:22:27.158242 20966 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:27.141417 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:27.152230 20910 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.690s	user 0.013s	sys 0.000s
I20260709 12:22:27.142938 20894 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:27.175953 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.248354 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.208420 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1821) took 1039 ms. Trace:
I20260709 12:22:27.105566 20972 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.102s
I20260709 12:22:27.209692 18093 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:27.245921 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:27.190946 18167 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:27.220384 20811 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.127s	user 0.016s	sys 0.001s
W20260709 12:22:27.218111 20846 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.090s	user 0.014s	sys 0.004s
I20260709 12:22:27.249747 18173 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:27.279114 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:27.253361 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 112098us
I20260709 12:22:27.275851 18095 rpcz_store.cc:276] 0709 12:22:26.168905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.188569 (+ 19664us) service_pool.cc:224] Handling call
0709 12:22:26.188896 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.194116 (+  5220us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:26.194149 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.241582 (+ 47433us) log.cc:844] Serialized 26 byte log entry
0709 12:22:26.411057 (+169475us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:26.411111 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.411199 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.208186 (+796987us) raft_consensus.cc:1685] finished
0709 12:22:27.208210 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.208348 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7236,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":3782,"thread_start_us":126656,"threads_started":1,"wal-append.queue_time_us":245751}
I20260709 12:22:27.226902 20169 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:27.224887 20948 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:27.281922 18093 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:27.283232 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.017s	sys 0.037s
I20260709 12:22:27.278816 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:27.295017 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.017s	sys 0.037s
W20260709 12:22:27.295414 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 734754us
I20260709 12:22:27.229769 20945 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:27.288849 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.221308 20947 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.292492 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:27.306063 18164 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:27.309691 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:27.256901 20905 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.107s	user 0.008s	sys 0.011s
W20260709 12:22:27.304633 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.006s	sys 0.009s
W20260709 12:22:27.280951 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.003s	sys 0.011s
W20260709 12:22:27.314385 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.006s	sys 0.009s
W20260709 12:22:27.315377 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 608524us
I20260709 12:22:27.295420 18093 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:27.323262 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.323731 18093 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.272161 20895 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:22:27.313915 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.004s	sys 0.011s
I20260709 12:22:27.329064 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.089553 20965 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.281888 20910 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:27.340015 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 338) took 1267 ms. Trace:
I20260709 12:22:27.340520 18016 rpcz_store.cc:276] 0709 12:22:26.072686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.073803 (+  1117us) service_pool.cc:224] Handling call
0709 12:22:26.074198 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:26.074449 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:26.169780 (+ 95331us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.277078 (+1107298us) spinlock_profiling.cc:243] Waited 993 us on lock 0x7b1800322c00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:22:27.339516 (+ 62438us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:27.339604 (+    88us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:27.339676 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.339763 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.339934 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":43735,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"spinlock_wait_cycles":2284800,"thread_start_us":48596,"threads_started":1}
I20260709 12:22:27.346316 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:27.350224 18168 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:27.353022 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:27.354349 18166 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:27.348141 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 296573us
I20260709 12:22:27.357532 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 396) took 1027 ms. Trace:
I20260709 12:22:27.358848 18165 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:27.358175 18166 rpcz_store.cc:276] 0709 12:22:26.329748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.330040 (+   292us) service_pool.cc:224] Handling call
0709 12:22:26.385890 (+ 55850us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.350922 (+965032us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54002a2720. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662edc0 00007f6c166b91c0 00007f6c166ba231 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c
0709 12:22:27.356926 (+  6004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.357423 (+   497us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289872128}
I20260709 12:22:27.328186 20959 consensus_queue.cc:1243] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:27.327859 20948 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:27.369398 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.231376 20917 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.353518 20947 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:27.375406 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:27.281569 20811 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.225464 20957 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.258437 20888 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.339021 20169 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.364562 20972 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:27.377949 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49042.4us
I20260709 12:22:27.393510 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:27.359232 20905 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:27.394186 20984 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
W20260709 12:22:27.398103 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50011.1us
I20260709 12:22:27.413122 18169 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:22:27.245980 18806 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.383056 20604 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:27.400933 18166 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:27.401012 18093 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:22:27.422313 18093 pending_rounds.cc:85] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:27.422967 18171 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:22:27.423393 18171 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 3
I20260709 12:22:27.423667 18171 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 4.4
W20260709 12:22:27.418780 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53730.4us
W20260709 12:22:27.381672 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 864928us
I20260709 12:22:27.424083 18171 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:27.402364 20892 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.976s	user 0.016s	sys 0.001s
W20260709 12:22:27.421226 18780 connection.cc:256] RPC call timeout handler was delayed by -1.0013252599997031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
W20260709 12:22:27.395515 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.422595 18093 pending_rounds.cc:107] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:22:27.436000 18093 raft_consensus.cc:2889] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:27.411056 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.425343 20976 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:22:27.425499 20947 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:27.422580 20748 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:22:27.426411 20604 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.390046 20982 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:22:27.406579 20972 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.428110 20978 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:27.430192 18806 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.441638 18138 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:22:27.449432 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14d2c4b70e164856a205483966ca4963"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:27.394416 20959 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:27.418259 20895 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:27.472103 18095 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:27.459074 18171 raft_consensus.cc:606] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.474352 18171 raft_consensus.cc:614] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:27.462651 20972 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:27.467602 20898 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.553s	user 0.013s	sys 0.000s
W20260709 12:22:27.468577 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43922.8us
I20260709 12:22:27.403483 20169 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.486459 18090 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:22:27.486984 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 81078.4us
W20260709 12:22:27.487823 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 178229us
I20260709 12:22:27.508689 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:27.509310 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106804us
W20260709 12:22:27.491438 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.488098 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:27.510798 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:27.514976 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:27.498867 20896 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.517123 20975 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.486680 20895 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:27.521445 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.525167 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:27.527287 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.541186 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.541795 18064 leader_election.cc:400] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:27.543509 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.529412 20910 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:27.517685 20896 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.520989 20972 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:27.543805 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.521270 20895 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.526871 20894 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:27.539595 20842 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:27.538978 20748 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:27.536850 20916 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:27.544739 20806 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:27.563978 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:27.545276 20846 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:27.519793 20975 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.612560 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.551783 20905 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:27.614790 18171 raft_consensus.cc:606] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
W20260709 12:22:27.616353 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75252.8us
I20260709 12:22:27.617707 20973 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:27.631453 18171 raft_consensus.cc:614] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:27.618042 20811 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.617898 20981 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.159s
I20260709 12:22:27.609184 20947 consensus_queue.cc:1243] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:22:27.644848 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 119821us
I20260709 12:22:27.617558 20604 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:27.612179 20892 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:27.619937 20987 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:22:27.657191 20949 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:27.660918 20991 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20260709 12:22:27.662262 20917 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:27.662892 20986 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:22:27.663345 20998 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:22:27.650837 20962 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.191s
W20260709 12:22:27.676687 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44048.4us
W20260709 12:22:27.680269 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.642952 20726 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:27.701586 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.560667 20894 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.779973 20950 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:27.780843 20975 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.782379 20894 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:27.609903 20772 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:22:27.643383 20965 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:27.779995 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 167601us
I20260709 12:22:27.791258 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.802580 20896 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.803551 20570 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:27.804612 20842 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:27.805881 20977 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:27.806818 20977 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.811700 20933 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:27.710656 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:22:27.727279 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.825956 18171 raft_consensus.cc:606] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.779342 20726 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:27.782727 20169 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.826447 18171 raft_consensus.cc:614] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:27.797379 20854 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.721s	user 0.010s	sys 0.003s
I20260709 12:22:27.712421 20949 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:27.880451 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89304.8us
I20260709 12:22:27.880776 20949 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:27.711973 20811 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.181s
I20260709 12:22:27.882301 20937 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:27.881523 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:27.885138 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:27.886921 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:27.888176 20977 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:27.888475 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:27.889142 18171 raft_consensus.cc:606] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:27.889566 18171 raft_consensus.cc:614] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:27.963327 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:28.097088 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:28.288064 18096 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:28.225299 20997 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.288323 18813 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:28.288924 20997 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:27.942536 20806 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:28.097076 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:27.907186 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 359587us
I20260709 12:22:27.950729 20937 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:28.292702 20937 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:28.136075 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:28.293459 20937 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:28.074684 20854 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.294836 20854 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:28.086750 20948 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.704s	user 0.009s	sys 0.004s
I20260709 12:22:28.095062 20899 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:27.891307 20978 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:27.894639 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 242996us
I20260709 12:22:28.289484 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:27.961896 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 392) took 1633 ms (client timeout 1873 ms). Trace:
W20260709 12:22:28.302721 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.043679 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:28.356624 18813 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:28.084503 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:28.369390 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:28.327687 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.291923 20997 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:28.365957 18096 raft_consensus.cc:2376] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6. Re-sending same reply.
I20260709 12:22:28.372540 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.371713 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:28.368764 18091 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:28.378734 18164 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:28.373646 18016 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:28.390677 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:27.980720 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.609s (SENT) [suppressed 37 similar messages]
I20260709 12:22:28.392733 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:28.395732 18096 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:28.398130 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:28.399210 18020 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:28.400647 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:28.334731 20979 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.898s	user 0.000s	sys 0.005s
I20260709 12:22:27.950659 20570 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:28.394301 18164 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:27.942252 20996 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:28.404587 18015 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:28.405196 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:28.352722 18806 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.864s	user 0.012s	sys 0.004s
I20260709 12:22:28.407737 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:27.985865 20972 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:28.409513 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:28.403579 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.410444 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:27.962606 20976 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.470s
I20260709 12:22:27.899442 20898 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:27.962188 20960 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:28.335129 20772 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.039878 20959 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.414831 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.297917 20854 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:28.366600 18168 rpcz_store.cc:269] 0709 12:22:26.328081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.328254 (+   173us) service_pool.cc:224] Handling call
0709 12:22:27.961557 (+1633303us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b54003025d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:27.961768 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2090728320}
I20260709 12:22:28.418330 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.418766 18164 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:22:28.277765 20895 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.717s	user 0.012s	sys 0.000s
I20260709 12:22:28.097929 20982 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.531263 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.532109 18017 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:28.357623 20811 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.960266 20991 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.389770 20975 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.067909 20995 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.366210 20896 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.416046 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:28.418905 20613 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:28.490396 20806 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:28.542130 18166 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:28.425346 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:28.542923 18171 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:28.495217 18096 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:28.544135 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:28.450402 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1739) took 1938 ms. Trace:
I20260709 12:22:28.501888 18099 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:28.493444 20898 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:28.386723 20948 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:28.393538 20999 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.373545 20169 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:28.522185 18020 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:28.519101 20960 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:28.528064 20974 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:28.539783 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.015s	sys 0.030s
I20260709 12:22:28.540488 18168 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:28.560302 18096 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 12:22:28.562093 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.417328 20570 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:27.889595 20842 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.506312 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:28.545414 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:28.571486 18099 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:28.544757 18172 rpcz_store.cc:276] 0709 12:22:26.511626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.834283 (+322657us) service_pool.cc:224] Handling call
0709 12:22:26.834785 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.123334 (+288549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.449938 (+1326604us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54003e2bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:28.450288 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":661897984}
I20260709 12:22:28.495635 20996 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:28.549881 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:28.527561 20772 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.556618 20977 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.491221 20978 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.567126 18096 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:28.567922 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.016s	sys 0.030s
I20260709 12:22:28.423171 21005 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.561841 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:28.640226 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78510.4us
I20260709 12:22:28.533888 21002 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:28.570379 18168 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:28.557096 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2580) took 1801 ms. Trace:
I20260709 12:22:28.534483 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:28.554746 18806 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:28.577643 18024 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:28.589239 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:28.558682 21009 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.561403 20806 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:28.665872 20988 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.236s	user 0.001s	sys 0.004s
I20260709 12:22:28.666200 20806 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:28.537422 20975 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:28.667745 21001 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:22:28.668391 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1827) took 1966 ms. Trace:
I20260709 12:22:28.549046 20896 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.668973 18092 rpcz_store.cc:276] 0709 12:22:26.702083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.738288 (+ 36205us) service_pool.cc:224] Handling call
0709 12:22:26.738741 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.265331 (+526590us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800360e80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166afbfb 00007f6c166ba64b
0709 12:22:27.265619 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:27.265661 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:27.266544 (+   883us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.383785 (+117241us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:27.383827 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.383907 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:28.664955 (+1281048us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:28.668083 (+  3128us) raft_consensus.cc:1685] finished
0709 12:22:28.668118 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.668310 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2503,"raft.queue_time_us":129615,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":13321,"spinlock_wait_cycles":57700992,"thread_start_us":255392,"threads_started":2,"wal-append.queue_time_us":162751}
I20260709 12:22:28.671558 20896 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:28.668767 20975 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:28.672461 18092 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:28.563612 20169 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.618638 18171 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:28.644400 20957 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:28.674911 20896 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:28.675801 20169 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:28.683983 20937 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.533936 20895 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.641423 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:28.688345 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47055.6us
I20260709 12:22:28.650740 18097 rpcz_store.cc:276] 0709 12:22:26.755271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.841568 (+ 86297us) service_pool.cc:224] Handling call
0709 12:22:26.895735 (+ 54167us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.956122 (+ 60387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.956164 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.990737 (+ 34573us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.291510 (+300773us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:27.291561 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.291641 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:28.299101 (+1007460us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:28.543799 (+244698us) raft_consensus.cc:1685] finished
0709 12:22:28.543834 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.556950 (+ 13116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20923,"raft.queue_time_us":69239,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":43119,"spinlock_wait_cycles":34798464,"thread_start_us":213105,"threads_started":1,"wal-append.queue_time_us":382619}
I20260709 12:22:28.576612 20570 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:28.546708 20811 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:28.660882 18813 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:28.662132 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1735) took 2224 ms. Trace:
I20260709 12:22:28.704843 18813 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:28.706151 18813 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.704959 18173 rpcz_store.cc:276] 0709 12:22:26.424658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.441016 (+ 16358us) service_pool.cc:224] Handling call
0709 12:22:26.492683 (+ 51667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.276056 (+783373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.648381 (+1372325us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:28.648694 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1740364672}
I20260709 12:22:28.598413 20972 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.692164 18020 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:28.693785 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:28.700059 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 719359us
I20260709 12:22:28.700498 18017 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:28.702885 18019 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:28.641351 20898 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.656687 20977 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.763321 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:28.794162 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.766021 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 564) took 2179 ms. Trace:
I20260709 12:22:28.954421 18024 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:28.954943 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.955089 18806 rpcz_store.cc:276] 0709 12:22:26.584804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.602410 (+ 17606us) service_pool.cc:224] Handling call
0709 12:22:26.740937 (+138527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.658307 (+1917370us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.763707 (+105400us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:28.796706 20957 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.169s
I20260709 12:22:28.957103 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:28.957783 20985 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.632s	user 0.008s	sys 0.005s
I20260709 12:22:28.783934 18813 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:28.959915 20977 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:28.768847 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 632911us
W20260709 12:22:28.962255 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 268575us
I20260709 12:22:28.962953 21010 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:28.850903 20980 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.033s	sys 0.107s
I20260709 12:22:28.965979 20967 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:28.966622 20967 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:28.708792 20169 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.248s
I20260709 12:22:28.970252 18813 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:28.971499 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:28.972234 18171 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:28.972071 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:28.974557 18778 connection.cc:256] RPC call timeout handler was delayed by -1.2034322290001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.242s for a response
W20260709 12:22:28.966245 20980 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.039s	sys 0.110s
I20260709 12:22:28.977762 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:28.704952 20975 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:28.975185 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.512s (SENT)
I20260709 12:22:28.983183 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:28.984866 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.744s (SENT)
I20260709 12:22:28.985477 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:28.797077 18808 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEARNER]: Advancing to term 7
I20260709 12:22:28.972882 20958 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.989692 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:28.996161 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.830s (SENT)
I20260709 12:22:28.996830 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:28.997025 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:28.988005 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:29.003031 18171 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:28.657626 21005 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.979305 20977 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:28.994088 20613 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.980822 20806 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:28.537798 20995 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.537081 20991 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.539207 20982 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.574855 20974 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.643085 20960 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.561465 20999 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:28.590888 20996 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.702809 21009 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.969635 20773 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:28.973872 20966 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.993346 20912 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.974182 20948 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:28.981117 20895 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:29.014669 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:29.106854 18022 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:29.107069 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92553.5us
I20260709 12:22:29.038563 18165 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:29.100543 20966 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.984362 20975 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:28.997263 21006 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.383s	user 0.001s	sys 0.001s
I20260709 12:22:28.528661 20959 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:29.052382 20613 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:29.081043 20842 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.047920 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65173us
W20260709 12:22:29.062583 20971 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
W20260709 12:22:29.065518 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69400.9us
I20260709 12:22:29.036099 20977 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.045351 21004 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:28.995189 21010 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260709 12:22:29.076782 18813 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:29.102319 18024 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:29.104367 20811 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:29.011968 20965 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260709 12:22:28.995630 21020 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.088s
I20260709 12:22:29.042829 21008 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 12:22:29.071065 20854 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:29.060145 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2076) took 2401 ms. Trace:
I20260709 12:22:29.113665 18810 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:22:29.109810 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:29.122767 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 140243us
W20260709 12:22:29.123401 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 140065us
I20260709 12:22:29.130762 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:29.100930 20169 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:29.100190 20912 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:29.066365 21015 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 12:22:29.045516 20958 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:28.997727 20967 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:29.347445 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 212306us
I20260709 12:22:29.351322 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1839) took 1820 ms. Trace:
W20260709 12:22:29.246750 21011 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.478s	user 0.002s	sys 0.000s
I20260709 12:22:29.109992 18809 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:29.413969 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2570) took 1017 ms. Trace:
I20260709 12:22:29.417296 18024 rpcz_store.cc:276] 0709 12:22:28.396067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.691018 (+294951us) service_pool.cc:224] Handling call
0709 12:22:29.413893 (+722875us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.418695 18022 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:29.413060 18169 rpcz_store.cc:276] 0709 12:22:26.658376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.986115 (+327739us) service_pool.cc:224] Handling call
0709 12:22:27.037909 (+ 51794us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.418997 (+381088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.059591 (+1640594us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b540002add4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:29.059974 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":862157056}
I20260709 12:22:29.409134 18093 rpcz_store.cc:276] 0709 12:22:27.530759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.532544 (+  1785us) service_pool.cc:224] Handling call
0709 12:22:27.533082 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.547272 (+ 14190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:27.547307 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:27.554368 (+  7061us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.892191 (+337823us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:27.892262 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.892406 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.350939 (+1458533us) raft_consensus.cc:1685] finished
0709 12:22:29.350978 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.351218 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13704,"raft.queue_time_us":25632,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":52943,"thread_start_us":248350,"threads_started":1,"wal-append.queue_time_us":989995}
I20260709 12:22:29.185775 21036 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:29.110903 20976 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:29.110178 20772 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:29.316969 21019 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:22:29.429697 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:29.117928 20975 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:29.007822 21005 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:29.105870 20999 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:29.315618 21024 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:22:29.122114 20948 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:29.438349 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:29.391741 21035 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.111075 21012 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
W20260709 12:22:29.444955 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 254592us
I20260709 12:22:29.410599 20977 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:29.112185 20773 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:29.435511 20965 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:29.454082 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 459413us
I20260709 12:22:29.449607 20967 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:29.315734 20966 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:29.463213 18094 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:22:29.532886 18094 pending_rounds.cc:85] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:29.533365 18094 pending_rounds.cc:107] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:22:29.533727 18094 raft_consensus.cc:2889] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.110260 20889 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.335s
I20260709 12:22:29.472748 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:29.480193 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:29.478787 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:29.433336 21036 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:29.489778 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:29.434551 20978 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:29.491508 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 335) took 3421 ms (client timeout 2198 ms). Trace:
I20260709 12:22:29.492375 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1743) took 2826 ms. Trace:
I20260709 12:22:29.455229 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:29.482251 20811 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.503300 20995 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:29.471105 20772 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:29.503988 21037 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.504688 21032 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.506453 20613 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:29.508054 20773 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:29.509023 20959 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:29.510409 21009 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:29.511106 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2568) took 1115 ms. Trace:
W20260709 12:22:29.497786 21038 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:22:29.511644 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:29.512022 20912 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:29.452397 20976 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:29.428916 20806 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260709 12:22:29.550443 18167 rpcz_store.cc:276] 0709 12:22:26.666459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.002317 (+335858us) service_pool.cc:224] Handling call
0709 12:22:27.002698 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.275626 (+272928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.492108 (+2216482us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:29.492313 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432107392}
I20260709 12:22:29.545722 18022 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:29.418924 20169 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:29.139019 20842 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:30.125627 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.066s	sys 0.102s
W20260709 12:22:29.629803 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 115300us
I20260709 12:22:30.130345 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:29.549888 18018 rpcz_store.cc:269] 0709 12:22:26.070170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.070361 (+   191us) service_pool.cc:224] Handling call
0709 12:22:29.491206 (+3420845us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b5400268754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:29.491387 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2205868416}
W20260709 12:22:30.130833 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.067s	sys 0.102s
W20260709 12:22:30.131973 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.00124e+06us
I20260709 12:22:30.132367 18020 rpcz_store.cc:276] 0709 12:22:28.395509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.783491 (+387982us) service_pool.cc:224] Handling call
0709 12:22:28.783984 (+   493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.789080 (+  5096us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.789131 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.946855 (+157724us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.042381 (+ 95526us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:29.042445 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.042566 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.510850 (+468284us) raft_consensus.cc:1685] finished
0709 12:22:29.510871 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.511032 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4476,"raft.queue_time_us":4636,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":36178,"thread_start_us":45647,"threads_started":1,"wal-append.queue_time_us":414665}
I20260709 12:22:30.133584 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:30.137476 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2584) took 1170 ms. Trace:
W20260709 12:22:30.014667 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.024s	sys 0.056s
W20260709 12:22:30.045295 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.003s	sys 0.005s
W20260709 12:22:30.139972 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.030s	sys 0.072s
I20260709 12:22:30.138553 18022 rpcz_store.cc:276] 0709 12:22:28.966913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.444522 (+477609us) service_pool.cc:224] Handling call
0709 12:22:30.137371 (+692849us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:30.140295 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.003s	sys 0.006s
W20260709 12:22:30.142827 18138 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.578s (SENT)
I20260709 12:22:30.143536 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:29.555238 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:29.936749 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.145809 18094 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:30.155720 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0462442210000518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20260709 12:22:30.157938 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:30.160621 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:29.503990 20982 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:30.125895 20570 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.390s	user 0.017s	sys 0.000s
I20260709 12:22:29.514801 20991 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:29.678011 21024 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 12:22:30.080813 20948 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:29.899197 21025 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:22:30.184473 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:30.211768 18018 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:29.499440 21030 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.051971 21013 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.842s	user 0.004s	sys 0.006s
W20260709 12:22:29.851918 18807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.018s	sys 0.044s
I20260709 12:22:30.175849 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2583) took 1209 ms. Trace:
I20260709 12:22:30.167028 21032 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.136790 20772 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:30.224972 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.209600 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:30.171849 20773 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:30.203387 20959 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:29.493295 21035 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:29.550163 20974 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.177382 20966 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:22:30.219714 18020 rpcz_store.cc:276] 0709 12:22:28.966033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.157421 (+1191388us) service_pool.cc:224] Handling call
0709 12:22:30.175295 (+ 17874us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.204183 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:30.250622 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2585) took 1061 ms. Trace:
I20260709 12:22:30.251183 18022 rpcz_store.cc:276] 0709 12:22:29.189450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.159825 (+970375us) service_pool.cc:224] Handling call
0709 12:22:30.250498 (+ 90673us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.084838 20958 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.225903 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:30.104032 21036 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:29.680160 20811 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:29.856231 21042 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.553s
W20260709 12:22:30.239133 18808 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.226s	user 0.017s	sys 0.000s
I20260709 12:22:30.239375 21016 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:29.628845 20978 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:29.550802 20965 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:30.224731 20989 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 2.576s	user 0.006s	sys 0.004s
I20260709 12:22:30.242280 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:30.323792 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2013) took 1343 ms. Trace:
I20260709 12:22:30.215981 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2012) took 1235 ms. Trace:
I20260709 12:22:30.253005 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:30.255280 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121205us
I20260709 12:22:30.324554 18018 rpcz_store.cc:276] 0709 12:22:28.980207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.156893 (+1176686us) service_pool.cc:224] Handling call
0709 12:22:30.215878 (+ 58985us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.324970 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2582) took 1359 ms. Trace:
I20260709 12:22:30.177146 20995 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:22:30.225420 21029 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.331660 18098 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:30.183270 20570 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.235008 20773 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:30.274345 18093 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:30.276103 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:30.281626 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 124834us
I20260709 12:22:30.237721 20959 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:30.223688 18807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.112s	user 0.022s	sys 0.049s
I20260709 12:22:30.295848 20966 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:30.346333 20773 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:30.242242 21058 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:30.325323 18020 rpcz_store.cc:276] 0709 12:22:28.965368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.249411 (+1284043us) service_pool.cc:224] Handling call
0709 12:22:30.324901 (+ 75490us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.324107 18021 rpcz_store.cc:276] 0709 12:22:28.980467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.241931 (+1261464us) service_pool.cc:224] Handling call
0709 12:22:30.323721 (+ 81790us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.331674 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2577) took 1871 ms. Trace:
I20260709 12:22:30.343083 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 647) took 2250 ms. Trace:
I20260709 12:22:30.221441 21030 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.209360 21037 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:30.350274 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2938442209997447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2574s for a response
I20260709 12:22:30.229908 20948 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:22:30.349414 18808 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEARNER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:30.361549 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2009) took 1382 ms. Trace:
I20260709 12:22:30.298573 21049 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.366637 18020 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:30.383711 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2578) took 1854 ms. Trace:
I20260709 12:22:30.309957 21031 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.222s
I20260709 12:22:30.366036 18018 rpcz_store.cc:276] 0709 12:22:28.460392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.330597 (+1870205us) service_pool.cc:224] Handling call
0709 12:22:30.330981 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.331221 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.331250 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.331300 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:30.331345 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.331443 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.331597 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.371589 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.390079 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:30.392869 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.394119 18095 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
W20260709 12:22:30.371567 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.25923e+06us
I20260709 12:22:30.374127 18021 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:22:30.402346 18021 pending_rounds.cc:85] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 3
I20260709 12:22:30.402779 18021 pending_rounds.cc:107] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:22:30.403120 18021 raft_consensus.cc:2889] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:30.354758 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.000s	sys 0.011s
I20260709 12:22:30.366178 18811 rpcz_store.cc:276] 0709 12:22:28.092799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.093049 (+   250us) service_pool.cc:224] Handling call
0709 12:22:28.093665 (+   616us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.143519 (+ 49854us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:28.143593 (+    74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.271804 (+128211us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.619017 (+347213us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:28.619080 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.619232 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.050792 (+1431560us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:30.330367 (+279575us) raft_consensus.cc:1685] finished
0709 12:22:30.330393 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.342966 (+ 12573us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45047,"raft.queue_time_us":426711,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":15132,"spinlock_wait_cycles":39359872,"thread_start_us":346543,"threads_started":1,"wal-append.queue_time_us":513050}
W20260709 12:22:30.404241 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.985s	user 0.000s	sys 0.012s
W20260709 12:22:30.354044 21026 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.131s	user 0.000s	sys 0.002s
I20260709 12:22:30.380070 18022 rpcz_store.cc:276] 0709 12:22:28.979228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.252138 (+1272910us) service_pool.cc:224] Handling call
0709 12:22:30.361456 (+109318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.370618 20966 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.410048 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.352819 20959 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.384020 18020 rpcz_store.cc:276] 0709 12:22:28.529808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.365897 (+1836089us) service_pool.cc:224] Handling call
0709 12:22:30.366365 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.383410 (+ 17045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.383650 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:30.365777 18138 leader_election.cc:336] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.008s (SENT)
I20260709 12:22:30.370720 20995 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.380100 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:30.381170 18809 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.912s	user 0.006s	sys 0.010s
I20260709 12:22:30.408793 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2580) took 1849 ms. Trace:
I20260709 12:22:30.412192 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.425290 18097 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:22:30.410112 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2579) took 1864 ms. Trace:
I20260709 12:22:30.377321 20997 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:30.326228 20772 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.421878 18093 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:30.430338 18022 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:30.423020 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 577) took 1010 ms. Trace:
I20260709 12:22:30.431254 18811 rpcz_store.cc:276] 0709 12:22:29.412882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.409640 (+996758us) service_pool.cc:224] Handling call
0709 12:22:30.422913 (+ 13273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.311147 21016 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.423833 18018 rpcz_store.cc:276] 0709 12:22:28.559289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.389508 (+1830219us) service_pool.cc:224] Handling call
0709 12:22:30.408102 (+ 18594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.408357 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.408397 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.408450 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:30.408494 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.408562 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.408715 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:30.406584 21057 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:22:30.425742 18021 rpcz_store.cc:276] 0709 12:22:28.545416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.373300 (+1827884us) service_pool.cc:224] Handling call
0709 12:22:30.373690 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.409765 (+ 36075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.410040 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.396760 20842 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:30.439810 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2008) took 1460 ms. Trace:
I20260709 12:22:30.440621 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 762) took 1750 ms. Trace:
I20260709 12:22:30.440405 18022 rpcz_store.cc:276] 0709 12:22:28.978941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.411716 (+1432775us) service_pool.cc:224] Handling call
0709 12:22:30.430133 (+ 18417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.435474 (+  5341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.439734 (+  4260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.441172 18811 rpcz_store.cc:276] 0709 12:22:28.690737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.439618 (+1748881us) service_pool.cc:224] Handling call
0709 12:22:30.440005 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.440237 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.440270 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.440321 (+    51us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.440361 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.440422 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.440557 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.443176 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:30.312137 20976 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.342190 21029 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.447422 18018 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
W20260709 12:22:30.448994 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83267.2us
I20260709 12:22:30.454085 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:30.455047 18093 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.456784 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2014) took 1473 ms. Trace:
I20260709 12:22:30.457489 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:30.457289 18018 rpcz_store.cc:276] 0709 12:22:28.983070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.446642 (+1463572us) service_pool.cc:224] Handling call
0709 12:22:30.447118 (+   476us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:30.456360 (+  9242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.456706 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.458441 18811 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:30.427937 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 757) took 1797 ms. Trace:
I20260709 12:22:30.423199 20982 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:30.467267 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:30.468101 18097 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.423552 18809 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:30.431073 20991 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.445096 20967 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.894s	user 0.009s	sys 0.007s
I20260709 12:22:30.420140 20966 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:30.479164 21050 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
W20260709 12:22:30.480365 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.428231 20995 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.452035 21032 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.438371 20997 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.462554 20169 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.480666 20948 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.494190 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.468879 20772 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.487816 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.493721 18095 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:30.396134 21049 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.508409 18811 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:30.477142 20982 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.487146 21009 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.521350 18093 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:22:30.493769 21031 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.464711 18808 rpcz_store.cc:276] 0709 12:22:28.615864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.639106 (+ 23242us) service_pool.cc:224] Handling call
0709 12:22:28.708935 (+ 69829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.379866 (+1670931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.412594 (+ 32728us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.526243 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.526618 21051 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:30.526862 20982 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.527133 18808 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:30.527290 21051 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:30.528319 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.528422 21051 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:30.530035 21053 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.212s	user 0.000s	sys 0.002s
I20260709 12:22:30.529085 18808 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:30.531494 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1762) took 1539 ms. Trace:
I20260709 12:22:30.532102 18168 rpcz_store.cc:276] 0709 12:22:28.992216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.998193 (+  5977us) service_pool.cc:224] Handling call
0709 12:22:28.998596 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.117016 (+118420us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.117124 (+   108us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.125085 (+  7961us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.631973 (+506888us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:29.632050 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.632177 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.531180 (+899003us) raft_consensus.cc:1685] finished
0709 12:22:30.531213 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.531412 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":245993,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":46967,"thread_start_us":532469,"threads_started":2,"wal-append.queue_time_us":1062929}
I20260709 12:22:30.536490 18171 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:30.540474 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.484176 20991 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.514617 20978 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.532866 21051 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.551155 20991 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:30.551316 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 576) took 1845 ms. Trace:
I20260709 12:22:30.516192 20948 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.560191 21030 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.561107 21037 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.562955 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1844) took 2263 ms. Trace:
W20260709 12:22:30.556990 20977 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.007s	user 0.012s	sys 0.000s
I20260709 12:22:30.569727 18096 rpcz_store.cc:276] 0709 12:22:28.299949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.612579 (+312630us) service_pool.cc:224] Handling call
0709 12:22:28.613087 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.613805 (+   718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.614001 (+   196us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.614710 (+   709us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.240327 (+1625617us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800199660. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:22:30.373430 (+133103us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:30.373476 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.373562 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.562610 (+189048us) raft_consensus.cc:1685] finished
0709 12:22:30.562636 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.562872 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25817,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":6979,"spinlock_wait_cycles":27219840,"thread_start_us":132652,"threads_started":1,"wal-append.queue_time_us":1801413}
I20260709 12:22:30.543740 20811 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:30.568127 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1841) took 2272 ms (client timeout 1822 ms). Trace:
W20260709 12:22:30.574798 18091 rpcz_store.cc:269] 0709 12:22:28.295928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.562368 (+2266440us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:30.568035 (+  5667us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:30.563385 21035 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:30.538440 21044 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:30.558851 20975 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.077s	user 0.012s	sys 0.004s
I20260709 12:22:30.576576 21049 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.577021 20975 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.578204 20966 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.579116 20965 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.579344 20975 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:30.572887 21072 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.580433 20977 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.581099 20948 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.581270 21072 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.582950 20977 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:30.577438 21056 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
I20260709 12:22:30.578310 18169 raft_consensus.cc:606] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:30.597765 18169 raft_consensus.cc:614] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:30.531857 21009 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:30.563050 21046 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.201s	user 0.001s	sys 0.001s
I20260709 12:22:30.565619 18097 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:30.584339 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 764) took 1819 ms. Trace:
I20260709 12:22:30.566531 18808 rpcz_store.cc:276] 0709 12:22:28.705511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.550182 (+1844671us) service_pool.cc:224] Handling call
0709 12:22:30.550619 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.550881 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.550918 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.550972 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:30.551019 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.551095 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.551235 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.587112 20947 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:30.587585 21016 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:30.587826 21041 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
W20260709 12:22:30.590997 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 265619us
W20260709 12:22:30.592001 21063 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:22:30.560194 20995 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:30.612145 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.613284 18811 rpcz_store.cc:276] 0709 12:22:28.764900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.546251 (+1781351us) service_pool.cc:224] Handling call
0709 12:22:30.572256 (+ 26005us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.583834 (+ 11578us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.583879 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.583949 (+    70us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:30.584009 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.584093 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.584254 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.548053 21031 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.608322 21030 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:30.623775 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:30.628409 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1751) took 1657 ms. Trace:
I20260709 12:22:30.629413 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.630278 18167 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:30.630520 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1760) took 1640 ms. Trace:
I20260709 12:22:30.630514 20982 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:30.631453 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.631719 21030 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.632385 18167 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:30.631004 18172 rpcz_store.cc:276] 0709 12:22:28.990059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.990220 (+   161us) service_pool.cc:224] Handling call
0709 12:22:28.990598 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.072219 (+ 81621us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.072298 (+    79us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.073303 (+  1005us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.412535 (+339232us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:29.412596 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.412693 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.630301 (+1217608us) raft_consensus.cc:1685] finished
0709 12:22:30.630325 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.630457 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":197090,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":82658,"thread_start_us":358780,"threads_started":2,"wal-append.queue_time_us":1148966}
I20260709 12:22:30.633392 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.633512 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.634574 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:30.635927 18172 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:30.628971 18166 rpcz_store.cc:276] 0709 12:22:28.970761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.970905 (+   144us) service_pool.cc:224] Handling call
0709 12:22:28.987947 (+ 17042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.988358 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.988399 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.998129 (+  9730us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.261941 (+263812us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:29.261998 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.262121 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.628122 (+1366001us) raft_consensus.cc:1685] finished
0709 12:22:30.628148 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.628329 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":186690,"threads_started":1,"wal-append.queue_time_us":1240700}
I20260709 12:22:30.614240 20947 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:30.629413 20977 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:30.639629 21072 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.640728 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 765) took 1871 ms. Trace:
I20260709 12:22:30.640789 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 648) took 2546 ms. Trace:
I20260709 12:22:30.638705 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.688505 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.728992 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.681694 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.650697 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.751487 18091 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:30.751942 18172 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:30.751403 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.728538 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.656589 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.703271 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.755002 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:30.690881 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.751488 18099 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:30.756002 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.648908 20971 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:30.758253 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.758855 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.758401 18096 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:30.759747 18091 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:30.758316 21030 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.760720 20971 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:30.708248 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:30.767344 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:30.768519 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60425.9us
I20260709 12:22:30.769187 18099 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 12:22:30.771191 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:30.771790 18167 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:30.769977 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:30.774024 18815 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.775713 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.776486 18093 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 1.
I20260709 12:22:30.777932 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.779721 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:30.769344 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:30.780838 18166 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:30.783319 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.787700 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.788229 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.788525 18172 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:30.647047 20811 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:30.777689 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.791685 18167 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:22:30.708750 20772 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.709766 20896 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.651374 21009 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.794140 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:30.810495 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:30.795255 21066 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:30.791059 21030 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:30.798709 18172 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:22:30.631337 20982 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.825237 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.775408 21037 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:30.812485 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.798564 20958 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:22:30.815392 20948 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:30.816175 20983 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 3.371s	user 0.024s	sys 0.055s
I20260709 12:22:30.752985 18807 rpcz_store.cc:276] 0709 12:22:28.094425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.226392 (+131967us) service_pool.cc:224] Handling call
0709 12:22:28.326008 (+ 99616us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.832690 (+506682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.849276 (+ 16586us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.011531 (+162255us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.364696 (+1353165us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:30.374757 (+ 10061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.380421 (+  5664us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.640532 (+260111us) raft_consensus.cc:1685] finished
0709 12:22:30.640553 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.640713 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1332,"raft.queue_time_us":196703,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":10739,"thread_start_us":1697493,"threads_started":2,"wal-append.queue_time_us":1154227}
I20260709 12:22:30.776063 20773 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.817241 21049 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:30.815603 21071 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.767839 20977 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:22:30.789429 21072 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:30.823676 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.824671 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.799875 18167 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:22:30.830194 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:30.801076 21042 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:30.833484 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.833631 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.834089 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.859081 18093 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:30.833484 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.860267 18094 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:30.816495 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:30.860749 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:30.819636 20959 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.854624 21067 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.838141 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.856418 18099 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:30.864810 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48445.1us
I20260709 12:22:30.866819 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:30.831182 21037 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.869433 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:30.869858 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.868530 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.826203 20965 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.809408 20772 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.872895 18095 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:30.893347 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.895567 18815 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:30.871233 20977 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:30.753959 18808 rpcz_store.cc:276] 0709 12:22:28.769665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.622900 (+1853235us) service_pool.cc:224] Handling call
0709 12:22:30.639830 (+ 16930us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.640165 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.640217 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.640288 (+    71us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.640345 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.640440 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.640627 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.868446 20966 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.876401 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:30.877139 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2077) took 4218 ms. Trace:
I20260709 12:22:30.877254 20169 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:30.907909 21066 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:30.907802 20966 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.908571 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.909660 18172 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:30.813380 20896 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.864039 21042 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.914116 21042 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.914898 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:30.915859 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 646) took 2823 ms. Trace:
I20260709 12:22:30.905689 18098 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:30.836198 21031 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.871942 21071 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.852128 20948 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.882319 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:30.883742 20982 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.853101 21049 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.886901 20947 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.938161 21049 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:30.938493 20947 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.887567 20773 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.885933 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.889519 20959 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:30.890302 20976 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:30.941840 20937 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.944254 20976 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.944864 20772 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.944914 20976 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:30.890100 21067 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:30.896761 18095 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:30.873467 21075 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:30.943146 18090 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:30.903136 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 649) took 2768 ms (client timeout 1955 ms). Trace:
I20260709 12:22:30.869441 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.977476 20948 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.977851 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.978248 21044 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:30.978823 21044 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.905282 20977 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:30.897141 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 645) took 2805 ms (client timeout 1609 ms). Trace:
I20260709 12:22:30.906023 18170 rpcz_store.cc:276] 0709 12:22:26.658660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.995568 (+336908us) service_pool.cc:224] Handling call
0709 12:22:26.996002 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.021004 (+ 25002us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:27.021068 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:27.021786 (+   718us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.285433 (+263647us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:27.346910 (+ 61477us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.347191 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.347301 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.774313 (+3427012us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:30.876775 (+102462us) raft_consensus.cc:1685] finished
0709 12:22:30.876896 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.877054 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8797,"prepare.queue_time_us":62155,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":7855,"raft.queue_time_us":33671,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":22214,"spinlock_wait_cycles":16677760,"thread_start_us":170570,"threads_started":1,"wal-append.queue_time_us":357334}
I20260709 12:22:30.877790 21037 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:30.878796 20888 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.919947 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 759) took 2303 ms. Trace:
I20260709 12:22:30.946383 20966 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.916301 18812 rpcz_store.cc:276] 0709 12:22:28.092076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.092374 (+   298us) service_pool.cc:224] Handling call
0709 12:22:28.099193 (+  6819us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.130427 (+ 31234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.130484 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.131346 (+   862us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.915116 (+2783770us) spinlock_profiling.cc:243] Waited 7.12 ms on lock 0x7b1800419920. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1
0709 12:22:30.915320 (+   204us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:30.915364 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.915442 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.915493 (+    51us) raft_consensus.cc:1685] finished
0709 12:22:30.915506 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.915786 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6416,"raft.queue_time_us":347778,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":35380,"spinlock_wait_cycles":16378624,"thread_start_us":318565,"threads_started":1,"wal-append.queue_time_us":578585,"wal-append.run_cpu_time_us":22404,"wal-append.run_wall_time_us":1743208}
I20260709 12:22:30.922830 21071 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.923852 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:30.926419 20896 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:30.928632 20997 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:30.832666 20971 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.965176 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:30.966171 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:30.946708 18807 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4. Re-sending same reply.
I20260709 12:22:30.970993 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:30.904875 20965 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:30.974617 21075 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.834988 21009 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:30.976078 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:31.118712 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.061761 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.130789 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1756) took 2146 ms. Trace:
I20260709 12:22:31.135731 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:31.131371 18092 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:31.137385 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.137485 20842 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:31.140857 18090 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:31.131494 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 160573us
I20260709 12:22:31.021981 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:30.980018 18815 rpcz_store.cc:269] 0709 12:22:28.135307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.135501 (+   194us) service_pool.cc:224] Handling call
0709 12:22:30.769742 (+2634241us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:30.903006 (+133264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26600320}
I20260709 12:22:31.127215 18095 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:31.143074 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:31.141742 18165 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.014s	sys 0.047s
W20260709 12:22:31.148463 18165 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.014s	sys 0.048s
I20260709 12:22:31.149305 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:31.135650 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.134508 18173 rpcz_store.cc:276] 0709 12:22:28.984242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.998300 (+ 14058us) service_pool.cc:224] Handling call
0709 12:22:28.998619 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.998853 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:29.116399 (+117546us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.589368 (+1472969us) spinlock_profiling.cc:243] Waited 9.25 ms on lock 0x7b18003e84e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:22:30.589645 (+   277us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.130176 (+540531us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.130287 (+   111us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:31.130372 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.130482 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.130707 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1175608,"prepare.queue_time_us":1617228,"prepare.run_cpu_time_us":170,"prepare.run_wall_time_us":8894,"raft.queue_time_us":102075,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":128,"spinlock_wait_cycles":21951360,"thread_start_us":879432,"threads_started":3,"wal-append.queue_time_us":364542,"wal-append.run_cpu_time_us":1388,"wal-append.run_wall_time_us":1018751}
I20260709 12:22:31.126886 18813 rpcz_store.cc:276] 0709 12:22:28.616726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.994128 (+377402us) service_pool.cc:224] Handling call
0709 12:22:28.994616 (+   488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.076574 (+ 81958us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:29.455143 (+378569us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.520917 (+ 65774us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.521016 (+    99us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:29.521091 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.919633 (+1398542us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b5400494154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:30.919692 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.919863 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28707,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":119481,"spinlock_wait_cycles":187529472,"thread_start_us":65211,"threads_started":1,"wal-append.queue_time_us":771498}
I20260709 12:22:30.995402 20977 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:31.139381 21066 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:31.127324 21075 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:31.157745 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.127562 20726 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.162590 18172 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260709 12:22:31.140296 20811 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:31.154465 21074 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
W20260709 12:22:31.193104 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.166352 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:31.183655 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.190857 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.154610 21071 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:31.154259 20966 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:31.199003 18094 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:31.161772 20842 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:31.156256 21015 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:22:31.216054 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:31.203835 21071 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.219265 18095 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:22:31.126874 18809 rpcz_store.cc:269] 0709 12:22:28.091669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.096505 (+  4836us) service_pool.cc:224] Handling call
0709 12:22:30.896877 (+2800372us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b540034fdd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:30.897052 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1858744448}
I20260709 12:22:31.202860 21067 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.222592 21071 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:31.223506 18095 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:22:31.224058 21009 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.224834 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.231618 20811 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:31.207765 20965 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.246119 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:31.244449 18095 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 2
I20260709 12:22:31.251294 18095 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:22:31.251366 18094 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:31.251713 18095 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:31.253264 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:31.238016 21029 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.254279 18095 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:22:31.254289 20811 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.257759 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.267104 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.267498 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.254040 21009 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.253399 21042 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.268224 18095 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:31.245504 20976 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.271937 18094 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:31.274158 21009 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.274158 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.276364 18812 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:31.277056 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 766) took 2330 ms. Trace:
I20260709 12:22:31.277405 18812 rpcz_store.cc:276] 0709 12:22:28.946460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.275760 (+2329300us) service_pool.cc:224] Handling call
0709 12:22:31.276141 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.276785 (+   644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.276990 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.278906 18812 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:31.281453 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 767) took 2312 ms. Trace:
I20260709 12:22:31.281534 21032 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:31.282310 21032 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:31.283483 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2561) took 2891 ms. Trace:
I20260709 12:22:31.284200 21020 consensus_queue.cc:1243] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:31.284096 21032 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:31.284766 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.86723e+06us
I20260709 12:22:31.284950 21020 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:31.285421 21020 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:31.285600 20995 raft_consensus.cc:2955] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:31.286361 21020 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.288164 21020 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.289342 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.290051 20895 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:31.290776 20897 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:31.291628 21040 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:31.292402 20897 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:31.294981 18022 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:31.295218 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2010) took 2315 ms. Trace:
I20260709 12:22:31.283991 18016 rpcz_store.cc:276] 0709 12:22:28.392295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.392436 (+   141us) service_pool.cc:224] Handling call
0709 12:22:28.415503 (+ 23067us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:28.951900 (+536397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.951957 (+    57us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:28.982077 (+ 30120us) log.cc:844] Serialized 608 byte log entry
0709 12:22:29.067689 (+ 85612us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:29.099653 (+ 31964us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.282735 (+2183082us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b180033c760. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:22:31.282985 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.283074 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.283120 (+    46us) raft_consensus.cc:1685] finished
0709 12:22:31.283206 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.283373 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77798,"prepare.queue_time_us":44977,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":2324963,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":7564,"spinlock_wait_cycles":4911488,"thread_start_us":440122,"threads_started":2,"wal-append.queue_time_us":360945}
I20260709 12:22:31.295603 18020 rpcz_store.cc:276] 0709 12:22:28.979476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.412279 (+1432803us) service_pool.cc:224] Handling call
0709 12:22:30.412612 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.294804 (+882192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.294836 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.294904 (+    68us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:31.294948 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.295023 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.295155 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":881772,"raft.queue_time_us":883183,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93}
I20260709 12:22:31.297358 21032 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:31.298429 18815 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:31.299664 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.80494e+06us
I20260709 12:22:31.300030 20967 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:31.300587 20967 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:31.297930 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.312515 18812 rpcz_store.cc:276] 0709 12:22:28.969422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.278300 (+2308878us) service_pool.cc:224] Handling call
0709 12:22:31.278677 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.281138 (+  2461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.281351 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.270958 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.305897 20570 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.271857 20842 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.309284 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.328332 18022 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:31.301730 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:31.310302 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 33) took 1139 ms. Trace:
I20260709 12:22:31.330219 18169 rpcz_store.cc:276] 0709 12:22:30.170778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.171282 (+   504us) service_pool.cc:224] Handling call
0709 12:22:31.303570 (+1132288us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54003e2454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:31.310231 (+  6661us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":809110912}
I20260709 12:22:31.310880 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 768) took 2338 ms. Trace:
I20260709 12:22:31.298164 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.315519 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.293299 21040 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:31.312891 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:31.326821 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.332012 18815 rpcz_store.cc:276] 0709 12:22:28.972901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.297929 (+2325028us) service_pool.cc:224] Handling call
0709 12:22:31.298231 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.298872 (+   641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.310815 (+ 11943us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.335875 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.350327 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.358333 18016 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:22:31.362679 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49891us
I20260709 12:22:31.363955 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:31.367058 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.374154 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.351500 21085 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:22:31.378962 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:31.382088 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:31.270548 21044 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.382591 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.385895 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:31.387568 17987 connection.cc:256] RPC call timeout handler was delayed by -1.3656356940000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260709 12:22:31.391467 17987 connection.cc:256] RPC call timeout handler was delayed by -1.1604356940001708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.205s for a response
W20260709 12:22:31.391901 17987 connection.cc:256] RPC call timeout handler was delayed by -1.1145356940000966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
I20260709 12:22:31.392566 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:31.316166 20442 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:22:31.316371 20949 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:22:31.385007 18809 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:31.395680 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 94027.2us
I20260709 12:22:31.405786 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:31.319337 21049 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.396546 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:31.393967 18808 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 2: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:31.376788 20997 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:31.403012 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:31.382318 20959 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:31.412901 18095 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 3: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:31.393378 20800 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.419142 20977 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.421017 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.410823 20999 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:31.397197 20982 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:31.335975 21058 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:31.465943 20971 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:31.426575 21005 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:31.427372 20974 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:31.428508 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.428988 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:31.429204 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.429804 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.495151 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2556) took 3123 ms (client timeout 2008 ms). Trace:
I20260709 12:22:31.430997 20895 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:22:31.495695 18019 rpcz_store.cc:269] 0709 12:22:28.371568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.371712 (+   144us) service_pool.cc:224] Handling call
0709 12:22:31.494886 (+3123174us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b540024d954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:31.495054 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95686016}
I20260709 12:22:31.431906 20899 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:31.434404 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.436513 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.510389 18171 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:31.437148 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.512403 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:31.437510 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:31.516021 18169 raft_consensus.cc:2368] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4.
W20260709 12:22:31.438663 21087 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.081s	user 0.001s	sys 0.001s
I20260709 12:22:31.438881 18168 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:31.439903 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.441107 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.523766 18173 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:31.449477 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.414032 20997 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.452195 21008 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:31.452490 21003 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.460819 20996 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:31.397579 21022 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:31.462445 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2554) took 3091 ms (client timeout 1578 ms). Trace:
I20260709 12:22:31.540730 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.541345 18023 rpcz_store.cc:269] 0709 12:22:28.370835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.371016 (+   181us) service_pool.cc:224] Handling call
0709 12:22:31.462016 (+3091000us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:31.462309 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3123062016}
I20260709 12:22:31.541896 18016 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:31.542629 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.467481 20982 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:22:31.470623 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.496716 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67827.7us
I20260709 12:22:31.498454 18091 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:31.549134 18091 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:22:31.401679 20811 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.442754 21015 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:31.505487 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.565840 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:31.566615 18022 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:31.444907 20806 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.441171 21049 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.506765 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.501219 20949 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:31.518369 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.576737 18019 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
W20260709 12:22:31.577353 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.533516 20899 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:31.582329 20899 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:31.583402 20899 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.584028 21014 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.584748 21014 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.584874 20899 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:31.528414 20997 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:31.463128 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.542526 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.609594 18018 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:31.463788 18020 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:22:31.554026 20961 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:31.555708 20977 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.493309 20974 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:31.569459 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.548650 21008 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.618424 18023 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:31.619923 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:31.573581 18015 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
W20260709 12:22:31.622643 21090 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:22:31.622871 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.624640 21008 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.530040 21002 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:31.621608 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.549415 21003 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.626714 20976 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.626894 18023 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:31.628196 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.629137 21003 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.580675 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 573) took 3163 ms. Trace:
I20260709 12:22:31.629953 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:31.629875 18810 rpcz_store.cc:276] 0709 12:22:28.417224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.417790 (+   566us) service_pool.cc:224] Handling call
0709 12:22:28.964121 (+546331us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:28.964365 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.420536 (+456171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.580287 (+2159751us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b540009bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:31.580582 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":280178304}
I20260709 12:22:31.581322 20967 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.454806 20800 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.633935 20800 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.576812 20806 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.575563 21015 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.585371 21014 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.639178 20965 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:31.640244 21015 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.594687 20811 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:31.641927 21037 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.509s	user 0.004s	sys 0.009s
I20260709 12:22:31.584460 21071 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.642541 21037 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.643494 18096 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:31.643571 20811 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:31.644261 20811 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.645140 18172 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:31.647974 20811 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.648886 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:31.585422 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:31.614944 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.653609 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.655121 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.578796 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 182345us
I20260709 12:22:31.617401 20966 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.618054 20974 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:31.568090 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:31.624568 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.662387 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 94435.3us
I20260709 12:22:31.663481 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.664484 18015 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:31.493959 20947 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.555s	user 0.007s	sys 0.008s
I20260709 12:22:31.625849 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.666079 20966 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.667807 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.668291 21044 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.668731 18019 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:31.669358 20966 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.550873 20982 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.672538 20811 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:31.673290 20811 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:31.675110 20800 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:31.675633 20800 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.676047 20800 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:31.676541 20811 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.644860 21037 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.691960 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.606070 18022 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:31.600826 21075 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:31.588498 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.644876 21071 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.656723 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:31.637511 20806 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:31.631229 18023 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:31.756093 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 99464.2us
I20260709 12:22:31.757068 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:31.678088 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:31.758908 18019 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:31.665736 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.577037 20999 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:22:31.682421 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.695050 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.771179 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.771955 18022 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:22:31.706538 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:31.749853 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.760226 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:31.774585 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.775424 18015 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 3: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:31.769006 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.783090 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.783946 18015 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:31.785312 18022 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:31.786599 18022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:22:31.789691 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:31.797030 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.805352 20997 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.776324 20947 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:31.775926 20976 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:31.784238 20811 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:31.823666 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.828908 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.830570 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 220421us
I20260709 12:22:31.839747 18019 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
W20260709 12:22:31.834913 18138 connection.cc:256] RPC call timeout handler was delayed by -1.202086256000257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17900000000000002s for a response
I20260709 12:22:31.799285 21003 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:31.845762 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.848629 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.849159 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.810012 21092 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:22:31.879364 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:31.887343 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49621.9us
I20260709 12:22:31.828657 20976 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.820925 20947 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:31.905103 18015 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:31.913800 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.915711 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.916471 18168 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:31.919219 20950 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:31.921944 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.923723 18018 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:31.917524 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.924566 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:31.939388 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.942993 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:31.943742 18015 raft_consensus.cc:2376] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:31.934322 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:31.946403 18018 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:31.919806 20950 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:31.900252 20976 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.949939 18099 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:31.951047 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:31.952579 18090 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:31.961325 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.902626 20947 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:31.894838 20895 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 12:22:31.973965 20965 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.974611 20965 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.975939 21092 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:31.977000 20965 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.978657 18806 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:31.980717 20959 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.537s	user 0.012s	sys 0.001s
I20260709 12:22:31.981340 20959 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:31.926623 20949 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:31.986325 20949 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:31.987561 21032 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.000s
I20260709 12:22:31.988992 20949 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.888021 21003 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.962226 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:31.992169 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:31.992900 18090 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:31.994632 21003 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:31.956435 21075 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:31.967718 20950 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:31.968057 21014 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.969859 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:32.000177 18099 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:31.971284 20997 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:31.971397 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:32.003532 18164 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:32.005012 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:31.973388 20800 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:31.963615 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.006847 18015 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:32.006956 18018 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:32.010555 18168 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
W20260709 12:22:32.012094 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 247101us
I20260709 12:22:32.015591 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:32.016505 18812 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:32.024761 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.025879 18018 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:32.028100 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:32.029721 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:32.031210 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:31.990986 21010 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.013959 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:32.042997 18815 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: [11.12-11.12]
I20260709 12:22:32.056146 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.060254 18019 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:32.066185 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:32.069711 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:32.028962 20950 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:32.074657 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.070995 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:32.067592 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.076122 18164 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:32.077173 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:32.080148 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:32.081991 18018 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:32.083381 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.126761 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:32.125329 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:32.127827 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:32.145409 18164 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:32.127552 20971 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.657s	user 0.009s	sys 0.004s
I20260709 12:22:32.153682 18811 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:32.123136 20997 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:32.128492 20959 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:32.157310 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.159497 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:32.059414 21010 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.163933 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:32.168430 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 327228us
I20260709 12:22:32.124092 21075 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
W20260709 12:22:32.171433 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.150681 18812 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:22:32.165072 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.157330 20971 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:32.153030 20982 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.152678 20947 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.171587 21002 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:32.179013 21036 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:32.227782 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:32.228291 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.189175 20974 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.530s	user 0.014s	sys 0.000s
I20260709 12:22:32.245708 18810 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:32.246963 18022 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Received request to transfer leadership
I20260709 12:22:32.188566 21107 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:32.201623 20997 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:32.224678 21054 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:32.226334 21075 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:32.235289 21002 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:32.262727 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 643) took 4172 ms. Trace:
I20260709 12:22:32.267879 18814 rpcz_store.cc:276] 0709 12:22:28.090811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.097195 (+  6384us) service_pool.cc:224] Handling call
0709 12:22:28.097783 (+   588us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:28.098380 (+   597us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.098416 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:28.324664 (+226248us) log.cc:844] Serialized 52 byte log entry
0709 12:22:28.704692 (+380028us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:31.564993 (+2860301us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b1800063b80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf
0709 12:22:31.565173 (+   180us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.256646 (+691473us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.262083 (+  5437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.262233 (+   150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.262374 (+   141us) raft_consensus.cc:1685] finished
0709 12:22:32.262390 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.262645 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18837,"prepare.queue_time_us":409950,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":5816,"raft.queue_time_us":452296,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":11256,"spinlock_wait_cycles":8640384,"thread_start_us":1180015,"threads_started":5,"wal-append.queue_time_us":991943,"wal-append.run_cpu_time_us":15322,"wal-append.run_wall_time_us":2212999}
I20260709 12:22:32.251507 21021 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:32.260231 21107 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.280439 18809 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:32.254634 21042 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:32.265018 21052 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:32.302271 21101 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
I20260709 12:22:32.278779 21075 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:32.285670 20800 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:32.293622 21014 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:32.251298 21097 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260709 12:22:32.351181 20910 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:32.252285 20974 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.351784 20910 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:32.352809 20910 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.353926 20974 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:32.354916 18023 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:32.259169 20842 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 12:22:32.345854 21075 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:32.358316 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 487) took 1657 ms. Trace:
I20260709 12:22:32.358847 18098 rpcz_store.cc:276] 0709 12:22:30.701135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.702850 (+  1715us) service_pool.cc:224] Handling call
0709 12:22:32.358040 (+1655190us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54002c9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.358210 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346508032}
W20260709 12:22:32.339073 21100 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:22:32.349807 20947 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:32.364720 20898 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:32.385208 21109 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:22:32.365446 20910 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:32.365945 21108 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:22:32.326457 21107 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:32.345803 21037 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:22:32.394721 20982 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:32.395596 21114 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260709 12:22:32.396440 21125 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
W20260709 12:22:32.388547 21098 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 12:22:32.257860 21095 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.141s
W20260709 12:22:32.399345 21084 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.169s	user 0.005s	sys 0.002s
I20260709 12:22:32.346685 21052 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.405198 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.406975 21010 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:32.408077 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.406939 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.287349 21054 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.327296 20997 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.456043 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:32.514788 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:32.576260 18093 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:32.540745 20976 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.573s	user 0.014s	sys 0.000s
I20260709 12:22:32.570365 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2576) took 4146 ms. Trace:
I20260709 12:22:32.525518 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:32.500569 20961 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.454198 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:32.440798 20967 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:32.321192 21071 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:32.435143 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.528343 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:32.639297 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111036us
I20260709 12:22:32.640126 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:32.575637 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.428217 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.580649 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:32.582049 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:32.644690 18098 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:32.585253 21118 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
I20260709 12:22:32.577829 18017 rpcz_store.cc:276] 0709 12:22:28.423774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.955279 (+531505us) service_pool.cc:224] Handling call
0709 12:22:31.496382 (+2541103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.496818 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.496853 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.497410 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.821423 (+324013us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:32.134872 (+313449us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.408001 (+273129us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b180006b140. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:22:32.408237 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.408368 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.408417 (+    49us) raft_consensus.cc:1685] finished
0709 12:22:32.408517 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.570086 (+161569us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b540024d950. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.570272 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7790,"prepare.queue_time_us":88060,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":275771,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":16032,"spinlock_wait_cycles":5847880320,"thread_start_us":568232,"threads_started":2,"wal-append.queue_time_us":613926}
I20260709 12:22:32.647598 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:32.585953 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:32.648738 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 351077us
I20260709 12:22:32.649154 18095 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:32.421120 20811 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.591s	user 0.015s	sys 0.008s
I20260709 12:22:32.590130 21123 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.590332 20889 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:32.596778 18166 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:32.604274 21002 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.608428 21052 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:32.609534 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 500) took 1476 ms. Trace:
I20260709 12:22:32.609434 20976 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.613327 20912 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.619705 20997 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:32.582664 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.635924 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:32.688506 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.644778 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64254.8us
I20260709 12:22:32.644877 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:32.647265 21058 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:32.650244 20898 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:32.633761 21066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.406s	user 0.016s	sys 0.039s
I20260709 12:22:32.652935 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 360) took 1219 ms. Trace:
I20260709 12:22:32.636699 20967 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:32.692340 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.692276 18016 rpcz_store.cc:276] 0709 12:22:31.427992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.428142 (+   150us) service_pool.cc:224] Handling call
0709 12:22:32.646853 (+1218711us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b5400268754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.647022 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109949312}
I20260709 12:22:32.693454 18090 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:32.677556 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:32.693964 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.681031 18096 rpcz_store.cc:276] 0709 12:22:31.132987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.225273 (+ 92286us) service_pool.cc:224] Handling call
0709 12:22:31.225569 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.644010 (+418441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.609235 (+965225us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400410fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:32.609451 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":960577280}
I20260709 12:22:32.695047 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:32.686820 18093 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:32.689567 18091 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:32.650275 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.689870 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:32.694948 21066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.468s	user 0.017s	sys 0.039s
I20260709 12:22:32.697657 18022 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:32.683238 20975 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:22:32.691648 21058 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.698259 21058 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:32.676460 21035 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.698693 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.699411 18022 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:32.699257 21035 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.700454 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.700974 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.702409 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.703016 21054 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:32.703119 18016 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:32.694727 20912 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.704278 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.646415 21036 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:32.695237 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:32.704440 20811 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:32.693176 20967 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.730324 20811 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:32.731629 20811 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.734194 18171 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:32.701683 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.713441 18091 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:32.713442 18024 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:32.714880 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.716935 21097 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:32.644367 21071 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.717015 21040 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:32.720314 20958 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:32.648344 20961 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.682308 21123 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.722282 20975 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:32.723196 20960 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:32.723507 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:32.855630 18098 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:32.725780 20976 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:32.728487 21083 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.517s	user 0.001s	sys 0.005s
I20260709 12:22:32.712424 21036 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.870205 21036 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:32.874349 21035 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:32.735670 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:32.737725 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 581) took 1866 ms (client timeout 1778 ms). Trace:
W20260709 12:22:32.875375 18808 rpcz_store.cc:269] 0709 12:22:30.871646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.914521 (+ 42875us) service_pool.cc:224] Handling call
0709 12:22:32.737401 (+1822880us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b5400616054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.737599 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":965989760}
W20260709 12:22:32.739863 21127 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.472s	user 0.006s	sys 0.000s
I20260709 12:22:32.876526 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.710196 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:32.879256 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 169156us
I20260709 12:22:32.879576 20975 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:32.880079 20975 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.880458 20975 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:32.883443 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2099) took 1198 ms. Trace:
I20260709 12:22:32.884123 18171 rpcz_store.cc:276] 0709 12:22:31.684689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.878895 (+194206us) service_pool.cc:224] Handling call
0709 12:22:32.883137 (+1004242us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.883312 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1834241920}
I20260709 12:22:32.740482 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1774) took 1282 ms. Trace:
I20260709 12:22:32.845471 20958 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:32.885704 18172 rpcz_store.cc:276] 0709 12:22:31.458020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.531664 (+ 73644us) service_pool.cc:224] Handling call
0709 12:22:31.570943 (+ 39279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.645640 (+ 74697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.740157 (+1094517us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b540007c454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:32.740405 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170025344}
I20260709 12:22:32.886812 21037 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.743563 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 358) took 1353 ms. Trace:
I20260709 12:22:32.887609 21037 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.755724 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:32.887959 18020 rpcz_store.cc:276] 0709 12:22:31.389893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.390174 (+   281us) service_pool.cc:224] Handling call
0709 12:22:31.463268 (+ 73094us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.610167 (+146899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.743339 (+1133172us) spinlock_profiling.cc:243] Waited 72.5 ms on lock 0x7b5400268250. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.743487 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166739200}
W20260709 12:22:32.889071 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 133466us
I20260709 12:22:32.748775 21075 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:32.756732 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.757649 21020 raft_consensus.cc:993] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:22:32.759533 21002 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:32.771821 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 590) took 1124 ms. Trace:
I20260709 12:22:32.775658 20967 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:32.779994 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.880463 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:32.781790 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 761) took 4124 ms. Trace:
I20260709 12:22:32.893584 18806 rpcz_store.cc:276] 0709 12:22:28.657228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.241330 (+584102us) service_pool.cc:224] Handling call
0709 12:22:29.445733 (+204403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.979132 (+2533399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.781360 (+802228us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:32.781686 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5823969152}
I20260709 12:22:32.773275 21058 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:32.777081 21107 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.903448 18813 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:32.808694 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:32.809705 20999 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.811547 20912 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:32.819271 18093 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:32.831079 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.834877 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:32.909260 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:32.841480 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 98791.9us
I20260709 12:22:32.844837 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2027) took 1331 ms. Trace:
I20260709 12:22:32.911103 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:32.866210 21029 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:32.866425 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143017us
W20260709 12:22:32.867760 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2093) took 2237 ms (client timeout 2632 ms). Trace:
W20260709 12:22:32.916234 18166 rpcz_store.cc:269] 0709 12:22:30.630799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.783023 (+152224us) service_pool.cc:224] Handling call
0709 12:22:32.867569 (+2084546us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b5400030a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.867689 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4162486656}
I20260709 12:22:32.917100 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:32.893163 20950 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:32.695384 20898 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.887308 20958 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.744879 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1853) took 1433 ms. Trace:
I20260709 12:22:32.889648 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.895113 18017 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:32.892352 18815 rpcz_store.cc:276] 0709 12:22:31.647310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.647906 (+   596us) service_pool.cc:224] Handling call
0709 12:22:31.648368 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.043646 (+395278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.771469 (+727823us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540009bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:32.771739 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":906773888}
I20260709 12:22:32.898653 18015 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:32.796401 18811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 657) took 2127 ms (client timeout 1955 ms). Trace:
I20260709 12:22:32.846288 21097 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:32.906483 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98000us
W20260709 12:22:32.906698 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0059563560002971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
W20260709 12:22:32.912025 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77233.6us
I20260709 12:22:32.910812 18023 rpcz_store.cc:276] 0709 12:22:31.513038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.768635 (+255597us) service_pool.cc:224] Handling call
0709 12:22:32.844602 (+1075967us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.844762 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1327446400}
I20260709 12:22:32.971462 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:32.971715 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:32.865527 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 483) took 2189 ms. Trace:
I20260709 12:22:32.869495 21071 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:32.914330 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:32.967872 18811 rpcz_store.cc:269] 0709 12:22:30.669047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.812164 (+143117us) service_pool.cc:224] Handling call
0709 12:22:32.796057 (+1983893us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54005f1c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.796270 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2979175296}
I20260709 12:22:32.969539 18020 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:32.969928 18016 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:22:32.969695 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2593) took 1064 ms. Trace:
I20260709 12:22:32.973682 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:32.974370 18019 rpcz_store.cc:276] 0709 12:22:31.905277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.921629 (+ 16352us) service_pool.cc:224] Handling call
0709 12:22:32.969437 (+1047808us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540010aed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.969625 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315456896}
I20260709 12:22:32.969027 18094 rpcz_store.cc:276] 0709 12:22:31.311100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.559037 (+247937us) service_pool.cc:224] Handling call
0709 12:22:32.360124 (+801087us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.360736 (+   612us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.360781 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.360845 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:32.360888 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.360970 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.744686 (+383716us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54002c9f50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.744811 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1840961920}
I20260709 12:22:32.976538 18020 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 12:22:32.934656 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 239490us
I20260709 12:22:32.908984 20968 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:32.982787 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:32.981146 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:32.981619 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2007) took 4003 ms (client timeout 1795 ms). Trace:
I20260709 12:22:32.905809 21095 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:32.900913 21040 raft_consensus.cc:1081] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
W20260709 12:22:32.948820 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.912621 20966 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.006012 18018 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:22:31.653317 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:32.973282 18097 rpcz_store.cc:276] 0709 12:22:30.676662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.709612 (+ 32950us) service_pool.cc:224] Handling call
0709 12:22:30.722021 (+ 12409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.766402 (+ 44381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.766468 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.773959 (+  7491us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.055514 (+281555us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:31.190316 (+134802us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.191060 (+   744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.191192 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.723841 (+1532649us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:32.865128 (+141287us) raft_consensus.cc:1685] finished
0709 12:22:32.865247 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.865433 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":188892,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":1373,"raft.queue_time_us":7567,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":5354,"spinlock_wait_cycles":5596032,"thread_start_us":415175,"threads_started":2,"wal-append.queue_time_us":389631}
I20260709 12:22:33.007861 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.906484 21119 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:32.993889 18024 rpcz_store.cc:269] 0709 12:22:28.978486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.445062 (+466576us) service_pool.cc:224] Handling call
0709 12:22:32.981406 (+3536344us) spinlock_profiling.cc:243] Waited 3.17 s on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:32.981535 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7290423680}
I20260709 12:22:32.995000 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:32.917636 21032 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:33.015839 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.913136 21035 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.930769 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:32.920470 21029 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:33.004263 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:32.912665 21020 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:33.005187 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:32.928227 21052 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:32.934552 21042 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:32.981433 21137 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.933871 20613 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:33.035163 18017 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:33.013639 20966 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:32.998193 20442 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:32.988354 18780 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.024s (SENT) [suppressed 35 similar messages]
I20260709 12:22:32.949781 20950 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:32.906297 20974 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260709 12:22:33.000777 20897 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:33.034982 20613 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.010397 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:32.989661 20968 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:32.934036 21054 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260709 12:22:32.927793 20965 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20260709 12:22:33.005712 20958 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:33.000471 20811 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:32.989332 20898 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:32.912770 20999 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.009317 21095 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:33.057477 18018 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Received request to transfer leadership
W20260709 12:22:33.057902 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47621.4us
I20260709 12:22:33.061731 20975 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.063265 21107 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:33.000691 21071 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:22:33.064649 20968 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:33.033128 21042 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:33.091044 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.031672 21020 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:22:33.106033 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.071650 21032 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:22:33.050458 21120 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.741s	user 0.009s	sys 0.000s
I20260709 12:22:33.125168 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:33.050843 21138 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:33.134325 17989 leader_election.cc:400] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4.
I20260709 12:22:33.103526 21075 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:33.147120 21042 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:33.147764 20613 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:33.152302 20961 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:33.123657 21005 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:33.149567 20999 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:33.099949 20442 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:33.070909 21037 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:33.030725 21119 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.044422 21137 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.166911 18806 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
W20260709 12:22:33.247761 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 232256us
I20260709 12:22:33.043726 21035 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.209594 17989 leader_election.cc:403] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:22:33.248917 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 334633us
I20260709 12:22:33.032356 21029 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:33.249796 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:33.209606 18021 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:33.218528 21058 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:33.222913 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:33.252982 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
W20260709 12:22:33.227342 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.794s (SENT)
I20260709 12:22:33.160768 21020 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:33.204469 20974 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.211283 21138 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:33.249498 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115420us
I20260709 12:22:33.251986 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:33.255333 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.450s (SENT)
I20260709 12:22:33.259682 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:33.168883 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2594) took 1263 ms. Trace:
I20260709 12:22:33.219573 21005 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:33.213101 21143 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:33.262349 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:33.260697 18017 rpcz_store.cc:276] 0709 12:22:31.905584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.653182 (+747598us) service_pool.cc:224] Handling call
0709 12:22:33.168655 (+515473us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54001d1354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:33.168811 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458192256}
I20260709 12:22:33.234084 21071 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:33.266222 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 292665us
I20260709 12:22:33.284708 18815 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:22:33.285244 18815 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 5
I20260709 12:22:33.285482 18815 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:22:33.257424 21037 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:33.286079 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:33.265666 21143 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:33.287014 21143 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:33.218698 21095 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.288285 21150 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:33.288928 21150 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:33.289400 21150 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:33.273448 21058 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:33.286262 21037 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:33.246345 21145 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:33.252830 20999 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4.
I20260709 12:22:33.253017 20962 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.285763 18815 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:33.072628 20897 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.261490 21138 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:33.206314 20950 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.273557 21107 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:33.303170 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.319042 18166 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:33.305981 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43726.2us
I20260709 12:22:33.310088 21071 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:33.316605 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:33.324347 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:33.319232 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 389931us
I20260709 12:22:33.328384 18812 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:33.329255 18098 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:33.331704 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:33.296547 21002 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:33.104285 21097 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.299209 20613 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.335153 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:33.335491 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 363) took 1906 ms (client timeout 2024 ms). Trace:
W20260709 12:22:33.336442 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 348134us
I20260709 12:22:33.336541 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:33.354465 18021 rpcz_store.cc:269] 0709 12:22:31.429142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.433961 (+  4819us) service_pool.cc:224] Handling call
0709 12:22:33.335295 (+1901334us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54000fee54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:33.335424 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3912215936}
I20260709 12:22:33.355122 18806 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:33.263088 20974 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.338196 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:33.339428 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:33.283809 21032 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:33.345021 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.344151 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.358296 18811 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:33.358722 18165 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.351379 18094 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:33.358311 18173 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:33.357551 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:33.360834 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1770) took 1911 ms. Trace:
I20260709 12:22:33.358167 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.365396 18169 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:33.361290 18167 rpcz_store.cc:276] 0709 12:22:31.449208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.449341 (+   133us) service_pool.cc:224] Handling call
0709 12:22:31.449753 (+   412us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:31.820593 (+370840us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.820679 (+    86us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:31.835525 (+ 14846us) log.cc:844] Serialized 78 byte log entry
0709 12:22:32.176500 (+340975us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:32.176559 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.176666 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.018079 (+841413us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b24001bd898. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:22:33.360516 (+342437us) raft_consensus.cc:1685] finished
0709 12:22:33.360540 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.360766 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":43989,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":17921,"spinlock_wait_cycles":3832064,"thread_start_us":672835,"threads_started":2,"wal-append.queue_time_us":376560}
I20260709 12:22:33.371084 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.371496 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:33.372262 18812 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:33.374684 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.375650 18164 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:33.382978 18170 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:33.386314 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:33.334417 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 339596us
I20260709 12:22:33.405611 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:33.393615 18023 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:33.343465 20442 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.432183 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:33.303474 21145 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.362738 20950 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:33.429574 18024 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:33.434284 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:33.381383 21071 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:33.440368 18173 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
W20260709 12:22:33.404335 21036 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.006s
I20260709 12:22:33.315657 20962 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.354450 20996 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 12:22:33.374331 18022 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:33.370795 20806 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:33.470635 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:33.387086 21002 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:33.483155 18016 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:33.485061 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:33.486325 18173 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:33.501214 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:33.511389 18024 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:33.530746 21002 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.522646 20999 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:22:33.404712 21075 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260709 12:22:33.380416 20613 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:33.359175 20961 consensus_queue.cc:1175] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ae9641d5c3184572a102d51a75f670ee" responder_term: 5 status { last_received { term: 5 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 5 index: 4 } }
I20260709 12:22:33.471621 21155 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.463230 21137 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:33.477764 18809 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:33.444445 20442 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.472949 21071 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:33.531790 20996 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:33.377235 21032 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.354668 21005 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.663779 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:33.626618 20997 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:33.462167 21037 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:33.523185 21151 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:33.680433 18815 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:33.664247 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:33.476226 20966 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:33.489601 21036 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.646883 20996 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:33.488250 21152 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.159s
I20260709 12:22:33.590971 21157 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:33.523989 20806 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:33.675668 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 344069us
W20260709 12:22:33.722474 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 384203us
I20260709 12:22:33.746946 18024 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:33.802525 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:33.803830 18814 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:33.674702 21035 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:33.683602 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 350012us
W20260709 12:22:33.666549 21147 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.428s	user 0.002s	sys 0.000s
W20260709 12:22:33.705574 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.642361 21155 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:33.695098 21075 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:33.544047 18165 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:22:33.684710 20965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.017s	sys 0.006s
I20260709 12:22:33.676452 21040 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.811329 18172 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.9-6.9]
W20260709 12:22:33.708490 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.815137 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:33.669781 20997 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:33.850382 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:33.679040 21151 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:33.677388 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:33.895053 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:33.900117 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 510996us
I20260709 12:22:33.897126 18165 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:33.914819 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.007s	sys 0.013s
W20260709 12:22:33.915634 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.007s	sys 0.013s
W20260709 12:22:33.916352 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.719408 21157 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:33.722106 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:33.941329 21157 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:33.941653 18809 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:22:33.946754 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.952140 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:33.953601 18165 raft_consensus.cc:2368] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6.
W20260709 12:22:33.853356 20965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.017s	sys 0.006s
I20260709 12:22:33.955790 20965 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:33.956514 20965 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.957130 20965 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:33.897066 21075 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:33.995303 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 321221us
I20260709 12:22:33.893476 21148 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:33.832809 21149 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.696615 21071 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:33.987121 21153 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:22:33.949965 21119 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:33.896112 21040 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.002547 21058 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:33.989948 21032 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:33.968524 21123 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.036583 21092 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.036247 21107 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:34.058974 21164 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:33.987385 20996 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:33.995206 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 211735us
I20260709 12:22:34.280052 20996 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:34.280817 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:34.177532 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 393136us
I20260709 12:22:34.037850 20997 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.283494 20997 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:34.285073 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.179116 18021 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.008s	sys 0.020s
W20260709 12:22:34.179805 18020 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.008s	sys 0.004s
W20260709 12:22:34.286254 18020 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.009s	sys 0.005s
W20260709 12:22:34.286134 18021 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.009s	sys 0.020s
I20260709 12:22:34.288550 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 373) took 1286 ms. Trace:
W20260709 12:22:34.179972 18022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.000s	sys 0.009s
W20260709 12:22:34.290163 18022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.000s	sys 0.010s
W20260709 12:22:34.179190 18019 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.010s	sys 0.017s
W20260709 12:22:34.292168 18019 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.010s	sys 0.017s
I20260709 12:22:34.292505 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:34.179204 18018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.004s	sys 0.008s
W20260709 12:22:34.204546 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.008s	sys 0.018s
W20260709 12:22:34.293717 18018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.004s	sys 0.009s
W20260709 12:22:34.293798 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.008s	sys 0.018s
W20260709 12:22:34.294608 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 889037us
I20260709 12:22:34.296203 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:34.207469 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1869) took 1672 ms. Trace:
I20260709 12:22:33.966326 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:34.289196 18020 rpcz_store.cc:276] 0709 12:22:33.002006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.010952 (+  8946us) service_pool.cc:224] Handling call
0709 12:22:33.011616 (+   664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.011979 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:33.034596 (+ 22617us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.287467 (+1252871us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:34.287649 (+   182us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:34.287788 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.287993 (+   205us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.288286 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30542,"prepare.queue_time_us":22965,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":645435,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":26377,"thread_start_us":1231941,"threads_started":2,"wal-append.queue_time_us":647497}
I20260709 12:22:34.297964 18096 rpcz_store.cc:276] 0709 12:22:32.535080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.695560 (+160480us) service_pool.cc:224] Handling call
0709 12:22:32.695976 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:32.929687 (+233711us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.929743 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:32.968105 (+ 38362us) log.cc:844] Serialized 52 byte log entry
0709 12:22:33.368290 (+400185us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:33.368349 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.368429 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.206480 (+838051us) raft_consensus.cc:1685] finished
0709 12:22:34.206582 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.207112 (+   530us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":94991,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":112,"thread_start_us":587765,"threads_started":2,"wal-append.queue_time_us":713852}
W20260709 12:22:34.218916 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.003s	sys 0.031s
W20260709 12:22:34.303092 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.004s	sys 0.032s
W20260709 12:22:33.991802 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 209475us
I20260709 12:22:34.016147 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:34.230238 18170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.031s	sys 0.009s
W20260709 12:22:34.310503 18170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.032s	sys 0.010s
I20260709 12:22:34.312481 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:34.243749 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 772) took 1340 ms. Trace:
I20260709 12:22:34.301118 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:34.036131 20442 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.316082 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:34.244858 20613 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:34.244911 18810 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.003s	sys 0.020s
I20260709 12:22:34.084064 21145 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.318653 20613 raft_consensus.cc:2764] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 12:22:34.350419 18810 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.003s	sys 0.020s
I20260709 12:22:34.307848 20912 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:22:34.352092 20442 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:34.352949 21040 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:34.434657 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 142533us
W20260709 12:22:34.356287 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 541055us
W20260709 12:22:34.357296 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77126.7us
I20260709 12:22:34.092450 21107 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.313294 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:34.134966 21146 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:34.066815 21148 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:22:34.045082 21149 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:34.502532 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.032s	sys 0.021s
W20260709 12:22:34.132248 21154 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.243s	user 0.000s	sys 0.002s
I20260709 12:22:34.139344 21155 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:34.108078 21156 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.262s	user 0.000s	sys 0.002s
I20260709 12:22:34.075453 21162 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:34.081341 21168 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:22:34.117146 21167 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
W20260709 12:22:34.426151 18164 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.009s	user 0.035s	sys 0.029s
W20260709 12:22:34.435014 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.537225 18164 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:34.538230 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 425) took 1199 ms. Trace:
I20260709 12:22:34.538682 18164 rpcz_store.cc:276] 0709 12:22:33.340335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.347094 (+  6759us) service_pool.cc:224] Handling call
0709 12:22:34.538110 (+1191016us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.539041 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 775) took 1635 ms. Trace:
I20260709 12:22:34.418692 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:34.540202 20897 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.314090 18808 rpcz_store.cc:276] 0709 12:22:32.903039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.906295 (+  3256us) service_pool.cc:224] Handling call
0709 12:22:32.989459 (+ 83164us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.047154 (+ 57695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.050754 (+  3600us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.213404 (+162650us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.816527 (+603123us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:33.834059 (+ 17532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.863283 (+ 29224us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.242749 (+379466us) raft_consensus.cc:1685] finished
0709 12:22:34.242807 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.243339 (+   532us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10061,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":6333,"thread_start_us":386268,"threads_started":1,"wal-append.queue_time_us":812216}
W20260709 12:22:34.546669 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85926.4us
W20260709 12:22:34.548749 20968 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.402s	user 0.019s	sys 0.016s
I20260709 12:22:34.541224 18164 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:22:34.550930 20962 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:34.553850 20974 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.563102 21149 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:34.549149 21005 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:34.567281 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1782) took 1019 ms. Trace:
I20260709 12:22:34.567889 18164 rpcz_store.cc:276] 0709 12:22:33.547843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.539963 (+992120us) service_pool.cc:224] Handling call
0709 12:22:34.540674 (+   711us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:34.549755 (+  9081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.567150 (+ 17395us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.117635 21075 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.461217 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:34.577428 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1786) took 1015 ms. Trace:
I20260709 12:22:34.578083 18165 rpcz_store.cc:276] 0709 12:22:33.561934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.317807 (+755873us) service_pool.cc:224] Handling call
0709 12:22:34.577312 (+259505us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.580181 18165 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:34.582226 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.591259 18168 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:22:34.592429 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1783) took 1031 ms. Trace:
I20260709 12:22:34.594344 17989 leader_election.cc:400] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6.
I20260709 12:22:34.594875 21040 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:34.595520 21040 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:34.603183 21020 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.338s	user 0.023s	sys 0.007s
I20260709 12:22:34.604367 21020 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.539701 18807 rpcz_store.cc:276] 0709 12:22:32.903833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.906134 (+  2301us) service_pool.cc:224] Handling call
0709 12:22:32.968612 (+ 62478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.013535 (+ 44923us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:33.013574 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.014322 (+   748us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.308317 (+293995us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:33.308385 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.308479 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.538423 (+1229944us) raft_consensus.cc:1685] finished
0709 12:22:34.538483 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.538783 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3134,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":10639,"thread_start_us":290750,"threads_started":1,"wal-append.queue_time_us":656014}
I20260709 12:22:34.609228 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2614) took 1236 ms. Trace:
I20260709 12:22:34.609740 18022 rpcz_store.cc:276] 0709 12:22:33.373257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.373405 (+   148us) service_pool.cc:224] Handling call
0709 12:22:33.373834 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.374143 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:33.572072 (+197929us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.452705 (+880633us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.501527 (+ 48822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:34.521358 (+ 19831us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:34.608712 (+ 87354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.608856 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.609117 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21592,"raft.queue_time_us":35297,"raft.run_cpu_time_us":561,"raft.run_wall_time_us":7778,"thread_start_us":871286,"threads_started":2,"wal-append.queue_time_us":558264}
I20260709 12:22:34.594947 17989 leader_election.cc:403] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260709 12:22:34.617224 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.618440 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 418) took 1314 ms. Trace:
I20260709 12:22:34.619009 18172 rpcz_store.cc:276] 0709 12:22:33.303969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.304123 (+   154us) service_pool.cc:224] Handling call
0709 12:22:33.333351 (+ 29228us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.868121 (+534770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.617738 (+749617us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400050ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:34.618182 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1098213632}
I20260709 12:22:34.620029 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.621219 18808 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:34.621567 18165 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:34.593022 18168 rpcz_store.cc:276] 0709 12:22:33.560993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.541477 (+980484us) service_pool.cc:224] Handling call
0709 12:22:34.590640 (+ 49163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.591913 (+  1273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.592306 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.623272 21040 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.624542 20806 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:34.627014 21168 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:34.627576 21168 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:34.625455 21040 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:34.631343 20975 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:34.633127 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.631351 21167 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:34.636922 21107 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.637032 21167 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:34.637312 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 106315us
I20260709 12:22:34.638854 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:34.639019 21167 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.641044 21167 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:34.642613 20960 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:34.643561 21159 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.655s	user 0.000s	sys 0.005s
I20260709 12:22:34.641222 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:34.644572 20967 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.747s	user 0.012s	sys 0.023s
I20260709 12:22:34.645064 21188 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:34.646814 18017 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:34.648036 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:34.628995 21168 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:34.475579 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.478402 18170 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:34.652042 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1780) took 1118 ms. Trace:
I20260709 12:22:34.652552 18170 rpcz_store.cc:276] 0709 12:22:33.536771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.312730 (+775959us) service_pool.cc:224] Handling call
0709 12:22:34.651957 (+339227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.482167 21146 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:34.653146 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:34.653553 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2609) took 1313 ms. Trace:
W20260709 12:22:34.655272 21150 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.241s	user 0.020s	sys 0.010s
I20260709 12:22:34.654779 21146 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.655970 21150 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.656600 21146 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:34.657014 21160 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.669s	user 0.002s	sys 0.000s
I20260709 12:22:34.482877 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 774) took 1579 ms. Trace:
W20260709 12:22:34.492569 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 52600.3us
I20260709 12:22:34.658051 18810 rpcz_store.cc:276] 0709 12:22:32.903615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.905406 (+  1791us) service_pool.cc:224] Handling call
0709 12:22:33.327091 (+421685us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.327554 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.327587 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.343596 (+ 16009us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.482143 (+1138547us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:34.482264 (+   121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.482425 (+   161us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.482505 (+    80us) raft_consensus.cc:1685] finished
0709 12:22:34.482524 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.482790 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4315,"raft.queue_time_us":6342,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7620,"thread_start_us":1008567,"threads_started":1,"wal-append.queue_time_us":759498}
I20260709 12:22:34.661350 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:34.509135 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.510345 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:34.642613 20961 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6.
W20260709 12:22:34.668685 21029 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.404s	user 0.031s	sys 0.015s
W20260709 12:22:34.669060 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 159130us
W20260709 12:22:34.464676 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 168098us
I20260709 12:22:34.515882 21145 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:34.669611 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2608) took 1331 ms (client timeout 1743 ms). Trace:
I20260709 12:22:34.670444 21145 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.673223 21164 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:34.517257 21189 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:34.518110 21002 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.034s	sys 0.043s
I20260709 12:22:33.960139 21151 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.678297 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:34.679019 21151 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.680794 18810 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:34.682156 20897 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:34.682750 20897 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.683185 20897 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:34.549497 20968 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.683867 18808 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:34.686111 21188 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:34.687659 21002 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.035s	sys 0.043s
I20260709 12:22:34.687796 21021 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:34.688210 21188 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:34.688822 21189 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:34.688673 21021 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:34.535666 21162 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:34.673661 21145 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:34.670221 18017 rpcz_store.cc:269] 0709 12:22:33.338843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.339096 (+   253us) service_pool.cc:224] Handling call
0709 12:22:34.649707 (+1310611us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:34.669482 (+ 19775us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2964737920}
I20260709 12:22:34.690965 21162 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.649894 21186 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:34.089583 21092 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.692564 21186 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.693078 21186 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:34.693408 21021 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:34.669382 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.695353 21092 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:34.698370 20898 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.563s	user 0.025s	sys 0.015s
W20260709 12:22:34.530344 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.034s	sys 0.022s
I20260709 12:22:34.699167 20960 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:34.699054 20898 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.699657 20960 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:34.700824 20960 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.701681 18023 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 4: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:34.663591 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.663684 21071 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:34.664981 20958 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.562s	user 0.029s	sys 0.011s
I20260709 12:22:34.654268 18018 rpcz_store.cc:276] 0709 12:22:33.340002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.340251 (+   249us) service_pool.cc:224] Handling call
0709 12:22:33.341116 (+   865us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.343826 (+  2710us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.343867 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.344428 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.294724 (+950296us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:34.294844 (+   120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.295027 (+   183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.653199 (+358172us) raft_consensus.cc:1685] finished
0709 12:22:34.653233 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.653445 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2002,"raft.queue_time_us":209044,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":53549,"thread_start_us":863596,"threads_started":1,"wal-append.queue_time_us":1276203}
W20260709 12:22:34.514094 18015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.008s	sys 0.042s
I20260709 12:22:34.689285 21189 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.692103 20999 raft_consensus.cc:1064] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:22:34.692428 21162 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:34.663028 21179 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:34.704355 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2615) took 1328 ms. Trace:
I20260709 12:22:34.703279 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2618) took 1326 ms. Trace:
W20260709 12:22:34.717778 18015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.008s	sys 0.044s
I20260709 12:22:34.714834 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.654878 21168 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:34.717484 18023 rpcz_store.cc:276] 0709 12:22:33.376897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.421267 (+ 44370us) service_pool.cc:224] Handling call
0709 12:22:34.702938 (+1281671us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400108bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:34.703179 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2907810048}
W20260709 12:22:34.703483 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 800695us
I20260709 12:22:34.703056 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:34.704048 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:34.724357 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:34.705643 21143 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.418s	user 0.020s	sys 0.009s
W20260709 12:22:34.723511 20999 raft_consensus.cc:1068] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Unable to promote non-voter 910f8c8d787e4abbb092413c1f90ebf3: Service unavailable: leader transfer in progress
I20260709 12:22:34.733551 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.701786 20898 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.671654 17918 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4), now has a pending config: VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1), VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3), VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:34.704216 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.741513 18168 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:34.737416 17921 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 291) took 9803 ms. Trace:
I20260709 12:22:34.718664 20968 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.669369 21029 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.708979 21189 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:34.715027 18019 rpcz_store.cc:276] 0709 12:22:33.375780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.375952 (+   172us) service_pool.cc:224] Handling call
0709 12:22:33.376336 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:34.293301 (+916965us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:34.293369 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:34.296141 (+  2772us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.468569 (+172428us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:34.468612 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.497539 (+ 28927us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.675171 (+177632us) raft_consensus.cc:1685] finished
0709 12:22:34.675205 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.704262 (+ 29057us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33083,"raft.queue_time_us":613880,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":143,"thread_start_us":1054920,"threads_started":2,"wal-append.queue_time_us":277460}
I20260709 12:22:34.756336 18814 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:34.749783 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51008.1us
I20260709 12:22:34.769989 21145 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:34.771215 21145 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:34.773835 21092 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:34.774645 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.774780 21092 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:34.755265 20960 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:34.782696 18164 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:34.787063 21036 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:34.813798 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172751us
I20260709 12:22:34.745064 21143 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.793283 18808 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:34.773180 21145 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:34.812245 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:34.828872 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.832993 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2610) took 1490 ms. Trace:
I20260709 12:22:34.776631 21092 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.809258 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:34.836794 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:34.837368 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:34.848165 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.819192 20967 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:34.844933 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:34.836424 17918 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 3 to 4, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:34.835310 21042 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.479s	user 0.015s	sys 0.009s
W20260709 12:22:34.863302 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 134316us
I20260709 12:22:34.866074 21020 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:34.743444 17921 rpcz_store.cc:276] 0709 12:22:24.922850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.923926 (+  1076us) service_pool.cc:224] Handling call
0709 12:22:25.358718 (+434792us) spinlock_profiling.cc:243] Waited 955 us on lock 0x7b5400002490. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13593640 00007f6c1c46a846 00007f6c1c57490e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:22:34.717170 (+9358452us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:27.430267 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:27.431966 (+  1699us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:27.432041 (+    75us) write_op.cc:435] Acquired schema lock
0709 12:22:27.432079 (+    38us) tablet.cc:662] Decoding operations
0709 12:22:27.799152 (+367073us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:27.799258 (+   106us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:27.799292 (+    34us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:22:28.087388 (+288096us) tablet.cc:701] Row locks acquired
0709 12:22:28.087451 (+    63us) write_op.cc:260] PREPARE: finished
0709 12:22:28.087773 (+   322us) write_op.cc:270] Start()
0709 12:22:28.088067 (+   294us) write_op.cc:276] Timestamp: P: 1783599748087657 usec, L: 0
0709 12:22:28.088104 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 12:22:28.226063 (+137959us) log.cc:844] Serialized 9093 byte log entry
0709 12:22:30.214018 (+1987955us) op_driver.cc:464] REPLICATION: finished
0709 12:22:30.514545 (+300527us) write_op.cc:301] APPLY: starting
0709 12:22:30.514666 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:30.550101 (+ 35435us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:30.550261 (+   160us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:30.553254 (+  2993us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:30.553770 (+   516us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0709 12:22:30.559007 (+  5237us) write_op.cc:312] APPLY: finished
0709 12:22:30.586501 (+ 27494us) log.cc:844] Serialized 158 byte log entry
0709 12:22:30.587530 (+  1029us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:30.587779 (+   249us) write_op.cc:454] Released schema lock
0709 12:22:30.587879 (+   100us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:30.588390 (+   511us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:30.588418 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10535,"reported_tablets":35,"spinlock_wait_cycles":2195840,"child_traces":[["op",{"apply.queue_time_us":300086,"apply.run_cpu_time_us":7112,"apply.run_wall_time_us":73834,"num_ops":23,"prepare.queue_time_us":322040,"prepare.run_cpu_time_us":162568,"prepare.run_wall_time_us":1621574,"raft.queue_time_us":969643,"raft.run_cpu_time_us":60424,"raft.run_wall_time_us":135044,"replication_time_us":2125651,"thread_start_us":1634628,"threads_started":4,"wal-append.queue_time_us":787612,"wal-append.run_cpu_time_us":53544,"wal-append.run_wall_time_us":2552070}]]}
I20260709 12:22:34.868024 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:34.868939 18169 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.438s	user 0.024s	sys 0.027s
I20260709 12:22:34.869181 18023 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:34.869910 21097 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:22:34.870468 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68788us
I20260709 12:22:34.871174 20961 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:34.871851 21188 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:34.872143 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:34.873448 18093 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:34.854812 20967 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:34.874955 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:34.880939 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1784) took 1316 ms. Trace:
I20260709 12:22:34.841262 21150 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:34.843991 21107 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:34.874432 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:34.895391 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 190591us
I20260709 12:22:34.900540 18017 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:34.921868 18164 rpcz_store.cc:276] 0709 12:22:33.561328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.573152 (+1011824us) service_pool.cc:224] Handling call
0709 12:22:34.574542 (+  1390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.789008 (+214466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.877407 (+ 88399us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b540007bf54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:34.877591 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478126336}
I20260709 12:22:34.922963 21119 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:22:34.923097 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63588.4us
W20260709 12:22:34.923528 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48723.8us
I20260709 12:22:34.923893 21177 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:34.855001 21165 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:34.928052 21052 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.853s	user 0.020s	sys 0.015s
I20260709 12:22:34.929857 21029 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:34.933858 21097 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:35.012233 21052 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:35.012795 21097 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:35.014794 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.935166 21143 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:35.015877 18096 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:22:35.016995 18096 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:35.017443 18096 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:35.018270 18813 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:35.020052 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 667) took 1665 ms. Trace:
I20260709 12:22:35.020452 18813 rpcz_store.cc:276] 0709 12:22:33.355182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.355356 (+   174us) service_pool.cc:224] Handling call
0709 12:22:33.355732 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.018838 (+1663106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.019668 (+   830us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400557154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:35.019974 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3822451584}
W20260709 12:22:34.936686 20950 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.468s	user 0.004s	sys 0.044s
W20260709 12:22:34.937089 20965 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.979s	user 0.007s	sys 0.020s
I20260709 12:22:35.022477 20965 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.025228 20965 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:34.938410 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64125.4us
I20260709 12:22:34.939062 20999 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:35.029212 18020 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:22:35.033479 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2611) took 1690 ms. Trace:
I20260709 12:22:34.892402 20961 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.942401 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:34.942885 18166 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.468s	user 0.023s	sys 0.028s
I20260709 12:22:34.942265 17918 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 3 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 5 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:34.833676 18015 rpcz_store.cc:276] 0709 12:22:33.342350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.342619 (+   269us) service_pool.cc:224] Handling call
0709 12:22:33.368810 (+ 26191us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:33.690469 (+321659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.690516 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:33.691778 (+  1262us) log.cc:844] Serialized 52 byte log entry
0709 12:22:34.740884 (+1049106us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:34.781860 (+ 40976us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.803982 (+ 22122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.815473 (+ 11491us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.815529 (+    56us) raft_consensus.cc:1685] finished
0709 12:22:34.815542 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.832902 (+ 17360us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17461,"prepare.queue_time_us":19034,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2365,"raft.queue_time_us":296946,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":25396,"spinlock_wait_cycles":7534336,"thread_start_us":1191877,"threads_started":2,"wal-append.queue_time_us":792018}
I20260709 12:22:34.946359 21107 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:34.948064 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:34.948392 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:34.950850 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.966812 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 173629us
I20260709 12:22:35.005561 18169 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:35.008714 18023 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:34.940791 20897 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.044548 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:35.035677 20997 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.751s	user 0.005s	sys 0.019s
I20260709 12:22:35.053097 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:35.057300 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.060818 18811 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:35.061060 21054 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:35.061782 21054 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.063862 21107 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:35.056259 18166 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:35.064327 21143 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:22:35.065120 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 417) took 1762 ms (client timeout 1507 ms). Trace:
I20260709 12:22:35.065207 20842 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:35.065861 18024 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:35.066481 18167 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:35.067642 18809 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:35.067945 20897 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:35.069753 21157 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.127s	user 0.017s	sys 0.007s
I20260709 12:22:35.070351 21157 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.072510 20960 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:35.073549 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:35.074505 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:35.074127 20960 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:35.076427 20960 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:35.065541 21143 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:35.077345 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 423) took 1763 ms (client timeout 1643 ms). Trace:
I20260709 12:22:35.077792 20997 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:35.078277 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:35.078935 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76128.4us
I20260709 12:22:35.080227 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.080708 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.034147 18021 rpcz_store.cc:276] 0709 12:22:33.371874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.372612 (+   738us) service_pool.cc:224] Handling call
0709 12:22:33.373020 (+   408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:34.288034 (+915014us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:34.288112 (+    78us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:34.376753 (+ 88641us) log.cc:844] Serialized 52 byte log entry
0709 12:22:34.600385 (+223632us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:22:34.800196 (+199811us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:34.816698 (+ 16502us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.823730 (+  7032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.847558 (+ 23828us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.944092 (+ 96534us) raft_consensus.cc:1685] finished
0709 12:22:34.951389 (+  7297us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.991528 (+ 40139us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b18000eda60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:35.033395 (+ 41867us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35544,"prepare.queue_time_us":25244,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":282649,"raft.run_cpu_time_us":5050,"raft.run_wall_time_us":952351,"spinlock_wait_cycles":35901440,"thread_start_us":1230618,"threads_started":2,"wal-append.queue_time_us":465811}
I20260709 12:22:35.072346 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:35.065580 18166 rpcz_store.cc:269] 0709 12:22:33.302679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.302847 (+   168us) service_pool.cc:224] Handling call
0709 12:22:35.064992 (+1762145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.074513 18091 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 2.
I20260709 12:22:35.086270 21054 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:35.072129 18097 raft_consensus.cc:2955] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:35.083643 21157 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:35.011368 21177 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.047885 20961 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:35.098531 18169 rpcz_store.cc:269] 0709 12:22:33.314201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.317664 (+  3463us) service_pool.cc:224] Handling call
0709 12:22:35.077238 (+1759574us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.088958 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:34.893040 21042 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:35.115633 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.112689 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.111750 21035 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:35.116500 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.119897 18098 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 3.
I20260709 12:22:35.121052 18809 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:35.117566 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.098269 21143 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:35.109365 20996 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.828s	user 0.013s	sys 0.020s
I20260709 12:22:35.054430 21188 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.042905 21165 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.126595 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:35.368608 18166 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:35.146492 18019 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:35.367268 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.373214 18170 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:35.169572 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68961.6us
I20260709 12:22:35.215551 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:35.182813 17918 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.229450 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1775) took 1843 ms. Trace:
I20260709 12:22:35.113359 20965 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:35.363900 18138 leader_election.cc:336] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.743s (SENT)
I20260709 12:22:35.141868 18096 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:35.353950 21071 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.615s	user 0.011s	sys 0.004s
W20260709 12:22:35.299918 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 116299us
I20260709 12:22:35.253239 18173 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:22:35.192203 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:35.183077 20842 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.114797 21177 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:35.295338 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1878) took 1974 ms. Trace:
I20260709 12:22:35.209497 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:35.176949 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.400813 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:35.385335 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.367919 21189 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.601s	user 0.005s	sys 0.009s
W20260709 12:22:35.240166 21151 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.560s	user 0.013s	sys 0.003s
W20260709 12:22:35.398808 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 183342us
I20260709 12:22:35.364252 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:35.115860 20997 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:35.394799 18167 rpcz_store.cc:276] 0709 12:22:33.386212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.431237 (+ 45025us) service_pool.cc:224] Handling call
0709 12:22:35.065803 (+1634566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.067019 (+  1216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.182682 (+115663us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400273dd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:35.229354 (+ 46672us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3758003840}
I20260709 12:22:35.190814 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:35.422286 18096 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:35.254264 21171 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:35.363547 21042 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:35.422962 18172 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:35.359108 21123 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.122082 21054 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:35.143580 20968 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:35.364624 21107 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:35.431967 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:35.143710 18024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:22:35.474303 18024 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Received request to transfer leadership
I20260709 12:22:35.436280 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:35.454236 21145 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.009s	sys 0.021s
I20260709 12:22:35.442082 18164 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.413065 20960 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:35.430867 17918 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.412648 21119 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:35.228725 20974 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:35.426477 18093 rpcz_store.cc:276] 0709 12:22:33.320684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.322820 (+  2136us) service_pool.cc:224] Handling call
0709 12:22:33.351512 (+ 28692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.039704 (+1688192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.294784 (+255080us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b5400052654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:35.295223 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3444780416}
W20260709 12:22:35.456794 21186 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.756s	user 0.011s	sys 0.005s
I20260709 12:22:35.182367 21035 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.486531 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:35.471030 21146 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.814s	user 0.013s	sys 0.010s
I20260709 12:22:35.493991 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:35.412266 20965 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.123562 20967 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.473043 20968 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:35.502193 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.482085 18091 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:35.480389 21145 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.010s	sys 0.022s
W20260709 12:22:35.506608 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 142742us
I20260709 12:22:35.501578 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:35.450506 21189 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.463999 20842 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.484045 21107 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.475209 21123 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.510076 17918 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.509879 18095 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:22:35.626775 18095 pending_rounds.cc:85] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:35.552556 18168 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:35.517371 18098 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:35.613780 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:35.629539 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 343335us
I20260709 12:22:35.575738 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:35.516834 21171 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.550908 20997 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:35.511117 21167 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.869s	user 0.015s	sys 0.003s
I20260709 12:22:35.519155 21143 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.637218 18024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:22:35.658243 18024 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Received request to transfer leadership
W20260709 12:22:35.641405 21162 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.919s	user 0.007s	sys 0.009s
I20260709 12:22:35.611884 18166 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:35.494577 20960 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.483402 20996 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:35.551024 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1778) took 2018 ms. Trace:
I20260709 12:22:35.511152 21151 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.501543 21119 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.646750 18172 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:22:35.655252 21071 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.566560 21054 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:35.652318 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 155173us
I20260709 12:22:35.511919 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:35.550145 21200 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
W20260709 12:22:35.638088 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68340.3us
I20260709 12:22:35.660916 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:35.550889 21138 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 2.027s	user 0.015s	sys 0.019s
W20260709 12:22:35.663537 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.775s (SENT)
I20260709 12:22:35.701478 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:35.660633 17918 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 3 to 4, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.657667 21092 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:35.666293 18169 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:35.648593 18164 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:35.683349 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.671636 20960 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:35.688751 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.627345 18095 pending_rounds.cc:107] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:22:35.718571 18095 raft_consensus.cc:2889] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:35.503499 21186 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.673765 18173 rpcz_store.cc:276] 0709 12:22:33.533058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.557004 (+ 23946us) service_pool.cc:224] Handling call
0709 12:22:33.557637 (+   633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.430965 (+1873328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.550623 (+119658us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:35.550932 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3741321600}
I20260709 12:22:35.694420 21052 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.632803 21040 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:22:35.704634 18093 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 12:22:35.713369 18097 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.585s	user 0.007s	sys 0.002s
I20260709 12:22:35.706738 21092 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:35.677790 20996 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:35.588799 20974 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.751859 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:35.722324 21097 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.773983 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:35.727684 18167 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:22:35.776458 18172 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:35.778255 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:35.722711 21143 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.748104 20996 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:35.802556 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:35.789963 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:35.694201 21151 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.806119 18166 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: [11.12-11.12]
I20260709 12:22:35.809988 18164 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:35.811679 18167 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:35.818243 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:35.831662 18091 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:35.731770 21071 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.806764 17918 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 3, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:35.835640 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:35.755689 21138 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.837759 18173 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:35.841898 18096 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:35.848672 18173 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 12:22:35.836369 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56493.5us
I20260709 12:22:35.819275 21188 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:35.778048 21097 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.865903 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:35.867533 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.868958 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.765497 20968 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:35.801640 21052 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:35.887879 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.876880 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48377.8us
I20260709 12:22:35.888270 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:35.893669 18096 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:35.898149 18171 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:35.905346 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 787) took 1154 ms. Trace:
W20260709 12:22:35.912882 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.917232 17918 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:35.907718 20961 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.788s	user 0.012s	sys 0.006s
W20260709 12:22:35.932946 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.905023 20965 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:35.939109 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 223019us
I20260709 12:22:35.912299 21054 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:35.948007 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:35.952512 18810 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.953555 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:35.908812 18815 rpcz_store.cc:276] 0709 12:22:34.745600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.774048 (+ 28448us) service_pool.cc:224] Handling call
0709 12:22:34.774757 (+   709us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:34.775529 (+   772us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:34.775568 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:34.776448 (+   880us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.847527 (+ 71079us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:34.878288 (+ 30761us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.878654 (+   366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.878764 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:35.899991 (+1021227us) raft_consensus.cc:1685] finished
0709 12:22:35.900024 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.900248 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":441954,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":11104,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":11090,"thread_start_us":91776,"threads_started":2,"wal-append.queue_time_us":283525}
I20260709 12:22:35.922093 21189 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:35.968187 18091 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:35.965198 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:35.970014 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:35.943910 21107 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:35.882987 21092 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.941591 21171 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:35.987771 17918 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.960304 20968 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:35.994522 18096 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:35.945214 21150 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:22:35.978313 21123 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:35.943260 20961 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:35.992959 21186 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.013104 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:35.993860 20965 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.001210 21171 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:36.030028 21107 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.030932 21107 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:36.032829 21214 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:22:36.034379 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:36.035198 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 81780.9us
I20260709 12:22:36.036824 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:36.010864 17918 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 3 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:36.047415 18091 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:36.048645 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.063158 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 117571us
W20260709 12:22:36.063860 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.071380 17918 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 9 to 10, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 10 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:22:36.074126 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.070160 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:36.075551 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 289844us
I20260709 12:22:36.078079 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:36.041414 20974 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.034850 21220 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:36.101965 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1801) took 1024 ms. Trace:
I20260709 12:22:36.102720 18169 rpcz_store.cc:276] 0709 12:22:35.077546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.467182 (+389636us) service_pool.cc:224] Handling call
0709 12:22:36.101614 (+634432us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:36.101854 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":372880128}
I20260709 12:22:36.099439 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:36.102860 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46576.5us
I20260709 12:22:36.109510 17918 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 4 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:36.116425 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:36.085096 21208 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.283s	user 0.000s	sys 0.002s
W20260709 12:22:36.116228 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1791) took 2466 ms (client timeout 2128 ms). Trace:
I20260709 12:22:36.118422 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:36.074716 21151 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.085295 21188 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:36.141176 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:36.150197 18091 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:36.143337 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:36.126461 18171 rpcz_store.cc:269] 0709 12:22:33.649722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.311637 (+661915us) service_pool.cc:224] Handling call
0709 12:22:36.115935 (+1804298us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54001b5654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:36.116131 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3626572160}
I20260709 12:22:36.087839 20965 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.160557 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1797) took 1084 ms. Trace:
I20260709 12:22:36.164350 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:36.115320 21054 raft_consensus.cc:2764] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:36.165486 18096 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:36.173472 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:36.180231 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:36.121115 21186 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:36.181915 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2652) took 1095 ms. Trace:
I20260709 12:22:36.182886 18090 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:36.156876 21167 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:36.171299 18165 rpcz_store.cc:276] 0709 12:22:35.076226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.081250 (+  5024us) service_pool.cc:224] Handling call
0709 12:22:35.081657 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.084539 (+  2882us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:35.348549 (+264010us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.080167 (+731618us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.103541 (+ 23374us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.159210 (+ 55669us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.159461 (+   251us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:36.159617 (+   156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.159868 (+   251us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.160292 (+   424us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22792,"prepare.queue_time_us":826381,"prepare.run_cpu_time_us":192,"prepare.run_wall_time_us":204,"raft.queue_time_us":55043,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":19634,"thread_start_us":953816,"threads_started":4,"wal-append.queue_time_us":526262}
I20260709 12:22:36.185555 18094 rpcz_store.cc:276] 0709 12:22:35.086427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.102108 (+ 15681us) service_pool.cc:224] Handling call
0709 12:22:35.114122 (+ 12014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.484908 (+370786us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.484992 (+    84us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.510911 (+ 25919us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.987694 (+476783us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:35.987761 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.987910 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:36.181646 (+193736us) raft_consensus.cc:1685] finished
0709 12:22:36.181664 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.181796 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":265504,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":17315,"thread_start_us":698562,"threads_started":2,"wal-append.queue_time_us":478817}
I20260709 12:22:36.191787 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:36.208093 20897 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:36.209375 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:36.220276 21052 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:36.221951 21217 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.257s	user 0.000s	sys 0.001s
I20260709 12:22:36.222326 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1807) took 1005 ms. Trace:
I20260709 12:22:36.222720 21138 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.223930 21054 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:36.190690 21151 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:36.226135 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 125845us
I20260709 12:22:36.183208 20965 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.185410 21186 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:36.230911 21232 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:36.303195 21232 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:36.226191 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 562674us
I20260709 12:22:36.304822 20965 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.242698 21107 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.243003 20961 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.229645 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:36.237588 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2650) took 1167 ms. Trace:
W20260709 12:22:36.248730 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1795) took 1173 ms (client timeout 1537 ms). Trace:
W20260709 12:22:36.249776 21177 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.779s	user 0.011s	sys 0.007s
I20260709 12:22:36.256417 21071 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.257895 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:36.259372 21229 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:22:36.269409 18172 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:36.274092 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84030.8us
I20260709 12:22:36.273223 18170 rpcz_store.cc:276] 0709 12:22:35.216847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.496991 (+280144us) service_pool.cc:224] Handling call
0709 12:22:35.517201 (+ 20210us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.551339 (+ 34138us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.551375 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.552081 (+   706us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.990544 (+438463us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:35.990576 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.990632 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:36.221926 (+231294us) raft_consensus.cc:1685] finished
0709 12:22:36.221959 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.222224 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1695,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":7656,"spinlock_wait_cycles":874240,"thread_start_us":413059,"threads_started":1,"wal-append.queue_time_us":615195}
W20260709 12:22:36.294771 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150507us
I20260709 12:22:36.301069 18810 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:36.294615 18091 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
W20260709 12:22:36.307171 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.892s (SENT)
I20260709 12:22:36.297570 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:36.324088 18090 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:36.235821 21042 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:36.311509 18168 rpcz_store.cc:269] 0709 12:22:35.075576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.077945 (+  2369us) service_pool.cc:224] Handling call
0709 12:22:36.248422 (+1170477us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54003e2bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:36.248619 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1043108224}
I20260709 12:22:36.311193 18097 rpcz_store.cc:276] 0709 12:22:35.070108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.070870 (+   762us) service_pool.cc:224] Handling call
0709 12:22:35.071319 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.071705 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:35.732361 (+660656us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.176456 (+444095us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.236830 (+ 60374us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b1800288e80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:22:36.237066 (+   236us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.237145 (+    79us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:36.237220 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.237322 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.237500 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80094,"prepare.queue_time_us":32347,"raft.queue_time_us":68400,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":13056,"spinlock_wait_cycles":13462784,"thread_start_us":378450,"threads_started":1}
I20260709 12:22:36.318558 18093 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:22:36.320525 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91023.6us
I20260709 12:22:36.277158 21054 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.320353 18807 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:36.338253 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:36.338717 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.299647 21040 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:36.343947 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:36.307644 21186 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:36.325055 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:36.356487 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.311194 21167 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:36.367779 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 191718us
I20260709 12:22:36.363358 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:36.369889 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:36.370373 18019 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:36.298231 21097 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:36.373754 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:36.339272 21151 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.330862 21042 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:36.394984 18024 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:36.350279 21054 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:36.428876 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48901.4us
I20260709 12:22:36.401384 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.361992 21232 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.400028 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.367120 20897 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.329348 21177 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:36.463896 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 152893us
I20260709 12:22:36.459152 18098 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.467005 18021 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:36.399945 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:36.367714 21165 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.366112 21029 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:36.429304 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78265.4us
I20260709 12:22:36.402051 21042 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.517484 18091 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 4
W20260709 12:22:36.466078 21143 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.653s	user 0.018s	sys 0.000s
I20260709 12:22:36.395349 21107 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:36.515753 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2656) took 1001 ms. Trace:
I20260709 12:22:36.533618 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.440474 21052 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:36.395110 21040 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:36.360636 21171 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:36.581362 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 101387us
I20260709 12:22:36.379410 21035 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:36.473470 20996 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.653s	user 0.016s	sys 0.000s
I20260709 12:22:36.428598 21233 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.395499 20961 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.428215 21242 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.150s
I20260709 12:22:36.575009 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:36.605185 18807 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 12:22:36.575754 18023 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:36.575866 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:36.576741 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:36.577057 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 269922us
I20260709 12:22:36.462411 21235 raft_consensus.cc:2764] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20260709 12:22:36.581583 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.540969 18018 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:22:36.583771 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:36.585173 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:36.441382 21071 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.586131 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:36.590674 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.611316 18024 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:36.611687 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:36.593084 18019 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:36.596500 18099 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:36.380586 21167 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:36.476563 20960 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.704s	user 0.016s	sys 0.002s
I20260709 12:22:36.609351 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:36.615710 18095 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:36.571640 18092 rpcz_store.cc:276] 0709 12:22:35.188878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.256129 (+ 67251us) service_pool.cc:224] Handling call
0709 12:22:35.256729 (+   600us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.257146 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:35.599455 (+342309us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.181321 (+581866us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.181393 (+    72us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:36.190045 (+  8652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.190160 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.190443 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":104527,"prepare.queue_time_us":560822,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":64,"raft.queue_time_us":75437,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":11962,"thread_start_us":756903,"threads_started":2,"wal-append.queue_time_us":605882}
I20260709 12:22:36.438256 21151 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.610441 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:36.618805 21143 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:36.619469 21143 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:36.620007 21143 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:36.586453 21165 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.517253 20897 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.527976 21123 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 12:22:36.586691 21107 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:36.644624 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68852.3us
I20260709 12:22:36.656929 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:36.659209 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.660264 18021 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:36.670805 18023 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:36.671265 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:36.671981 18811 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:36.674242 21177 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.621543 20960 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.626184 21233 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.680747 21246 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:36.681428 21246 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.681808 21246 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.664740 18020 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:36.684019 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110221us
I20260709 12:22:36.670855 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.600260 20996 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.610651 21171 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.611287 20961 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.701702 18810 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:36.710428 18807 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:36.711465 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 110458us
I20260709 12:22:36.712960 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:36.717130 18021 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:36.621958 21071 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:36.721925 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 103370us
I20260709 12:22:36.724789 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:36.726040 18810 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:36.670931 20842 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:36.739296 18023 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:36.683439 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:36.739471 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.527s (SENT)
I20260709 12:22:36.746115 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:36.723944 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:36.650069 21165 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.689023 21247 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.751893 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 791) took 1758 ms. Trace:
I20260709 12:22:36.723346 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:36.735752 20960 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.778316 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:36.751122 21225 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.747042 21052 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:36.763077 21097 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.674316 21107 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.693239 21040 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.758602 18809 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:36.796581 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 150852us
W20260709 12:22:36.805498 21188 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.639s	user 0.012s	sys 0.002s
I20260709 12:22:36.806357 21233 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.806146 21097 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.827587 18811 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:36.811419 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:36.844969 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99250.7us
I20260709 12:22:36.855790 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:36.790947 21052 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.848840 21188 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:36.786578 21123 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:36.788417 21177 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:22:36.868186 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 115497us
I20260709 12:22:36.811961 20842 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:36.861479 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:36.876572 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.878302 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.784003 21247 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:36.890610 21042 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.855985 21167 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.895011 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:36.895860 21167 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:36.896373 21167 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.896791 21167 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:36.896703 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:36.866165 21123 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:36.898952 21123 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.900611 21123 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:36.906090 21243 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:36.906703 21243 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:36.907148 21243 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:36.907423 21052 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.908128 21188 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:36.908725 21188 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.911003 21188 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.854951 21138 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.929314 18807 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:36.893491 21042 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:36.946368 18098 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:36.951812 18097 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:36.881482 21247 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:36.962811 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:36.963771 18015 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:36.965086 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:36.966082 20967 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:36.970156 21119 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.978971 21042 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:36.875620 18021 raft_consensus.cc:2376] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5. Re-sending same reply.
I20260709 12:22:36.987058 20961 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:36.952296 20842 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:36.989499 20842 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.992707 21143 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:36.951889 21138 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.040726 21138 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:37.047662 21143 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.000989 21247 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:37.054677 21186 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.000s
I20260709 12:22:37.055186 21186 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.006696 20996 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.007570 21242 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.056598 21242 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.009001 21171 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.009840 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 792) took 1959 ms. Trace:
I20260709 12:22:37.009716 21177 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.854595 21225 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.012337 21151 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.026412 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.027395 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:37.028280 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:37.030383 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:37.037147 21054 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.005389 21165 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.031827 18021 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:37.033408 18022 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:37.035311 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2674) took 1001 ms. Trace:
I20260709 12:22:37.036590 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:37.033301 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.000034 21097 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.054909 21247 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:37.052960 20968 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.052s	user 0.010s	sys 0.004s
I20260709 12:22:37.064670 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:37.206763 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:37.219877 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.220008 18098 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:37.076934 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.079422 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.224184 18090 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:37.218954 18024 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:37.143899 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 404436us
W20260709 12:22:37.207187 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.034s	sys 0.025s
W20260709 12:22:37.184218 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.226166 18097 rpcz_store.cc:276] 0709 12:22:36.033470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.434243 (+400773us) service_pool.cc:224] Handling call
0709 12:22:36.951532 (+517289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.952445 (+   913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.034969 (+ 82524us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b540011dd50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:37.035205 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1095782784}
I20260709 12:22:36.763863 18814 rpcz_store.cc:276] 0709 12:22:34.993423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.033083 (+ 39660us) service_pool.cc:224] Handling call
0709 12:22:35.055227 (+ 22144us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.109938 (+ 54711us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:35.566627 (+456689us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.621283 (+1054656us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b1800075880. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287
0709 12:22:36.722981 (+101698us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.724161 (+  1180us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.737843 (+ 13682us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.737904 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:36.737962 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.738055 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.751783 (+ 13728us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12348,"prepare.queue_time_us":173088,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":107497,"raft.run_cpu_time_us":571,"raft.run_wall_time_us":71745,"spinlock_wait_cycles":25510784,"thread_start_us":962965,"threads_started":4,"wal-append.queue_time_us":524804}
I20260709 12:22:37.219574 18807 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:37.075122 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.190126 18020 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:37.072667 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:37.182713 18091 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.607s	user 0.015s	sys 0.000s
I20260709 12:22:37.230038 18094 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.230293 18091 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:37.225054 18018 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.227288 18016 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:37.076396 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:37.225459 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.534s (SENT)
W20260709 12:22:37.073308 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2653) took 1986 ms (client timeout 2243 ms). Trace:
I20260709 12:22:37.083128 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:37.226188 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.035s	sys 0.025s
I20260709 12:22:37.243083 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.259217 18093 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:37.259866 18091 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 12:22:37.272593 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 196261us
I20260709 12:22:37.273885 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:37.275931 18099 rpcz_store.cc:269] 0709 12:22:35.087080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.088098 (+  1018us) service_pool.cc:224] Handling call
0709 12:22:37.072959 (+1984861us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54000523d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:37.073181 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3374023552}
I20260709 12:22:37.275619 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:37.278824 18023 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:22:37.279090 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:37.283730 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.014s	sys 0.010s
W20260709 12:22:37.284226 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.014s	sys 0.010s
I20260709 12:22:37.276595 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.295317 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.296540 18097 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:37.297808 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:37.301565 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.207283 21254 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.078593 21234 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.078734 21092 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:37.067901 20897 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.057711 21186 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.374552 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.280725 21143 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:37.065105 21167 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.386623 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.387699 18097 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:37.386677 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.392279 18165 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:37.362185 20997 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:37.220840 18813 rpcz_store.cc:276] 0709 12:22:35.050432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.050706 (+   274us) service_pool.cc:224] Handling call
0709 12:22:35.098429 (+ 47723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.098735 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:35.606173 (+507438us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.009215 (+1403042us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b18003deac0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287
0709 12:22:37.009340 (+   125us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.009402 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:37.009470 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.009561 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.009743 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13819,"prepare.queue_time_us":191119,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":175534,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":8850,"spinlock_wait_cycles":41373952,"thread_start_us":815376,"threads_started":3,"wal-append.queue_time_us":492047}
I20260709 12:22:37.328027 18019 raft_consensus.cc:2376] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:37.063190 21071 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.331712 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.102632 20996 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:37.339044 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.411535 18164 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:37.309374 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:37.310945 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:37.278535 21157 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.008s	sys 0.020s
I20260709 12:22:37.321135 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:37.345646 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.338284 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.349934 18168 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.260720 21168 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:37.368880 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.375985 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:37.423962 18173 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:37.424222 18172 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:37.085124 21241 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.246s
I20260709 12:22:37.368533 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.375471 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:37.426520 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.379000 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 153564us
W20260709 12:22:37.430217 18778 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.636s (SENT)
I20260709 12:22:37.430766 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:37.317857 21177 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:37.463169 21177 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:37.464303 21177 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.467237 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.468225 18019 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:37.469372 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.371706 21268 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:37.421947 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 111103us
I20260709 12:22:37.363126 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.422642 18093 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:37.378479 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:37.425055 21256 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.434036 21261 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:37.434684 21054 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:37.435019 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.426002 18171 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:37.435894 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:37.435716 20996 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.436513 21168 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.438553 18164 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:22:37.424431 21157 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.009s	sys 0.021s
I20260709 12:22:37.441114 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:37.446234 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.401679 21143 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.447615 20897 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.450824 21040 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:37.451532 20961 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.451982 20967 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.452415 20974 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.453691 21225 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.457034 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:37.471753 21188 raft_consensus.cc:993] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:22:37.407109 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.359848 21165 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 12:22:37.484845 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1891) took 1297 ms. Trace:
I20260709 12:22:37.410481 21242 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:37.489918 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.070s	sys 0.088s
I20260709 12:22:37.429008 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:37.322824 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 426205us
W20260709 12:22:37.495028 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.366091 21234 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.359244 21254 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.677284 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:37.690990 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.691509 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.076s	sys 0.097s
I20260709 12:22:37.648645 18173 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:37.692517 18020 raft_consensus.cc:2376] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
W20260709 12:22:37.523016 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 201975us
I20260709 12:22:37.591812 18018 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:37.582698 18019 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:37.695910 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:37.695009 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 266086us
I20260709 12:22:37.472981 21151 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:37.550541 18017 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:37.697144 18016 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:37.557333 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.700069 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:37.561172 21246 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.829s	user 0.014s	sys 0.002s
I20260709 12:22:37.570168 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.776684 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:37.714015 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.787959 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.704108 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.794759 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:37.774605 18019 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49522
I20260709 12:22:37.746093 18098 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:37.693990 18095 rpcz_store.cc:276] 0709 12:22:36.187236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.640947 (+453711us) service_pool.cc:224] Handling call
0709 12:22:36.663120 (+ 22173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.663597 (+   477us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:37.162782 (+499185us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.484110 (+321328us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.484223 (+   113us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:37.484314 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.484470 (+   156us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.484722 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":380107,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":13085,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":61966,"thread_start_us":685737,"threads_started":2,"wal-append.queue_time_us":482264}
I20260709 12:22:37.732546 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:37.779682 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 338683us
I20260709 12:22:37.794795 18168 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:37.574326 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144145us
I20260709 12:22:37.485708 21107 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:22:37.799119 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.643s (SENT)
I20260709 12:22:37.701650 18021 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:37.773984 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:37.808028 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:37.696118 21243 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.788s	user 0.012s	sys 0.006s
I20260709 12:22:37.813695 18099 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:37.787693 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:37.759418 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:37.810564 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 110562us
I20260709 12:22:37.529538 21268 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:37.796729 18166 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:37.681519 21247 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.813743 18173 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:37.802433 18093 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:37.583029 21071 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:37.701598 20897 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.819021 21273 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:37.835373 21263 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:37.833616 18167 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:37.700873 21171 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.701188 21151 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.837199 18171 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
W20260709 12:22:37.838281 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.806s (SENT)
I20260709 12:22:37.702301 21241 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:37.702246 21138 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:37.854339 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:37.551920 21029 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:37.820609 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88173.2us
I20260709 12:22:37.838744 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:37.840384 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:37.843937 21263 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.736167 21242 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.101s
I20260709 12:22:37.834496 18090 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:37.758845 21265 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:37.844158 21071 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.853240 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:37.736562 21036 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.106s
W20260709 12:22:37.857420 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110735us
I20260709 12:22:37.839885 21054 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:37.893786 21071 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:37.842706 18137 connection.cc:256] RPC call timeout handler was delayed by -1.08709989900035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2135s for a response
I20260709 12:22:37.851068 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:37.894817 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.757469 21040 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.700950 21256 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.860947 21168 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.761726 21225 raft_consensus.cc:1081] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Signalling peer ae9641d5c3184572a102d51a75f670ee to start an election
I20260709 12:22:37.701704 20967 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.836242 21268 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.889041 21165 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:37.861213 21254 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:37.865126 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:37.712284 20974 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:37.701155 20961 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.771000 21167 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.801244 21246 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.879326 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:37.883368 20897 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.883670 20968 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:37.883481 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:37.853050 21151 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.834780 21243 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.944808 18096 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:37.843448 21267 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.267s	user 0.000s	sys 0.002s
I20260709 12:22:37.859755 21138 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:37.895860 18137 connection.cc:256] RPC call timeout handler was delayed by -1.0444632889998502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260709 12:22:37.897440 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.887430 21263 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:37.954313 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70940us
I20260709 12:22:37.910020 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:37.913870 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75594.8us
W20260709 12:22:37.864084 18094 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.005s
I20260709 12:22:37.878738 21029 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:37.919350 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2630) took 1401 ms (client timeout 1835 ms). Trace:
W20260709 12:22:37.958230 18017 rpcz_store.cc:269] 0709 12:22:36.517894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.518058 (+   164us) service_pool.cc:224] Handling call
0709 12:22:37.918997 (+1400939us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54001d0e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:37.919235 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1575602560}
I20260709 12:22:37.928180 21165 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.853654 21171 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:37.930231 18024 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:37.890133 21264 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:37.937969 21270 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:22:37.911777 21123 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:22:37.939625 21246 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.940670 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.941586 18015 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:37.941926 20968 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.904506 21092 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:37.957186 18094 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:37.959169 18022 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
W20260709 12:22:37.902227 21274 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260709 12:22:37.959968 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.960253 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 231932us
I20260709 12:22:37.960659 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:37.961040 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:37.967550 18093 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:37.955778 21138 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.974242 18099 pending_rounds.cc:85] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:37.980368 18017 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:37.981135 18099 pending_rounds.cc:107] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:22:37.981137 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:37.982618 18095 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:37.986791 18019 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Received request to transfer leadership
I20260709 12:22:37.961524 21029 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:37.981452 18099 raft_consensus.cc:2889] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:37.961390 20960 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:37.990206 18778 connection.cc:256] RPC call timeout handler was delayed by -1.030803855999693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
I20260709 12:22:37.966125 21165 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:37.963213 20996 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:38.002032 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:37.966768 21251 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.977406 21071 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:37.998988 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.001683 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:38.021991 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:37.994280 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.030s (SENT) [suppressed 29 similar messages]
I20260709 12:22:37.968288 20961 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.005240 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:38.023839 18023 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:37.972427 20968 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:37.998569 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 119292us
I20260709 12:22:37.964350 20897 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:22:38.003281 20842 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.011811 21143 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.013629 21243 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:37.981580 21040 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:38.042284 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.383s (SENT)
I20260709 12:22:38.000584 20960 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:38.044693 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:38.048784 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 138960us
I20260709 12:22:38.039269 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:38.003526 21254 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.031688 21246 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.072964 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:38.062644 18813 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:38.058677 21157 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:38.087615 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63347us
W20260709 12:22:38.097092 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71320.7us
I20260709 12:22:38.098111 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.075591 20960 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:38.087209 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.117955 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:38.118676 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:38.067413 21254 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:38.123443 18815 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:38.078907 21261 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:38.124532 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.132090 18811 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:38.125941 18814 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:38.127664 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:38.131050 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105582us
I20260709 12:22:38.144035 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.145098 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.143121 18807 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:38.140961 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 152038us
I20260709 12:22:38.106114 21242 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.099788 21256 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.110119 21036 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:38.077762 20842 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
W20260709 12:22:38.159011 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71933.3us
I20260709 12:22:38.131210 21254 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.084986 21246 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.130954 21243 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:38.162607 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 89771.8us
I20260709 12:22:38.140164 21123 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.163852 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:38.168982 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.170440 18810 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:38.155176 21242 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.152557 21261 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.218705 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:38.168497 21271 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.505s	user 0.008s	sys 0.000s
W20260709 12:22:38.166898 21255 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.026s	sys 0.032s
I20260709 12:22:38.239529 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:38.240980 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.011s (SENT)
W20260709 12:22:38.200302 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.241744 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.172385 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.157050 21234 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.246925 18811 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:38.164530 21243 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:38.249004 21261 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.252065 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.218598 18809 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:38.219761 21242 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.170226 21157 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.168291 21036 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.253705 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.161803 21107 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.081s
W20260709 12:22:38.254705 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84980.3us
I20260709 12:22:38.161921 21092 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 12:22:38.296277 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:38.251153 21255 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.027s	sys 0.033s
I20260709 12:22:38.274274 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:38.260609 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69652.3us
W20260709 12:22:38.287988 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 124222us
I20260709 12:22:38.275631 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:38.298765 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144434us
I20260709 12:22:38.303066 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.276233 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.289727 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:38.278389 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.241s (SENT)
I20260709 12:22:38.292874 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.294798 21040 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.298091 18811 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:38.281275 21234 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:38.260766 21143 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:38.299610 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2683) took 1085 ms. Trace:
I20260709 12:22:38.256810 21243 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.305598 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:38.306048 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.293030 20974 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.309486 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.311476 18096 rpcz_store.cc:276] 0709 12:22:37.214583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.229220 (+ 14637us) service_pool.cc:224] Handling call
0709 12:22:37.271985 (+ 42765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.945224 (+673239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.299316 (+354092us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:38.299538 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546779904}
I20260709 12:22:38.314477 18164 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:38.308799 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:38.315521 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.316033 18166 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:38.275125 21280 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:22:38.316586 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.320315 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.317911 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.320952 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.316154 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:38.259416 21123 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.051s
W20260709 12:22:38.330109 21279 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.059s	user 0.002s	sys 0.000s
I20260709 12:22:38.323972 21288 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:22:38.301647 18808 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:38.324949 21247 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:38.331240 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52899.9us
I20260709 12:22:38.295118 21036 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.358582 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.326620 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.335206 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:38.323794 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:38.325538 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.300176 21269 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:38.273715 21281 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:22:38.321333 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:22:38.347466 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.343886 20974 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.350035 20997 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:38.351274 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.351931 18173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:50512
I20260709 12:22:38.346316 21247 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.342381 21290 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:38.354471 21277 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.271s	user 0.002s	sys 0.000s
W20260709 12:22:38.371747 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46333.2us
I20260709 12:22:38.379539 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:38.337560 20842 raft_consensus.cc:2764] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:22:38.381013 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:38.382514 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64029.4us
I20260709 12:22:38.388433 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:38.389246 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84194.3us
I20260709 12:22:38.392298 21225 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.373450 21256 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.393867 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.373201 20974 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:38.375411 21071 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.395412 18171 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:22:38.396580 21288 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:38.408385 21288 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.417228 21288 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.398132 21247 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.419298 21247 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:38.400573 21289 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
W20260709 12:22:38.423278 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99613.5us
W20260709 12:22:38.426550 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.866s (SENT)
I20260709 12:22:38.382884 21269 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.451092 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:38.451360 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71967.5us
I20260709 12:22:38.448791 21298 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:38.456833 21029 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.461014 18095 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:38.465060 18169 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:38.451059 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62759.7us
I20260709 12:22:38.452355 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.469058 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.470826 20960 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.455749 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.456604 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.483326 18171 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:38.406589 21071 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:38.491655 21071 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.493093 21246 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.494448 21040 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:38.494625 20960 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.494982 21040 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:38.411378 21241 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.004s
I20260709 12:22:38.491165 21251 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.386446 21290 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:38.495730 21241 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:38.462874 21284 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:22:38.472467 21269 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.496580 21290 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.498399 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49522 (request call id 89) took 1101 ms. Trace:
I20260709 12:22:38.498836 18019 rpcz_store.cc:276] 0709 12:22:37.396723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.743995 (+347272us) service_pool.cc:224] Handling call
0709 12:22:38.497273 (+753278us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002ef2a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c16634520 00007f6c166b1652 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:38.498125 (+   852us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:38.498281 (+   156us) inbound_call.cc:177] Queueing failure response
Metrics: {"spinlock_wait_cycles":676961920}
I20260709 12:22:38.407012 21294 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:38.479931 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:38.507352 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 119996us
I20260709 12:22:38.449501 21299 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:38.508687 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:38.453887 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.405112 20974 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.472427 21242 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.500552 20961 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:38.514108 20961 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:38.507416 21294 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.514837 21294 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.485996 18022 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:38.516281 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2686) took 1282 ms. Trace:
I20260709 12:22:38.488514 20997 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.495239 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:38.475077 21273 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.622s	user 0.014s	sys 0.000s
W20260709 12:22:38.476678 21268 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.542s	user 0.014s	sys 0.000s
I20260709 12:22:38.503800 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.500612 18173 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 12:22:38.505125 21263 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.529s	user 0.007s	sys 0.007s
W20260709 12:22:38.477155 21151 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.534s	user 0.005s	sys 0.005s
I20260709 12:22:38.486105 21256 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.527232 21256 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:38.526949 21263 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.527880 21256 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.530721 18166 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:38.532048 21263 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.527015 21151 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.538026 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:38.539059 18808 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.544672 21246 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.546316 21256 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.546978 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:38.548671 21256 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.549362 21256 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.549739 21256 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.549855 21151 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.516695 18091 rpcz_store.cc:276] 0709 12:22:37.233705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.272553 (+ 38848us) service_pool.cc:224] Handling call
0709 12:22:37.345520 (+ 72967us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.592356 (+246836us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.592389 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:37.693054 (+100665us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.955177 (+262123us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:37.955221 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.955306 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:38.353696 (+398390us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:38.503175 (+149479us) raft_consensus.cc:1685] finished
0709 12:22:38.503199 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.516185 (+ 12986us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24395,"raft.queue_time_us":72261,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":21318,"spinlock_wait_cycles":23067520,"thread_start_us":377037,"threads_started":2,"wal-append.queue_time_us":262766}
I20260709 12:22:38.526682 18173 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:38.550161 21256 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.510279 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:38.558518 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48288.5us
I20260709 12:22:38.559759 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.560768 21296 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:38.561259 21296 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.561595 21296 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.562124 21246 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.566772 18173 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:38.569357 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:38.413324 20968 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.570955 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:38.573187 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47618.8us
I20260709 12:22:38.573553 18165 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:38.494488 21143 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.572635 21157 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.515151 20961 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:38.516891 21299 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:38.604096 18138 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.604s (SENT)
I20260709 12:22:38.515786 21242 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.513666 21225 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:38.582034 21092 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:38.582438 21254 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.516793 20960 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:38.571225 21273 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:38.575302 21268 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:38.599548 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 89976.9us
I20260709 12:22:38.593743 21246 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:38.577030 18016 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:38.576164 21040 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.605551 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.609450 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:38.574693 21177 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:38.610548 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:38.615015 18095 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:38.616887 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:38.619144 18094 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:38.622876 18099 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:38.623023 18092 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:38.623842 21268 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.624728 21040 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.624599 21071 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.617146 21092 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.627269 21071 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.628712 21092 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.630504 18098 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:38.632587 20960 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:38.632822 21300 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 12:22:38.634768 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:38.608044 21123 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.635671 21123 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.636826 21177 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:38.637996 21177 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.618909 21263 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 12:22:38.642932 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:38.644173 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.645180 21143 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:38.657011 18094 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:38.657744 18172 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:38.606696 21299 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.607350 21251 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:22:38.659890 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55853.4us
I20260709 12:22:38.660321 18093 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:38.664474 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:38.665529 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:38.666109 21285 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:38.664893 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:38.666239 18017 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:38.668514 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.633314 20960 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.617507 21246 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:38.672758 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101869us
I20260709 12:22:38.673357 18019 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:38.649794 21273 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.682236 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.683347 21123 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.658684 21269 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:38.685144 21251 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.686151 20960 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.687672 21251 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.681744 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.674189 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:38.675731 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 65296.5us
I20260709 12:22:38.649595 21143 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.677901 21157 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:38.679400 21285 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.695403 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.681203 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:38.696163 18017 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.696583 21285 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.686847 21167 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:38.697849 21167 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:38.698786 21167 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.700044 21167 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:38.666126 21171 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.679412 21268 raft_consensus.cc:2764] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:22:38.659031 21300 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.689749 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:38.724630 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:38.672803 21071 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:22:38.689904 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:38.693499 21143 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.691026 18808 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:38.680295 20967 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.681958 21029 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:38.736800 21283 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.669451 20968 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:38.675662 21054 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:38.737694 21283 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.676651 21246 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.738777 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:38.739468 18168 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:38.739845 21272 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.306s	user 0.014s	sys 0.000s
I20260709 12:22:38.740702 21283 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:38.692977 20960 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:38.727713 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49923.6us
I20260709 12:22:38.741194 21263 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:38.741596 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2653) took 1693 ms. Trace:
I20260709 12:22:38.742149 18024 rpcz_store.cc:276] 0709 12:22:37.048316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.456632 (+408316us) service_pool.cc:224] Handling call
0709 12:22:38.741358 (+1284726us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54001cf554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:38.741522 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551042816}
I20260709 12:22:38.743629 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.743853 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:38.744377 18024 raft_consensus.cc:2376] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7. Re-sending same reply.
I20260709 12:22:38.745088 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:38.745556 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:38.741410 20960 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.746568 20960 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:38.726879 21288 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:38.658803 21290 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.693015 21273 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:38.720108 21165 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.688s	user 0.012s	sys 0.005s
I20260709 12:22:38.735368 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.693588 21251 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:22:38.693073 21242 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
W20260709 12:22:38.708969 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.738184 21054 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.703248 21241 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 12:22:38.718456 18017 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:22:38.721982 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58131.1us
I20260709 12:22:38.766912 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:38.727921 18019 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:38.745455 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.735646 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:38.758836 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.762413 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.767881 18024 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.772766 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.760653 21151 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.773761 18019 raft_consensus.cc:2376] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:38.760417 21165 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.782825 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:38.783046 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:38.785027 18167 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:38.786160 21165 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:38.786953 21165 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.787386 21165 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.787739 21165 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.794066 21296 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:38.797993 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.804518 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.805141 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:38.751322 21288 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.728963 21261 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:22:38.814358 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:38.776098 21138 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.776s	user 0.014s	sys 0.000s
I20260709 12:22:38.815378 18173 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:38.820140 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:38.821107 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.822082 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:38.865149 18017 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:38.877878 18810 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:38.776881 21269 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:38.885733 18169 raft_consensus.cc:606] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:38.778851 21268 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:22:38.887408 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 199389us
I20260709 12:22:38.864089 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:38.889359 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91934.4us
W20260709 12:22:38.889365 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78595.9us
I20260709 12:22:38.891708 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:38.747596 21294 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.893929 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:38.816129 21288 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:38.895476 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:38.896134 18093 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:38.923797 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:38.730444 21300 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.925585 18810 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:38.890332 21269 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:38.810490 21029 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
W20260709 12:22:38.926965 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 181492us
I20260709 12:22:38.896289 21296 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.900223 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.935952 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:38.938388 18097 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.938570 18092 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:38.906405 20897 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.905233 21268 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:38.942317 18094 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2.
I20260709 12:22:38.883200 21138 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:38.944219 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:38.926277 21123 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.945487 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:38.948319 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:38.956384 18096 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
I20260709 12:22:38.917145 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:38.949529 18094 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:38.929073 21254 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:38.958701 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:38.943030 21247 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.012s
I20260709 12:22:38.997406 18171 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:22:39.003520 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.054597 18092 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.962407 21029 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:38.917124 21054 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.944264 21242 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
W20260709 12:22:39.011018 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 119434us
I20260709 12:22:39.070657 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:39.009366 21269 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:38.957379 21263 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:38.995186 18809 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:22:39.040285 21317 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:22:39.040993 21306 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260709 12:22:38.939985 21283 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:39.053637 18095 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 12 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:39.062559 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.063060 18168 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:38.949682 20897 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.097199 21143 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.123625 18090 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
I20260709 12:22:38.954236 20974 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.071156 18021 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:38.969921 21123 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.079993 21269 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:39.080559 21294 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:39.060911 21029 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:39.086141 21262 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:39.086639 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 632838us
I20260709 12:22:39.179620 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.180696 18095 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:39.087745 21316 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:22:39.077886 21054 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:39.092943 21256 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.000s
W20260709 12:22:39.195235 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.097405 21263 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:39.098965 21300 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:39.060786 21268 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.102196 21036 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:39.103477 21138 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:39.110508 21165 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.085484 18024 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:39.118016 18097 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:22:39.148904 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:39.207859 21285 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.007s	sys 0.013s
W20260709 12:22:39.124087 21243 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.804s	user 0.013s	sys 0.000s
I20260709 12:22:39.168710 18094 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:39.222886 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.223989 18094 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:39.224954 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:39.181957 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.182912 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.188351 18023 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:39.180564 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:39.194285 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2644) took 2155 ms (client timeout 2236 ms). Trace:
I20260709 12:22:39.096622 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:39.127045 21177 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:39.198527 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.199764 18809 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:39.230980 18098 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:39.232342 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.230224 18094 raft_consensus.cc:2368] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
W20260709 12:22:39.200506 18065 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45032 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20260709 12:22:39.202553 21123 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:39.203548 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.240804 18095 raft_consensus.cc:2368] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
I20260709 12:22:39.149363 21283 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:39.204499 18091 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.012s	sys 0.011s
I20260709 12:22:39.219707 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.154336 18092 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:39.245786 18097 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260709 12:22:39.228820 18015 rpcz_store.cc:269] 0709 12:22:37.024375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.025960 (+  1585us) service_pool.cc:224] Handling call
0709 12:22:39.179475 (+2153515us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54002ffb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:39.179655 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1653652736}
W20260709 12:22:39.221392 21285 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.007s	sys 0.013s
I20260709 12:22:39.205307 21256 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:39.242905 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 146375us
I20260709 12:22:39.284017 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:39.243166 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.286840 18098 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:22:39.229035 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48563.7us
I20260709 12:22:39.250195 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.253799 21165 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:39.254499 18091 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.012s	sys 0.012s
I20260709 12:22:39.233878 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.253615 21283 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.254006 20974 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:39.261619 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.230521 21036 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:39.271970 21263 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.240890 21262 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:39.277899 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.287678 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:39.259227 21151 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:39.234314 21288 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.246428 21242 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:39.351177 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.351737 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.352826 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.327764 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.353642 18170 raft_consensus.cc:2368] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:22:39.328078 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.354525 18096 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:39.331233 18022 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:39.332070 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44495.9us
I20260709 12:22:39.356313 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1819) took 1053 ms. Trace:
I20260709 12:22:39.339004 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.356968 18167 rpcz_store.cc:276] 0709 12:22:38.303060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.310056 (+  6996us) service_pool.cc:224] Handling call
0709 12:22:39.355989 (+1045933us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b540007b554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:39.356215 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1065115904}
I20260709 12:22:39.343369 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.290189 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:39.263058 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.345719 18020 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.359720 18095 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:39.359928 18166 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.350184 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.233690 18065 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45032 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 523)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 527)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 531)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 530)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 1916)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 517)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1907)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1910)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 539)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 541)
I20260709 12:22:39.361644 18168 raft_consensus.cc:2368] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6.
I20260709 12:22:39.358940 18099 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:39.230801 21243 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.375072 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.373241 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.371699 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.376619 18099 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:39.377199 18173 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:39.379191 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88798.8us
W20260709 12:22:39.379779 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89676.3us
I20260709 12:22:39.381210 18022 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:39.382406 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:39.312906 21138 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:39.384919 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:39.384930 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.386070 18099 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:39.388027 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.389968 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.333545 21281 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:39.393219 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.393529 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.402359 18095 raft_consensus.cc:2376] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 2. Re-sending same reply.
W20260709 12:22:39.374377 21246 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.635s	user 0.014s	sys 0.000s
I20260709 12:22:39.242084 21177 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.412421 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.397086 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:39.330694 21262 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.346758 21256 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:39.404775 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.406458 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:39.406734 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.241104 21029 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.429329 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.343987 21288 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:39.430408 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172707us
I20260709 12:22:39.406160 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.414956 18813 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
I20260709 12:22:39.384093 21071 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:39.427348 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.427966 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:39.389410 21242 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:39.451030 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.391135 21285 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:39.409857 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.437204 18090 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 3.
W20260709 12:22:39.418417 20960 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.665s	user 0.001s	sys 0.015s
I20260709 12:22:39.455411 18093 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:39.456938 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.413854 21143 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:39.414345 21246 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.323853 21332 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20260709 12:22:39.391638 21138 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:39.443477 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.422205 21262 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:39.444902 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.445560 18015 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.477612 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:39.462637 18064 connection.cc:256] RPC call timeout handler was delayed by -4.9220800219995908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
I20260709 12:22:39.475152 18091 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:39.478725 18093 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 2.
W20260709 12:22:39.478932 18064 connection.cc:256] RPC call timeout handler was delayed by -4.9049800219995632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
W20260709 12:22:39.479566 18064 connection.cc:256] RPC call timeout handler was delayed by -4.7744800219998069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
W20260709 12:22:39.480047 18064 connection.cc:256] RPC call timeout handler was delayed by -4.7483800219997647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
W20260709 12:22:39.480504 18064 connection.cc:256] RPC call timeout handler was delayed by -4.68988002199967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260709 12:22:39.480968 18064 connection.cc:256] RPC call timeout handler was delayed by -4.1921800219997749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24830000000000002s for a response
I20260709 12:22:39.480067 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.484532 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.495095 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.496219 18015 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.497085 18164 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 7. Ops: []
I20260709 12:22:39.498230 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.477337 21138 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.506552 21246 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:39.497159 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.519681 18093 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.464228 21292 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:39.491348 21299 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.813s	user 0.011s	sys 0.004s
W20260709 12:22:39.520545 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43895.1us
I20260709 12:22:39.524258 18024 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:39.453953 21281 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:22:39.545394 21292 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:39.545723 21242 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:39.547122 21292 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.547665 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.550558 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.554051 18096 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:39.554765 18024 raft_consensus.cc:2376] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:39.444342 21036 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:39.458608 20960 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:39.530664 21335 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:22:39.564289 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.564679 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.564291 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.565963 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.566102 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.566758 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.575932 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.579572 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.492061 21285 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:39.641348 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.740095 18167 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:39.740727 18096 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:39.741840 18023 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.682994 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:39.643893 18091 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:39.744261 18024 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:39.712296 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 355621us
I20260709 12:22:39.683004 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:39.745990 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:39.746843 18168 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:39.741411 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:39.755405 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 189418us
I20260709 12:22:39.755821 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:39.757320 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:39.758687 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.758587 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.760730 18169 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:39.761490 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.762236 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:39.763942 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:39.759624 18090 raft_consensus.cc:2376] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6. Re-sending same reply.
I20260709 12:22:39.771909 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.764073 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.775535 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.774536 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.778657 18024 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:39.778681 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.587141 21177 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:39.785986 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:39.788125 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 268392us
I20260709 12:22:39.789376 18169 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:39.790134 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:39.786036 18093 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:39.790380 18024 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:39.757548 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:39.543251 21251 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.232s
I20260709 12:22:39.774943 21315 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:39.794555 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77472.6us
I20260709 12:22:39.796432 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.544617 21299 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:39.799187 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.797506 18090 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:39.805578 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.806532 18096 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:39.806895 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.807597 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:39.800841 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:39.797439 18173 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:39.797411 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.811215 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.811589 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:39.773018 21243 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:39.807808 18090 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:39.784799 20961 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:39.809305 18023 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:39.793392 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 382202us
I20260709 12:22:39.812050 18093 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:39.819202 21298 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:39.825901 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:39.814031 21292 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.845736 18022 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:39.839255 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:39.840677 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2670) took 1178 ms. Trace:
I20260709 12:22:39.831079 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.847429 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:39.852113 18023 raft_consensus.cc:2368] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:39.865111 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:39.851747 18096 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:39.869149 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.869791 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:39.809674 21246 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:39.872720 21246 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.873257 21246 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:39.873951 18024 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.825799 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.878515 20974 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:39.878350 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.879163 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.880097 18093 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:39.882576 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.830885 21256 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:39.887976 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:39.847151 18016 rpcz_store.cc:276] 0709 12:22:38.662495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.220876 (+558381us) service_pool.cc:224] Handling call
0709 12:22:39.236284 (+ 15408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:39.236787 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.236825 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:39.288166 (+ 51341us) log.cc:844] Serialized 52 byte log entry
0709 12:22:39.508920 (+220754us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:39.550157 (+ 41237us) log.cc:844] Serialized 14 byte log entry
0709 12:22:39.550434 (+   277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.550514 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:39.840283 (+289769us) raft_consensus.cc:1685] finished
0709 12:22:39.840322 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.840564 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30395,"prepare.queue_time_us":41551,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"raft.queue_time_us":518,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":9178,"thread_start_us":170574,"threads_started":1,"wal-append.queue_time_us":436270}
I20260709 12:22:39.879316 18023 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:39.887355 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.892655 18093 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:39.849009 21243 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:39.837375 21298 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:39.891430 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.896126 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:39.921470 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.921861 18098 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:39.921880 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.849855 21342 raft_consensus.cc:2764] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:22:39.826498 21343 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:39.896145 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.930590 18096 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:39.923524 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:39.931623 18023 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:39.879510 21300 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:39.933121 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 166484us
I20260709 12:22:39.935070 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.941612 18098 raft_consensus.cc:2376] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7. Re-sending same reply.
W20260709 12:22:39.940385 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60021.5us
I20260709 12:22:39.942240 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:39.842427 21292 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:39.963857 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.965271 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:39.971354 18093 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 12 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:39.965306 18023 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:39.984759 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:39.924983 21311 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.878s	user 0.005s	sys 0.002s
I20260709 12:22:39.993309 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:39.997234 18024 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:39.920008 21298 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.000000 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.005748 18024 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 6. Ops: [4.3-4.3]
I20260709 12:22:40.009447 18023 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:39.939188 21177 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:40.014688 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:40.014779 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 257534us
I20260709 12:22:40.015945 18094 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:40.018332 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.022029 18024 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:40.026590 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:39.995141 21242 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:40.007949 21298 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.898990 21285 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.004984 21342 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.044234 18023 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [3.3-6.5]
I20260709 12:22:40.010037 21341 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:40.055913 18098 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.9-6.9]
I20260709 12:22:40.058750 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 515) took 1017 ms. Trace:
W20260709 12:22:40.061458 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 196445us
I20260709 12:22:40.035072 20960 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:40.071331 18780 connection.cc:256] RPC call timeout handler was delayed by -1.5808248280000043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260709 12:22:40.063587 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.059332 18098 rpcz_store.cc:276] 0709 12:22:39.040843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.039759 (+998916us) service_pool.cc:224] Handling call
0709 12:22:40.055332 (+ 15573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.058130 (+  2798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.058637 (+   507us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.035449 21242 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:40.077700 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.080044 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2660) took 1428 ms. Trace:
I20260709 12:22:40.080579 18020 rpcz_store.cc:276] 0709 12:22:38.651328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.651589 (+   261us) service_pool.cc:224] Handling call
0709 12:22:40.079760 (+1428171us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.079962 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1564257152}
W20260709 12:22:40.081955 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.047626 21251 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.042701 21292 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:40.091543 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 221852us
I20260709 12:22:40.096287 18098 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:40.056907 21341 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.112110 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 517) took 1070 ms. Trace:
I20260709 12:22:40.112916 18098 rpcz_store.cc:276] 0709 12:22:39.041414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.073598 (+1032184us) service_pool.cc:224] Handling call
0709 12:22:40.095996 (+ 22398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.096893 (+   897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.111994 (+ 15101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.105510 21251 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.115710 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1907) took 1024 ms. Trace:
I20260709 12:22:40.116317 18098 rpcz_store.cc:276] 0709 12:22:39.091439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.114329 (+1022890us) service_pool.cc:224] Handling call
0709 12:22:40.114886 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.115190 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.115231 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.115298 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:40.115348 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.115433 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.115610 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.107870 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.094357 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.075099 21352 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:40.080202 21242 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.094440 20960 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:40.128563 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64330.3us
W20260709 12:22:40.128434 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64937.3us
W20260709 12:22:40.130477 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57543.9us
I20260709 12:22:40.132494 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:40.133165 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.109664 21177 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.054283 21342 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.112314 21341 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:40.139631 18098 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:40.145381 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1910) took 1053 ms. Trace:
I20260709 12:22:40.145936 18098 rpcz_store.cc:276] 0709 12:22:39.092400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.117764 (+1025364us) service_pool.cc:224] Handling call
0709 12:22:40.139195 (+ 21431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.144940 (+  5745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.145300 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.125003 21251 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:40.147845 18098 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:40.128826 20960 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.133898 21300 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:40.183108 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 526) took 1089 ms. Trace:
I20260709 12:22:40.135059 21359 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:40.191526 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59106.6us
I20260709 12:22:40.189291 18098 rpcz_store.cc:276] 0709 12:22:39.093234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.147006 (+1053772us) service_pool.cc:224] Handling call
0709 12:22:40.147460 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.151011 (+  3551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.182999 (+ 31988us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.144518 21299 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.126703 21351 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:22:40.188810 20960 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:40.211627 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 103878us
I20260709 12:22:40.223300 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.228118 21359 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:40.230269 21359 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:40.231210 21360 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:40.232400 21360 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:40.236109 21359 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.229732 21364 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:40.245172 21364 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:40.254297 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.799s (SENT)
I20260709 12:22:40.256305 18090 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:40.257109 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 511) took 1217 ms. Trace:
I20260709 12:22:40.254915 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.257601 18090 rpcz_store.cc:276] 0709 12:22:39.039549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.806501 (+766952us) service_pool.cc:224] Handling call
0709 12:22:40.256999 (+450498us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.260047 18098 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 12. (term mismatch)
W20260709 12:22:40.262799 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 125965us
I20260709 12:22:40.195609 21365 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.264720 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:40.130932 21352 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:40.257517 21350 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.319s	user 0.000s	sys 0.002s
I20260709 12:22:40.263157 18098 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 11
I20260709 12:22:40.278569 18098 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 11.12
I20260709 12:22:40.278929 18098 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:40.256024 21234 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:40.150195 21356 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260709 12:22:40.270900 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:40.267160 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:40.283043 18814 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:40.262547 21287 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:40.270071 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:40.260042 21165 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.291903 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 148618us
I20260709 12:22:40.292248 21371 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 12:22:40.292495 21165 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.295142 21165 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.296789 21300 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.298095 21366 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:40.299045 21366 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.299611 21366 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:40.302147 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.248s (SENT)
I20260709 12:22:40.283111 18090 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:40.302793 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.275584 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:40.262287 21299 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.284835 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 528) took 1182 ms. Trace:
I20260709 12:22:40.307431 18098 rpcz_store.cc:276] 0709 12:22:39.102910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.202301 (+1099391us) service_pool.cc:224] Handling call
0709 12:22:40.259522 (+ 57221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.284459 (+ 24937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.284756 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.258577 21143 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.318374 21143 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:40.317595 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47650us
I20260709 12:22:40.320636 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.296109 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:40.306521 21369 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.317502 21177 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:40.287397 21348 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
I20260709 12:22:40.281016 21234 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:40.333990 18064 leader_election.cc:400] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6.
I20260709 12:22:40.335045 18807 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:22:40.308231 21138 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.801s	user 0.017s	sys 0.000s
I20260709 12:22:40.338956 21369 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.334570 21299 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:40.354166 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.305944 21359 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:40.345566 18815 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260709 12:22:40.275367 21365 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.366633 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:40.358570 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:40.344502 18064 leader_election.cc:403] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:40.343809 18098 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:40.346699 18811 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:40.359376 18090 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:40.291592 21287 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:40.379953 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 539) took 1226 ms. Trace:
I20260709 12:22:40.389894 18098 rpcz_store.cc:276] 0709 12:22:39.154046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.308482 (+1154436us) service_pool.cc:224] Handling call
0709 12:22:40.343247 (+ 34765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.379583 (+ 36336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.379866 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.384104 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82006.1us
W20260709 12:22:40.393023 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.030s (SENT)
I20260709 12:22:40.333139 21367 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:40.393195 18090 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:40.396950 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 541) took 1242 ms. Trace:
I20260709 12:22:40.347259 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:40.347321 20961 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.522s	user 0.013s	sys 0.000s
I20260709 12:22:40.384658 18814 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:40.360632 20974 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:40.366155 21262 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.873s	user 0.015s	sys 0.002s
I20260709 12:22:40.372789 21366 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.359912 21234 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:40.346880 21054 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.141s	user 0.015s	sys 0.001s
I20260709 12:22:40.340961 18808 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:40.353060 21143 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:40.394733 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65120.3us
I20260709 12:22:40.396365 21367 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:40.433111 18020 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:40.282147 21364 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.433845 18064 leader_election.cc:400] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:22:40.434394 18064 leader_election.cc:403] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:22:40.371589 21165 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.393728 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.397500 18090 rpcz_store.cc:276] 0709 12:22:39.154858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.392625 (+1237767us) service_pool.cc:224] Handling call
0709 12:22:40.393003 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.396555 (+  3552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.396866 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.345783 21138 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.360143 21369 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.348832 21352 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.366932 21356 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:22:40.443621 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50659.3us
I20260709 12:22:40.420991 21368 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.379592 21299 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.365772 21375 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:40.345332 21342 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.361523 21177 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:40.421751 18098 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:40.460213 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2724) took 1181 ms. Trace:
I20260709 12:22:40.405710 21346 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6.
I20260709 12:22:40.409756 18016 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:22:40.439112 18090 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:40.444823 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.445048 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:40.445430 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.035s (SENT)
I20260709 12:22:40.490831 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:40.490947 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45988.2us
I20260709 12:22:40.431754 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:40.453300 18172 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:22:40.439388 21256 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.537s	user 0.012s	sys 0.002s
W20260709 12:22:40.442231 21246 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.568s	user 0.015s	sys 0.000s
I20260709 12:22:40.460650 18098 rpcz_store.cc:276] 0709 12:22:39.279059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.390858 (+1111799us) service_pool.cc:224] Handling call
0709 12:22:40.415830 (+ 24972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.459884 (+ 44054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.460144 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.462364 21342 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.457037 21368 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.466727 21375 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:40.471004 21369 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.472893 21177 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:40.474725 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.475732 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 390) took 3129 ms. Trace:
I20260709 12:22:40.502838 18018 rpcz_store.cc:276] 0709 12:22:37.346773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.695017 (+348244us) service_pool.cc:224] Handling call
0709 12:22:38.747843 (+1052826us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.765292 (+ 17449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.765330 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:38.765391 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:38.765425 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.892487 (+1127062us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b54001cf554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:39.892554 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.475463 (+582909us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001cf550. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.475622 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2459745024}
I20260709 12:22:40.446694 21143 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:22:40.475780 20974 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.441253 21165 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.478935 21138 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.422963 21262 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.496587 21299 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:40.439471 21378 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:22:40.496961 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51565.8us
I20260709 12:22:40.494181 21029 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.509159 18015 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:40.339617 21353 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:40.509574 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.516306 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:40.496405 21256 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:40.497761 21343 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:40.521276 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:40.522854 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.527s (SENT)
I20260709 12:22:40.525611 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:40.526435 18806 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:40.510494 21300 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.530601 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:40.512723 21143 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.500311 21246 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.541560 21300 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.510584 18808 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:40.550575 21246 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.551553 18017 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:22:40.551954 21371 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.511056 21356 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:22:40.552850 21356 consensus_queue.cc:1243] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:40.552666 21371 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.555253 21300 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.556545 21341 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.544173 21375 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:40.558434 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2661) took 1906 ms (client timeout 1709 ms). Trace:
W20260709 12:22:40.558887 18021 rpcz_store.cc:269] 0709 12:22:38.651817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.672716 (+ 20899us) service_pool.cc:224] Handling call
0709 12:22:40.558185 (+1885469us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.558348 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":247967104}
I20260709 12:22:40.561131 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.561908 21365 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:40.562335 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 514) took 1521 ms (client timeout 1892 ms). Trace:
W20260709 12:22:40.562920 18092 rpcz_store.cc:269] 0709 12:22:39.040557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.560716 (+1520159us) service_pool.cc:224] Handling call
0709 12:22:40.562260 (+  1544us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.562706 21242 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.563730 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:40.494392 21287 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.519025 21298 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.568673 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.571445 18095 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:40.532801 18814 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:40.509992 21369 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.538805 21368 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.501650 20960 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.544956 18018 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:40.523299 21363 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.542835 21262 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.539216 20897 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.546346 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.607158 20897 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:40.547825 21343 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:40.607898 20897 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.608395 20897 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:40.608174 21343 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.553522 21356 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:40.627410 21356 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:40.631610 21341 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.599582 21363 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
W20260709 12:22:40.607055 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60845.1us
I20260709 12:22:40.569777 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.610535 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2723) took 1666 ms. Trace:
I20260709 12:22:40.567443 21242 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:40.659093 18094 rpcz_store.cc:276] 0709 12:22:38.943945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.025344 (+1081399us) service_pool.cc:224] Handling call
0709 12:22:40.041871 (+ 16527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.078959 (+ 37088us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.079022 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:40.105075 (+ 26053us) log.cc:844] Serialized 26 byte log entry
0709 12:22:40.500748 (+395673us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:40.500827 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.500910 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.610178 (+109268us) raft_consensus.cc:1685] finished
0709 12:22:40.610209 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.610415 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22321,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":14264,"thread_start_us":364240,"threads_started":1,"wal-append.queue_time_us":413031}
W20260709 12:22:40.660544 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2050) took 1248 ms (client timeout 1516 ms). Trace:
W20260709 12:22:40.665206 18022 rpcz_store.cc:269] 0709 12:22:39.411763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.412011 (+   248us) service_pool.cc:224] Handling call
0709 12:22:40.660230 (+1248219us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.660439 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":991683840}
I20260709 12:22:40.520303 21378 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.568207 21287 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:40.678596 21356 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.686905 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:40.687016 21165 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.690924 18020 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:40.691525 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 246764us
I20260709 12:22:40.692864 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:40.695019 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.699259 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.700793 21384 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:40.702279 21384 consensus_queue.cc:1243] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:40.703111 21387 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:40.703593 21387 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:40.704663 21387 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.706264 21387 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.707948 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.708824 18097 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
W20260709 12:22:40.709409 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 532) took 1605 ms (client timeout 1799 ms). Trace:
W20260709 12:22:40.709875 18097 rpcz_store.cc:269] 0709 12:22:39.104301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.707583 (+1603282us) service_pool.cc:224] Handling call
0709 12:22:40.709332 (+  1749us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.714402 21299 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:40.718516 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.687286 21382 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.723233 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.726969 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:40.730545 18020 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:40.731585 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:40.734175 18094 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:40.690047 21343 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:40.689929 21378 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.746876 18170 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:40.730072 21143 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.723225 21298 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.722844 21346 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:40.762959 18166 raft_consensus.cc:606] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
W20260709 12:22:40.767012 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52591.7us
I20260709 12:22:40.769301 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:40.747800 21356 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:40.770118 18173 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.770840 21345 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:40.771893 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:40.776515 18165 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:40.773171 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:40.777660 18167 raft_consensus.cc:2368] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5.
I20260709 12:22:40.778566 18168 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 6. Ops: []
I20260709 12:22:40.779774 18167 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:40.780331 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:40.781292 21288 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:40.781572 18171 raft_consensus.cc:2368] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 5.
I20260709 12:22:40.781737 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:40.791975 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.788371 21345 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:40.781775 21288 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:40.795940 18094 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
I20260709 12:22:40.795713 21143 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.738511 21382 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:40.809860 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.805002 18173 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:22:40.841683 20897 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:40.843853 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.811791 21143 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.805346 21234 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.814694 20960 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.789450 21299 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.836506 20842 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:40.856731 21273 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.857210 20842 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:40.857465 21273 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.858312 20842 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.859443 20842 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:40.859894 21273 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.862845 18812 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:40.870841 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.872287 21385 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.872562 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:40.872995 21385 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.874054 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:40.874960 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.875597 21385 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.876168 21242 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:40.878947 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:40.865504 21273 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.862504 21234 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.868750 21346 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.877043 21029 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.876201 21385 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.878727 21242 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:40.900930 20897 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:40.887233 21177 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.847728 21288 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.884618 21285 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:22:40.885289 21241 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 12:22:40.834791 21345 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.891345 21273 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.897946 21029 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.909214 21241 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:40.909682 21241 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:40.910543 21273 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.910812 21029 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:40.912974 21357 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.723s	user 0.004s	sys 0.000s
I20260709 12:22:40.914464 21242 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.916396 21256 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:40.916632 21370 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.917506 21368 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.917301 21370 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:40.917317 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.917727 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 194648us
I20260709 12:22:40.919772 21370 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.920077 20897 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:40.920617 20960 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.922235 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.924259 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:40.924839 21387 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:40.925246 18020 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:40.925539 21387 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:40.925936 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:40.926692 21387 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.932562 21143 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.903079 21382 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.933617 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.897266 21346 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.933997 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2053) took 1444 ms. Trace:
I20260709 12:22:40.934346 21382 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:40.934487 18015 rpcz_store.cc:276] 0709 12:22:39.489942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.496845 (+  6903us) service_pool.cc:224] Handling call
0709 12:22:40.933736 (+1436891us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.933908 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2105524992}
I20260709 12:22:40.935259 21298 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:40.935703 21298 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:40.936173 21298 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:40.936362 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.938861 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:40.935861 21346 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.939862 18023 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10.
I20260709 12:22:40.921229 20960 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.944334 18780 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10.
I20260709 12:22:40.945866 18780 leader_election.cc:403] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20260709 12:22:40.944182 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.726s (SENT)
I20260709 12:22:40.945266 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:40.949025 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:40.910733 21241 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:40.951225 21241 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:40.951315 21288 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:40.952144 20960 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:40.954545 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2119) took 1646 ms. Trace:
I20260709 12:22:40.954972 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:40.955884 18023 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:40.955019 18164 rpcz_store.cc:276] 0709 12:22:39.308272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.333762 (+ 25490us) service_pool.cc:224] Handling call
0709 12:22:39.496838 (+163076us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.497601 (+   763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.954306 (+1456705us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400383450. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.954473 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373575936}
I20260709 12:22:40.956108 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:40.954286 21177 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:40.957630 21367 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:40.946273 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.947358 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:40.965711 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 35) took 1209 ms. Trace:
I20260709 12:22:40.966400 18166 rpcz_store.cc:276] 0709 12:22:39.751570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.763782 (+ 12212us) service_pool.cc:224] Handling call
0709 12:22:40.960564 (+1196782us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:40.960686 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2295517952}
I20260709 12:22:40.972071 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.972795 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.971305 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:40.973461 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.974375 18097 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:40.976794 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:40.976979 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:40.977555 18015 raft_consensus.cc:2376] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7. Re-sending same reply.
I20260709 12:22:40.967873 21341 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.011s
I20260709 12:22:40.979748 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:40.978559 21281 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:40.981729 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:40.981931 21343 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:40.982358 21399 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:40.982580 18092 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:40.983294 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:40.989290 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:40.990283 18019 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:40.990698 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2126) took 1612 ms. Trace:
I20260709 12:22:40.991158 18170 rpcz_store.cc:276] 0709 12:22:39.378389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.378553 (+   164us) service_pool.cc:224] Handling call
0709 12:22:39.378942 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.762576 (+1383634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.990433 (+227857us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:40.990623 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3145351424}
I20260709 12:22:40.994647 21299 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:40.966172 21388 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.002s
I20260709 12:22:40.998590 21299 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.999812 21299 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:41.002915 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:41.019435 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:41.020853 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.021179 18812 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:22:40.968176 21234 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:22:41.019522 18093 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:22:41.021407 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.019871 18018 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:41.027400 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:41.034878 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:41.036099 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64888.7us
I20260709 12:22:41.043157 18064 leader_election.cc:400] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:41.043545 18064 leader_election.cc:403] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:22:41.044701 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.046661 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:41.046394 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:41.047501 18814 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:41.020455 20960 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:41.054888 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.021759 21399 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:41.058483 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.059546 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:41.071789 18017 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:41.033901 21242 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.019865 21071 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:41.039191 21384 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10.
W20260709 12:22:41.075625 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.077535 18023 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:41.042560 21143 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:41.083462 18017 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:41.048969 21346 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:41.087250 18811 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:22:41.087697 18811 pending_rounds.cc:85] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 6
I20260709 12:22:41.049556 21385 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:41.087929 18811 pending_rounds.cc:107] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:22:41.093215 18811 raft_consensus.cc:2889] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.094882 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:41.096467 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:41.099498 21346 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:41.099743 18097 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:41.103672 18806 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:41.073356 21177 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:41.105577 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:41.085666 21071 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.111511 18166 raft_consensus.cc:606] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.074035 21396 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:41.079311 20960 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:41.115665 21029 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 12:22:41.128942 21367 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:41.140033 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:41.140940 18809 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:41.142751 21363 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:41.144094 20897 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:41.109927 21177 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:41.112172 21157 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:22:41.146349 21177 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:41.146770 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:36144 (request call id 10) took 1311 ms. Trace:
I20260709 12:22:41.147189 18024 rpcz_store.cc:276] 0709 12:22:39.835173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.023144 (+187971us) service_pool.cc:224] Handling call
0709 12:22:41.146434 (+1123290us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bd735 00007f6c1b67dd29 00007f6c14bfc1d4 00007f6c14bfc162 00007f6c14bfc091 00007f6c14bfc00c 00007f6c14bfb2c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:41.146649 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":858324224}
I20260709 12:22:41.111967 20842 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:22:41.148380 20842 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:41.148757 20842 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:41.084378 21242 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.150758 18092 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260709 12:22:41.151394 21399 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.152117 21242 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:41.114686 21036 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:22:41.153084 18097 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:41.154181 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:41.155611 18097 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:41.156024 21299 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.156988 18090 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:22:41.157363 18090 pending_rounds.cc:85] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 2
I20260709 12:22:41.157605 18090 pending_rounds.cc:107] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:22:41.095839 21369 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:41.157861 18090 raft_consensus.cc:2889] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.122349 21364 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:41.160424 18090 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:41.160812 18813 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:22:41.162364 17970 auto_rebalancer.cc:893] Removing replace marker failed: Replica cab4a2f5d3ca48a2a803ee8ec3906e1d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:22:41.162940 17970 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for f9731c63bc7f406db2c77f5ef0415a78 already in progress
I20260709 12:22:41.152002 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:41.203213 18092 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:41.182845 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.111596 21269 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:41.120903 21368 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 12:22:41.200531 20960 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.206496 18023 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20260709 12:22:41.245939 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 187541us
I20260709 12:22:41.187698 21298 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:41.248693 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.208175 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:41.234272 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.203001 17970 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for f9731c63bc7f406db2c77f5ef0415a78 already in progress
I20260709 12:22:41.263271 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:41.265187 18018 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:41.268458 18092 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:41.270660 18022 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:41.277099 18022 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:41.279678 21157 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:41.281649 21029 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:41.282315 21029 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:41.283545 21029 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:41.207945 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45447.1us
I20260709 12:22:41.288435 18097 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:41.294548 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 130481us
I20260709 12:22:41.293123 20842 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.302150 20842 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.302727 21256 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:41.303165 21298 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:41.308655 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.248778 21269 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:41.323523 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:41.324746 18137 connection.cc:256] RPC call timeout handler was delayed by -1.059980545000144s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:22:41.326581 18814 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3.
W20260709 12:22:41.327842 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48468.1us
I20260709 12:22:41.322973 21385 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:22:41.334287 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:41.335186 18092 raft_consensus.cc:2376] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:22:41.338263 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.340411 18015 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:41.323725 21351 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.006s	user 0.004s	sys 0.010s
W20260709 12:22:41.341153 21351 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.023s	user 0.004s	sys 0.011s
I20260709 12:22:41.341296 18099 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:41.342362 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55722us
I20260709 12:22:41.346994 21315 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:41.348050 21343 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:41.350252 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.353436 18814 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:41.357323 18166 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:41.358076 18164 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 9. Ops: []
I20260709 12:22:41.361056 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:41.361428 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.361073 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.363583 18164 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:41.364037 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.364697 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.349069 21356 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:41.367311 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.367878 18064 leader_election.cc:400] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:41.369910 18166 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:41.377300 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:41.378544 18064 leader_election.cc:403] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:41.378636 18097 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:22:41.378218 18173 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:41.381855 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.383616 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:41.331153 21371 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:41.393956 18164 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:41.404783 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:41.394415 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:41.407066 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.394753 18814 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:22:41.409123 18814 pending_rounds.cc:85] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 8
I20260709 12:22:41.409250 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.397644 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:41.398139 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.399139 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:41.396224 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.395381 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.411929 18168 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:41.409559 18814 pending_rounds.cc:107] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:22:41.415047 18814 raft_consensus.cc:2889] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.376724 21341 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.350337 21143 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:41.420197 18810 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:41.368428 21054 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:41.427676 21143 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:41.322965 21313 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 12:22:41.438895 18171 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:41.439407 21143 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.440963 21143 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:41.421296 21345 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.432672 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:41.443198 18815 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:41.437443 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.378351 21356 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:41.382323 21389 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:41.449322 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.337991 21285 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 12:22:41.351397 21036 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:22:41.464872 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.433552 21251 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:41.366602 21315 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260709 12:22:41.482293 18018 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:41.467588 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.485937 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:41.376127 21363 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:22:41.422304 21406 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7.
I20260709 12:22:41.424726 21346 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:41.486209 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.382432 21399 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260709 12:22:41.438130 21269 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.494438 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:41.504961 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.508450 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:41.493348 21036 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:41.468046 21345 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.570120 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.574568 18168 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:41.466432 21071 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:41.594879 21404 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260709 12:22:41.603085 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:41.604096 18092 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:41.484786 21389 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:41.614607 21408 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:22:41.617574 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.645103 18094 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:41.646672 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:41.647132 21419 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:22:41.647486 18094 raft_consensus.cc:2376] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7. Re-sending same reply.
I20260709 12:22:41.649255 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:41.623812 21409 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:22:41.619922 21410 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:22:41.654983 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.587397 21036 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:41.657768 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.658814 18094 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:41.659758 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:41.567202 21356 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.669940 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:41.675989 18097 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:41.680198 18096 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:41.443416 21341 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.688076 18017 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:41.689734 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:41.691695 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.694936 21420 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.695688 21420 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.696784 18094 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:22:41.696916 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2137) took 2131 ms (client timeout 1944 ms). Trace:
I20260709 12:22:41.697676 18810 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:41.697569 18172 rpcz_store.cc:269] 0709 12:22:39.565641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.565783 (+   142us) service_pool.cc:224] Handling call
0709 12:22:41.696652 (+2130869us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:41.696788 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1578392960}
I20260709 12:22:41.699162 18094 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:41.700397 18094 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.703246 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d440327a63754b889f7edd91e1373942"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:41.705705 18168 raft_consensus.cc:606] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:41.706749 18168 raft_consensus.cc:614] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:41.728798 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.729679 18094 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:41.746500 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 281949us
W20260709 12:22:41.755357 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.775921 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:41.781859 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.790268 18091 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:41.793921 18094 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 12:22:41.795238 18094 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:22:41.796258 18172 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:41.795617 18094 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:22:41.799953 18094 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.800413 18092 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:41.777953 21298 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:41.779309 21299 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:41.809303 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.810887 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.811718 18164 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:41.786592 21356 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.799319 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.817517 18172 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:41.791445 21071 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:41.826514 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.808738 21315 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 12:22:41.815821 21241 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 12:22:41.787565 21269 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:22:41.794246 21157 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:41.868053 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.804730 21251 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:22:41.875063 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.885706 18164 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:22:41.888398 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 112643us
I20260709 12:22:41.888713 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.890523 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.895991 18064 leader_election.cc:400] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5.
I20260709 12:22:41.893013 21341 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:41.897001 21313 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:41.899019 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.900007 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.898187 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.903097 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:41.891973 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.815954 21298 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.905391 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:41.870887 21364 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:41.900494 21177 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.754s	user 0.012s	sys 0.007s
I20260709 12:22:41.919251 18167 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:41.923744 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:41.924839 21241 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:41.925428 21241 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:41.926123 21177 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:41.926424 21241 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:41.929148 21177 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:41.931457 18170 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.009s
I20260709 12:22:41.931936 18170 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:41.933189 21157 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:41.876040 21036 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:41.892859 21292 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:41.909888 21054 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:41.934548 21251 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:41.902395 21233 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:41.943437 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.908296 21420 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:41.910019 21285 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:41.953161 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:41.914208 21242 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:41.959340 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:41.941479 20842 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.639s	user 0.017s	sys 0.005s
I20260709 12:22:41.915297 21389 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:41.967825 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.943877 21054 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.974327 18168 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 12:22:41.975299 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.943887 21251 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:41.979976 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82358.6us
I20260709 12:22:41.983946 18064 leader_election.cc:400] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 5.
I20260709 12:22:41.984828 18064 leader_election.cc:403] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:41.964834 21288 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.013s	user 0.014s	sys 0.010s
W20260709 12:22:41.973006 18809 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.831s	user 0.007s	sys 0.016s
I20260709 12:22:41.943073 21364 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.990418 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:41.953773 21352 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:22:41.966094 21285 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:41.953845 21157 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:22:41.980903 21054 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:41.976215 21233 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.964879 21399 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:22:41.964488 21396 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:22:41.975119 21143 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 12:22:42.004029 21251 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.005587 21251 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.992077 18809 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:42.011302 18167 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:42.018330 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:42.019109 18164 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.022346 18813 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:42.031889 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.029656 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:42.036020 18167 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:42.036489 18814 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:42.037524 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53735us
I20260709 12:22:42.038558 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:42.054157 18164 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:42.057837 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:42.058621 18167 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.062528 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:42.064584 21256 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:42.065527 21256 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.066941 21359 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.067739 21359 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.068657 21256 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:42.075955 18022 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:42.055498 21384 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.087321 21384 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:42.089148 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50706.5us
I20260709 12:22:42.090160 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:42.091149 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:42.091864 18810 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:42.093523 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:42.083231 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.102221 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.084018 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:42.082664 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:42.094578 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.099753 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2706) took 1010 ms. Trace:
I20260709 12:22:42.102342 18814 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:42.105787 18023 rpcz_store.cc:276] 0709 12:22:41.089310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.090057 (+   747us) service_pool.cc:224] Handling call
0709 12:22:41.101343 (+ 11286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.253275 (+151932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.099459 (+846184us) spinlock_profiling.cc:243] Waited 61.2 ms on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:42.099677 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140845312}
W20260709 12:22:42.107304 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.112613 18021 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:42.002753 21233 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:42.090579 21437 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:22:42.081544 21435 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:42.090572 21422 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:42.089710 21384 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.127772 21233 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:42.139585 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46163us
I20260709 12:22:42.147611 21151 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:42.148383 21151 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:42.149495 21151 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.129352 21396 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:22:42.152021 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:42.160351 18023 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:42.160336 20960 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.162410 20960 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.164788 20960 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.168104 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:42.170004 21285 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.172554 21422 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:42.120744 21436 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:42.174743 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:42.177282 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:42.191229 18023 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
W20260709 12:22:42.195036 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.197417 18015 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 6. Ops: [4.3-4.3]
I20260709 12:22:42.175336 21143 consensus_queue.cc:1243] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:22:42.199831 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:42.218638 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:42.194754 21436 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:42.198979 21359 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:42.225206 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.227273 18164 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:42.227990 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.228478 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.228986 18172 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:42.230180 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.230360 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:42.231398 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.229686 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:42.235809 18172 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:42.237736 18172 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:42.242369 21352 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.243050 21352 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.243300 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:42.245357 21352 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:42.247388 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:42.247646 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:42.264240 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45751.5us
I20260709 12:22:42.270952 21435 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 3. Current term is 5.
I20260709 12:22:42.218881 21313 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.225396 21157 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.383229 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.508745 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.645433 20960 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.280134 21177 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:42.345381 21436 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.249038 21434 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.123s
I20260709 12:22:42.225557 21353 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
W20260709 12:22:42.655486 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.666450 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 400165us
W20260709 12:22:42.671577 18167 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.590s	user 0.014s	sys 0.003s
I20260709 12:22:42.677201 18167 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:22:42.665997 21364 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.661s	user 0.016s	sys 0.012s
W20260709 12:22:42.684278 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.673930 21356 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.848s	user 0.006s	sys 0.017s
I20260709 12:22:42.657997 21384 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:42.662850 21383 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:42.686442 21352 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.708621 21352 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:42.709203 21448 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:22:42.687683 21399 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.665951 21157 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.669144 20960 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.665900 21313 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.692610 21432 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:42.717943 21143 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.715219 21359 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:42.750263 21445 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:22:42.838829 21455 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:42.713093 21383 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.759686 21432 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.751587 21399 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.718255 21454 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260709 12:22:42.896551 21366 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.705111 21384 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.760582 18164 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 4: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:42.723225 21435 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:42.777356 21143 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.912992 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 551) took 1461 ms. Trace:
I20260709 12:22:42.919389 21315 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.919788 21346 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:42.920125 18024 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:42.920783 21352 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:42.921061 21285 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.923660 21054 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.913s	user 0.018s	sys 0.010s
I20260709 12:22:42.923918 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:42.924064 21269 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.919420 18021 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:42.924119 21359 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.928141 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:42.928695 21345 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.694470 21364 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.976231 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:42.978065 18019 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:42.915570 21456 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.194s	user 0.000s	sys 0.005s
I20260709 12:22:42.981806 21384 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:42.898414 21177 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:42.978380 21364 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.984493 21364 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.907469 21427 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:42.945475 18095 rpcz_store.cc:276] 0709 12:22:41.451596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.789525 (+337929us) service_pool.cc:224] Handling call
0709 12:22:41.793807 (+  4282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.885996 (+ 92189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.886031 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.886799 (+   768us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.336261 (+449462us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:42.336324 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.336431 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.912552 (+576121us) raft_consensus.cc:1685] finished
0709 12:22:42.912591 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.912866 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3156,"raft.queue_time_us":81903,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":18273,"thread_start_us":480146,"threads_started":2,"wal-append.queue_time_us":736867}
I20260709 12:22:42.985314 21427 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.986104 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:42.986337 21269 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.988233 21399 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:42.981807 21359 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.951586 21435 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.989679 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:42.984494 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2754) took 1011 ms. Trace:
I20260709 12:22:42.961956 18093 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:42.948706 21366 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.991463 18092 rpcz_store.cc:276] 0709 12:22:41.973037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.975622 (+  2585us) service_pool.cc:224] Handling call
0709 12:22:41.990875 (+ 15253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.092298 (+101423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.092334 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.092885 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.335745 (+242860us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:42.335813 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.335944 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.984239 (+648295us) raft_consensus.cc:1685] finished
0709 12:22:42.984259 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.984404 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107817,"raft.queue_time_us":121664,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":11876,"thread_start_us":206216,"threads_started":1,"wal-append.queue_time_us":853137}
I20260709 12:22:42.992650 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2753) took 1020 ms. Trace:
I20260709 12:22:42.957398 21315 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.993702 21366 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.994767 18092 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:42.995209 18092 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:42.995704 21315 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:42.996163 21364 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.953305 21346 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:42.984198 21177 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:42.987663 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:42.954888 18170 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:42.915586 21457 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.988308 18015 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:22:42.993258 18091 rpcz_store.cc:276] 0709 12:22:41.972338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.972510 (+   172us) service_pool.cc:224] Handling call
0709 12:22:41.987890 (+ 15380us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:42.080245 (+ 92355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.080283 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:42.119310 (+ 39027us) log.cc:844] Serialized 52 byte log entry
0709 12:22:42.318463 (+199153us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:42.646767 (+328304us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.666631 (+ 19864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.666867 (+   236us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.992355 (+325488us) raft_consensus.cc:1685] finished
0709 12:22:42.992387 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.992575 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87200,"prepare.queue_time_us":327747,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":122562,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":17885,"thread_start_us":130143,"threads_started":1,"wal-append.queue_time_us":792895}
I20260709 12:22:42.972126 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.016258 18015 raft_consensus.cc:606] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:43.016695 18015 raft_consensus.cc:614] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:43.018314 18095 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:42.987883 21427 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:42.997605 18092 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.021080 18092 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:42.978199 21383 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.019878 18094 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:43.014396 21251 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.005s	user 0.006s	sys 0.016s
I20260709 12:22:42.991677 21269 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:43.036799 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.975091 21375 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.038638 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.039513 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.044339 18091 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:43.045535 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:42.972476 21054 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:42.972298 21285 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.047035 18091 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:43.048672 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.050643 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.042905 21243 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.042766 21353 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.067982 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.054234 18095 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:43.054838 18092 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:43.070053 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.042523 21427 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.072189 21243 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:43.072511 21427 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.072955 21364 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.074072 21463 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:43.053377 21467 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:43.074918 21427 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.047372 21375 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.076009 21435 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.077888 21375 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.078003 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:43.078794 18807 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:43.079617 21273 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:43.079701 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.081846 21054 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.083088 18165 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:22:43.083770 21157 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:43.084136 21399 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:43.085438 21432 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.087806 21285 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:43.088541 21454 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:43.088737 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:43.090955 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.064703 18093 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:43.094739 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:43.095772 18814 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:43.094818 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:43.098731 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.846s (SENT) [suppressed 22 similar messages]
I20260709 12:22:43.023005 21457 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.050836 21313 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.066354 21436 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:43.051290 20960 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.108089 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:43.115049 21313 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:43.071657 21298 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.091256 21243 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.125733 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:43.126159 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:43.126708 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.120128 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.134747 18018 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:43.134829 18020 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:43.136018 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.122092 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.129180 21468 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:43.138839 18024 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:43.139125 21468 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:43.139587 18024 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:22:43.139645 21468 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:43.124287 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.124699 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.141777 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:43.142520 18023 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:43.100493 21353 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.145344 21353 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.146003 21384 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.119182 18015 raft_consensus.cc:1217] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-6.5]   Dedup: 6.5->[]
I20260709 12:22:43.146780 18015 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [3.3-6.5]
I20260709 12:22:43.146832 21458 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:43.147800 21364 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:43.149919 21384 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.151716 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:43.152742 21384 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:43.152892 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:43.100073 21464 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:43.154044 21375 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:43.154635 21375 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:43.154798 21298 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:43.155290 21298 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.155675 21298 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:43.155706 21375 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:43.156774 18780 connection.cc:256] RPC call timeout handler was delayed by -1.6812319629998456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
I20260709 12:22:43.128371 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:43.158847 18169 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:43.167320 18015 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:43.169008 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.128691 21143 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.110954 21436 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:43.170569 21436 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:43.171000 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.171876 18023 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:43.123912 21457 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.053030 21346 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.174578 21346 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:22:43.175213 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.127559 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:43.176834 18167 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:43.177625 18167 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:43.125523 18170 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 6. Ops: []
W20260709 12:22:43.131703 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.136099 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.112135 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:43.180088 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 68541.1us
I20260709 12:22:43.142127 21029 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:43.181802 21461 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:43.182405 21432 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.183035 21432 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.130996 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.181802 21313 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:22:43.146345 21383 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:22:43.182428 21461 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:43.188041 21461 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:43.138101 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.156853 21421 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:43.190315 21421 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:43.150425 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:43.191308 21421 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.186728 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.192966 18093 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:43.152966 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2070) took 1698 ms. Trace:
I20260709 12:22:43.197592 18017 rpcz_store.cc:276] 0709 12:22:41.454558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.454733 (+   175us) service_pool.cc:224] Handling call
0709 12:22:41.687868 (+233135us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.688659 (+   791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.152742 (+1464083us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54000ff850. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:43.152895 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":535200128}
I20260709 12:22:43.186753 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:43.129709 18171 raft_consensus.cc:1217] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Deduplicated request from leader. Original: 3.3->[4.4-6.5]   Dedup: 6.5->[]
I20260709 12:22:43.201001 18171 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260709 12:22:43.202993 21356 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.203714 21356 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.185423 21432 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:43.211364 21407 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.557s	user 0.000s	sys 0.005s
I20260709 12:22:43.212384 21356 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.213006 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1946) took 2239 ms. Trace:
W20260709 12:22:43.157567 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3866537079998125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260709 12:22:43.214092 18780 connection.cc:256] RPC call timeout handler was delayed by -1.2433558239999911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
W20260709 12:22:43.214887 18780 connection.cc:256] RPC call timeout handler was delayed by -1.1041394829999263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
I20260709 12:22:43.213570 18098 rpcz_store.cc:276] 0709 12:22:40.973468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.973601 (+   133us) service_pool.cc:224] Handling call
0709 12:22:40.973960 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.035211 (+ 61251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.035237 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.050715 (+ 15478us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.337744 (+287029us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:41.773491 (+435747us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.850171 (+ 76680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.850304 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.210583 (+1360279us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:43.212594 (+  2011us) raft_consensus.cc:1685] finished
0709 12:22:43.212731 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.212895 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17520,"prepare.queue_time_us":231955,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":82839,"raft.run_cpu_time_us":15724,"raft.run_wall_time_us":47434,"spinlock_wait_cycles":2987264,"thread_start_us":714104,"threads_started":3,"wal-append.queue_time_us":442462}
I20260709 12:22:43.215812 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 451) took 1221 ms. Trace:
I20260709 12:22:43.216246 18093 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:22:43.216952 18093 raft_consensus.cc:606] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:43.217317 18093 raft_consensus.cc:614] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:43.218797 18093 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:43.220371 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:43.221280 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1852) took 1812 ms (client timeout 1663 ms). Trace:
I20260709 12:22:43.221706 18093 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:43.223230 18093 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:43.224030 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2721) took 1129 ms. Trace:
I20260709 12:22:43.224447 18019 rpcz_store.cc:276] 0709 12:22:42.094762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.094956 (+   194us) service_pool.cc:224] Handling call
0709 12:22:42.095263 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.978602 (+883339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.223762 (+245160us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b540024de54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:43.223955 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2029736320}
I20260709 12:22:43.227442 18096 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:22:43.216297 18165 rpcz_store.cc:276] 0709 12:22:41.994325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.994511 (+   186us) service_pool.cc:224] Handling call
0709 12:22:42.001210 (+  6699us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.083555 (+1082345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.215469 (+131914us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000424d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:43.215725 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2487367936}
I20260709 12:22:43.228922 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:43.229610 18096 raft_consensus.cc:2376] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8. Re-sending same reply.
W20260709 12:22:43.231389 21406 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260709 12:22:43.221650 18173 rpcz_store.cc:269] 0709 12:22:41.408675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.408893 (+   218us) service_pool.cc:224] Handling call
0709 12:22:43.217411 (+1808518us) spinlock_profiling.cc:243] Waited 8.41 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592e9c 00007f6c13592a5b 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:43.221199 (+  3788us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19350400}
I20260709 12:22:43.209266 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.227445 18094 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:43.234520 21421 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:43.236025 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.238610 21384 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.239320 21384 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:43.242254 21447 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.534s	user 0.000s	sys 0.003s
I20260709 12:22:43.253432 21356 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.254266 21356 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.264406 21287 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:43.265202 21287 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:43.222287 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:43.266471 21287 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.267086 21036 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:43.267684 21036 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.267740 21287 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:43.268118 21036 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:43.269975 18166 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:22:43.270928 18812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 818) took 3004 ms (client timeout 2062 ms). Trace:
W20260709 12:22:43.271533 18812 rpcz_store.cc:269] 0709 12:22:40.266908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.267085 (+   177us) service_pool.cc:224] Handling call
0709 12:22:43.270574 (+3003489us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54006f2054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:43.270768 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1281555712}
I20260709 12:22:43.274371 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.275269 18019 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:43.278756 21363 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.282230 21363 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:43.288522 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.291677 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.291756 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1851) took 1905 ms. Trace:
I20260709 12:22:43.321746 18020 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:22:43.294688 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.322443 18019 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:43.298589 18094 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:43.295321 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.323700 18017 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 4.
W20260709 12:22:43.297389 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.300523 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:43.266958 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44789.3us
I20260709 12:22:43.302423 18098 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:43.300156 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.310637 21438 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.313894 21384 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:43.314766 17921 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20260709 12:22:43.321305 18166 rpcz_store.cc:276] 0709 12:22:41.386219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.408956 (+ 22737us) service_pool.cc:224] Handling call
0709 12:22:43.291531 (+1882575us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400031bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:43.291688 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4276452608}
W20260709 12:22:43.322602 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.299418 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:43.304523 21363 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:43.419657 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.387521 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:43.615693 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:43.606698 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.614223 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.681674 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:43.461043 17918 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 177) took 18319 ms (client timeout 15000 ms). Trace:
W20260709 12:22:43.626839 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.462049 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2757) took 1229 ms. Trace:
I20260709 12:22:43.663276 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:43.326072 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 227350us
I20260709 12:22:43.366714 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:43.670056 18024 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:43.618505 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 465359us
I20260709 12:22:43.682498 18015 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:43.324142 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:43.684711 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.725867 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44907.2us
I20260709 12:22:43.726359 18094 raft_consensus.cc:2376] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6. Re-sending same reply.
I20260709 12:22:43.728375 18094 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:43.727427 18090 rpcz_store.cc:276] 0709 12:22:42.232432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.248980 (+ 16548us) service_pool.cc:224] Handling call
0709 12:22:42.249404 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.249873 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.249917 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.250550 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.155744 (+905194us) spinlock_profiling.cc:243] Waited 9.73 ms on lock 0x7b180002d2e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:22:43.236875 (+ 81131us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:43.236921 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.237012 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.457107 (+220095us) raft_consensus.cc:1685] finished
0709 12:22:43.457144 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.461908 (+  4764us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17,"raft.queue_time_us":98,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":9667,"spinlock_wait_cycles":22387456,"thread_start_us":80910,"threads_started":1,"wal-append.queue_time_us":974372}
I20260709 12:22:43.728972 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:43.731312 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44135.2us
I20260709 12:22:43.669315 17921 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 6, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 6 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:43.732292 18093 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:43.684265 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 70138.7us
I20260709 12:22:43.678273 21285 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.742378 21285 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.745115 21285 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.746006 21490 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:43.740929 21287 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:22:43.746773 21490 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.747313 21490 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.747506 21029 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:43.747738 21292 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:43.748100 21287 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:43.749159 21029 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:43.646595 21427 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:43.256881 21356 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.581010 21151 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.581214 21463 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:43.744221 17921 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 4, VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2) evicted, no longer has a pending config: VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1), VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3), VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:43.738868 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:43.725087 17918 rpcz_store.cc:269] 0709 12:22:25.137721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.170033 (+ 32312us) service_pool.cc:224] Handling call
0709 12:22:43.457249 (+18287216us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:37.028130 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:37.029227 (+  1097us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:37.029308 (+    81us) write_op.cc:435] Acquired schema lock
0709 12:22:37.029339 (+    31us) tablet.cc:662] Decoding operations
0709 12:22:37.075332 (+ 45993us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:37.075428 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:37.075460 (+    32us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:22:37.076809 (+  1349us) tablet.cc:701] Row locks acquired
0709 12:22:37.111333 (+ 34524us) write_op.cc:260] PREPARE: finished
0709 12:22:37.147020 (+ 35687us) write_op.cc:270] Start()
0709 12:22:37.217299 (+ 70279us) write_op.cc:276] Timestamp: P: 1783599757137711 usec, L: 0
0709 12:22:37.217357 (+    58us) op_driver.cc:348] REPLICATION: starting
0709 12:22:37.442354 (+224997us) log.cc:844] Serialized 5337 byte log entry
0709 12:22:37.893877 (+451523us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b
0709 12:22:38.532238 (+638361us) op_driver.cc:464] REPLICATION: finished
0709 12:22:38.534307 (+  2069us) write_op.cc:301] APPLY: starting
0709 12:22:38.534435 (+   128us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:38.534793 (+   358us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:38.534837 (+    44us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:38.540035 (+  5198us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:38.540461 (+   426us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0709 12:22:38.540513 (+    52us) write_op.cc:312] APPLY: finished
0709 12:22:38.540974 (+   461us) log.cc:844] Serialized 95 byte log entry
0709 12:22:38.542033 (+  1059us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:38.542214 (+   181us) write_op.cc:454] Released schema lock
0709 12:22:38.542295 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:38.544861 (+  2566us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:38.544903 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":40527,"reported_tablets":25,"spinlock_wait_cycles":3456,"child_traces":[["op",{"apply.queue_time_us":1733,"apply.run_cpu_time_us":7710,"apply.run_wall_time_us":10503,"num_ops":13,"prepare.queue_time_us":349569,"prepare.run_cpu_time_us":75310,"prepare.run_wall_time_us":1295647,"raft.queue_time_us":67854,"raft.run_cpu_time_us":1663,"raft.run_wall_time_us":4794,"replication_time_us":1298845,"spinlock_wait_cycles":85994240,"thread_start_us":1063570,"threads_started":4,"wal-append.queue_time_us":711146,"wal-append.run_cpu_time_us":24114,"wal-append.run_wall_time_us":1393540}]]}
I20260709 12:22:43.670251 21273 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:43.750833 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57423.6us
I20260709 12:22:43.794322 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:43.381572 21346 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:43.791595 21285 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.783849 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56138.1us
I20260709 12:22:43.680212 21384 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:43.688001 21353 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:43.683679 21138 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.695430 21464 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.426208 21468 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.749619 21029 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:43.696125 21389 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.475586 21036 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:43.815068 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.835815 17989 connection.cc:256] RPC call timeout handler was delayed by -1.4207534460001625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20260709 12:22:43.836254 17989 connection.cc:256] RPC call timeout handler was delayed by -1.4047759999998561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
W20260709 12:22:43.836602 17989 connection.cc:256] RPC call timeout handler was delayed by -1.3967423539997981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
W20260709 12:22:43.836895 17989 connection.cc:256] RPC call timeout handler was delayed by -1.3831759999998212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
W20260709 12:22:43.837178 17989 connection.cc:256] RPC call timeout handler was delayed by -1.3363760000002003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
W20260709 12:22:43.837473 17989 connection.cc:256] RPC call timeout handler was delayed by -1.2886760000001232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21150000000000002s for a response
I20260709 12:22:43.808317 17921 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 5, term changed from 5 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to <none>, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:43.746052 21177 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.774430 18090 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:43.750526 21481 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:43.749626 21490 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.426513 21436 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.449358 21461 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.696046 21054 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
I20260709 12:22:43.843454 17921 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 6, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 6 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:43.807380 21420 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.366319 21298 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:43.851874 17989 connection.cc:256] RPC call timeout handler was delayed by -1.2220760000000155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20260709 12:22:43.858974 17989 connection.cc:256] RPC call timeout handler was delayed by -1.1896759999999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
I20260709 12:22:43.857437 21353 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:43.860918 18095 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:43.754606 20960 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.112s
I20260709 12:22:43.817474 21384 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.669412 21438 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.851312 17921 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:43.819394 21287 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:43.863214 21353 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.854204 21375 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.869093 18096 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:43.807286 21273 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.810252 21151 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.873034 21036 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:43.874001 21292 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.874229 18016 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:43.814733 21346 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.813419 21285 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.854669 21458 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:43.874779 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 550) took 2423 ms. Trace:
I20260709 12:22:43.827580 21138 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.885356 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1960) took 1163 ms. Trace:
I20260709 12:22:43.799062 21463 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.860134 21420 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:43.876932 21470 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:22:43.835211 21464 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.922737 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033364
I20260709 12:22:43.851298 21177 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.924546 21436 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.927510 21461 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.928805 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:43.929545 18806 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:43.902243 18095 rpcz_store.cc:276] 0709 12:22:42.721746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.131952 (+410206us) service_pool.cc:224] Handling call
0709 12:22:43.132327 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.861432 (+729105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.884911 (+ 23479us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b540040bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:43.885246 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1669968768}
I20260709 12:22:43.873642 21298 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.898397 18097 rpcz_store.cc:276] 0709 12:22:41.451285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.718047 (+266762us) service_pool.cc:224] Handling call
0709 12:22:41.785370 (+ 67323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.281783 (+1496413us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b18000ddda0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166afbfb 00007f6c166ba64b
0709 12:22:43.293974 (+ 12191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.294013 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.302886 (+  8873us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:22:43.303543 (+   657us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.618072 (+314529us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:43.618135 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.618243 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.874512 (+256269us) raft_consensus.cc:1685] finished
0709 12:22:43.874537 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.874700 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63685,"raft.queue_time_us":148750,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":6819,"spinlock_wait_cycles":28134016,"thread_start_us":264160,"threads_started":2,"wal-append.queue_time_us":421910}
I20260709 12:22:43.890021 21287 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.850867 21389 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:43.893471 21353 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:43.907931 21384 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:43.897214 21375 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.915925 18023 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:43.916625 21427 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.917213 21421 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.936585 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65549.5us
I20260709 12:22:43.920671 21420 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:43.965801 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:43.974597 18812 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:43.899632 17921 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 4, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 4 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:43.940584 21287 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:43.891225 21029 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.945363 21468 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.012089 17921 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:43.903044 21458 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:43.901494 21292 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:44.009833 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57465.4us
I20260709 12:22:44.047075 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:44.021028 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 292256us
I20260709 12:22:44.022228 18812 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:44.050155 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:43.953161 21427 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:44.024151 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.010329 21177 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.057493 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.008915 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1957) took 1288 ms. Trace:
I20260709 12:22:43.967434 21421 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.013756 21363 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20260709 12:22:44.022917 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 246396us
I20260709 12:22:44.060000 20960 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:44.022907 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 635498us
I20260709 12:22:44.060824 18092 rpcz_store.cc:276] 0709 12:22:42.720545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.080972 (+360427us) service_pool.cc:224] Handling call
0709 12:22:43.081372 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.081779 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.081811 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.082507 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.575841 (+493334us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:43.575923 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.576076 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.979444 (+403368us) raft_consensus.cc:1685] finished
0709 12:22:43.979489 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.008814 (+ 29325us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2106,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":101,"thread_start_us":492750,"threads_started":1,"wal-append.queue_time_us":588784}
I20260709 12:22:44.045398 21029 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:44.071599 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:44.072417 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.073529 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:44.074388 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.076894 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:44.015035 21285 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:44.080334 18815 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:44.061589 21298 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:44.064853 21487 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:22:44.077874 21356 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:44.091110 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.091960 18812 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:44.034211 17921 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:44.092936 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.093632 18814 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:44.059873 21420 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.094343 17921 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:44.078202 21177 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:44.099958 21177 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:44.100420 21421 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:44.100751 21458 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:44.096158 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 357359us
I20260709 12:22:44.082607 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:44.064829 20960 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.109942 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.078054 21359 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:44.111404 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:44.112529 18807 raft_consensus.cc:2376] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10. Re-sending same reply.
I20260709 12:22:44.091818 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:44.079975 21463 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.077003 20974 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:44.109458 18806 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:44.054630 21464 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:44.130821 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.112680 17921 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 6, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 6 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:44.106702 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.117547 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.120182 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.135663 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.120229 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:44.136230 18168 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:44.136690 18171 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:22:44.120263 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.137722 18165 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:44.136761 18810 raft_consensus.cc:2376] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 3. Re-sending same reply.
I20260709 12:22:44.078279 21490 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:22:44.092083 21493 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:44.140444 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.095830 21356 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:44.142593 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.107151 21420 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.146430 21420 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:44.147490 21292 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:44.131270 21384 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:44.150215 21389 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.150638 21273 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.106600 21438 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.151465 21138 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:44.151957 21356 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:44.157635 21494 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:44.120636 20960 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.158856 21375 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:44.160565 20960 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.101107 21177 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.141163 21346 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.117461 21492 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:44.166760 21151 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.097764 21298 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:44.168413 21298 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.176216 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.179392 18814 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:44.174793 18813 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
W20260709 12:22:44.181727 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 373479us
I20260709 12:22:44.185624 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.185711 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.186745 18815 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:44.188505 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.189527 18090 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
W20260709 12:22:44.193401 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51187.9us
W20260709 12:22:44.196861 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44156.6us
I20260709 12:22:44.133167 17921 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:44.203066 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:44.214105 17921 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 3, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:44.216631 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.217103 21427 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:44.182127 21464 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.218151 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:44.218317 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.222122 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.230515 18018 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:44.223965 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.224462 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.225868 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.236011 18172 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:44.238842 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.239068 18023 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:44.239661 18172 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:44.240119 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:44.243121 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:44.247023 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.248672 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.249382 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:44.250342 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.250979 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:44.251132 18020 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:44.256443 18172 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:44.255753 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:44.259164 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:44.261198 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:44.261885 18806 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:44.262305 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.262710 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.263700 18173 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:44.263895 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:44.217770 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.267189 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.267954 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:44.269619 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.270915 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.271957 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.232699 21468 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.278074 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.282240 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.277019 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:22:44.229027 17921 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:44.261775 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.289918 17921 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:44.260782 21427 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:44.263223 21346 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:44.292788 18092 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:44.293061 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:44.293787 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.310316 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:44.310703 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:44.294121 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.294214 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.311914 18020 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:44.293897 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:44.275153 21436 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:44.301225 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.287717 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:44.275779 21438 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:44.313334 18022 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:44.320582 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:44.322657 18099 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.017s	sys 0.037s
I20260709 12:22:44.324065 21359 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:44.324792 21359 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:44.322939 21420 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:44.322700 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71827.7us
I20260709 12:22:44.326035 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:44.326889 21420 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:44.327353 21420 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:44.327517 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:44.327874 21420 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:44.331247 21036 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:44.333207 21036 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:44.336036 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.326124 21359 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.333690 21036 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.347163 18806 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:44.350517 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:44.360230 18812 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:44.364866 18810 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:44.371901 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.266876 20960 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:22:44.290354 18809 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:44.312901 21427 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.312144 21363 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.372825 18092 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:44.312644 21346 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:44.411413 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:44.417919 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.362150 21389 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:44.436061 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.425387 18811 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:44.433113 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.438704 18165 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:44.433959 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.363317 21438 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:44.439893 18168 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:44.311523 21285 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260709 12:22:44.291723 21468 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.369340 21036 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:44.311080 21251 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.098s
I20260709 12:22:44.459419 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.460021 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.457525 18172 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:44.511385 18173 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:44.389427 21364 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:44.460333 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.517083 18166 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:44.458920 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:44.519300 18168 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:44.548434 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.422379 21427 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:44.536188 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.434808 21346 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:44.561292 18166 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:44.562873 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:44.461158 20960 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:44.564320 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45475.8us
I20260709 12:22:44.447713 21463 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.499315 21499 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.567580 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.420357 21138 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:44.447919 21438 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:44.578061 18092 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:44.559026 21389 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:44.549793 21054 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:44.608398 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.460863 20974 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20260709 12:22:44.629760 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.638897 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:44.579344 21138 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:44.650422 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.003s	sys 0.011s
I20260709 12:22:44.578316 20960 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:44.660624 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:44.661125 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.003s	sys 0.012s
I20260709 12:22:44.665649 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:44.670145 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 561048us
I20260709 12:22:44.678874 18810 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:22:44.611207 21054 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:44.692301 18812 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:44.376224 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:44.697223 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 321101us
I20260709 12:22:44.578363 21463 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:44.704586 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 377126us
I20260709 12:22:44.705080 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:44.638011 21501 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.601019 21029 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:22:44.574479 21499 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.693739 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.602984 21461 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 12:22:44.706238 18811 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:44.667577 21138 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:44.726686 18806 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:22:44.746124 18806 pending_rounds.cc:85] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 7
I20260709 12:22:44.746510 18806 pending_rounds.cc:107] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:22:44.746824 18806 raft_consensus.cc:2889] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:44.746863 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 378880us
I20260709 12:22:44.747804 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:44.720041 21492 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.004s	sys 0.022s
I20260709 12:22:44.718777 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:44.755829 18092 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:44.728744 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 398413us
I20260709 12:22:44.748378 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:44.749194 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.750633 18168 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:44.749215 21436 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:44.766171 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47566.5us
I20260709 12:22:44.766448 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:44.767009 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:44.767108 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:44.770793 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.764575 21436 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.762656 18811 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEARNER]: Advancing to term 9
I20260709 12:22:44.795136 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:44.797098 18809 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:44.719058 21177 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.734234 21481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.011s	sys 0.030s
W20260709 12:22:44.760066 21492 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.004s	sys 0.023s
W20260709 12:22:44.841176 21514 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:22:44.841503 21468 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:44.841558 21492 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:44.842487 21468 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:44.842911 21468 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.843281 21492 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.844786 21499 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:44.789742 21508 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.216s	user 0.002s	sys 0.000s
W20260709 12:22:44.817289 18780 connection.cc:256] RPC call timeout handler was delayed by -1.4862205340004948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
W20260709 12:22:44.840504 21506 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
I20260709 12:22:44.840921 21497 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:44.846439 21481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.012s	sys 0.030s
W20260709 12:22:44.853739 18780 connection.cc:256] RPC call timeout handler was delayed by -1.4862205340004948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
I20260709 12:22:44.860368 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2783) took 1099 ms. Trace:
I20260709 12:22:44.843281 21468 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:44.875919 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 278905us
I20260709 12:22:44.837795 21029 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:44.850359 21498 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:44.858697 21507 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
W20260709 12:22:44.888015 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3854205340003318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
I20260709 12:22:44.856609 21364 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:44.895080 18780 connection.cc:256] RPC call timeout handler was delayed by -1.0785205340002904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2523s for a response
I20260709 12:22:44.733311 21501 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:44.898741 18780 connection.cc:256] RPC call timeout handler was delayed by -1.0785205340002904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2523s for a response
I20260709 12:22:44.846761 21496 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
W20260709 12:22:44.888800 21359 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.020s
I20260709 12:22:44.859242 21497 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:44.907702 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:44.881398 21029 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.851841 21177 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.862525 20974 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
W20260709 12:22:44.909390 21359 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.000s	sys 0.021s
W20260709 12:22:44.911960 21353 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.945s	user 0.016s	sys 0.000s
I20260709 12:22:44.862421 20960 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.905984 21498 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.906951 21509 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:44.913115 21497 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:44.914860 21029 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.862977 21054 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.937883 18811 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:44.917070 21526 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:44.909406 21513 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:22:44.951964 21509 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.957197 21509 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:44.959569 21420 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.631s	user 0.016s	sys 0.000s
I20260709 12:22:44.960142 21420 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.960029 21458 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:44.967832 21536 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:22:44.981192 21532 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:44.981716 21532 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:44.982432 21420 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:44.982744 21532 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:44.984301 21532 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.986881 21501 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:44.987439 21363 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.526s	user 0.012s	sys 0.001s
I20260709 12:22:44.988180 21363 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:44.988458 21520 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
W20260709 12:22:44.989485 21512 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
W20260709 12:22:44.970216 21356 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.817s	user 0.016s	sys 0.000s
I20260709 12:22:44.968868 21509 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:44.947368 21353 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:44.973519 21458 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:44.999073 21463 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:44.997723 21521 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:44.998803 21509 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:45.027051 21517 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:22:45.028880 21527 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:22:45.022892 21509 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.020874 21521 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:45.037055 21521 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:45.038955 21521 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:45.078233 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.002s	sys 0.017s
W20260709 12:22:45.079648 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.002s	sys 0.017s
W20260709 12:22:45.082996 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 646897us
W20260709 12:22:45.086205 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 387518us
I20260709 12:22:45.087983 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:45.088802 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:45.096355 18062 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.010s	sys 0.025s
W20260709 12:22:45.097198 18062 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.010s	sys 0.025s
W20260709 12:22:45.107160 18090 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.001s	sys 0.005s
W20260709 12:22:45.108006 18090 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.002s	sys 0.005s
I20260709 12:22:45.114527 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:44.876873 18094 rpcz_store.cc:276] 0709 12:22:43.760821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.929100 (+168279us) service_pool.cc:224] Handling call
0709 12:22:43.929478 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.931724 (+  2246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.931782 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.932680 (+   898us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.265053 (+332373us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:44.265125 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.265225 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.860011 (+594786us) raft_consensus.cc:1685] finished
0709 12:22:44.860046 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.860274 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6390,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":8494,"thread_start_us":331836,"threads_started":1,"wal-append.queue_time_us":595398}
I20260709 12:22:45.122215 18099 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:45.123178 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2785) took 1358 ms. Trace:
I20260709 12:22:45.125460 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2784) took 1364 ms. Trace:
I20260709 12:22:45.126281 18095 rpcz_store.cc:276] 0709 12:22:43.761144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.931601 (+170457us) service_pool.cc:224] Handling call
0709 12:22:43.932734 (+  1133us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.939655 (+  6921us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.939692 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.944334 (+  4642us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.153154 (+208820us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:22:44.186820 (+ 33666us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:44.186871 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.186947 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.124463 (+937516us) raft_consensus.cc:1685] finished
0709 12:22:45.124534 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.125145 (+   611us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43381,"raft.queue_time_us":8291,"raft.run_cpu_time_us":3260,"raft.run_wall_time_us":208126,"spinlock_wait_cycles":10322688,"thread_start_us":175178,"threads_started":1,"wal-append.queue_time_us":528188}
I20260709 12:22:45.123860 18097 rpcz_store.cc:276] 0709 12:22:43.764907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.933843 (+168936us) service_pool.cc:224] Handling call
0709 12:22:43.934281 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.939684 (+  5403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.939719 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.940594 (+   875us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.263404 (+322810us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:44.263558 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.263678 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.122482 (+858804us) raft_consensus.cc:1685] finished
0709 12:22:45.122524 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.122899 (+   375us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4988,"raft.queue_time_us":32189,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":5915,"thread_start_us":320311,"threads_started":1,"wal-append.queue_time_us":639843}
I20260709 12:22:45.142946 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:45.144822 18097 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:45.149694 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:45.151125 18095 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:22:45.154659 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 388090us
I20260709 12:22:45.164335 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:45.172931 21036 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.623s	user 0.018s	sys 0.004s
I20260709 12:22:44.975513 21054 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.174515 21233 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:45.180008 21233 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.180294 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:45.180819 21233 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.181478 21233 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.183151 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:45.183838 21177 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:45.184585 21273 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:45.186141 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:45.184698 21177 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:45.000429 21356 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:45.190518 18808 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:45.193193 21356 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:45.198391 21364 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:45.202461 18098 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:45.204252 21375 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:45.204901 21359 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:45.205075 21375 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:45.205490 21359 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.206502 21359 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.207275 21375 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:45.201222 21353 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:45.187171 21177 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:45.197222 21363 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.009s
W20260709 12:22:45.256953 21438 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.006s	sys 0.013s
I20260709 12:22:45.257433 21363 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:45.230916 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57210.6us
I20260709 12:22:45.261232 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:45.232016 21287 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.210s	user 0.018s	sys 0.003s
I20260709 12:22:45.241559 21458 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:45.262753 21287 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:45.264313 21464 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.010s	user 0.006s	sys 0.015s
I20260709 12:22:45.264911 18170 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:45.265017 21464 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:45.263350 21458 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.218956 21353 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:45.266563 21353 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:45.271570 21420 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:45.272233 21420 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.272689 21420 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:45.273307 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.274762 21287 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:45.276189 21287 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:45.276890 21287 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.280361 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:45.285015 21346 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.710s	user 0.009s	sys 0.010s
W20260709 12:22:45.293326 21138 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.012s
I20260709 12:22:45.296799 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:45.298368 21273 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:45.299034 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:45.300845 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:45.284392 21468 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:45.302441 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:45.329654 21177 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:45.386401 18022 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:22:45.387153 21138 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:45.400542 21533 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.390303 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 110117us
I20260709 12:22:45.390677 21522 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.398563 21375 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:45.325649 21287 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.293184 21251 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.085s
I20260709 12:22:45.375351 21273 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.384403 21285 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
W20260709 12:22:45.573544 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.466374 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 377864us
W20260709 12:22:45.576408 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 393428us
I20260709 12:22:45.257673 21438 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:45.578877 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 276596us
I20260709 12:22:45.435942 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 846) took 1171 ms. Trace:
W20260709 12:22:45.884445 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.133771 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 460) took 1889 ms (client timeout 2070 ms). Trace:
I20260709 12:22:46.131865 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 639) took 2031 ms. Trace:
I20260709 12:22:46.219170 18808 rpcz_store.cc:276] 0709 12:22:44.100004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.101403 (+  1399us) service_pool.cc:224] Handling call
0709 12:22:46.131466 (+2030063us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004a60d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:46.131703 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2423392128}
W20260709 12:22:46.216006 18098 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.012s	user 0.017s	sys 0.017s
I20260709 12:22:46.230822 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:22:45.641149 18171 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.007s	sys 0.005s
W20260709 12:22:45.640648 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.006s	sys 0.005s
W20260709 12:22:46.218991 18170 rpcz_store.cc:269] 0709 12:22:44.243770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.246624 (+  2854us) service_pool.cc:224] Handling call
0709 12:22:46.133328 (+1886704us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:46.133569 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2337851648}
W20260709 12:22:45.646214 18065 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.011s	sys 0.000s
W20260709 12:22:45.659129 21389 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.039s	user 0.012s	sys 0.010s
W20260709 12:22:45.881006 21427 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.306s	user 0.015s	sys 0.007s
I20260709 12:22:45.173652 21036 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.223284 21458 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:46.208674 18815 rpcz_store.cc:276] 0709 12:22:44.232602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.232760 (+   158us) service_pool.cc:224] Handling call
0709 12:22:44.269134 (+ 36374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.269660 (+   526us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.269710 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.270445 (+   735us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.788973 (+518528us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:44.789025 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.789112 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.403438 (+614326us) raft_consensus.cc:1685] finished
0709 12:22:45.403494 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.403743 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":595022,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":19380,"thread_start_us":424386,"threads_started":1,"wal-append.queue_time_us":1105729}
W20260709 12:22:46.206755 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.09252e+06us
I20260709 12:22:45.759999 21384 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:45.574038 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.001s	sys 0.010s
I20260709 12:22:45.447773 21353 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.087261 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2777) took 2328 ms. Trace:
I20260709 12:22:45.926417 21543 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:45.783234 21468 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:45.758635 21502 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:46.226356 21503 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.724s	user 0.004s	sys 0.011s
I20260709 12:22:45.756850 21511 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.644731 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.006s	sys 0.004s
W20260709 12:22:45.645994 18024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.003s	sys 0.014s
W20260709 12:22:45.650444 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 442842us
W20260709 12:22:45.648684 17921 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.001s	sys 0.016s
W20260709 12:22:45.763576 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.209234 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.574347 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.001s	sys 0.010s
W20260709 12:22:45.705546 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51775.8us
W20260709 12:22:46.240469 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.479s	user 0.018s	sys 0.045s
W20260709 12:22:45.926285 21054 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.623s	user 0.012s	sys 0.005s
W20260709 12:22:46.205936 21548 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.318s	user 0.001s	sys 0.001s
W20260709 12:22:45.924479 21233 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.742s	user 0.009s	sys 0.005s
W20260709 12:22:46.243582 21498 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.011s	sys 0.002s
I20260709 12:22:45.741513 21533 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:45.649960 21298 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.005s	sys 0.005s
W20260709 12:22:46.313953 21298 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.610s	user 0.016s	sys 0.006s
W20260709 12:22:45.887017 18096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.919s	user 0.000s	sys 0.012s
W20260709 12:22:46.203835 21540 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.811s	user 0.000s	sys 0.004s
W20260709 12:22:46.245458 18065 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.458s	user 0.023s	sys 0.011s
W20260709 12:22:46.202584 21535 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.809s	user 0.003s	sys 0.000s
W20260709 12:22:46.247519 18171 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.481s	user 0.030s	sys 0.033s
I20260709 12:22:45.871981 21526 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:45.922415 21029 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.744s	user 0.017s	sys 0.001s
W20260709 12:22:45.827574 21364 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.012s	sys 0.018s
I20260709 12:22:45.643981 20960 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.604s
I20260709 12:22:45.577217 21375 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:45.391326 21346 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.324996 18096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.536s	user 0.005s	sys 0.022s
W20260709 12:22:45.884891 21497 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.930s	user 0.013s	sys 0.002s
I20260709 12:22:46.247594 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:45.738631 21534 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:45.710614 21285 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.280519 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:46.276852 21427 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.281554 21138 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:46.281773 21421 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.374s	user 0.011s	sys 0.055s
W20260709 12:22:46.283847 21521 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.244s	user 0.011s	sys 0.008s
I20260709 12:22:46.028111 21522 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:45.666524 21539 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.383523 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2787) took 2086 ms. Trace:
I20260709 12:22:46.287006 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1982) took 1971 ms. Trace:
W20260709 12:22:46.288635 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.521s	user 0.002s	sys 0.026s
I20260709 12:22:45.655019 21359 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.275645 21384 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:46.280663 21389 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:46.379395 21421 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.988s	user 0.012s	sys 0.055s
W20260709 12:22:46.302944 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.535s	user 0.005s	sys 0.024s
I20260709 12:22:46.303261 18167 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260709 12:22:46.309229 18024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.594s	user 0.014s	sys 0.022s
W20260709 12:22:46.306633 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.579s	user 0.006s	sys 0.020s
W20260709 12:22:46.302639 17921 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.576s	user 0.008s	sys 0.021s
I20260709 12:22:46.290223 21458 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:46.295002 21490 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.360203 21564 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:46.304863 21503 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.838s	user 0.004s	sys 0.011s
W20260709 12:22:46.311946 21498 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.011s	sys 0.002s
I20260709 12:22:46.297737 21502 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:22:46.318053 21364 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.118s	user 0.028s	sys 0.041s
I20260709 12:22:46.327253 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.329073 21534 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.290975 18091 rpcz_store.cc:276] 0709 12:22:43.758330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.758517 (+   187us) service_pool.cc:224] Handling call
0709 12:22:43.758955 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.904693 (+145738us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:44.295407 (+390714us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.116675 (+821268us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:45.116848 (+   173us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:45.117027 (+   179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.086537 (+969510us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:46.086623 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.086931 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1534507,"prepare.queue_time_us":696196,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":65,"raft.queue_time_us":148173,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"spinlock_wait_cycles":334465920,"thread_start_us":1196402,"threads_started":3,"wal-append.queue_time_us":662478,"wal-append.run_cpu_time_us":1457,"wal-append.run_wall_time_us":1058393}
W20260709 12:22:46.325103 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.56398e+06us
W20260709 12:22:46.364596 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84138.4us
W20260709 12:22:46.323572 21523 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.700s	user 0.003s	sys 0.000s
W20260709 12:22:46.374195 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 132675us
I20260709 12:22:46.363691 21298 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:46.383023 21541 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.016s	sys 0.009s
W20260709 12:22:46.383229 21557 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.462s	user 0.002s	sys 0.020s
I20260709 12:22:46.314713 21233 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.384121 21539 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.372538 21509 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.198s	user 0.019s	sys 0.000s
I20260709 12:22:46.299090 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2095) took 1647 ms. Trace:
I20260709 12:22:46.384114 18096 rpcz_store.cc:276] 0709 12:22:44.297014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.297259 (+   245us) service_pool.cc:224] Handling call
0709 12:22:44.297777 (+   518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:44.298189 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:44.788323 (+490134us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.375904 (+1587581us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.382970 (+  7066us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:46.383089 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.383228 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.383418 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1469159,"prepare.queue_time_us":344717,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":51,"raft.queue_time_us":10150,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":14336,"thread_start_us":1985396,"threads_started":2,"wal-append.queue_time_us":1130691}
I20260709 12:22:46.303244 21511 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.401859 21346 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.409531 21490 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.410938 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.412168 21538 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:22:46.413215 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.64614e+06us
I20260709 12:22:46.414144 21533 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:46.415131 21552 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.495s	user 0.023s	sys 0.017s
I20260709 12:22:46.415205 21427 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.378997 21522 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:46.426019 21036 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:46.412325 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.64514e+06us
I20260709 12:22:46.380402 21285 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.384516 18090 rpcz_store.cc:276] 0709 12:22:44.315901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.114079 (+798178us) service_pool.cc:224] Handling call
0709 12:22:45.114834 (+   755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.115642 (+   808us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:46.138153 (+1022511us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.284713 (+146560us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.284785 (+    72us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:46.284845 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.284931 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.286916 (+  1985us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1021543,"prepare.queue_time_us":1064438,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":58,"raft.queue_time_us":40340,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":16007,"thread_start_us":106040,"threads_started":1}
I20260709 12:22:46.444519 21534 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:46.517506 21560 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:22:46.518474 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2789) took 2213 ms. Trace:
I20260709 12:22:46.519755 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:46.329262 21029 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.522148 21298 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:46.522555 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:46.523871 21029 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.524883 21547 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:22:46.447767 21541 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.017s	sys 0.010s
I20260709 12:22:46.328303 21543 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.525101 21029 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:46.528069 21543 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.482409 18097 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:22:46.531281 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:46.532240 18808 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 3.
I20260709 12:22:46.533061 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:46.534240 21533 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:46.510202 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2175) took 2137 ms. Trace:
I20260709 12:22:46.534794 21533 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.512117 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:46.535346 21533 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:46.536136 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:46.534987 18171 rpcz_store.cc:276] 0709 12:22:44.373179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.373384 (+   205us) service_pool.cc:224] Handling call
0709 12:22:44.433690 (+ 60306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:44.433983 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:44.765895 (+331912us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.470562 (+1704667us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.499806 (+ 29244us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:46.503783 (+  3977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.503913 (+   130us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.510063 (+  6150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":477619,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":3430,"raft.queue_time_us":2126,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"spinlock_wait_cycles":2387200,"thread_start_us":1994005,"threads_started":3,"wal-append.queue_time_us":1534832}
I20260709 12:22:46.538033 21353 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.538949 18814 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:46.538864 21353 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.539004 21534 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.541680 21534 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.519004 18092 rpcz_store.cc:276] 0709 12:22:44.304783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.881949 (+577166us) service_pool.cc:224] Handling call
0709 12:22:44.882536 (+   587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.118725 (+236189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.118801 (+    76us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:45.120277 (+  1476us) log.cc:844] Serialized 26 byte log entry
0709 12:22:45.576155 (+455878us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:45.576209 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.646121 (+ 69912us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54002968d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:45.646210 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.284688 (+638478us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:46.517951 (+233263us) raft_consensus.cc:1685] finished
0709 12:22:46.517977 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.518216 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":125363,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":330,"spinlock_wait_cycles":617688704,"thread_start_us":331471,"threads_started":1,"wal-append.queue_time_us":799822}
I20260709 12:22:46.547079 18090 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.512609 21490 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.491289 21375 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.515007 21285 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:46.549031 21569 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
W20260709 12:22:46.431025 21565 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.366s	user 0.001s	sys 0.020s
I20260709 12:22:46.416234 21438 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.362633 21497 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:46.550282 21543 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.533226 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:46.519506 21551 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:46.541496 21353 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.416558 21526 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.402558 21458 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.500348 21420 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.223s	user 0.021s	sys 0.000s
I20260709 12:22:46.471086 18015 rpcz_store.cc:276] 0709 12:22:44.227532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.227673 (+   141us) service_pool.cc:224] Handling call
0709 12:22:44.264417 (+ 36744us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.264783 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.264813 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.265416 (+   603us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.759656 (+494240us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:45.551439 (+791783us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.788972 (+237533us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.816312 (+ 27340us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.824009 (+  7697us) raft_consensus.cc:1685] finished
0709 12:22:45.824033 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.874701 (+ 50668us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":617358,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":105,"raft.queue_time_us":222190,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":12439,"thread_start_us":826022,"threads_started":2,"wal-append.queue_time_us":628047}
W20260709 12:22:46.506752 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2786) took 2185 ms (client timeout 2091 ms). Trace:
I20260709 12:22:46.525923 18094 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:22:46.559068 18099 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:46.525418 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:46.563122 18098 rpcz_store.cc:269] 0709 12:22:44.292239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.292457 (+   218us) service_pool.cc:224] Handling call
0709 12:22:46.469227 (+2176770us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b5400214154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:46.477032 (+  7805us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2051643136}
W20260709 12:22:46.556373 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 155527us
W20260709 12:22:46.563562 21273 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.913s	user 0.012s	sys 0.004s
I20260709 12:22:46.582913 18097 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:22:46.583297 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71416.2us
I20260709 12:22:46.525504 21547 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:46.597906 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 152908us
I20260709 12:22:46.485219 21564 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.605310 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:46.606333 18809 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:46.610939 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:46.612293 21556 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.613101 21556 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.614871 21573 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:22:46.616714 21556 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.617540 21556 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:46.618151 21556 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.618589 21359 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.620623 21556 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.621395 21556 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:46.621770 21556 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.622215 21556 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:46.622597 18812 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 721) took 1934 ms (client timeout 1846 ms). Trace:
W20260709 12:22:46.623252 18812 rpcz_store.cc:269] 0709 12:22:44.688499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.693407 (+  4908us) service_pool.cc:224] Handling call
0709 12:22:46.622088 (+1928681us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:46.622338 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4269047168}
I20260709 12:22:46.484525 21502 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:46.627105 21554 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.232s	user 0.002s	sys 0.000s
I20260709 12:22:46.630888 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2174) took 2258 ms. Trace:
I20260709 12:22:46.631415 18167 rpcz_store.cc:276] 0709 12:22:44.372838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.372982 (+   144us) service_pool.cc:224] Handling call
0709 12:22:44.434746 (+ 61764us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.409785 (+1975039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.630520 (+220735us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b540029dbd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:46.630809 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4262957824}
W20260709 12:22:46.632810 21571 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.245s	user 0.002s	sys 0.000s
I20260709 12:22:46.638679 21363 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:46.639186 21363 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:46.639251 21526 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.500743 21509 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.641393 21558 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.257s	user 0.000s	sys 0.002s
I20260709 12:22:46.641870 21509 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.643110 21364 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:46.643831 21568 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:22:46.644747 21562 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 12:22:46.650489 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 844) took 2532 ms. Trace:
I20260709 12:22:46.650699 21522 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.652576 21522 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:46.651353 18807 rpcz_store.cc:276] 0709 12:22:44.118334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.204858 (+ 86524us) service_pool.cc:224] Handling call
0709 12:22:44.499654 (+294796us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:22:44.499912 (+   258us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:44.912221 (+412309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.912256 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:44.926422 (+ 14166us) log.cc:844] Serialized 52 byte log entry
0709 12:22:45.641652 (+715230us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:45.641779 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.647959 (+  6180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.640419 (+992460us) spinlock_profiling.cc:243] Waited 7.73 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:46.645773 (+  5354us) raft_consensus.cc:1685] finished
0709 12:22:46.645799 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.650161 (+  4362us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4209,"raft.queue_time_us":377387,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":13364,"spinlock_wait_cycles":49297024,"thread_start_us":851737,"threads_started":2,"wal-append.queue_time_us":1338868}
I20260709 12:22:46.654402 21359 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.655299 21359 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:46.656847 21532 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.672s	user 0.016s	sys 0.006s
I20260709 12:22:46.658141 21498 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:46.658655 21497 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:46.658861 21559 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.729s	user 0.005s	sys 0.011s
I20260709 12:22:46.659680 21461 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:46.564406 21420 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.663045 21233 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.572919 21468 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.645349 21364 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.665807 21468 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.666129 21364 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.667814 21564 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.668357 21468 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.603549 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2094) took 2367 ms. Trace:
I20260709 12:22:46.668963 21364 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:46.670444 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91358.9us
I20260709 12:22:46.656955 21375 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.627100 21502 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:46.605326 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:46.691535 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78116us
I20260709 12:22:46.699177 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:46.701087 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.704504 18097 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:46.658710 21498 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.747846 21359 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:46.747184 18810 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.003s	sys 0.014s
W20260709 12:22:46.752321 18810 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.003s	sys 0.014s
W20260709 12:22:46.754931 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.749617 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1983) took 2383 ms. Trace:
I20260709 12:22:46.505708 21384 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.758849 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.588146 21543 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.760643 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79059.8us
I20260709 12:22:46.682160 21511 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.767277 18814 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:46.669219 18024 rpcz_store.cc:276] 0709 12:22:44.225394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.226554 (+  1160us) service_pool.cc:224] Handling call
0709 12:22:44.262766 (+ 36212us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.264612 (+  1846us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:44.714845 (+450233us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.201569 (+486724us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:22:46.464033 (+1262464us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.514119 (+ 50086us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.514329 (+   210us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:46.514372 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.514504 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.553774 (+ 39270us) raft_consensus.cc:1685] finished
0709 12:22:46.562831 (+  9057us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.592645 (+ 29814us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16360,"prepare.queue_time_us":1126300,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":1074,"raft.queue_time_us":28264,"raft.run_cpu_time_us":14001,"raft.run_wall_time_us":951869,"spinlock_wait_cycles":12085632,"thread_start_us":2137510,"threads_started":2,"wal-append.queue_time_us":1100095}
I20260709 12:22:46.757776 18094 rpcz_store.cc:276] 0709 12:22:44.365835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.122901 (+757066us) service_pool.cc:224] Handling call
0709 12:22:45.123977 (+  1076us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.573129 (+1449152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.749155 (+176026us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002cae54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:46.749513 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2926709504}
I20260709 12:22:46.763331 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:46.765895 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.592015 21273 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:46.799547 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.799998 21273 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.800987 18812 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:46.801963 21438 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:46.770934 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:46.762696 21420 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:46.756728 21498 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:46.766589 21287 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.545s	user 0.015s	sys 0.000s
I20260709 12:22:46.818785 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:46.828841 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:46.834589 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:46.836346 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.844806 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:46.846295 18811 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:46.780021 21364 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:46.850276 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.852639 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:46.827338 21384 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.859503 18090 raft_consensus.cc:2376] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8. Re-sending same reply.
I20260709 12:22:46.869279 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.831535 21511 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:46.870674 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 345418us
W20260709 12:22:46.872845 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 267719us
W20260709 12:22:46.873778 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 173728us
I20260709 12:22:46.883409 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:46.855793 21233 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:46.890053 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106584us
I20260709 12:22:46.832679 21287 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.892194 21346 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:46.892807 21493 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.904721 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:46.898689 21346 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.910223 21346 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:46.910995 21363 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.912672 21363 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.902456 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:46.908164 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71908.4us
I20260709 12:22:46.918233 18812 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:46.906801 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:46.918658 18811 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:22:46.922719 18809 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 12:22:46.919147 21532 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:46.924516 21575 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.928113 18095 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:46.929229 18097 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:46.932756 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.884768 21539 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:46.935483 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:46.894232 21509 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:22:46.942010 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.921965 21493 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.949185 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:46.950070 21577 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:22:46.960553 21298 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.968608 21298 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.970451 21493 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.953239 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:46.971170 21298 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:46.953365 21036 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:46.972465 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:46.972337 21036 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.973277 18167 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:46.954135 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:46.953380 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:46.946487 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:46.975895 18170 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:22:46.959838 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:46.976591 21526 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:46.977137 21526 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:46.977174 18094 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:46.977721 21526 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:46.979007 21576 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:22:46.983927 21538 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:46.987576 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:46.990310 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:46.991179 18020 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:46.991276 18097 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:46.962838 21389 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:46.946630 21564 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:46.951504 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:46.999357 18098 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:46.966710 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:46.952210 21359 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:46.962347 18095 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:46.943303 21438 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:46.908561 21511 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.989637 21526 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.926000 21233 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:46.975135 21036 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:47.002355 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58315.8us
I20260709 12:22:47.003780 18099 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:22:47.005409 18090 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:47.007571 21493 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.007294 21420 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:47.007772 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 717) took 2325 ms. Trace:
I20260709 12:22:47.008596 18168 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:47.003499 21359 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.888454 21547 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:46.938902 21427 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:22:46.925251 21575 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.004304 21438 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.042376 18099 pending_rounds.cc:85] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 5
I20260709 12:22:46.948702 21539 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20260709 12:22:47.067087 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:47.033577 21564 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:47.068789 18015 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:47.069680 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:47.076215 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 182132us
I20260709 12:22:47.073553 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:47.070161 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:47.082834 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:47.067127 18099 pending_rounds.cc:107] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:22:47.084677 18099 raft_consensus.cc:2889] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:47.057336 21359 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:47.089530 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:47.091578 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 142600us
I20260709 12:22:47.069048 21438 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.051134 18810 rpcz_store.cc:276] 0709 12:22:44.682699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.685019 (+  2320us) service_pool.cc:224] Handling call
0709 12:22:44.685614 (+   595us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.886357 (+1200743us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b18003cd660. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166afbfb 00007f6c166ba64b
0709 12:22:45.886659 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.886691 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:45.887666 (+   975us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.752919 (+865253us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:46.752976 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.761148 (+  8172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.998599 (+237451us) raft_consensus.cc:1685] finished
0709 12:22:46.998641 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.007680 (+  9039us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":112,"raft.queue_time_us":223634,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":2442,"spinlock_wait_cycles":33684224,"thread_start_us":1097845,"threads_started":2,"wal-append.queue_time_us":713458}
I20260709 12:22:47.094617 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:47.096452 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:47.073316 21543 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:47.106668 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.096511 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:47.107976 18167 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:47.051786 21389 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.106489 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.116886 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:47.118106 18098 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:47.115912 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:47.120631 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:47.121342 21364 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:47.121891 21364 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:47.065755 21493 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.122627 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:47.123318 21364 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.124507 21493 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:47.077083 21562 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:22:47.124965 21364 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.125885 21582 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:47.126413 21582 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.126811 21582 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.131613 18168 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:47.132042 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:47.135730 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:47.106546 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.138056 18809 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:22:47.139079 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.139915 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:47.141211 21029 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:47.141685 21029 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:47.142135 21287 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:47.142481 21556 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.520s	user 0.014s	sys 0.000s
I20260709 12:22:47.143028 21556 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.142802 21029 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:47.143828 21499 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:47.144436 21029 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:47.145435 21556 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:47.146257 18813 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:47.146874 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:47.147166 21458 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:22:47.151721 17988 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.601s (SENT)
W20260709 12:22:47.152814 21581 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:22:47.142179 21490 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:47.154413 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.146736 21588 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:47.139261 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 104822us
I20260709 12:22:47.121986 21420 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:47.121665 21511 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:47.155617 21526 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:47.125393 21562 consensus_queue.cc:1243] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:47.154336 21346 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.156457 21543 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:47.182024 21427 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:47.157071 21539 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:47.157554 21499 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:47.157605 21564 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.162261 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:47.166337 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:47.120805 18021 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.166587 21233 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:47.173761 18171 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:47.179069 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:47.178979 18020 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:47.180513 18094 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:47.122567 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.114591 21534 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:47.214911 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:47.215324 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 118903us
I20260709 12:22:47.218528 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:47.162544 21389 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:47.279016 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:47.221478 18137 leader_election.cc:336] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.846s (SENT)
I20260709 12:22:47.280329 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:47.223263 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:47.121894 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.178498 21490 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.281952 18019 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:47.230958 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.199242 21503 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:47.227866 21502 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.537s	user 0.011s	sys 0.004s
I20260709 12:22:47.291101 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:47.292021 18811 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:47.240738 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.251125 18170 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:47.200523 21273 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:47.223095 21420 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:47.252056 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:47.283313 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46392.4us
W20260709 12:22:47.286831 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 125409us
I20260709 12:22:47.296752 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:47.297627 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:47.232709 21427 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:47.314375 21273 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.300222 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:47.236462 21539 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:47.306813 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:47.311751 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:47.312022 21498 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.312614 18092 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:47.312420 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:47.288738 21490 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:47.313674 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:47.314271 21499 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.226593 21543 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:47.350935 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.283528 21534 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.291540 21497 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:47.361171 21497 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.361696 21497 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:47.343971 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50582.6us
I20260709 12:22:47.349917 18167 raft_consensus.cc:606] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:47.314337 21427 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:47.353946 21587 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:47.359150 21551 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:47.359913 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 208285us
I20260709 12:22:47.360306 21273 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:47.367703 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.637s (SENT)
W20260709 12:22:47.374305 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60778us
I20260709 12:22:47.378727 18021 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:47.321261 21539 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:47.383172 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 131011us
I20260709 12:22:47.384006 18809 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:22:47.338672 21503 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:47.354172 21543 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:47.375005 21375 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.685s	user 0.011s	sys 0.004s
I20260709 12:22:47.364329 21534 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:47.389986 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:47.378167 21533 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.842s	user 0.016s	sys 0.000s
W20260709 12:22:47.399278 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 184546us
W20260709 12:22:47.399326 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.381655 21575 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:47.374397 21502 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:47.378012 21551 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:47.395380 21522 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.742s	user 0.016s	sys 0.002s
I20260709 12:22:47.404379 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:47.399045 21287 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:47.402311 21389 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:47.440119 18169 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
W20260709 12:22:47.442559 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59324us
I20260709 12:22:47.406378 21511 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:47.447537 18807 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:47.421787 21273 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:22:47.449925 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45669.3us
W20260709 12:22:47.450860 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97343.9us
I20260709 12:22:47.449745 21287 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:47.453047 21375 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:47.454636 21287 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:47.460290 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.429123 21551 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:47.474164 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:47.443837 21389 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.414234 21533 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:47.491947 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.494514 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.447932 21511 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:22:47.503902 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55117.9us
W20260709 12:22:47.504258 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 123516us
I20260709 12:22:47.502820 21511 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:47.516171 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:47.517035 18814 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:47.516155 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 121492us
I20260709 12:22:47.519025 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:47.487681 21389 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:47.521346 18021 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.522053 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:47.486832 21273 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:47.497689 21346 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:47.544467 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 97122.8us
I20260709 12:22:47.545002 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:47.546370 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76619us
W20260709 12:22:47.546926 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.515564 21375 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:47.447748 21420 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.570703 18090 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.571882 18094 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.576918 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:47.581200 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:47.536509 21298 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:47.554877 21591 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:47.585615 18806 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:47.593772 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.598493 18018 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.598765 21490 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.584103 18092 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:47.607738 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:47.608111 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.609865 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:47.610525 21598 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:22:47.610967 18168 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:22:47.584404 21498 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:47.618666 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.641350 18170 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:47.645015 18167 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:22:47.605504 21375 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:47.505741 21503 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:47.600121 21590 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.254s	user 0.001s	sys 0.002s
I20260709 12:22:47.588379 21591 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.669274 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:47.684540 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:47.713409 18019 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:47.665458 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:47.666850 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:47.716957 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 151931us
W20260709 12:22:47.666320 21595 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
W20260709 12:22:47.612360 21592 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:22:47.661255 21273 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:47.713120 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 158761us
W20260709 12:22:47.672397 21582 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.000s
I20260709 12:22:47.707233 21591 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:47.715011 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:47.746774 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:47.747532 18018 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:47.743335 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:47.752089 18023 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:47.677546 21497 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.735620 21603 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.763404 18019 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:47.760314 21363 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.844s	user 0.015s	sys 0.003s
I20260709 12:22:47.782732 18015 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:47.791035 18023 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:22:47.780324 21359 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.003s
I20260709 12:22:47.798586 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:47.754587 21582 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.806734 18814 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:47.792812 21533 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:47.826853 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:47.852965 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 178452us
I20260709 12:22:47.808005 21359 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:47.786158 21603 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.835670 21556 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:47.874912 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.882431 21603 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:47.848915 21490 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:47.889106 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:47.889604 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.891245 21547 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:47.891785 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 100702us
I20260709 12:22:47.893687 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:47.902216 21029 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.757s	user 0.017s	sys 0.010s
I20260709 12:22:47.904412 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.914616 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:47.917438 18023 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:47.874303 21438 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.699s	user 0.012s	sys 0.003s
I20260709 12:22:47.905284 18170 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:47.920548 18098 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.866017 21556 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:47.929958 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:47.892482 21611 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.135s	user 0.001s	sys 0.001s
I20260709 12:22:47.930946 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:47.935473 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:47.937350 18168 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:47.918685 21577 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:22:47.951615 18099 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:47.968873 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:47.970528 18815 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:47.905855 21363 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:47.933163 21490 raft_consensus.cc:2764] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260709 12:22:47.983700 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:47.936652 21562 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.950076 21533 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:48.003260 21605 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
W20260709 12:22:48.004369 21614 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:22:47.959836 21497 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.010193 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.013375 21389 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:48.018707 21612 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:22:47.934369 21438 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:48.025094 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.031806 18167 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:48.034576 21359 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.037071 18019 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:48.039052 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
W20260709 12:22:48.047425 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.056665 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 101063us
I20260709 12:22:48.000504 21617 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:22:48.068181 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.072059 18023 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:48.075444 18095 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:48.011952 21562 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.025122 21587 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:22:48.074870 21623 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:48.086674 21582 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.096424 21526 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:48.080902 21298 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
W20260709 12:22:48.129535 21618 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
W20260709 12:22:48.129806 21539 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.724s	user 0.009s	sys 0.006s
I20260709 12:22:48.130604 21539 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.132122 21526 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:48.137342 21594 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.711s	user 0.009s	sys 0.000s
I20260709 12:22:48.114607 21623 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.142894 21438 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.114445 21497 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.147078 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 1995) took 1143 ms. Trace:
I20260709 12:22:48.150643 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.118131 21587 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:22:48.112952 21629 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.097s
I20260709 12:22:48.165779 18023 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:48.213593 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.331620 18097 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:48.331506 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.216854 18018 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:22:48.228302 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 124404us
I20260709 12:22:48.333155 18095 rpcz_store.cc:276] 0709 12:22:47.003753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.004044 (+   291us) service_pool.cc:224] Handling call
0709 12:22:47.033118 (+ 29074us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.075979 (+1042861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.146684 (+ 70705us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540035f2d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:48.146980 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2396365696}
I20260709 12:22:48.336017 21497 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.329504 21539 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.342242 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.153538 21526 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.344413 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:48.152019 21511 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.644s	user 0.013s	sys 0.000s
W20260709 12:22:48.191428 21543 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.748s	user 0.014s	sys 0.001s
W20260709 12:22:48.212180 21364 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.087s	user 0.017s	sys 0.003s
W20260709 12:22:48.166244 21551 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.669s	user 0.009s	sys 0.005s
W20260709 12:22:48.305112 21534 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.859s	user 0.010s	sys 0.005s
W20260709 12:22:48.230983 21427 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.834s	user 0.011s	sys 0.005s
I20260709 12:22:48.350348 21298 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:22:48.351125 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.351881 21503 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:48.201095 21036 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.350409 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.283365 21603 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.359051 18019 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:48.357059 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.359890 18094 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:48.365554 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.352070 21551 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.371933 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.375061 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.376436 18019 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:48.376128 18018 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:48.351925 21497 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.357226 21562 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:48.366650 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1888605439999083s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260709 12:22:48.385335 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1024605439997686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
W20260709 12:22:48.385737 18064 connection.cc:256] RPC call timeout handler was delayed by -1.957560543999989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18580000000000002s for a response
W20260709 12:22:48.386082 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8441605439998057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
W20260709 12:22:48.386498 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7640605440001309s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
I20260709 12:22:48.376128 21503 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:48.386816 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6740605439999854s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21730000000000002s for a response
W20260709 12:22:48.387172 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6065605439998762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
W20260709 12:22:48.387470 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3113605439998537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2576s for a response
I20260709 12:22:48.388782 21503 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.390859 18019 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:48.349987 21511 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.393379 21623 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:48.393692 18019 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:48.394173 18019 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:48.397389 18019 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.403791 18019 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:48.349120 21543 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.397993 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 471) took 1300 ms. Trace:
I20260709 12:22:48.407722 18169 rpcz_store.cc:276] 0709 12:22:47.097300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.122071 (+ 24771us) service_pool.cc:224] Handling call
0709 12:22:48.397714 (+1275643us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54003ba1d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:48.397921 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":371932032}
W20260709 12:22:48.398322 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 725) took 1595 ms (client timeout 1984 ms). Trace:
W20260709 12:22:48.409349 18807 rpcz_store.cc:269] 0709 12:22:46.802696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.805983 (+  3287us) service_pool.cc:224] Handling call
0709 12:22:48.398070 (+1592087us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:48.398226 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1189989120}
I20260709 12:22:48.399466 21498 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:48.399819 21564 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:48.389575 21503 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.358577 21427 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.355494 21587 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:22:48.368290 21438 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.396345 21591 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.632s	user 0.015s	sys 0.000s
I20260709 12:22:48.418377 18807 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:48.434875 21591 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.420876 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:48.421778 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:48.423157 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2743) took 1098 ms. Trace:
I20260709 12:22:48.441556 18809 raft_consensus.cc:2376] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8. Re-sending same reply.
I20260709 12:22:48.441378 18016 rpcz_store.cc:276] 0709 12:22:47.325181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.325376 (+   195us) service_pool.cc:224] Handling call
0709 12:22:47.389147 (+ 63771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.443612 (+ 54465us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:47.458257 (+ 14645us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.141094 (+682837us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b18000a1f20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:22:48.421580 (+280486us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b18000a1f20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c1a58b58f 00007f6c135ca571 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd
0709 12:22:48.422144 (+   564us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.422588 (+   444us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.422743 (+   155us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:48.422792 (+    49us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:48.422840 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.422920 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.423083 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":914970,"prepare.queue_time_us":922450,"prepare.run_cpu_time_us":145,"prepare.run_wall_time_us":148,"raft.queue_time_us":11717,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":247,"spinlock_wait_cycles":19808000,"thread_start_us":33634,"threads_started":1,"wal-append.queue_time_us":30280}
I20260709 12:22:48.445062 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.425746 18815 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:48.438380 21591 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:48.421447 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.426836 18018 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:48.427055 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:48.447088 18814 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:48.355711 21534 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.364490 21036 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.455582 18090 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:48.458825 21547 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.544s	user 0.015s	sys 0.000s
I20260709 12:22:48.420877 21498 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:48.467144 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.468442 18164 raft_consensus.cc:606] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:48.469172 18164 raft_consensus.cc:614] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:48.470126 18019 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 12:22:48.470916 18019 pending_rounds.cc:85] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 3
I20260709 12:22:48.471297 18019 pending_rounds.cc:107] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:22:48.471662 18019 raft_consensus.cc:2889] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:48.411106 21603 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.474536 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.478713 18022 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:48.480285 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:48.471652 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75143.5us
I20260709 12:22:48.453969 21562 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.485769 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:22:48.486356 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.433802 21503 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.481897 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:22:48.489918 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72012.5us
I20260709 12:22:48.462433 21511 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.471586 21498 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:48.480712 21606 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:48.469839 21438 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.512414 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.513744 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.513926 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.514941 18016 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:48.515499 21642 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.516176 21642 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.517292 21603 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:48.519049 21036 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.522413 21503 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:48.523725 21503 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:48.524369 21503 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.524900 21503 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.526715 21534 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:48.527524 21497 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:48.490711 21562 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.528283 21497 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:48.528947 21497 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:48.530478 21562 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:48.530831 21631 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:22:48.532998 21636 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:48.533496 21642 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.527602 21534 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.534806 21534 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.538935 21534 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:48.542629 21633 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
W20260709 12:22:48.562115 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80408.5us
I20260709 12:22:48.571219 21538 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:48.572140 21538 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:48.573762 18137 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.984s (SENT)
I20260709 12:22:48.574554 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.575938 21538 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.577942 21538 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:48.511204 21637 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:22:48.542936 21539 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:48.619728 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:48.620319 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46627.6us
I20260709 12:22:48.623080 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.621063 18016 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:48.619171 21640 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:48.628314 21641 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:48.631168 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.633060 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.640887 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.522612 21498 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.642228 18169 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:48.646565 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.653754 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.655754 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.656325 18016 raft_consensus.cc:2368] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8.
I20260709 12:22:48.668062 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.665918 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 39) took 1062 ms. Trace:
I20260709 12:22:48.671911 18164 rpcz_store.cc:276] 0709 12:22:47.602371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.665038 (+ 62667us) service_pool.cc:224] Handling call
0709 12:22:48.664241 (+999203us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54000505d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:48.664375 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1685327616}
I20260709 12:22:48.673965 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.664902 21498 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:48.681217 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.683769 21551 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.658650 21427 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.681217 18016 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:48.689975 21438 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.700383 21556 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.705132 21582 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:48.710376 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.713409 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45458us
I20260709 12:22:48.713191 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.715368 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:48.718813 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.727039 18019 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:48.727274 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:48.728235 18164 raft_consensus.cc:606] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:48.730085 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:48.730576 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.646s (SENT) [suppressed 16 similar messages]
I20260709 12:22:48.717729 21582 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:48.721841 21648 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:48.741285 21427 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:48.742309 21427 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:48.743391 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.743559 21427 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.744380 18092 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:48.745507 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.745641 21606 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:48.746325 21606 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.746784 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.746719 21539 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:48.747766 18097 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:48.747967 21539 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:48.748328 21539 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:48.748803 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.750183 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.751328 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.751576 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.753036 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:48.753582 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.753861 18169 raft_consensus.cc:606] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
W20260709 12:22:48.754513 21647 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:22:48.755339 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:48.755939 21650 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.116s	user 0.002s	sys 0.000s
I20260709 12:22:48.756235 18167 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:48.753144 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.757663 18171 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:48.758097 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.758827 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.759703 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2756) took 1004 ms. Trace:
I20260709 12:22:48.750190 21539 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.760145 18017 rpcz_store.cc:276] 0709 12:22:47.758914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.053365 (+294451us) service_pool.cc:224] Handling call
0709 12:22:48.758615 (+705250us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:48.759631 (+  1016us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2681216}
I20260709 12:22:48.754415 18016 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:48.766332 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.767177 18172 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.754316 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.768707 18090 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:48.754385 18169 raft_consensus.cc:614] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:48.765583 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:48.746344 18092 raft_consensus.cc:2368] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
I20260709 12:22:48.778153 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.775919 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:48.783035 21384 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:48.784057 21384 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:48.789003 18095 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:48.786793 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66342.4us
I20260709 12:22:48.797375 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.800825 18017 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:22:48.805368 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.806357 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.809986 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.809809 18167 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:48.787180 21420 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:48.812641 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82095.6us
I20260709 12:22:48.796175 21629 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.816314 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.822402 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.790500 21497 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.829571 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.833568 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d440327a63754b889f7edd91e1373942"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:48.834343 18169 raft_consensus.cc:606] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:48.834683 18169 raft_consensus.cc:614] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:48.834419 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.835534 18095 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260709 12:22:48.814313 21438 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:48.785663 21384 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.822690 21641 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:48.839259 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.846526 18164 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:48.849715 21497 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.856142 21503 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:48.857296 21654 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:22:48.887706 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:48.888072 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.888581 18811 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:48.889652 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.890254 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.890466 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.890834 18164 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:48.892455 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:48.860406 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.825486 21629 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.894759 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.895718 18090 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:48.896386 21629 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:48.861549 18092 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:22:48.889707 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.900141 21582 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.860675 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:48.862913 21656 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:22:48.865331 18097 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:48.868376 18096 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:48.862358 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.846103 21649 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:48.873381 18099 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:48.870734 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.844655 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.837137 21640 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:48.902174 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.831297 20960 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:48.939963 20960 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:48.941061 20960 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:48.941597 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.941762 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.942308 20960 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:48.942638 18164 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:22:48.944108 18164 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:22:48.917349 18807 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:22:48.892707 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.879612 21438 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:48.946309 21498 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.946961 21498 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:48.947593 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.918773 18168 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:22:48.948366 18092 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:48.950428 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.953150 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.902791 18814 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:48.922866 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:48.954646 18024 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:48.903507 18019 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:48.915556 18016 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:48.904158 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.904569 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:48.910524 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:48.880559 21543 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.958854 18173 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:22:48.913724 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:48.961200 18166 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.914582 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:48.907289 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.915120 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:48.923439 18020 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 12:22:48.904783 21623 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.915314 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 420) took 1892 ms (client timeout 1754 ms). Trace:
I20260709 12:22:48.947993 18165 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:48.971195 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.903286 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.972469 18094 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:48.920255 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:48.926627 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59338us
I20260709 12:22:48.924643 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.942651 18167 raft_consensus.cc:2368] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:22:48.949546 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.925257 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:48.944959 21657 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
W20260709 12:22:48.926740 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62841.5us
I20260709 12:22:48.923748 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:48.841630 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:48.928290 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.964951 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:48.966964 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:48.923126 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:48.910303 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:48.969744 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:48.925160 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:48.912775 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:48.972949 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:49.997762 18095 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:48.930075 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:49.998194 18019 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:49.998380 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2832) took 1089 ms. Trace:
I20260709 12:22:49.998588 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 574) took 1105 ms. Trace:
W20260709 12:22:49.998980 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.06897e+06us
I20260709 12:22:49.998898 18096 rpcz_store.cc:276] 0709 12:22:48.908550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.912466 (+  3916us) service_pool.cc:224] Handling call
0709 12:22:49.998264 (+1085798us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.999111 21556 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:49.999837 18098 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:50.000092 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:50.000321 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2835) took 1090 ms. Trace:
I20260709 12:22:50.000792 18098 rpcz_store.cc:276] 0709 12:22:48.909432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.923459 (+ 14027us) service_pool.cc:224] Handling call
0709 12:22:50.000249 (+1076790us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:48.970894 18022 rpcz_store.cc:269] 0709 12:22:47.023438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.023609 (+   171us) service_pool.cc:224] Handling call
0709 12:22:48.915072 (+1891463us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:48.915200 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3225854336}
I20260709 12:22:50.001435 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2017) took 1141 ms. Trace:
I20260709 12:22:49.968111 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.001888 18096 rpcz_store.cc:276] 0709 12:22:48.860466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.000444 (+1139978us) service_pool.cc:224] Handling call
0709 12:22:50.000872 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.001098 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.001125 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.001168 (+    43us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:50.001209 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.001265 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.001375 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.173187 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.003198 18093 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:22:50.003741 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2841) took 1060 ms. Trace:
I20260709 12:22:50.003808 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 569) took 1112 ms. Trace:
I20260709 12:22:50.004215 18094 rpcz_store.cc:276] 0709 12:22:48.943538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.030935 (+ 87397us) service_pool.cc:224] Handling call
0709 12:22:50.003648 (+972713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.933245 18017 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:50.054690 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2122) took 1195 ms. Trace:
I20260709 12:22:49.274852 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.055188 18016 rpcz_store.cc:276] 0709 12:22:48.859252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.966543 (+107291us) service_pool.cc:224] Handling call
0709 12:22:50.054606 (+1088063us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.056325 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1898) took 1103 ms. Trace:
W20260709 12:22:50.056648 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.1464e+06us
I20260709 12:22:50.056810 18171 rpcz_store.cc:276] 0709 12:22:48.952540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.274497 (+321957us) service_pool.cc:224] Handling call
0709 12:22:50.056251 (+781754us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.057093 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:49.068403 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91208us
W20260709 12:22:50.017119 18170 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.056s	user 0.013s	sys 0.000s
I20260709 12:22:49.999078 18095 rpcz_store.cc:276] 0709 12:22:48.893580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.924383 (+ 30803us) service_pool.cc:224] Handling call
0709 12:22:49.998520 (+1074137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.059341 18170 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:22:49.164882 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.060336 18095 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:49.851684 18024 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.881s	user 0.014s	sys 0.000s
I20260709 12:22:50.004324 18093 rpcz_store.cc:276] 0709 12:22:48.891764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.209114 (+317350us) service_pool.cc:224] Handling call
0709 12:22:50.003734 (+794620us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.271934 18166 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:50.062000 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:22:50.063074 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2204) took 1173 ms. Trace:
W20260709 12:22:49.162169 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.014871 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2745) took 1609 ms. Trace:
I20260709 12:22:48.901784 21582 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:50.004421 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2828) took 1097 ms. Trace:
I20260709 12:22:49.071754 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 424) took 1955 ms. Trace:
I20260709 12:22:49.013476 18806 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:50.065631 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1886) took 1124 ms. Trace:
W20260709 12:22:49.833885 18090 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.937s	user 0.015s	sys 0.000s
I20260709 12:22:50.066308 18809 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:50.066613 18090 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:50.066442 18167 rpcz_store.cc:276] 0709 12:22:48.941315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.941455 (+   140us) service_pool.cc:224] Handling call
0709 12:22:50.065527 (+1124072us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.066448 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:49.356801 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.060941 18169 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:49.024940 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.064714 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2124) took 1204 ms. Trace:
I20260709 12:22:49.253341 18172 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:50.068809 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2208) took 1106 ms. Trace:
I20260709 12:22:49.067646 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:49.997614 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.065641 18096 rpcz_store.cc:276] 0709 12:22:48.907278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.003293 (+1096015us) service_pool.cc:224] Handling call
0709 12:22:50.003898 (+   605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.004093 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.004119 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.004162 (+    43us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:50.004196 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.004261 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.004365 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.150820 18018 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:50.069274 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2118) took 1248 ms. Trace:
I20260709 12:22:50.033767 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2827) took 1126 ms. Trace:
I20260709 12:22:50.071902 18017 rpcz_store.cc:276] 0709 12:22:48.820727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.822011 (+  1284us) service_pool.cc:224] Handling call
0709 12:22:50.069088 (+1247077us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:50.069202 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162497920}
I20260709 12:22:50.072194 18098 rpcz_store.cc:276] 0709 12:22:48.906933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.001652 (+1094719us) service_pool.cc:224] Handling call
0709 12:22:50.033084 (+ 31432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.033343 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.033372 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.033425 (+    53us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:50.033470 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.033529 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.033678 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.063436 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2123) took 1203 ms. Trace:
I20260709 12:22:50.069334 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 487) took 1304 ms. Trace:
I20260709 12:22:50.067389 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 570) took 1175 ms. Trace:
I20260709 12:22:50.074471 18098 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:50.074878 18090 rpcz_store.cc:276] 0709 12:22:48.892157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.894324 (+  2167us) service_pool.cc:224] Handling call
0709 12:22:50.067251 (+1172927us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.062245 18099 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:50.071048 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.073894 18024 rpcz_store.cc:276] 0709 12:22:48.859595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.922558 (+ 62963us) service_pool.cc:224] Handling call
0709 12:22:50.063354 (+1140796us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.068630 18019 rpcz_store.cc:276] 0709 12:22:48.859890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.972658 (+112768us) service_pool.cc:224] Handling call
0709 12:22:50.064648 (+1091990us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.073905 18172 rpcz_store.cc:276] 0709 12:22:48.765093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.765982 (+   889us) service_pool.cc:224] Handling call
0709 12:22:50.069267 (+1303285us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:48.926218 21645 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.237s	user 0.000s	sys 0.002s
I20260709 12:22:49.028275 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:50.060595 18091 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:49.024199 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.060079 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2197) took 1178 ms. Trace:
I20260709 12:22:50.063514 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:22:49.671680 20960 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.729s	user 0.015s	sys 0.002s
W20260709 12:22:49.337615 21588 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:22:50.064119 21609 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:49.339805 21539 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:49.336972 21522 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:48.948030 21438 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.124614 21522 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:48.922539 21389 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:48.914038 21511 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:50.077419 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.125422 21511 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:50.125805 18171 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:50.125952 21511 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:50.125720 21522 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.127074 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.127743 18171 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:50.128794 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.129654 18171 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:22:49.246152 21503 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:49.292502 21498 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:50.064404 21363 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:50.115962 20960 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:50.150633 21389 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:50.151108 20960 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.151511 21363 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.152130 21503 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:50.152626 21503 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.152810 18020 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:50.153023 21503 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:50.127079 21522 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:50.154070 21498 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:50.075317 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2834) took 1166 ms. Trace:
I20260709 12:22:50.154209 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2128) took 1290 ms. Trace:
I20260709 12:22:50.105561 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:50.155148 18098 rpcz_store.cc:276] 0709 12:22:48.909120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.073730 (+1164610us) service_pool.cc:224] Handling call
0709 12:22:50.074113 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.075025 (+   912us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.075256 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.064940 18021 rpcz_store.cc:276] 0709 12:22:47.407904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.419653 (+ 11749us) service_pool.cc:224] Handling call
0709 12:22:47.420476 (+   823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.522434 (+101958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.014493 (+1492059us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400268754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:49.014769 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":231433984}
I20260709 12:22:50.155370 18020 rpcz_store.cc:276] 0709 12:22:48.863638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.015728 (+152090us) service_pool.cc:224] Handling call
0709 12:22:50.154008 (+1138280us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b540024e354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:50.154138 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161640832}
I20260709 12:22:50.065896 18015 rpcz_store.cc:276] 0709 12:22:47.116650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.117146 (+   496us) service_pool.cc:224] Handling call
0709 12:22:49.071430 (+1954284us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:49.071651 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1478095616}
I20260709 12:22:49.027042 21623 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:49.013191 21649 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 1.134s
I20260709 12:22:48.918388 21538 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.077344 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1883) took 1137 ms. Trace:
I20260709 12:22:50.167681 18169 rpcz_store.cc:276] 0709 12:22:48.940236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.969267 (+ 29031us) service_pool.cc:224] Handling call
0709 12:22:50.077272 (+1108005us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:50.141079 21638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.008s	sys 0.016s
W20260709 12:22:49.805414 21384 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s	user 0.011s	sys 0.014s
I20260709 12:22:50.102233 18094 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:50.169199 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2829) took 1261 ms. Trace:
I20260709 12:22:50.169569 18094 rpcz_store.cc:276] 0709 12:22:48.907653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.054612 (+1146959us) service_pool.cc:224] Handling call
0709 12:22:50.101770 (+ 47158us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.168870 (+ 67100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.169129 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43136}
I20260709 12:22:50.010104 17919 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 7, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.084802 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2840) took 1141 ms. Trace:
I20260709 12:22:50.171121 18091 rpcz_store.cc:276] 0709 12:22:48.943236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.024410 (+ 81174us) service_pool.cc:224] Handling call
0709 12:22:50.084704 (+1060294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.172636 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.083539 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 668) took 1168 ms. Trace:
I20260709 12:22:50.146348 21609 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:50.103119 18017 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:49.144488 21543 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:50.104683 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.098227 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.102031 18093 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:50.105020 18016 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 12:22:50.105046 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.170970 21638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.009s	sys 0.017s
I20260709 12:22:50.196630 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2831) took 1288 ms. Trace:
I20260709 12:22:50.197125 18093 rpcz_store.cc:276] 0709 12:22:48.908214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.062841 (+1154627us) service_pool.cc:224] Handling call
0709 12:22:50.101770 (+ 38929us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.196290 (+ 94520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.196556 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47616}
I20260709 12:22:50.081139 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2837) took 1139 ms. Trace:
I20260709 12:22:50.198614 18097 rpcz_store.cc:276] 0709 12:22:48.942144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.962860 (+ 20716us) service_pool.cc:224] Handling call
0709 12:22:50.081073 (+1118213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.069362 18164 rpcz_store.cc:276] 0709 12:22:48.961954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.024610 (+ 62656us) service_pool.cc:224] Handling call
0709 12:22:50.068740 (+1044130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.076145 18090 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:50.077138 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 666) took 1169 ms (client timeout 1557 ms). Trace:
I20260709 12:22:50.077045 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2830) took 1169 ms. Trace:
I20260709 12:22:50.086947 18170 rpcz_store.cc:276] 0709 12:22:48.881781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.903773 (+ 21992us) service_pool.cc:224] Handling call
0709 12:22:50.059943 (+1156170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.170615 21273 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.116603 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.200234 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.201380 18167 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:50.080950 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 573) took 1187 ms. Trace:
I20260709 12:22:50.086619 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:22:50.202553 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2206) took 1241 ms (client timeout 1557 ms). Trace:
I20260709 12:22:50.077239 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 742) took 1185 ms. Trace:
I20260709 12:22:50.200234 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2836) took 1290 ms. Trace:
I20260709 12:22:50.204696 18090 rpcz_store.cc:276] 0709 12:22:48.909745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.075616 (+1165871us) service_pool.cc:224] Handling call
0709 12:22:50.075956 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.199911 (+123955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.200160 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.200433 18095 rpcz_store.cc:276] 0709 12:22:48.907957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.059729 (+1151772us) service_pool.cc:224] Handling call
0709 12:22:50.060152 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.076748 (+ 16596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.076973 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.063477 18166 rpcz_store.cc:276] 0709 12:22:48.905994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.913418 (+  7424us) service_pool.cc:224] Handling call
0709 12:22:50.063005 (+1149587us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.147802 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1895) took 1196 ms. Trace:
I20260709 12:22:50.206610 18171 rpcz_store.cc:276] 0709 12:22:48.951604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.128509 (+1176905us) service_pool.cc:224] Handling call
0709 12:22:50.147716 (+ 19207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.151372 21389 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:50.064195 21497 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:22:50.213778 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1893) took 1262 ms. Trace:
I20260709 12:22:50.214430 18164 rpcz_store.cc:276] 0709 12:22:48.951008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.199931 (+1248923us) service_pool.cc:224] Handling call
0709 12:22:50.213680 (+ 13749us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.154821 21511 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:49.245116 21606 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.157701 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2108) took 1704 ms. Trace:
I20260709 12:22:50.163470 18809 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:22:50.153528 20960 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.077330 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2839) took 1134 ms. Trace:
I20260709 12:22:50.239331 18092 rpcz_store.cc:276] 0709 12:22:48.942906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.949214 (+  6308us) service_pool.cc:224] Handling call
0709 12:22:50.077256 (+1128042us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.166966 18098 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:50.173420 18815 rpcz_store.cc:276] 0709 12:22:48.914938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.924841 (+  9903us) service_pool.cc:224] Handling call
0709 12:22:50.083427 (+1158586us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19712}
I20260709 12:22:50.234835 18018 rpcz_store.cc:276] 0709 12:22:48.453690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.477445 (+ 23755us) service_pool.cc:224] Handling call
0709 12:22:49.072015 (+594570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.071610 (+999595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.149681 (+ 78071us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b540024ca50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:50.157539 (+  7858us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1342983936}
I20260709 12:22:50.103036 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2833) took 1194 ms. Trace:
I20260709 12:22:50.172109 21538 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.176906 21623 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:50.178726 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1899) took 1225 ms. Trace:
I20260709 12:22:50.211705 21551 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:50.241701 18169 rpcz_store.cc:276] 0709 12:22:48.952873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.172305 (+1219432us) service_pool.cc:224] Handling call
0709 12:22:50.178650 (+  6345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.242307 21551 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.179133 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2127) took 1316 ms. Trace:
I20260709 12:22:50.243791 18017 rpcz_store.cc:276] 0709 12:22:48.863237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.081835 (+1218598us) service_pool.cc:224] Handling call
0709 12:22:50.082210 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.178825 (+ 96615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.179069 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.192427 18172 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:50.244611 21551 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.244971 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1903) took 1018 ms. Trace:
I20260709 12:22:50.245324 18172 rpcz_store.cc:276] 0709 12:22:49.226740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.097885 (+871145us) service_pool.cc:224] Handling call
0709 12:22:50.244897 (+147012us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.216431 21346 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:22:50.212038 21543 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.193378 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2126) took 1331 ms. Trace:
I20260709 12:22:50.246413 21346 consensus_queue.cc:1243] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:50.246590 18807 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:50.246802 18016 rpcz_store.cc:276] 0709 12:22:48.861607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.077506 (+1215899us) service_pool.cc:224] Handling call
0709 12:22:50.104731 (+ 27225us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.192905 (+ 88174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.193284 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.248426 21543 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:50.248618 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2842) took 1304 ms. Trace:
I20260709 12:22:50.086771 18173 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:50.248988 18098 rpcz_store.cc:276] 0709 12:22:48.943816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.166204 (+1222388us) service_pool.cc:224] Handling call
0709 12:22:50.166761 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.248301 (+ 81540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.248549 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.249344 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2205) took 1288 ms. Trace:
I20260709 12:22:50.249742 18173 rpcz_store.cc:276] 0709 12:22:48.961001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.961253 (+   252us) service_pool.cc:224] Handling call
0709 12:22:50.249278 (+1288025us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.250465 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.251139 18173 pending_rounds.cc:85] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 3
I20260709 12:22:50.251500 18173 pending_rounds.cc:107] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:22:50.251740 18173 raft_consensus.cc:2889] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:50.243090 18169 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:50.208077 18166 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 4. Current term is 6. Ops: []
I20260709 12:22:50.240834 18096 rpcz_store.cc:276] 0709 12:22:48.908840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.071618 (+1162778us) service_pool.cc:224] Handling call
0709 12:22:50.102429 (+ 30811us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.102656 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.102683 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.102729 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:50.102777 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.102833 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.102962 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.253062 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1890) took 1303 ms. Trace:
I20260709 12:22:50.246615 18172 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:50.178259 21556 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:50.200047 18806 rpcz_store.cc:269] 0709 12:22:48.907855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.910199 (+  2344us) service_pool.cc:224] Handling call
0709 12:22:50.076885 (+1166686us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.209553 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.257783 18170 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:50.209412 21273 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:50.262337 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.201833 18099 rpcz_store.cc:276] 0709 12:22:48.893237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.919909 (+ 26672us) service_pool.cc:224] Handling call
0709 12:22:50.080870 (+1160961us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.262966 18814 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:50.210579 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1892) took 1259 ms. Trace:
W20260709 12:22:50.202925 18165 rpcz_store.cc:269] 0709 12:22:48.961346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.023846 (+ 62500us) service_pool.cc:224] Handling call
0709 12:22:50.202484 (+1178638us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:50.076359 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.263597 18167 rpcz_store.cc:276] 0709 12:22:48.950704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.082072 (+1131368us) service_pool.cc:224] Handling call
0709 12:22:50.210497 (+128425us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.213912 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.228482 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2843) took 1265 ms. Trace:
I20260709 12:22:50.268143 18094 rpcz_store.cc:276] 0709 12:22:48.963475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.194707 (+1231232us) service_pool.cc:224] Handling call
0709 12:22:50.227684 (+ 32977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.227968 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.228010 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.228075 (+    65us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:50.228127 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.228209 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.228381 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:50.230430 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 230525us
I20260709 12:22:50.233021 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:50.200935 21384 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.344s	user 0.012s	sys 0.019s
I20260709 12:22:50.210332 17919 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:22:50.274533 21556 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.276006 17919 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:50.262372 21669 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:50.277864 17919 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.279987 21649 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:50.280145 17919 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.280758 21649 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.233562 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.282424 18097 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:50.282554 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:50.282267 21609 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.284276 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:50.284361 17919 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 6 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.235723 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.286592 18171 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:50.287971 18171 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:50.286616 18093 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:50.235901 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 670) took 1318 ms (client timeout 1617 ms). Trace:
I20260709 12:22:50.237357 21389 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.292037 21498 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:50.292672 21498 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:50.238129 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.238481 21363 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.293265 21511 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.238615 21606 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.242983 21538 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.254535 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1897) took 1302 ms. Trace:
I20260709 12:22:50.294991 18172 rpcz_store.cc:276] 0709 12:22:48.952230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.246084 (+1293854us) service_pool.cc:224] Handling call
0709 12:22:50.246430 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.254243 (+  7813us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.254471 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.252650 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1896) took 1300 ms. Trace:
I20260709 12:22:50.296074 18169 rpcz_store.cc:276] 0709 12:22:48.951915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.242537 (+1290622us) service_pool.cc:224] Handling call
0709 12:22:50.242901 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.252317 (+  9416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.252562 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.255792 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:50.253407 18166 rpcz_store.cc:276] 0709 12:22:48.950088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.207600 (+1257512us) service_pool.cc:224] Handling call
0709 12:22:50.207927 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.252800 (+ 44873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.253007 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.260447 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:50.203635 18811 rpcz_store.cc:276] 0709 12:22:48.891937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.914843 (+ 22906us) service_pool.cc:224] Handling call
0709 12:22:50.077172 (+1162329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10112}
I20260709 12:22:50.297569 18091 raft_consensus.cc:2376] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6. Re-sending same reply.
W20260709 12:22:50.289152 18809 rpcz_store.cc:269] 0709 12:22:48.917789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.924808 (+  7019us) service_pool.cc:224] Handling call
0709 12:22:50.235788 (+1310980us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226944}
I20260709 12:22:50.259835 21666 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.302016 18167 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:50.302839 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2207) took 1341 ms. Trace:
I20260709 12:22:50.303156 18167 rpcz_store.cc:276] 0709 12:22:48.961647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.266742 (+1305095us) service_pool.cc:224] Handling call
0709 12:22:50.301753 (+ 35011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.302550 (+   797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.302774 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.304315 18170 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:50.306005 18169 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:50.307199 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:50.273083 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.30815e+06us
W20260709 12:22:50.285658 17921 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 276) took 13392 ms (client timeout 15000 ms). Trace:
I20260709 12:22:50.293643 21498 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.306157 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:50.320500 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.321460 18095 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:22:50.309588 21538 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.322645 18091 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:50.322495 21538 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:50.322853 21665 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
W20260709 12:22:50.307274 21661 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:22:50.294459 21363 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:50.329728 18095 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:50.306738 21346 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:50.306898 21497 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:50.307497 20960 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.331508 18166 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:50.332190 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:50.340560 18090 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 8
W20260709 12:22:50.332602 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.305228 21543 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:50.334444 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 119375us
I20260709 12:22:50.360561 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:50.305011 21539 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:22:50.331255 21511 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:50.361434 18814 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
W20260709 12:22:50.336773 18780 connection.cc:256] RPC call timeout handler was delayed by -1.0065515940000296s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2364s for a response
I20260709 12:22:50.360172 20960 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.330871 21606 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:50.395493 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1891) took 1414 ms. Trace:
I20260709 12:22:50.337736 21497 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.417951 18164 rpcz_store.cc:276] 0709 12:22:48.950404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.215420 (+1265016us) service_pool.cc:224] Handling call
0709 12:22:50.216004 (+   584us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:50.216584 (+   580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.216630 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:50.236128 (+ 19498us) log.cc:844] Serialized 52 byte log entry
0709 12:22:50.364203 (+128075us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:50.364262 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.364361 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.364425 (+    64us) raft_consensus.cc:1685] finished
0709 12:22:50.364443 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.364625 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10796,"raft.queue_time_us":9880,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":28286,"thread_start_us":119926,"threads_started":1,"wal-append.queue_time_us":98380}
I20260709 12:22:50.400374 21498 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:50.401609 21511 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:50.451056 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.60937e+06us
I20260709 12:22:50.362314 21543 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:50.466581 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1901) took 1513 ms. Trace:
I20260709 12:22:50.473873 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:22:50.477542 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 420479us
I20260709 12:22:50.485091 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:50.486476 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.316188 21666 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:50.490391 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 430804us
I20260709 12:22:50.483876 18165 rpcz_store.cc:276] 0709 12:22:48.953554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.263932 (+1310378us) service_pool.cc:224] Handling call
0709 12:22:50.264456 (+   524us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:50.289770 (+ 25314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.289806 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:50.290582 (+   776us) log.cc:844] Serialized 52 byte log entry
0709 12:22:50.324573 (+ 33991us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:50.416121 (+ 91548us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.465668 (+ 49547us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.465793 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.465893 (+   100us) raft_consensus.cc:1685] finished
0709 12:22:50.465912 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.466505 (+   593us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47477,"raft.queue_time_us":33676,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"thread_start_us":111273,"threads_started":2,"wal-append.queue_time_us":28923}
I20260709 12:22:50.510324 18090 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:50.402092 21623 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:50.434381 21497 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:50.471486 17919 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 5, term changed from 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:22:50.310595 17921 rpcz_store.cc:269] 0709 12:22:36.881088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.902452 (+ 21364us) service_pool.cc:224] Handling call
0709 12:22:50.262532 (+13360080us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:45.926654 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:45.927870 (+  1216us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:45.927945 (+    75us) write_op.cc:435] Acquired schema lock
0709 12:22:45.927994 (+    49us) tablet.cc:662] Decoding operations
0709 12:22:46.564947 (+636953us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:46.583313 (+ 18366us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:46.587868 (+  4555us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:22:46.812730 (+224862us) tablet.cc:701] Row locks acquired
0709 12:22:46.847816 (+ 35086us) write_op.cc:260] PREPARE: finished
0709 12:22:46.884639 (+ 36823us) write_op.cc:270] Start()
0709 12:22:46.917035 (+ 32396us) write_op.cc:276] Timestamp: P: 1783599766875340 usec, L: 0
0709 12:22:46.920844 (+  3809us) op_driver.cc:348] REPLICATION: starting
0709 12:22:47.009176 (+ 88332us) log.cc:844] Serialized 5960 byte log entry
0709 12:22:48.058281 (+1049105us) op_driver.cc:464] REPLICATION: finished
0709 12:22:48.277833 (+219552us) write_op.cc:301] APPLY: starting
0709 12:22:48.277979 (+   146us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:48.315019 (+ 37040us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:48.315214 (+   195us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:48.337168 (+ 21954us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:48.337636 (+   468us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=30
0709 12:22:48.337687 (+    51us) write_op.cc:312] APPLY: finished
0709 12:22:48.350484 (+ 12797us) log.cc:844] Serialized 107 byte log entry
0709 12:22:48.350718 (+   234us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:48.350932 (+   214us) write_op.cc:454] Released schema lock
0709 12:22:48.351036 (+   104us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:48.351764 (+   728us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:48.351801 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":179441,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":219028,"apply.run_cpu_time_us":7182,"apply.run_wall_time_us":73800,"num_ops":15,"prepare.queue_time_us":1199499,"prepare.run_cpu_time_us":26690,"prepare.run_wall_time_us":1161584,"raft.queue_time_us":205075,"raft.run_cpu_time_us":15966,"raft.run_wall_time_us":45898,"replication_time_us":1137185,"thread_start_us":1867531,"threads_started":4,"wal-append.queue_time_us":219444,"wal-append.run_cpu_time_us":16714,"wal-append.run_wall_time_us":2213289}]]}
I20260709 12:22:50.422578 21384 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.521728 21389 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.011s
W20260709 12:22:50.531582 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 413849us
I20260709 12:22:50.529848 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:50.483280 21511 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:50.400527 21346 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20260709 12:22:50.502755 21543 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:50.531926 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46937.2us
I20260709 12:22:50.395654 21363 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.522323 21606 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:50.559271 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.420941 21639 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260709 12:22:50.552951 17919 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.567152 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:50.568185 21678 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:50.569424 21678 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:50.569926 21678 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.570346 21678 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:50.573770 21677 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:50.574404 21677 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:50.526841 21551 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.541150 21538 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:50.593237 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.595818 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.576422 21606 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:50.608251 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:50.598745 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94375.4us
I20260709 12:22:50.602159 18098 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:50.613049 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.580931 21389 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:50.614308 18093 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:50.578689 21562 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:50.614427 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47377us
I20260709 12:22:50.615067 17919 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.541640 21384 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.622565 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.597543 21609 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:50.629020 18090 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:50.530359 21623 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.637769 17919 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 6, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.577138 21427 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:22:50.637898 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:50.641671 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:50.604447 21638 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:50.588896 21551 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.613490 21606 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:50.667137 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.640776 17919 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:22:50.654512 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124809us
I20260709 12:22:50.591104 21649 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:50.624276 21389 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:50.681293 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:50.682423 21438 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:50.646099 21609 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:50.714282 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.724545 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:50.727217 17919 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 2 to 5, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.735294 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:50.725571 18809 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:50.699396 21438 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:50.739295 21693 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 12:22:50.748160 21693 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:50.642738 21677 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.691546 21638 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:50.680047 21562 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.741400 18814 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:50.743114 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:50.711992 21346 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:50.804085 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 221271us
W20260709 12:22:50.800486 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.801395 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 208541us
W20260709 12:22:50.802124 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 171826us
I20260709 12:22:50.804680 17919 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 4 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:50.807245 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.807204 17919 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 6, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.808847 17919 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:22:50.809685 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:50.811440 18164 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:22:50.811786 21623 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:50.749774 21677 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:50.797855 18815 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10. Re-sending same reply.
W20260709 12:22:50.813176 21687 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:22:50.814119 21666 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.820360 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.821481 17919 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.822623 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.823772 18170 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:50.822803 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.824061 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.824671 18171 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:50.803138 21346 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:50.843199 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.739043 21667 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:50.851138 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.739013 20960 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:50.844669 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:50.742785 21685 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:22:50.854554 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:50.855420 18172 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:50.745934 21273 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:50.811790 21688 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:22:50.742574 21649 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:50.852684 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.854691 18166 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 12:22:50.850512 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43536.6us
I20260709 12:22:50.867144 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:50.859802 18169 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:50.845243 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:50.861678 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 180459us
I20260709 12:22:50.863080 18171 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:50.863929 18814 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:50.870060 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:50.874357 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 236540us
I20260709 12:22:50.818153 21682 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:50.875866 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:50.882813 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:50.885444 18172 pending_rounds.cc:85] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 4
I20260709 12:22:50.892851 18172 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:22:50.894037 18172 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:50.891234 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43560.1us
I20260709 12:22:50.866123 21649 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.743106 21438 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.874480 21666 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:50.888273 20960 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:50.908769 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:50.891804 21363 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:50.887439 21538 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:50.913700 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1900) took 1954 ms. Trace:
I20260709 12:22:50.903858 21649 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:50.931699 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61737.2us
I20260709 12:22:50.901796 21682 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:50.914808 18173 rpcz_store.cc:276] 0709 12:22:48.953232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.250488 (+1297256us) service_pool.cc:224] Handling call
0709 12:22:50.250891 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.263328 (+ 12437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.263365 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.263918 (+   553us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.423041 (+159123us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:50.680025 (+256984us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.795729 (+115704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.795837 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.907607 (+111770us) raft_consensus.cc:1685] finished
0709 12:22:50.907634 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.907815 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6110,"prepare.queue_time_us":190842,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":179732,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":20134,"thread_start_us":425922,"threads_started":3,"wal-append.queue_time_us":290057}
I20260709 12:22:50.847343 21389 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:50.932801 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:50.912184 20960 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.941053 21666 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:50.958145 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82340.3us
W20260709 12:22:50.952785 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 266321us
I20260709 12:22:50.967062 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:50.941745 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:50.914338 21711 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:50.933591 21438 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:50.930636 21363 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:22:50.952322 21704 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:22:50.973253 21690 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:50.974072 21717 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:50.974669 21670 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:22:51.007915 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99062.1us
W20260709 12:22:50.976750 21503 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.823s	user 0.012s	sys 0.002s
I20260709 12:22:50.921187 21638 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.946218 20960 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:50.933281 21384 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:50.978842 21539 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:50.979280 21682 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:50.942078 21582 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.858s	user 0.014s	sys 0.003s
I20260709 12:22:50.969846 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:51.022270 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52670.4us
I20260709 12:22:51.023823 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:50.928956 21538 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:50.990690 21711 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:51.011552 21685 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:22:51.042713 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 101058us
I20260709 12:22:51.026140 21539 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:51.043850 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.044276 18023 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4.
I20260709 12:22:51.006294 21551 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:51.011981 21713 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:50.935048 21346 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:51.014210 21666 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:51.017189 21503 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:51.072422 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 157052us
I20260709 12:22:51.043745 21582 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:51.042425 21711 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:51.076032 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.085606 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:51.070562 21539 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.087471 21539 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:51.091926 21538 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.086676 18093 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:22:51.092229 21522 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.938s	user 0.009s	sys 0.005s
I20260709 12:22:51.095109 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.093108 21582 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:51.100454 21703 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 12:22:51.102331 21674 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:22:51.103806 21538 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.104537 21538 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:51.106541 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2125) took 2246 ms. Trace:
I20260709 12:22:51.107169 21538 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:51.095654 21719 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:22:51.108070 21497 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.546s	user 0.013s	sys 0.000s
I20260709 12:22:51.108407 21719 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:51.109562 21719 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.113447 18168 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:51.073221 21683 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:51.115842 21503 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:51.121140 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.121415 21543 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.543s	user 0.013s	sys 0.000s
I20260709 12:22:51.097931 21713 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:51.123917 21678 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.553s	user 0.014s	sys 0.000s
I20260709 12:22:51.124580 21582 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:51.178134 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 339281us
I20260709 12:22:51.126528 21709 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:22:51.127812 21511 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.552s	user 0.013s	sys 0.000s
I20260709 12:22:51.128675 21562 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:51.130445 21716 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:51.138415 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.156169 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61125.5us
I20260709 12:22:51.154299 21522 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:51.166733 21543 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:51.167863 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.107031 18022 rpcz_store.cc:276] 0709 12:22:48.861202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.002171 (+1140969us) service_pool.cc:224] Handling call
0709 12:22:50.002572 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.520375 (+517803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.520421 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.532793 (+ 12372us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.823652 (+290859us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:50.823703 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.823777 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.106252 (+282475us) raft_consensus.cc:1685] finished
0709 12:22:51.106279 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.106460 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52749,"raft.queue_time_us":113321,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":9591,"thread_start_us":674673,"threads_started":2,"wal-append.queue_time_us":338115}
W20260709 12:22:51.130386 21498 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.660s	user 0.008s	sys 0.006s
I20260709 12:22:51.179216 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:51.108620 21497 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.164050 21363 consensus_queue.cc:1243] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:51.181253 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:51.182022 18090 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:51.182294 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:22:51.185938 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.130501 21346 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:51.220211 18808 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:51.220274 18093 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:51.221525 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 193024us
W20260709 12:22:51.218963 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 310280us
I20260709 12:22:51.195331 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:51.226177 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.200742 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:51.200974 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:51.164413 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.203498 18809 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:51.226330 18813 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:22:51.202615 21606 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.006s
W20260709 12:22:51.244704 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 63982.4us
I20260709 12:22:51.132843 21711 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.237960 21715 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:22:51.231010 21438 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:51.248824 21711 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.249042 21438 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:51.239053 21503 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260709 12:22:51.250407 21438 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.198287 21511 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.251421 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:51.251690 18097 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:51.252136 18090 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:51.253257 21511 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:22:51.171859 21698 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:51.241895 21675 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:22:51.254710 21698 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:51.254926 21675 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:51.255365 21675 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:51.255077 21606 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.239547 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:51.182940 21678 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.256484 21675 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:51.228518 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49384.4us
I20260709 12:22:51.240715 21497 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.247999 21649 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:51.229471 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.332263 18164 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:22:51.333324 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:51.383355 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.354682 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:51.384136 18019 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:51.385104 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.385391 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.385921 18090 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:51.266366 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:51.341213 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115286us
I20260709 12:22:51.376907 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:51.261988 21670 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:51.238296 21679 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:51.355084 18098 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [6.9-8.10]
W20260709 12:22:51.327070 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62413.8us
I20260709 12:22:51.388593 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.379954 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.355252 21711 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:51.354995 21723 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:51.380709 21675 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:51.329309 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:51.328167 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88728.1us
I20260709 12:22:51.386066 18019 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:51.400513 18091 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:51.401882 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.388175 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.405269 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:51.378178 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 278866us
I20260709 12:22:51.325587 21666 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:51.251474 21543 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:51.390619 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:51.231379 21700 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:22:51.390736 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.416844 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:51.417341 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:51.417620 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.417893 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.418521 18024 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:22:51.418860 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.420228 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.393810 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.420893 18022 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:51.390636 18015 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:51.326520 21715 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:51.263797 21683 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:51.304529 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45351.2us
I20260709 12:22:51.431169 18166 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:22:51.431459 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.228303 21707 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.180s
I20260709 12:22:51.246927 21668 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.151s
I20260709 12:22:51.408246 18020 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:51.376906 21539 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:22:51.386345 21606 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.414538 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:51.490070 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43681.6us
I20260709 12:22:51.376318 21551 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:22:51.414826 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.507956 18169 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:22:51.491545 18171 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:51.412691 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.412613 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.509647 18016 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:22:51.510197 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:51.511812 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:51.234211 21701 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.169s
I20260709 12:22:51.426746 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.390084 21697 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 12:22:51.483539 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet aef65d70106f4f458174493864b4ba3d with delete_type TABLET_DATA_TOMBSTONED (TS 768723648a9444fda53659e6a8cc3091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48546
I20260709 12:22:51.401976 21716 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.442142 18811 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:51.414702 21363 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:51.492146 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:51.490170 18170 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:51.501506 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.529692 18020 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:51.438536 21728 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:51.501513 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.533241 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.534087 18021 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:51.503790 21609 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:51.504892 21678 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.396863 20960 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.424407 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.557268 18017 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:51.426210 21715 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:51.491011 21666 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:22:51.427552 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.520277 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:51.512168 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:51.363801 21698 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:51.508265 21700 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:22:51.579206 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 151756us
I20260709 12:22:51.501789 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.420490 21690 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.096s
W20260709 12:22:51.541294 21727 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:22:51.541394 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.542755 21716 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.588670 18021 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:51.532939 21638 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:51.592458 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:51.593173 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 195071us
I20260709 12:22:51.593501 18021 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:51.544018 21511 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:51.402242 21679 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.560890 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:51.561611 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.518332 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.600725 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.566798 21715 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:51.565450 20960 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.522614 21683 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:51.527654 18024 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:51.586117 21678 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:51.530328 21606 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:51.583177 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.584082 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77367.3us
I20260709 12:22:51.586535 18022 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:51.621649 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.622684 18022 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:51.592454 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.540943 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.632823 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.633136 18020 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:51.633703 18017 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:22:51.587596 21728 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:51.596323 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:51.582990 21730 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:22:51.562649 21609 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:51.601557 18016 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:22:51.613888 21551 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:51.610567 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.610785 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.533914 21713 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:51.614573 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59500.5us
I20260709 12:22:51.616757 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.663246 18024 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:51.616624 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:51.628188 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.629261 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:51.667095 18091 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:51.614917 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:51.630766 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:51.670627 21716 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.671505 21716 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:51.671618 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:51.672036 21716 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.672482 18808 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:51.672518 21716 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:51.640524 21649 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.602065 21697 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:22:51.676033 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.631270 21606 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } attrs { replace: true } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.612362 21683 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.678207 21683 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:51.641542 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45351.7us
I20260709 12:22:51.611792 21707 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:51.679934 21734 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:22:51.604858 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:51.680269 21707 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.680963 21707 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:51.618392 21694 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 12:22:51.640524 18015 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:51.649884 21733 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.653424 21670 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.654323 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.684789 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.685372 21551 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.685724 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.681314 21588 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:51.684464 21670 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.619254 21690 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:22:51.689086 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:51.656379 21638 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:22:51.655979 20960 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:51.695415 20960 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:51.685503 18015 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:22:51.697242 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82432.3us
I20260709 12:22:51.670544 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:51.600538 21363 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:22:51.700402 21363 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:51.700855 21363 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:51.701112 21723 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:51.702066 21699 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:51.701843 21363 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:51.643463 21609 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:51.703315 21363 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:51.703358 21719 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:51.703871 21609 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:51.705024 21724 consensus_queue.cc:1175] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "768723648a9444fda53659e6a8cc3091" responder_term: 6 status { last_received { term: 6 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 6 index: 5 } }
I20260709 12:22:51.705438 21609 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:51.656324 21728 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:51.707468 21713 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.684346 21733 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.686903 18020 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:51.681144 21735 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:22:51.701687 21723 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:51.749398 18024 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:51.750801 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:51.720654 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50228.4us
I20260709 12:22:51.751907 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:51.752172 18090 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:51.697326 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.752915 18015 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:51.712559 21715 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:51.722558 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:51.727978 18016 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:51.756681 18098 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:51.756790 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:51.757275 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.696807 20960 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:51.719750 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.736850 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.760983 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.741027 21740 tablet_replica.cc:333] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:22:51.741297 21551 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.747658 18171 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:51.748786 21733 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:51.750392 21723 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:51.750996 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.774210 18020 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:51.763121 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:51.776017 18024 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10. Re-sending same reply.
I20260709 12:22:51.781919 18165 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:51.783485 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:51.763924 18016 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:51.789481 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.765659 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:51.766381 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:51.771025 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:51.791755 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:51.787771 18093 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:51.739379 21728 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:51.805436 18098 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:51.804348 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.808187 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.809732 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.810544 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:51.847146 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.848812 18020 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:51.861896 18169 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:22:51.865590 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.815224 18173 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:51.871410 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.872514 18173 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:51.874019 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.793206 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:51.762212 21713 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:51.774227 21694 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:52.008464 18173 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:52.011369 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.022185 18016 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:51.778402 21363 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:52.023882 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:51.999611 18019 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:51.780844 21729 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.526s	user 0.004s	sys 0.000s
I20260709 12:22:51.772993 21725 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:52.083240 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:51.810547 21731 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:52.086472 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.093016 18024 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:52.097440 18020 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:52.100394 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 376607us
I20260709 12:22:51.777084 21737 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.260s
I20260709 12:22:52.130295 18016 raft_consensus.cc:2368] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8.
I20260709 12:22:52.131712 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.134600 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:51.805876 21551 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.304s
I20260709 12:22:52.078540 21670 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:52.107755 21346 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.885s	user 0.010s	sys 0.005s
I20260709 12:22:51.802543 21724 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:52.113566 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2120) took 3235 ms (client timeout 2074 ms). Trace:
I20260709 12:22:52.140049 21733 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:51.809906 21690 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:52.168408 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.169087 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:52.167065 18023 rpcz_store.cc:269] 0709 12:22:48.858386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.861943 (+  3557us) service_pool.cc:224] Handling call
0709 12:22:51.998700 (+3136757us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400251554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:52.093343 (+ 94643us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4855831808}
I20260709 12:22:52.179138 18173 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:52.180117 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.181366 18015 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:52.182854 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.183713 18015 raft_consensus.cc:2376] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 4. Re-sending same reply.
I20260709 12:22:52.159101 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.189013 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:52.190151 18814 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:22:52.197188 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.185408 18020 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
W20260709 12:22:52.197924 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.202073 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.208048 18023 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:52.214239 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:52.109805 21363 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.168151 21670 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.231781 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:52.255288 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.219456 18165 raft_consensus.cc:2368] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:22:52.234334 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.292013 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.236667 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.289435 18015 raft_consensus.cc:2376] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8. Re-sending same reply.
I20260709 12:22:52.285640 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:52.291721 18020 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
W20260709 12:22:52.330155 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98501.1us
I20260709 12:22:52.320279 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.331514 18170 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:52.331741 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:52.230692 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:52.334964 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:52.336172 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:52.321544 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.008s	sys 0.034s
W20260709 12:22:52.338855 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.009s	sys 0.035s
I20260709 12:22:52.322036 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.299078 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:22:52.319526 20960 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.011s	sys 0.027s
W20260709 12:22:52.299535 18091 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.613s	user 0.015s	sys 0.000s
I20260709 12:22:52.341074 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:52.341703 18091 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:22:52.342679 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.338179 18814 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:52.343433 18169 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:52.214768 21438 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.232231 21606 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:52.343377 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:52.350277 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:52.360507 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.351377 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:52.362668 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 231016us
W20260709 12:22:52.340886 20960 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.011s	sys 0.031s
I20260709 12:22:52.370494 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.467253 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.468050 18015 raft_consensus.cc:2376] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6. Re-sending same reply.
I20260709 12:22:52.365147 21438 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.394094 21698 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.333957 21737 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.270493 21694 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 12:22:52.377914 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.236572 21363 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.492301 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.402204 18170 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:52.370021 21711 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:52.370456 21678 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.332978 21670 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.371913 21676 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.414176 18814 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:22:52.416880 18808 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.744s	user 0.014s	sys 0.001s
I20260709 12:22:52.424427 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:52.501339 18808 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:22:52.501708 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77438.3us
I20260709 12:22:52.334286 21744 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:52.470189 18167 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.013s	sys 0.052s
I20260709 12:22:52.510198 21744 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:52.510711 18167 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.013s	sys 0.052s
I20260709 12:22:52.510836 21744 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:52.511458 21719 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.807s	user 0.012s	sys 0.003s
I20260709 12:22:52.512362 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.513327 18167 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:52.514514 21707 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.833s	user 0.014s	sys 0.001s
I20260709 12:22:52.515177 21749 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:52.515379 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.515787 21749 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.516168 21749 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:52.516953 21363 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:52.517042 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.518620 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.519412 21670 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:52.204291 21733 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.497699 21678 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.520612 21678 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.521858 21733 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:52.522423 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.523285 18169 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 5: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:52.487962 21704 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:52.523869 21704 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.524205 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:52.489133 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137901us
W20260709 12:22:52.525313 21609 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.796s	user 0.010s	sys 0.000s
I20260709 12:22:52.491142 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.527631 18020 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:52.338729 21748 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.473556 21740 raft_consensus.cc:2243] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:52.482667 21760 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
W20260709 12:22:52.504177 21745 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
W20260709 12:22:52.508520 21716 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.835s	user 0.015s	sys 0.000s
I20260709 12:22:52.372833 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:52.519338 21438 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:52.526604 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.515139 21707 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:52.639408 18017 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:52.536556 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.497717 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.640882 18170 raft_consensus.cc:2368] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9.
I20260709 12:22:52.529418 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2022) took 1937 ms. Trace:
W20260709 12:22:52.641615 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 268860us
I20260709 12:22:52.538954 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:52.643528 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 104699us
I20260709 12:22:52.639122 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:52.479815 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 146460us
I20260709 12:22:52.477520 21754 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.141s
I20260709 12:22:52.494246 21737 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.544226 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.493414 21724 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.528234 21766 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:52.528635 21748 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.631737 21716 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.632342 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 869) took 1439 ms. Trace:
I20260709 12:22:52.558660 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.629426 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.694813 18167 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:52.695480 18016 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:52.696141 18808 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:52.635627 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.641709 18097 rpcz_store.cc:276] 0709 12:22:50.591765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.602431 (+ 10666us) service_pool.cc:224] Handling call
0709 12:22:50.623451 (+ 21020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.252148 (+628697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.529161 (+1277013us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b54000523d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:52.529352 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1444297600}
I20260709 12:22:52.534965 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:52.646274 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:52.563084 21715 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.809s	user 0.006s	sys 0.007s
W20260709 12:22:52.633370 21743 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.526s	user 0.000s	sys 0.003s
W20260709 12:22:52.716523 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.625476 21675 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.215s	user 0.012s	sys 0.006s
W20260709 12:22:52.672698 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.686640 18015 raft_consensus.cc:2376] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7. Re-sending same reply.
I20260709 12:22:52.644544 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.724180 18017 raft_consensus.cc:2376] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5. Re-sending same reply.
I20260709 12:22:52.543357 21551 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.534440 21649 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.692479 18814 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:52.728858 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82693.3us
I20260709 12:22:52.730489 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:22:52.614975 21751 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.002s
I20260709 12:22:52.534158 21768 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:52.693807 18811 rpcz_store.cc:276] 0709 12:22:51.193352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.193950 (+   598us) service_pool.cc:224] Handling call
0709 12:22:52.632106 (+1438156us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b5400590e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:52.632254 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493534464}
I20260709 12:22:52.714525 18166 raft_consensus.cc:2368] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:22:52.714874 18169 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:52.647259 21497 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.228s	user 0.014s	sys 0.039s
I20260709 12:22:52.716797 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.741552 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.717401 18093 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:52.646535 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.743304 18020 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:52.744621 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.746479 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.747212 18020 raft_consensus.cc:1813] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Handling vote request from an unknown peer 768723648a9444fda53659e6a8cc3091
I20260709 12:22:52.747821 18020 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:52.749148 18020 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:52.718664 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.674552 21522 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:52.752089 21522 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:52.719703 18173 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:22:52.654886 21704 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:52.755775 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:52.601610 21683 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.916s	user 0.016s	sys 0.000s
I20260709 12:22:52.757853 21683 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.561128 21711 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:52.762481 21715 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:52.761796 21711 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.753463 21522 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.737283 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.773017 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2778) took 1329 ms. Trace:
I20260709 12:22:52.778180 18017 rpcz_store.cc:276] 0709 12:22:51.443861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.771327 (+1327466us) service_pool.cc:224] Handling call
0709 12:22:52.771818 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.772159 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.772200 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.772269 (+    69us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:52.772332 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.772411 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.772902 (+   491us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.784796 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2135) took 1374 ms. Trace:
I20260709 12:22:52.787400 18020 rpcz_store.cc:276] 0709 12:22:51.410128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.748525 (+1338397us) service_pool.cc:224] Handling call
0709 12:22:52.748923 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.749590 (+   667us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.784687 (+ 35097us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.790040 18020 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:52.802300 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:52.792392 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50691us
I20260709 12:22:52.807416 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2780) took 1319 ms. Trace:
I20260709 12:22:52.786697 18017 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:52.718734 21740 raft_consensus.cc:2272] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:52.808681 18020 rpcz_store.cc:276] 0709 12:22:51.488026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.789029 (+1301003us) service_pool.cc:224] Handling call
0709 12:22:52.789508 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.806988 (+ 17480us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.807332 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.810719 18015 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:22:52.812505 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82131.9us
I20260709 12:22:52.817413 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 499) took 1104 ms. Trace:
I20260709 12:22:52.818110 18172 rpcz_store.cc:276] 0709 12:22:51.691751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.691946 (+   195us) service_pool.cc:224] Handling call
0709 12:22:51.692340 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.692766 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.692800 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:51.693375 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.114833 (+421458us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:52.114882 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.114988 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.796324 (+681336us) raft_consensus.cc:1685] finished
0709 12:22:52.796349 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.796528 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1165,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"thread_start_us":420914,"threads_started":1,"wal-append.queue_time_us":521766}
I20260709 12:22:52.820582 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2779) took 1376 ms. Trace:
I20260709 12:22:52.820839 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:52.821059 18017 rpcz_store.cc:276] 0709 12:22:51.444137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.785708 (+1341571us) service_pool.cc:224] Handling call
0709 12:22:52.786436 (+   728us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.820208 (+ 33772us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.820508 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.825557 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:52.793660 21497 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.375s	user 0.020s	sys 0.039s
W20260709 12:22:52.794728 21764 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:22:52.739552 21649 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:52.815539 21743 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:22:52.848897 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 129026us
I20260709 12:22:52.846899 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:52.851305 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 500) took 1153 ms. Trace:
W20260709 12:22:52.865044 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.864811 18171 rpcz_store.cc:276] 0709 12:22:51.698335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.698528 (+   193us) service_pool.cc:224] Handling call
0709 12:22:51.723496 (+ 24968us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.772337 (+ 48841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.851041 (+1078704us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:52.851236 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54524288}
I20260709 12:22:52.740159 21551 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.870048 21709 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:52.870946 21709 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.712116 21766 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.873986 21766 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:52.874969 21713 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.796s	user 0.011s	sys 0.004s
I20260709 12:22:52.875527 21721 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.875633 21713 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:52.876536 21758 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.103s	user 0.001s	sys 0.001s
I20260709 12:22:52.876930 21765 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.877393 21765 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:52.877853 21765 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.878314 21771 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:52.878825 21771 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:52.802480 21724 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 12:22:52.796015 21768 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:52.874104 21761 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.143s	user 0.001s	sys 0.001s
I20260709 12:22:52.900228 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2783) took 1397 ms. Trace:
I20260709 12:22:52.900763 18015 rpcz_store.cc:276] 0709 12:22:51.502788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.845809 (+1343021us) service_pool.cc:224] Handling call
0709 12:22:52.899627 (+ 53818us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.899840 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.899867 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.899913 (+    46us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:52.899962 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.900021 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.900154 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.878880 21701 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:52.793236 21769 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:52.880038 18813 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:52.881438 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2138) took 1470 ms. Trace:
I20260709 12:22:52.829195 21715 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.902874 18021 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:52.909898 18016 rpcz_store.cc:276] 0709 12:22:51.410815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.750698 (+1339883us) service_pool.cc:224] Handling call
0709 12:22:52.751147 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.880437 (+129290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.880474 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.880527 (+    53us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:52.880574 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.881176 (+   602us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54003bb5d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:52.881227 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.881371 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296728832}
I20260709 12:22:52.912961 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.913694 18016 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:52.915146 21757 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:52.915962 21757 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:52.918325 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:52.919340 18015 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:52.879865 21717 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:22:52.916772 21511 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.305s	user 0.003s	sys 0.021s
I20260709 12:22:52.886611 21706 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:52.879365 21771 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:52.933947 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2209) took 3965 ms (client timeout 2270 ms). Trace:
I20260709 12:22:52.935500 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1931) took 1133 ms. Trace:
I20260709 12:22:52.898483 21709 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:52.880508 21707 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:52.939709 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2856) took 1205 ms. Trace:
I20260709 12:22:52.842367 21704 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:22:52.906728 21701 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:52.815685 21770 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.124s
I20260709 12:22:52.883641 21678 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.986014 18096 rpcz_store.cc:276] 0709 12:22:51.733901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.734060 (+   159us) service_pool.cc:224] Handling call
0709 12:22:51.734477 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.112883 (+378406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.112950 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.131686 (+ 18736us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.471829 (+340143us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:52.471863 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.471927 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.939399 (+467472us) raft_consensus.cc:1685] finished
0709 12:22:52.939429 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.939627 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":284305,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":58012,"thread_start_us":572681,"threads_started":2,"wal-append.queue_time_us":324575}
I20260709 12:22:52.984122 18015 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:22:52.941741 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:22:52.984540 18168 rpcz_store.cc:269] 0709 12:22:48.962253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.081652 (+1119399us) service_pool.cc:224] Handling call
0709 12:22:52.927173 (+2845521us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b5400032ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:52.927413 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2084192000}
I20260709 12:22:52.931008 21717 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:52.952732 21740 ts_tablet_manager.cc:1916] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:22:52.973335 21511 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.366s	user 0.015s	sys 0.043s
I20260709 12:22:52.995359 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:52.926179 21715 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:52.997862 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2785) took 1494 ms. Trace:
I20260709 12:22:52.987217 21713 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.000221 18023 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:52.985013 21706 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:53.000962 18016 rpcz_store.cc:276] 0709 12:22:51.503547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.996961 (+1493414us) service_pool.cc:224] Handling call
0709 12:22:52.997248 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.997467 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.997493 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.997539 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:52.997577 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.997640 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.997746 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.995082 21716 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.023990 18811 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.009683 21757 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.010005 21784 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:53.010845 21748 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.013368 21649 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:53.010890 21752 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.015580 21678 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.018178 21715 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:53.018618 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1919) took 1220 ms (client timeout 1520 ms). Trace:
I20260709 12:22:53.019964 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2130) took 1612 ms. Trace:
W20260709 12:22:53.041143 18165 rpcz_store.cc:269] 0709 12:22:51.798145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.865059 (+ 66914us) service_pool.cc:224] Handling call
0709 12:22:53.018353 (+1153294us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.018535 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":803056384}
I20260709 12:22:53.041221 18019 rpcz_store.cc:276] 0709 12:22:51.407876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.428801 (+ 20925us) service_pool.cc:224] Handling call
0709 12:22:51.998943 (+570142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.066158 (+ 67215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.019738 (+953580us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b5400251550. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.019894 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308478208}
W20260709 12:22:53.020699 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2211) took 2200 ms (client timeout 1617 ms). Trace:
I20260709 12:22:52.996098 21778 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:53.026093 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2852) took 1303 ms. Trace:
I20260709 12:22:53.015597 21551 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.033721 18169 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:22:53.003276 21737 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.039233 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:53.042484 18164 rpcz_store.cc:269] 0709 12:22:50.819879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.820041 (+   162us) service_pool.cc:224] Handling call
0709 12:22:53.020485 (+2200444us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54001b6054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.020613 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":936588544}
I20260709 12:22:53.011689 21679 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.045490 21786 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:53.052110 18015 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:53.044848 18095 rpcz_store.cc:276] 0709 12:22:51.722465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.722633 (+   168us) service_pool.cc:224] Handling call
0709 12:22:51.738556 (+ 15923us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.174250 (+435694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.174288 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.600642 (+426354us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b1800392fe0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:22:52.601065 (+   423us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:22:52.632973 (+ 31908us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.990426 (+357453us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:52.990468 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.990541 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.025782 (+ 35241us) raft_consensus.cc:1685] finished
0709 12:22:53.025806 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.026014 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50694,"raft.queue_time_us":402444,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":22654,"spinlock_wait_cycles":69593728,"thread_start_us":774987,"threads_started":2,"wal-append.queue_time_us":381489}
I20260709 12:22:53.046015 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1930) took 1244 ms. Trace:
I20260709 12:22:53.056226 18169 rpcz_store.cc:276] 0709 12:22:51.801704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.525253 (+723549us) service_pool.cc:224] Handling call
0709 12:22:53.045938 (+520685us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.051628 21786 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.062541 21786 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.040547 21784 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.068877 21784 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.070322 21706 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.072163 21706 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:53.058815 21768 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.049721 21701 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.049239 21717 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:53.037925 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 212494us
I20260709 12:22:53.079450 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.081753 21717 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.083194 18167 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 12:22:53.084054 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2231) took 1311 ms. Trace:
I20260709 12:22:53.084524 18167 rpcz_store.cc:276] 0709 12:22:51.772813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.558185 (+785372us) service_pool.cc:224] Handling call
0709 12:22:53.083945 (+525760us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.086295 21682 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:53.086896 21682 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:53.087976 21682 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.048223 21752 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:53.098469 21749 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.582s	user 0.015s	sys 0.001s
I20260709 12:22:53.099123 21749 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:53.100780 21736 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.461s	user 0.004s	sys 0.007s
I20260709 12:22:53.101504 21752 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:53.086373 21744 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.003s
I20260709 12:22:53.102988 21744 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.105662 21744 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:53.101619 18811 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:53.107981 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2777) took 1664 ms. Trace:
I20260709 12:22:53.111099 18023 rpcz_store.cc:276] 0709 12:22:51.443551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.741246 (+1297695us) service_pool.cc:224] Handling call
0709 12:22:53.107685 (+366439us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400347bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.107908 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592831744}
I20260709 12:22:53.092724 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.108969 18020 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:53.110656 18015 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:22:53.222289 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2226) took 1485 ms (client timeout 1638 ms). Trace:
I20260709 12:22:52.985637 18170 rpcz_store.cc:276] 0709 12:22:51.802063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.661279 (+859216us) service_pool.cc:224] Handling call
0709 12:22:52.661874 (+   595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.662219 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:52.816845 (+154626us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.931135 (+114290us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.935168 (+  4033us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.935220 (+    52us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:52.935256 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.935315 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.935438 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19444,"prepare.queue_time_us":267719,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":2277,"raft.queue_time_us":9441,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":12965,"spinlock_wait_cycles":4472064,"thread_start_us":218800,"threads_started":2,"wal-append.queue_time_us":98087}
I20260709 12:22:53.278188 21715 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:22:53.256531 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:53.278250 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:53.225509 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2144) took 1702 ms (client timeout 1833 ms). Trace:
I20260709 12:22:53.198173 21679 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.281409 21679 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:53.280908 18021 rpcz_store.cc:269] 0709 12:22:51.523413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.594605 (+ 71192us) service_pool.cc:224] Handling call
0709 12:22:53.225317 (+1630712us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001d1854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.225437 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2475179776}
I20260709 12:22:53.284708 21679 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.286471 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.288545 18023 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:53.288684 18167 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:53.288761 18096 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:53.290779 18091 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:53.289477 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2143) took 1779 ms. Trace:
I20260709 12:22:53.290938 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:53.276137 18166 rpcz_store.cc:269] 0709 12:22:51.736688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.736822 (+   134us) service_pool.cc:224] Handling call
0709 12:22:53.221898 (+1485076us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b54003011d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.222178 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2160119040}
I20260709 12:22:53.291326 18023 rpcz_store.cc:276] 0709 12:22:51.509677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.287887 (+1778210us) service_pool.cc:224] Handling call
0709 12:22:53.288295 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.289130 (+   835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.289398 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.293800 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:53.294636 18170 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:53.295666 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:53.297101 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.298051 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:53.299517 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:53.301067 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 208458us
I20260709 12:22:53.302287 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.302975 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2791) took 1682 ms. Trace:
I20260709 12:22:53.269049 21719 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.304427 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:53.304888 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86798.8us
W20260709 12:22:53.305128 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.305202 18169 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:53.218907 21711 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:53.306730 18096 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:53.103236 21771 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:53.214172 21754 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:53.303586 18023 rpcz_store.cc:276] 0709 12:22:51.620793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.302164 (+1681371us) service_pool.cc:224] Handling call
0709 12:22:53.302480 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.302674 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.302704 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:53.302744 (+    40us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:53.302786 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.302836 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.302917 (+    81us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.285346 21755 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:53.299538 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.327392 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.308136 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.329933 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.310098 18015 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:53.312202 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:53.331142 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1932) took 1517 ms. Trace:
I20260709 12:22:53.331717 18097 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:53.332870 21539 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.331662 18172 rpcz_store.cc:276] 0709 12:22:51.813971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.845973 (+1032002us) service_pool.cc:224] Handling call
0709 12:22:52.905645 (+ 59672us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:52.906264 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.906309 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:52.913930 (+  7621us) log.cc:844] Serialized 52 byte log entry
0709 12:22:53.123198 (+209268us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:53.268238 (+145040us) log.cc:844] Serialized 14 byte log entry
0709 12:22:53.270069 (+  1831us) log.cc:844] Serialized 14 byte log entry
0709 12:22:53.270338 (+   269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.270445 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.330833 (+ 60388us) raft_consensus.cc:1685] finished
0709 12:22:53.330857 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.331045 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":145434,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":96,"raft.queue_time_us":1143,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":8221,"thread_start_us":137467,"threads_started":1,"wal-append.queue_time_us":383844}
I20260709 12:22:53.331171 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2787) took 1826 ms. Trace:
I20260709 12:22:53.334227 18015 rpcz_store.cc:276] 0709 12:22:51.504703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.285908 (+1781205us) service_pool.cc:224] Handling call
0709 12:22:53.309891 (+ 23983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.330907 (+ 21016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.331110 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.335819 18015 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:53.336422 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2793) took 1133 ms. Trace:
I20260709 12:22:53.336722 18015 rpcz_store.cc:276] 0709 12:22:52.204088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.335042 (+1130954us) service_pool.cc:224] Handling call
0709 12:22:53.335455 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.336154 (+   699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.336363 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.294426 21731 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:53.337953 21731 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.338491 18015 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:53.339664 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2794) took 1135 ms. Trace:
I20260709 12:22:53.340448 21731 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.340045 18015 rpcz_store.cc:276] 0709 12:22:52.204536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.337448 (+1132912us) service_pool.cc:224] Handling call
0709 12:22:53.337811 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.339352 (+  1541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.339594 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:53.303830 21744 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:53.305751 21649 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:53.305972 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:53.132395 21765 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.330106 21543 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:53.101612 21749 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.209122 21769 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:53.354151 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:53.407793 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 101941us
W20260709 12:22:53.408767 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54791.6us
I20260709 12:22:53.410045 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.356783 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.309098 21678 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:53.368528 21649 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.394536 21668 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:53.325165 21752 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.401569 21389 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:53.404649 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2155) took 1660 ms (client timeout 1887 ms). Trace:
I20260709 12:22:53.310791 21711 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:53.410542 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.293501 21715 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.309347 21719 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:53.395072 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2142) took 1902 ms. Trace:
I20260709 12:22:53.306500 21384 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:22:53.387063 21731 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.433408 21778 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.312547 21724 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260709 12:22:53.315035 21754 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:22:53.436949 18024 rpcz_store.cc:269] 0709 12:22:51.743704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.807789 (+ 64085us) service_pool.cc:224] Handling call
0709 12:22:53.404345 (+1596556us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b540024e354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.404547 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191249664}
I20260709 12:22:53.452996 18091 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:53.453708 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43851us
I20260709 12:22:53.472113 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:53.472915 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:53.472729 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.473985 18814 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:53.437162 21749 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.433279 21768 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.457692 18020 rpcz_store.cc:276] 0709 12:22:51.492686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.811805 (+1319119us) service_pool.cc:224] Handling call
0709 12:22:53.108729 (+296924us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.276635 (+167906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.394826 (+118191us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400347bd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.394998 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":679446400}
I20260709 12:22:53.459117 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:53.387509 21543 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:53.461172 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50773.8us
W20260709 12:22:53.461721 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 152631us
I20260709 12:22:53.463551 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.468276 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.430519 21438 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:53.433795 21744 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:53.454025 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.474988 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.430665 21649 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:53.473392 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:53.476397 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:53.485330 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.515952 21749 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.454682 21752 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.357100 21755 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.456519 21609 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:53.452402 21784 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:22:53.534962 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 178399us
I20260709 12:22:53.535355 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:53.535910 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.470196 21794 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.534814 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:53.492244 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:53.480507 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:53.537945 18167 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:53.538031 18091 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:53.479586 21728 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:53.488440 18094 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:53.516464 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.548597 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.468511 21709 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:53.485526 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.521150 18814 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.522179 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:53.558563 18099 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.525969 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.559615 18807 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:53.527273 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:53.527841 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.433176 21678 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:53.468210 21789 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:22:53.540081 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.538054 18164 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
W20260709 12:22:53.539914 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66589.2us
I20260709 12:22:53.546196 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.548887 18808 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:53.486356 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.548169 18165 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:53.552454 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.517683 21543 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.475090 21698 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:22:53.516347 21438 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:53.587007 18097 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:53.520738 21724 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:53.576596 18091 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:22:53.588500 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.585496 18020 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:22:53.519222 21744 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:53.468700 21731 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.633751 18015 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 4.
I20260709 12:22:53.519181 21768 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.593027 18024 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:53.630030 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 81546.1us
I20260709 12:22:53.667969 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.535775 21716 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.562090 18095 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:53.630182 18814 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:53.587581 18809 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:53.644021 21739 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.959s	user 0.000s	sys 0.014s
I20260709 12:22:53.642602 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.649952 18172 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:53.649864 18020 pending_rounds.cc:85] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 9
W20260709 12:22:53.652554 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64193.7us
I20260709 12:22:53.676378 18020 pending_rounds.cc:107] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:22:53.676851 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.633968 18096 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:53.676849 18020 raft_consensus.cc:2889] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:53.471169 21778 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.649634 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:53.657232 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.523841 21771 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:53.662585 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:53.522017 21798 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.179s
I20260709 12:22:53.653151 21606 raft_consensus.cc:2764] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 12:22:53.588155 18090 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:53.559209 21793 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:53.672075 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.675128 18094 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:22:53.655529 21438 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.557343 21784 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.662104 21744 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:53.710366 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.693778 21609 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:53.695920 21562 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:53.539060 21711 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.688042 18015 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:53.688021 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.685762 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.667291 21639 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:53.651978 21724 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:53.659578 21539 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:53.677977 21716 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:53.585526 21754 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.754869 21754 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.670313 21794 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:53.747236 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61568.8us
W20260709 12:22:53.658202 21691 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.116s	user 0.013s	sys 0.044s
I20260709 12:22:53.715641 21749 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:53.714592 21792 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:22:53.751849 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2037) took 1130 ms. Trace:
I20260709 12:22:53.787367 18093 rpcz_store.cc:276] 0709 12:22:52.621698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.621988 (+   290us) service_pool.cc:224] Handling call
0709 12:22:53.746561 (+1124573us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.751754 (+  5193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164946816}
I20260709 12:22:53.719635 21715 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.718835 21503 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:22:53.748425 21639 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:53.753089 21609 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:53.718966 21711 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:53.726673 21649 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.769788 21795 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:53.722807 21752 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:53.771804 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 83906.8us
I20260709 12:22:53.799317 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.799558 18809 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:53.773114 21740 ts_tablet_manager.cc:1929] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20260709 12:22:53.713755 21797 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:53.774922 21798 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:53.782428 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:53.714020 21787 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 12:22:53.807569 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:53.783206 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 246682us
I20260709 12:22:53.784098 18015 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:22:53.809213 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:53.715510 21717 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.633s	user 0.012s	sys 0.000s
I20260709 12:22:53.812059 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:53.783106 21691 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.262s	user 0.013s	sys 0.045s
I20260709 12:22:53.796983 18814 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:53.800768 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.815138 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:53.821208 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.824059 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:53.802394 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.793169 21639 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:53.792541 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123697us
I20260709 12:22:53.814646 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2789) took 2309 ms. Trace:
I20260709 12:22:53.829324 18021 rpcz_store.cc:276] 0709 12:22:51.506364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.283332 (+1776968us) service_pool.cc:224] Handling call
0709 12:22:53.283775 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.302157 (+ 18382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.302189 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:53.302767 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.390898 (+ 88131us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:53.390954 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.391173 (+   219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.814403 (+423230us) raft_consensus.cc:1685] finished
0709 12:22:53.814429 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.814578 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16943,"raft.queue_time_us":19944,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":2420,"thread_start_us":87749,"threads_started":1,"wal-append.queue_time_us":133002}
I20260709 12:22:53.816079 18811 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:53.816133 21741 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:53.816509 21539 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.816989 21346 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:53.816080 21740 log.cc:1199] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: Deleting WAL directory at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/wals/aef65d70106f4f458174493864b4ba3d
I20260709 12:22:53.818322 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2242) took 1103 ms. Trace:
W20260709 12:22:53.812284 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1926) took 2012 ms (client timeout 2069 ms). Trace:
I20260709 12:22:53.836336 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.786381 21709 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:53.843784 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.842016 18809 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:53.844496 18807 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:53.837525 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:53.837023 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.824612 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:53.842077 18173 rpcz_store.cc:269] 0709 12:22:51.799752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.466881 (+667129us) service_pool.cc:224] Handling call
0709 12:22:53.812098 (+1345217us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:53.812219 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":578551680}
I20260709 12:22:53.841495 18171 rpcz_store.cc:276] 0709 12:22:52.715331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.898663 (+183332us) service_pool.cc:224] Handling call
0709 12:22:52.899363 (+   700us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:52.920313 (+ 20950us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.920369 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:52.976432 (+ 56063us) log.cc:844] Serialized 52 byte log entry
0709 12:22:53.206034 (+229602us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:53.206105 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.206233 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.818027 (+611794us) raft_consensus.cc:1685] finished
0709 12:22:53.818056 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.818243 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1165,"raft.queue_time_us":1774,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":15454,"thread_start_us":207355,"threads_started":1,"wal-append.queue_time_us":319273}
I20260709 12:22:53.849241 17989 leader_election.cc:400] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9.
I20260709 12:22:53.837594 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.847795 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:53.843452 21539 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:53.857189 21539 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.862233 21649 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.862473 21709 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.863278 21438 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:53.806218 21724 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:53.866866 21759 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:53.845753 21609 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:53.867671 21609 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:53.868144 21609 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:53.868602 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.870164 18092 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:53.870360 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:53.836061 21741 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:53.871817 21741 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.872591 21691 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:53.874367 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.870218 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.877909 18021 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:53.878659 17989 leader_election.cc:400] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:22:53.879074 17989 leader_election.cc:403] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:53.873291 21691 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:53.884459 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:53.885581 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.888249 21691 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:53.840539 21725 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:53.895107 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47407us
I20260709 12:22:53.892066 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:53.895988 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:53.893360 21678 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:53.903882 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:53.904956 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.840842 21346 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:53.893764 21740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48546 (request call id 0) took 2514 ms. Trace:
I20260709 12:22:53.834303 21784 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:53.898950 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 432) took 2518 ms. Trace:
I20260709 12:22:53.899292 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:53.913486 18063 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.154s (SENT) [suppressed 59 similar messages]
I20260709 12:22:53.899595 21721 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:53.899806 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:22:53.916908 18022 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
W20260709 12:22:53.916525 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93093.4us
I20260709 12:22:53.920749 18018 rpcz_store.cc:276] 0709 12:22:51.380686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.380908 (+   222us) service_pool.cc:224] Handling call
0709 12:22:51.390980 (+ 10072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.391227 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:51.489540 (+ 98313us) log.cc:844] Serialized 14 byte log entry
0709 12:22:53.643149 (+2153609us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:53.887536 (+244387us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b18002e2be0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:22:53.888010 (+   474us) log.cc:844] Serialized 14 byte log entry
0709 12:22:53.898435 (+ 10425us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:53.898518 (+    83us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:53.898604 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.898690 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.898866 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13858,"prepare.queue_time_us":2336335,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":113,"raft.queue_time_us":14522,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":3803,"spinlock_wait_cycles":62610944,"thread_start_us":197100,"threads_started":2,"wal-append.queue_time_us":194507}
I20260709 12:22:53.887037 21438 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:53.920047 21346 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.920244 21389 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:22:53.931020 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:53.917685 21503 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:53.834224 21806 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:53.923414 18021 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:22:53.916940 21795 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:53.925153 21815 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:53.926117 21522 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:53.959014 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:53.961000 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:53.965484 21784 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:53.942464 21810 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:53.966596 21812 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.940999 21759 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:53.967343 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.991149 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.992236 18811 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:53.993040 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:53.993958 18814 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:53.903119 21698 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:22:53.899794 21744 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:53.973780 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99539.8us
I20260709 12:22:53.998826 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:53.977245 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:54.000278 21498 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.946816 21749 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:53.972438 21806 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:54.002976 21806 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:53.920433 21740 rpcz_store.cc:276] 0709 12:22:51.416590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.483060 (+ 66470us) service_pool.cc:224] Handling call
0709 12:22:51.740522 (+257462us) ts_tablet_manager.cc:1151] Deleting tablet aef65d70106f4f458174493864b4ba3d
0709 12:22:51.740642 (+   120us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:22:53.273799 (+1533157us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:53.772871 (+499072us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:53.893651 (+120780us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":430020,"tablet-delete.queue_time_us":188157,"thread_start_us":82128,"threads_started":1}
I20260709 12:22:53.948068 21539 raft_consensus.cc:2764] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:22:53.951015 21753 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:53.946594 21582 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.961477 21649 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:54.007519 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:54.009969 18064 leader_election.cc:400] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8.
I20260709 12:22:54.010442 18064 leader_election.cc:403] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:22:54.012331 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:53.982852 21522 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:53.996057 21707 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:54.026906 18809 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:54.026608 18814 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:53.979715 21815 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:22:54.006179 21582 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.019780 21715 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
W20260709 12:22:54.048928 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.009152 21753 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:54.017271 21698 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:54.026850 21771 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:54.028754 21522 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.029173 21707 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.041497 21778 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:54.128973 21820 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:54.100483 21771 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:54.141914 21771 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.144014 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:54.009698 21812 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.146148 21824 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:22:54.099262 21817 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:54.066979 21698 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.125252 21759 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:54.201596 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.098698 21707 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:54.312392 21498 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:54.166723 21543 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.528s	user 0.014s	sys 0.001s
I20260709 12:22:54.159622 21701 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:54.163269 21744 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:54.340301 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.147996 21812 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.322055 21682 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.326522 21698 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:54.337657 21639 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.395313 21582 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.336921 21543 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.409850 18095 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:54.384256 21754 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.629s	user 0.012s	sys 0.003s
I20260709 12:22:54.384898 21438 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8.
I20260709 12:22:54.411429 21543 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.396896 21639 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:54.381096 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.397778 18098 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:54.398875 21741 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:54.402556 21781 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:54.449620 21781 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:54.405740 21609 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.537s	user 0.013s	sys 0.002s
I20260709 12:22:54.408502 21668 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:54.424829 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 525566us
I20260709 12:22:54.462132 21668 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.463127 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:54.463440 21543 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:54.464169 21543 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.464437 21711 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:54.465101 21668 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:54.465567 21754 raft_consensus.cc:2764] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:22:54.426651 21670 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:54.466344 21670 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:54.427423 21795 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:22:54.429509 21389 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:54.469004 21721 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:54.436734 21807 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:54.470549 21807 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:54.471101 21711 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.469713 21721 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.471582 21807 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.471810 21711 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.472743 21807 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:54.442651 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.459889 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:54.463516 21765 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:54.433575 21798 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:54.469169 21389 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.470069 21754 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:54.475169 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.443962 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2168) took 2284 ms. Trace:
W20260709 12:22:54.484813 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 471727us
I20260709 12:22:54.469688 21795 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.476979 21543 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.469748 21755 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:54.482800 21678 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:54.150578 21753 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:54.473649 21721 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.490425 21389 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:54.523454 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2860) took 1089 ms. Trace:
I20260709 12:22:54.523914 18097 rpcz_store.cc:276] 0709 12:22:53.434334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.436546 (+  2212us) service_pool.cc:224] Handling call
0709 12:22:54.523230 (+1086684us) spinlock_profiling.cc:243] Waited 98.8 ms on lock 0x7b54001afc54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:54.523384 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227242880}
I20260709 12:22:54.490406 21582 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.476112 21609 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.487833 18023 rpcz_store.cc:276] 0709 12:22:52.159908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.315459 (+1155551us) service_pool.cc:224] Handling call
0709 12:22:53.315986 (+   527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.328616 (+ 12630us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.328675 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:53.329561 (+   886us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.397153 (+ 67592us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:53.712873 (+315720us) spinlock_profiling.cc:243] Waited 7.08 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:53.901244 (+188371us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.424093 (+522849us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b18002535a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:22:54.443417 (+ 19324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.443549 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.443615 (+    66us) raft_consensus.cc:1685] finished
0709 12:22:54.443635 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.443852 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":526932,"prepare.queue_time_us":505320,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2339,"raft.queue_time_us":641235,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":22462,"spinlock_wait_cycles":50802176,"thread_start_us":36170,"threads_started":1,"wal-append.queue_time_us":127164}
I20260709 12:22:54.526407 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:54.528452 18814 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:54.484238 21798 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.533756 18809 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:22:54.522276 21678 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:54.495388 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:54.511868 21711 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:54.512531 21725 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.513990 21683 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.517762 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:54.525144 18019 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:54.481451 21781 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:54.562026 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.007s	sys 0.022s
W20260709 12:22:54.563129 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.007s	sys 0.022s
W20260709 12:22:54.543814 18173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.011s	sys 0.015s
W20260709 12:22:54.564296 18173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.012s	sys 0.015s
W20260709 12:22:54.529866 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.550894 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:54.548597 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:54.566301 18171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.011s	sys 0.020s
W20260709 12:22:54.566685 18171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.011s	sys 0.020s
W20260709 12:22:54.553891 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58571.7us
I20260709 12:22:54.495254 21754 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.575554 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:54.551826 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.006s	sys 0.021s
I20260709 12:22:54.530329 21582 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:54.577020 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.007s	sys 0.021s
W20260709 12:22:54.578030 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 570213us
I20260709 12:22:54.580211 18812 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:54.565575 21709 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.046s	sys 0.084s
I20260709 12:22:54.506973 21670 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.584604 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:54.562479 21698 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:54.544041 21678 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.568900 21691 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:54.619289 21678 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.619554 21691 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.620179 21691 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:54.566670 21755 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:54.622084 21755 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.625079 21755 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:54.626744 21818 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:22:54.557082 21522 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.632342 21522 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:54.634845 21835 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:22:54.562636 21711 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:22:54.639398 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:54.586807 21709 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.047s	sys 0.085s
I20260709 12:22:54.583211 21543 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:22:54.581228 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:54.579450 21683 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:54.644270 21679 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.663667 21683 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:22:54.668221 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 89121.6us
W20260709 12:22:54.643005 21825 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:22:54.615780 21582 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:54.616937 21346 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.639s	user 0.011s	sys 0.003s
W20260709 12:22:54.643960 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 201377us
W20260709 12:22:54.683265 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.647228 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:54.648738 21498 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.649608 21749 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:54.615706 21609 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.651119 21831 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:54.586566 21781 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:54.555855 21725 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.669768 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:22:54.639487 21808 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.611719 21767 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:54.683948 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.638680 21715 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:54.692924 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:54.694744 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:54.681419 21697 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:54.715819 18023 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:54.689471 21639 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:54.684062 21522 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.658802 21707 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:54.722040 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 52400.4us
I20260709 12:22:54.665515 21683 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.663676 21679 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.689779 21346 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.712147 21749 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:22:54.758052 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 173306us
I20260709 12:22:54.695266 21498 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.760095 18023 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:54.727576 21609 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:54.771762 21798 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:54.761451 21639 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:54.773339 21749 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.774116 21639 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:54.776187 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90789us
I20260709 12:22:54.777216 18064 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:54.766850 21697 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:22:54.778537 21828 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:54.779023 21828 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:54.779083 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 443) took 2065 ms. Trace:
I20260709 12:22:54.721263 21808 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.780934 18020 rpcz_store.cc:276] 0709 12:22:52.713499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.479698 (+766199us) service_pool.cc:224] Handling call
0709 12:22:53.522080 (+ 42382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.679030 (+156950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.778739 (+1099709us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:54.778984 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144723840}
I20260709 12:22:54.725593 21715 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.806565 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:22:54.803948 18064 raft_consensus.cc:2686] T aef65d70106f4f458174493864b4ba3d P 768723648a9444fda53659e6a8cc3091: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:22:54.813516 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:54.812076 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.811874 18023 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:54.721577 21767 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.732019 21522 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.768589 21609 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:54.824496 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:54.821602 21543 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:54.780242 21828 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.828924 21346 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.828636 21697 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:54.830168 21389 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:22:54.870770 21701 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.523s	user 0.014s	sys 0.000s
I20260709 12:22:54.871898 21389 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.836593 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:54.840754 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:54.831151 21768 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:54.843369 21698 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:54.843806 21498 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:54.844609 21806 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.841s	user 0.014s	sys 0.004s
I20260709 12:22:54.851981 21522 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:54.824342 21798 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.855167 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:54.856413 21810 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.844s	user 0.008s	sys 0.005s
I20260709 12:22:54.861851 21795 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:54.835031 21609 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:54.864573 18018 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:54.871933 18166 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:22:54.841400 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:54.888120 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.852527 21678 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.895843 18020 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.860666 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2806) took 1799 ms. Trace:
I20260709 12:22:54.857662 21759 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:22:54.900147 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63650.5us
I20260709 12:22:54.901619 18023 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:54.823235 21711 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:22:54.935177 18813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 877) took 2637 ms (client timeout 2817 ms). Trace:
I20260709 12:22:54.910944 21678 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.938078 21389 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:54.911186 21697 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:54.903827 21503 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.901978 21810 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:54.966368 21522 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.900192 21798 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.015882 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:54.905699 21795 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:54.900946 21828 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.940150 18808 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:54.940120 21768 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.943864 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2878) took 1396 ms. Trace:
W20260709 12:22:54.973963 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66747.9us
W20260709 12:22:54.973198 18813 rpcz_store.cc:269] 0709 12:22:52.298224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.298739 (+   515us) service_pool.cc:224] Handling call
0709 12:22:54.911652 (+2612913us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:54.935025 (+ 23373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1239999744}
W20260709 12:22:54.968344 21834 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.291s	user 0.003s	sys 0.000s
I20260709 12:22:54.902508 18015 rpcz_store.cc:276] 0709 12:22:53.061411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.907766 (+846355us) service_pool.cc:224] Handling call
0709 12:22:53.939081 (+ 31315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.425777 (+486696us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.425877 (+   100us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.426671 (+   794us) log.cc:844] Serialized 26 byte log entry
0709 12:22:54.687141 (+260470us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:54.687193 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.687292 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.854719 (+167427us) raft_consensus.cc:1685] finished
0709 12:22:54.854764 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.860534 (+  5770us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":485866,"raft.queue_time_us":491565,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"thread_start_us":259930,"threads_started":1,"wal-append.queue_time_us":335915}
I20260709 12:22:54.897384 21806 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.013461 21522 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:54.889117 21754 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.048496 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.049242 18099 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:55.013931 21503 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.017362 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.019271 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:55.020352 18093 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:22:55.032806 18098 rpcz_store.cc:276] 0709 12:22:53.546935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.754687 (+207752us) service_pool.cc:224] Handling call
0709 12:22:53.755126 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.446785 (+691659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.943541 (+496756us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:54.943784 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1477253888}
W20260709 12:22:55.036242 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.056095 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3558112010000514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
W20260709 12:22:55.056711 18064 connection.cc:256] RPC call timeout handler was delayed by -1.882337202999679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
W20260709 12:22:55.057039 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5171287299999676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
W20260709 12:22:55.058292 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3551287300001604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
I20260709 12:22:55.013113 21678 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.043581 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.044727 18170 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:55.045351 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.045533 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.045727 18173 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:55.046484 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.040426 18097 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:55.048138 18169 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:55.064714 18167 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:55.058706 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3191287300001022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
I20260709 12:22:55.064985 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:54.980624 21749 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260709 12:22:54.988363 21389 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.042237 21701 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:55.010742 21716 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:55.040323 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88208.3us
I20260709 12:22:55.063481 18171 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:22:55.064200 18164 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
W20260709 12:22:55.065919 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2858287300000484s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.227s for a response
I20260709 12:22:55.089416 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.066535 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:54.983333 21697 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:55.067648 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.075150 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:22:55.081730 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 509) took 1591 ms. Trace:
I20260709 12:22:55.067173 18168 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:55.079679 18809 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:55.083314 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2812) took 1191 ms. Trace:
I20260709 12:22:55.101049 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.081853 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:54.980243 21711 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.042013 21756 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:55.170514 18098 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:55.171322 18018 rpcz_store.cc:276] 0709 12:22:53.891498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.033088 (+141590us) service_pool.cc:224] Handling call
0709 12:22:54.059339 (+ 26251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.900992 (+841653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.083043 (+182051us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b540004c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:55.083246 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1835452672}
I20260709 12:22:55.172171 18166 rpcz_store.cc:276] 0709 12:22:53.490386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.526943 (+ 36557us) service_pool.cc:224] Handling call
0709 12:22:55.081542 (+1554599us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54003b9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:55.081661 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3013316864}
I20260709 12:22:55.165439 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.053483 21759 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:55.076781 21522 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.048368 21707 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.190307 21716 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:55.175364 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2810) took 1426 ms. Trace:
I20260709 12:22:55.103840 21733 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.110s
I20260709 12:22:55.200294 21716 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:55.200251 18021 rpcz_store.cc:276] 0709 12:22:53.749089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.882786 (+133697us) service_pool.cc:224] Handling call
0709 12:22:55.175142 (+1292356us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b5400250b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:55.175295 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84120320}
I20260709 12:22:55.201475 21716 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.201977 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:22:55.189976 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114951us
I20260709 12:22:55.204308 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.205173 21707 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:55.049507 21808 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.172807 18172 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:55.207376 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.208772 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:55.190838 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84763.9us
I20260709 12:22:55.039744 21768 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.085413 21715 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.122789 21562 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.081498 21389 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.178406 21697 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.080891 21806 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.191907 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.193179 18167 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:55.215517 18020 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:55.085976 21701 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:55.104810 21678 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.235805 21771 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:55.235549 21678 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.236402 21771 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:55.236910 21771 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:55.242491 21781 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.531s	user 0.007s	sys 0.010s
I20260709 12:22:55.198959 21756 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.244974 21697 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.216584 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.218957 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.053328 21725 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.226504 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.256310 18173 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:55.069432 21798 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:22:55.211577 21741 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.762s	user 0.013s	sys 0.001s
I20260709 12:22:55.211879 21795 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.259469 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:22:55.084481 21807 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.611s	user 0.012s	sys 0.005s
I20260709 12:22:55.057857 21810 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:55.261399 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.224228 21562 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.272461 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:55.228714 21389 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.211097 21346 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:22:55.280344 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:55.279340 18096 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:55.281306 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.215205 21709 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:22:55.284191 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.248050 21701 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:55.276822 21691 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.656s	user 0.018s	sys 0.000s
I20260709 12:22:55.290853 18098 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:55.273922 21639 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:55.266716 21831 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:55.286458 21682 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.923s	user 0.014s	sys 0.002s
I20260709 12:22:55.307075 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:22:55.312776 21781 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:55.313436 21798 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.314039 21798 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.316500 21798 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.288725 21543 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:55.290303 21755 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:22:55.288614 21741 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.320801 21741 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.321527 21562 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.302208 21808 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:55.322418 21808 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.322906 21808 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:55.326658 21678 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.286752 21829 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:22:55.329262 21829 consensus_queue.cc:1243] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:55.286968 21699 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:22:55.299958 21691 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.334270 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.335107 21768 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.336118 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.337006 18098 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:22:55.338133 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 246390us
I20260709 12:22:55.341423 18017 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:55.343259 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.344015 18017 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:55.344547 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:55.345160 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:55.345944 18017 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.356530 21768 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:55.357344 21768 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:55.358508 21753 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.358374 21768 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.359133 21753 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.356606 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.362356 18018 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:55.294193 21749 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:55.291550 21756 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:55.356897 21849 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:22:55.372880 18017 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:55.381117 18096 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:55.385663 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.387579 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.388556 18096 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:55.389686 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.391772 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.393507 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.394516 18015 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:55.395975 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.397032 18098 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:55.304268 21831 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:22:55.390753 18096 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:55.400403 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.400969 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:22:55.402164 21698 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:55.402693 21698 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.403187 21698 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.392843 18023 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:55.311331 21639 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:55.414942 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.290251 21389 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } }
I20260709 12:22:55.390908 21582 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.415776 18018 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:22:55.423353 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:55.424235 18023 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:55.425539 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2804) took 2405 ms. Trace:
I20260709 12:22:55.426028 18024 rpcz_store.cc:276] 0709 12:22:53.020370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.668033 (+647663us) service_pool.cc:224] Handling call
0709 12:22:53.668638 (+   605us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.671582 (+  2944us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.671637 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:53.672395 (+   758us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.358856 (+1686461us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b180014c2c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:22:55.388381 (+ 29525us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:55.388429 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.388508 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:55.425152 (+ 36644us) raft_consensus.cc:1685] finished
0709 12:22:55.425175 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.425460 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28967,"raft.queue_time_us":4484,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":20060,"spinlock_wait_cycles":37392128,"thread_start_us":29289,"threads_started":1,"wal-append.queue_time_us":1737956}
I20260709 12:22:55.429093 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:55.430907 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.430655 21639 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.432582 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:55.435304 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2140) took 4001 ms (client timeout 2193 ms). Trace:
W20260709 12:22:55.436030 18022 rpcz_store.cc:269] 0709 12:22:51.434099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.675704 (+241605us) service_pool.cc:224] Handling call
0709 12:22:55.435002 (+3759298us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:55.435193 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5050154880}
I20260709 12:22:55.437778 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:22:55.448362 21837 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.600s	user 0.001s	sys 0.000s
I20260709 12:22:55.485239 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:55.486212 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:55.486775 18807 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:55.487720 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:55.488742 18815 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:55.485527 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:55.442445 21582 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.452701 18022 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.455407 21697 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.398016 21793 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:55.463580 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2877) took 1917 ms. Trace:
I20260709 12:22:55.504807 18095 rpcz_store.cc:276] 0709 12:22:53.546622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.720675 (+174053us) service_pool.cc:224] Handling call
0709 12:22:53.721070 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.410388 (+689318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.463249 (+1052861us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b5400052154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:55.463489 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1583438592}
I20260709 12:22:55.463902 21741 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.464617 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2876) took 1918 ms. Trace:
I20260709 12:22:55.465432 21711 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.466105 21691 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.293252 21759 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.466673 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.509318 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:55.510735 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.511700 18097 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:22:55.468016 21849 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:55.468302 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:55.469704 21850 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:22:55.407243 21831 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:55.507197 18094 rpcz_store.cc:276] 0709 12:22:53.546190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.717416 (+171226us) service_pool.cc:224] Handling call
0709 12:22:53.748020 (+ 30604us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.753910 (+  5890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.753946 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:53.754498 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.802626 (+ 48128us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:53.802666 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.802743 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.969456 (+166713us) raft_consensus.cc:1685] finished
0709 12:22:53.969489 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.464303 (+1494814us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b5400213c50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:55.464549 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5256,"raft.queue_time_us":6583,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":69244288,"thread_start_us":47630,"threads_started":1,"wal-append.queue_time_us":161687,"wal-append.run_cpu_time_us":2082,"wal-append.run_wall_time_us":1050422}
I20260709 12:22:55.516984 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:55.471338 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.519232 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.520124 18091 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:55.522500 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:55.401456 21781 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:22:55.534238 21828 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:55.534868 21828 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:55.536219 21828 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.444360 21389 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.539279 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:55.543648 21731 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.480469 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2786) took 3976 ms. Trace:
I20260709 12:22:55.483841 18809 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:55.487843 21855 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:55.442214 21771 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:55.445608 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.445273 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:22:55.472020 21802 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.701s	user 0.000s	sys 0.013s
I20260709 12:22:55.522809 18096 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:55.523366 18018 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.522136 21793 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.531108 21741 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.562465 21741 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.563936 21679 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.566069 18024 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:55.532322 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.568857 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.569558 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:22:55.549234 21582 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.551350 21771 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.570663 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.544982 18016 rpcz_store.cc:276] 0709 12:22:51.504370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.005264 (+1500894us) service_pool.cc:224] Handling call
0709 12:22:53.406279 (+401015us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.406545 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:53.948059 (+541514us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.428496 (+480437us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.428718 (+   222us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:54.428780 (+    62us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:54.428876 (+    96us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.428989 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.480117 (+1051128us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b540024e350. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:55.480326 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":936671,"prepare.queue_time_us":552111,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":96,"raft.queue_time_us":448961,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":921163520,"thread_start_us":83938,"threads_started":2,"wal-append.queue_time_us":111438,"wal-append.run_cpu_time_us":2868,"wal-append.run_wall_time_us":1369527}
I20260709 12:22:55.571575 18090 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:55.571812 21808 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.574092 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.549765 18098 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:55.546685 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.582896 18099 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:55.550192 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:55.584180 18173 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:55.584596 18173 raft_consensus.cc:614] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:55.586105 18090 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:55.475854 21753 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.546382 21855 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:55.547878 21842 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.572s	user 0.000s	sys 0.001s
W20260709 12:22:55.561450 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.575197 21719 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.575099 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:55.616763 21770 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.575868 21741 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:55.576963 18021 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:55.587992 18018 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:55.617580 21582 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.571880 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:55.590806 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121974us
I20260709 12:22:55.521351 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.559897 21831 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:55.649353 18021 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:55.597398 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.612803 18165 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:22:55.619252 18024 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:55.512672 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:22:55.623602 18090 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:55.597296 18022 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:22:55.469837 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.621688 18023 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:22:55.617385 21723 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.649887 21711 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.659662 21723 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.660012 21711 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:55.592864 21815 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.494s	user 0.060s	sys 0.111s
W20260709 12:22:55.662150 21815 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.592s	user 0.061s	sys 0.112s
I20260709 12:22:55.649938 21770 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.666036 21719 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.666850 21719 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.670498 18090 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:55.672063 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:55.672873 18173 raft_consensus.cc:606] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.673591 18093 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:55.629163 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.674350 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.618108 21806 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:55.639619 18091 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:22:55.680966 18094 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:55.681761 18091 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:55.683485 18091 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:55.685034 18091 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:55.655705 18097 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:55.687098 18097 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:55.632936 21701 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:55.632418 21609 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.721s	user 0.014s	sys 0.005s
I20260709 12:22:55.639099 18016 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:22:55.628136 18809 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:22:55.688508 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48991.1us
I20260709 12:22:55.683633 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:55.622949 21683 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.699491 18099 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:55.702764 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:55.707986 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:55.711740 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.712757 18166 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:55.670499 21817 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:55.673914 21741 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:22:55.727007 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.728637 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.729177 18166 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:55.729487 18171 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:55.732918 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.738799 18168 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:55.739117 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:55.732806 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.740767 18173 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:22:55.698534 21770 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:55.742858 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 124347us
I20260709 12:22:55.745918 21756 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:55.738569 21690 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:55.718936 21704 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:55.770203 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:55.649248 21582 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.718350 21562 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:55.724030 21817 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:22:55.727617 21808 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.692708 21806 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.787065 18815 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEARNER]: Advancing to term 11
I20260709 12:22:55.774580 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:22:55.737886 21719 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.698449 21682 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 12:22:55.741328 21716 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:55.788851 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:55.794816 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:55.719099 21753 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:22:55.703780 21609 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.716414 21683 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.724157 21679 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:22:55.762639 21522 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:22:55.879787 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.033551 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
W20260709 12:22:55.882638 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.010s	sys 0.029s
W20260709 12:22:56.035049 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.011s	sys 0.030s
I20260709 12:22:55.886452 18092 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:56.039237 21498 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.039868 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2042) took 1026 ms. Trace:
I20260709 12:22:56.040340 21498 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.889519 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.888651 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:22:56.037334 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 730361us
I20260709 12:22:56.042747 18091 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:56.043557 18064 leader_election.cc:400] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:22:55.906996 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.908800 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.794919 21704 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:55.910457 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.935693 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.937744 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.950366 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:55.066574 17906 catalog_manager.cc:5028] TS 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): tablet aef65d70106f4f458174493864b4ba3d (table test-workload [id=1d3d4360c24e4b96b93b2a24785bf894]) successfully deleted
I20260709 12:22:55.959602 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.964210 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:55.978225 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:22:55.982784 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.988168 18090 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:22:55.989053 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.004366 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.006160 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 705) took 1202 ms. Trace:
I20260709 12:22:56.008046 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:56.012743 18809 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:56.014438 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:56.018843 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 886) took 1480 ms. Trace:
I20260709 12:22:56.023972 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:56.040408 18092 rpcz_store.cc:276] 0709 12:22:55.013211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.018191 (+  4980us) service_pool.cc:224] Handling call
0709 12:22:55.018564 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.038425 (+1019861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.039413 (+   988us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54002963d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:22:56.039732 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993505664}
I20260709 12:22:55.901163 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:55.802129 21711 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:55.797250 21723 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.115271 21699 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:56.118144 18807 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:22:56.118685 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.118419 21649 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:22:56.119254 18096 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 10 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:22:56.119702 21498 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.120466 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.121760 18024 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:22:56.123706 21704 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.124832 18815 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:55.793016 21716 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:56.159770 21716 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:55.908696 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.162817 18170 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:22:56.163117 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.164077 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.164511 21701 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.165014 18170 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:56.166922 21806 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:55.787135 21868 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.837277 21871 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:56.074433 18806 rpcz_store.cc:276] 0709 12:22:54.803496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.804008 (+   512us) service_pool.cc:224] Handling call
0709 12:22:54.804404 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.804844 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.804877 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.667693 (+862816us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b2c000fafd8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:22:55.668274 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.794846 (+126572us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:55.794886 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.794991 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.005266 (+210275us) raft_consensus.cc:1685] finished
0709 12:22:56.005310 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.005774 (+   464us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":8466,"spinlock_wait_cycles":8095488,"thread_start_us":126155,"threads_started":1,"wal-append.queue_time_us":163424}
I20260709 12:22:56.094465 18097 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:22:56.168740 21871 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:56.095485 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:56.169536 21871 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:56.170051 18810 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:56.073963 18090 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:22:56.171083 18090 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:56.075306 18814 rpcz_store.cc:276] 0709 12:22:54.538461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.541400 (+  2939us) service_pool.cc:224] Handling call
0709 12:22:54.542064 (+   664us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.542745 (+   681us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:54.557573 (+ 14828us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.352584 (+795011us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b1800439420. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:22:55.735798 (+383214us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.736323 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.736520 (+   197us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:55.736557 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.736628 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.017650 (+281022us) raft_consensus.cc:1685] finished
0709 12:22:56.017929 (+   279us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.018484 (+   555us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31159,"prepare.queue_time_us":14445,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":163588,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":12025,"spinlock_wait_cycles":11196160,"thread_start_us":351461,"threads_started":2,"wal-append.queue_time_us":1022732}
W20260709 12:22:56.105736 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62513.6us
I20260709 12:22:56.097129 21609 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.176509 21765 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:56.097180 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.099404 21503 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.083313 18168 raft_consensus.cc:606] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:22:56.178357 21857 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:56.179144 18168 raft_consensus.cc:614] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:56.179322 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:56.180266 18015 raft_consensus.cc:2376] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9. Re-sending same reply.
I20260709 12:22:56.113265 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.180550 21857 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:55.786638 21562 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:56.127975 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:22:56.142863 21690 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:22:56.143573 18813 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:22:56.144220 17906 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.09505e+06us
I20260709 12:22:56.144311 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:22:56.146003 21389 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.147446 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:56.149566 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:55.777511 21698 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.150828 21582 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:56.151744 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.193507 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.151901 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.152745 18023 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11. Re-sending same reply.
I20260709 12:22:56.130368 18021 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:56.162220 21707 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:22:55.803355 21768 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:22:56.164153 18017 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:55.834401 21831 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.263361 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:22:56.095192 18173 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:22:56.168879 21868 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.177639 21765 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:55.829775 21855 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.118424 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:56.192613 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.173386 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:56.204739 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112833us
W20260709 12:22:56.188162 18020 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.047s	sys 0.100s
I20260709 12:22:56.217641 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.220299 18166 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:56.228159 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:56.281961 18170 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:22:56.282171 18023 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:56.282699 21683 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.225544 21562 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.283046 18023 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:22:56.283186 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.283844 18023 raft_consensus.cc:606] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:56.284801 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.289916 21831 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.290884 21831 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.246320 18022 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:56.291715 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:56.292387 18172 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.552s	user 0.015s	sys 0.012s
I20260709 12:22:56.292754 18167 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:22:56.292944 18172 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:22:56.246719 18021 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:56.296533 21831 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:56.298802 21771 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:56.299506 21724 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:56.300782 21771 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:56.301364 21771 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:56.229550 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:56.249878 21389 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:56.251144 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.252135 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.254644 18811 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:22:56.257153 18015 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:56.304840 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:56.257750 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.258296 18813 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:22:56.254873 17910 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.382s	user 0.003s	sys 0.004s
I20260709 12:22:56.291371 21868 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
W20260709 12:22:56.256022 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84758.9us
I20260709 12:22:56.316147 21389 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.309082 18016 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
W20260709 12:22:56.311697 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 126028us
I20260709 12:22:56.317034 18024 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:56.333698 18814 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:56.340377 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:56.342423 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:56.946115 18810 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:22:56.343837 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:22:56.947551 18806 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
W20260709 12:22:56.346127 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 173269us
I20260709 12:22:56.949930 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:22:56.346340 21768 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.952325 21768 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.956317 21879 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:56.956925 21879 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:56.957396 21879 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:56.958148 21868 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:56.247210 18091 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:56.961069 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2910) took 1073 ms. Trace:
I20260709 12:22:56.963275 18024 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:22:56.966171 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:22:56.966993 18024 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:56.358990 21683 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:56.970775 21683 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:22:56.971274 21683 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:56.973697 21831 raft_consensus.cc:2764] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:22:56.390738 21810 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:56.974908 21810 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:56.976115 21810 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:56.396318 21670 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:56.400319 21765 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.404605 21793 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.413138 21767 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:22:56.423766 18020 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.051s	sys 0.100s
I20260709 12:22:56.397979 21871 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.427280 21771 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.436646 21698 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:56.414583 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.309163 18169 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:22:56.984970 21898 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:59.783586 21898 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:22:59.784756 21898 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.899052 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 12.974s	user 0.257s	sys 0.097s Metrics: {"bytes_written":30861,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":322580,"dirs.run_cpu_time_us":614,"dirs.run_wall_time_us":1376,"drs_written":1,"lbm_read_time_us":526,"lbm_reads_lt_1ms":4,"lbm_write_time_us":324801,"lbm_writes_10-100_ms":10,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":340668,"threads_started":2,"update_count":31,"wal-append.queue_time_us":381049}
I20260709 12:22:57.033576 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:22:57.001443 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59356.4us
W20260709 12:22:59.887282 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 2.85383e+06us
I20260709 12:22:56.813676 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:57.120961 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:22:59.892935 18023 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:22:59.893618 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2849) took 2925 ms (client timeout 1731 ms). Trace:
W20260709 12:22:59.894055 18023 rpcz_store.cc:269] 0709 12:22:56.968407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.120569 (+152162us) service_pool.cc:224] Handling call
0709 12:22:59.893492 (+2772923us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:59.895864 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2850) took 2927 ms (client timeout 2186 ms). Trace:
I20260709 12:22:56.928987 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:22:56.961475 18091 rpcz_store.cc:276] 0709 12:22:55.887328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.887620 (+   292us) service_pool.cc:224] Handling call
0709 12:22:56.960957 (+1073337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":107904}
I20260709 12:22:59.898476 21831 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:59.899015 21831 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.947373 21907 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.899354 21831 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.899734 21907 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:59.902468 21907 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:59.252090 21909 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:22:59.902976 18023 rpcz_store.cc:269] 0709 12:22:56.968780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.895622 (+2926842us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:59.895761 (+   139us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:59.903483 21909 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.906513 21909 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:57.945279 21908 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.907229 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2844) took 2945 ms (client timeout 2878 ms). Trace:
I20260709 12:22:59.907665 21908 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.906713 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:22:57.040503 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59243.8us
I20260709 12:22:56.396239 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:22:59.917721 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1945) took 3963 ms (client timeout 1967 ms). Trace:
W20260709 12:22:59.918260 18166 rpcz_store.cc:269] 0709 12:22:55.954685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.130312 (+175627us) service_pool.cc:224] Handling call
0709 12:22:59.917606 (+3787294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:59.919432 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:22:57.053102 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:59.921506 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1960) took 3918 ms (client timeout 2086 ms). Trace:
W20260709 12:22:59.921972 18164 rpcz_store.cc:269] 0709 12:22:56.049170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.052613 (+1003443us) service_pool.cc:224] Handling call
0709 12:22:59.921411 (+2868798us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:59.923722 18164 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:59.924414 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1950) took 3967 ms. Trace:
I20260709 12:22:59.924729 18164 rpcz_store.cc:276] 0709 12:22:55.999121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.922755 (+3923634us) service_pool.cc:224] Handling call
0709 12:22:59.923260 (+   505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.924110 (+   850us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.924314 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.359088 18167 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:22:59.926752 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1959) took 3923 ms (client timeout 3111 ms). Trace:
W20260709 12:22:59.927160 18167 rpcz_store.cc:269] 0709 12:22:56.003123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.293502 (+290379us) service_pool.cc:224] Handling call
0709 12:22:59.926661 (+3633159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:59.928493 18167 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:59.929209 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1958) took 3926 ms. Trace:
I20260709 12:22:59.232589 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:56.943367 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:22:59.930943 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1947) took 3975 ms (client timeout 4450 ms). Trace:
W20260709 12:22:59.931461 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1951) took 3931 ms (client timeout 3332 ms). Trace:
W20260709 12:22:59.931382 18170 rpcz_store.cc:269] 0709 12:22:55.955856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.295337 (+339481us) service_pool.cc:224] Handling call
0709 12:22:59.232397 (+2937060us) spinlock_profiling.cc:243] Waited 9.28 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:59.930866 (+698469us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21344640}
W20260709 12:22:59.932021 18172 rpcz_store.cc:269] 0709 12:22:55.999789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.293851 (+294062us) service_pool.cc:224] Handling call
0709 12:22:56.943030 (+649179us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:22:59.931369 (+2988339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24669440}
I20260709 12:22:56.265733 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:22:57.984493 18168 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
W20260709 12:22:56.692292 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 193837us
W20260709 12:22:57.002506 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51108.1us
W20260709 12:22:56.815061 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.037351 18807 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:22:59.943061 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 899) took 3617 ms (client timeout 3111 ms). Trace:
I20260709 12:22:56.745993 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 889) took 1984 ms. Trace:
I20260709 12:22:59.945446 18812 rpcz_store.cc:276] 0709 12:22:54.761189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.761344 (+   155us) service_pool.cc:224] Handling call
0709 12:22:54.764600 (+  3256us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.765055 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:54.798453 (+ 33398us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.017137 (+1218684us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b1800392b00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:22:56.647306 (+630169us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.702303 (+ 54997us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.725620 (+ 23317us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.725935 (+   315us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:56.725992 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.734738 (+  8746us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.744279 (+  9541us) raft_consensus.cc:1685] finished
0709 12:22:56.744312 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.745284 (+   972us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":487759,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":5825,"raft.queue_time_us":11186,"raft.run_cpu_time_us":882,"raft.run_wall_time_us":125324,"spinlock_wait_cycles":59647360,"thread_start_us":474814,"threads_started":2,"wal-append.queue_time_us":1450311,"wal-append.run_cpu_time_us":42168,"wal-append.run_wall_time_us":3582016}
I20260709 12:22:56.774047 21389 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:22:56.987095 21683 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.504081 21716 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.955083 21683 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:59.955618 21716 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:59.956394 21716 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:56.983044 21724 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:22:56.785771 21746 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:57.040146 21768 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:57.042682 21771 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:57.035511 21793 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:56.721899 21808 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:56.989735 21810 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:56.797501 21828 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:56.925670 21865 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.624s	user 0.000s	sys 0.001s
I20260709 12:22:57.039791 21871 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:57.106756 21879 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:56.734915 21877 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:22:57.955821 18065 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.000s	sys 0.012s
I20260709 12:22:59.972952 21877 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:57.962886 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.64516e+06us
I20260709 12:22:59.973961 21879 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:22:59.974740 21879 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:58.068483 21875 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.760s	user 0.005s	sys 0.011s
I20260709 12:22:59.976315 21808 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:22:59.976680 21877 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:59.977387 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2191) took 4633 ms. Trace:
I20260709 12:22:59.977536 21808 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:59.979218 21808 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.977953 18018 rpcz_store.cc:276] 0709 12:22:55.343634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.631476 (+287842us) service_pool.cc:224] Handling call
0709 12:22:55.631974 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.632508 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.632548 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.651190 (+ 18642us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.140866 (+489676us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:56.141545 (+   679us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.142109 (+   564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.142278 (+   169us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.067595 (+1925317us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:59.976800 (+1909205us) raft_consensus.cc:1685] finished
0709 12:22:59.976947 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.977132 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16785,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":8519,"raft.run_cpu_time_us":596,"raft.run_wall_time_us":43103,"spinlock_wait_cycles":21303680,"thread_start_us":205102,"threads_started":1,"wal-append.queue_time_us":539136}
W20260709 12:22:59.682894 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1948) took 3726 ms (client timeout 2201 ms). Trace:
W20260709 12:22:59.745847 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 478949us
W20260709 12:22:59.980877 18169 rpcz_store.cc:269] 0709 12:22:55.956359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.237211 (+280852us) service_pool.cc:224] Handling call
0709 12:22:59.682750 (+3445539us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:59.981045 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:22:59.749387 18138 connection.cc:256] RPC call timeout handler was delayed by -2.4975649259999955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20260709 12:22:59.752629 21704 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 12:22:57.144737 21905 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.185949 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08000be100 after lost signal to thread 16792
I20260709 12:22:59.771390 21857 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.956238 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2062) took 2007 ms. Trace:
W20260709 12:22:59.812839 18165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 3.499s	user 0.021s	sys 0.071s
I20260709 12:22:59.819391 21715 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:22:59.825327 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 8490 bytes on disk
I20260709 12:22:59.997991 21715 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.999552 21670 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:00.000120 21670 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:00.000993 21715 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.001230 21670 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.001979 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 458) took 4663 ms. Trace:
I20260709 12:23:00.002821 21670 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.991142 21857 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.004735 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1953) took 4004 ms. Trace:
I20260709 12:23:00.002431 18020 rpcz_store.cc:276] 0709 12:22:55.339027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.339197 (+   170us) service_pool.cc:224] Handling call
0709 12:22:55.339621 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.340090 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.340130 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.340690 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.503293 (+162603us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b18002c2060. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:22:56.924538 (+1421245us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:22:57.033141 (+108603us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:57.034151 (+  1010us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.123824 (+ 89673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.129374 (+  5550us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.001583 (+2872209us) raft_consensus.cc:1685] finished
0709 12:23:00.001606 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.001870 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":239,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":44713,"raft.run_cpu_time_us":428,"raft.run_wall_time_us":30829,"spinlock_wait_cycles":80848000,"thread_start_us":1614465,"threads_started":2,"wal-append.queue_time_us":842873}
I20260709 12:23:00.008621 18020 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 6
I20260709 12:23:00.008961 18020 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:23:00.009212 18020 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:59.990943 21905 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.826337 21898 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:00.012068 21905 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.012976 21879 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.843585 21679 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:00.014241 21679 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:00.015722 21879 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.015412 21679 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.016731 21679 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.858276 21913 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:59.860622 21690 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.018857 21690 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:22:59.990535 18090 rpcz_store.cc:276] 0709 12:22:55.949308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.968628 (+ 19320us) service_pool.cc:224] Handling call
0709 12:22:55.987330 (+ 18702us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.172098 (+184768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.172141 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.172880 (+   739us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.271999 (+ 99119us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:56.272052 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.272143 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.955923 (+1683780us) raft_consensus.cc:1685] finished
0709 12:22:57.955948 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.956138 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":98504,"threads_started":1,"wal-append.queue_time_us":239373,"wal-append.run_cpu_time_us":17194,"wal-append.run_wall_time_us":1641607}
I20260709 12:23:00.023524 18019 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:22:59.861567 21817 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:22:59.869017 21709 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:22:58.107393 18194 debug-util.cc:398] Leaking SignalData structure 0x7b080016ea40 after lost signal to thread 16795
I20260709 12:23:00.026404 21907 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:59.997618 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":737,"lbm_reads_lt_1ms":4}
W20260709 12:23:00.028316 18194 debug-util.cc:398] Leaking SignalData structure 0x7b080034ee00 after lost signal to thread 17956
W20260709 12:23:00.028374 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121835us
I20260709 12:23:00.029686 21913 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:00.030174 18194 debug-util.cc:398] Leaking SignalData structure 0x7b080016ef00 after lost signal to thread 18048
I20260709 12:22:57.143483 21904 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.031157 21904 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:00.031994 18194 debug-util.cc:398] Leaking SignalData structure 0x7b08002f40c0 after lost signal to thread 18123
W20260709 12:23:00.032347 18194 debug-util.cc:398] Leaking SignalData structure 0x7b080037cc40 after lost signal to thread 18124
W20260709 12:22:59.261432 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.033759 18194 debug-util.cc:398] Leaking SignalData structure 0x7b08003f70c0 after lost signal to thread 18197
I20260709 12:22:57.102346 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:00.025573 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2824) took 3902 ms (client timeout 3363 ms). Trace:
I20260709 12:23:00.036517 18016 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:00.036561 18019 rpcz_store.cc:269] 0709 12:22:56.123403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.135619 (+ 12216us) service_pool.cc:224] Handling call
0709 12:22:56.813315 (+677696us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:00.025244 (+3211929us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400347bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:00.025458 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":308851968}
I20260709 12:23:00.038872 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2843) took 3078 ms. Trace:
I20260709 12:23:00.039386 18019 rpcz_store.cc:276] 0709 12:22:56.960944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.037774 (+3076830us) service_pool.cc:224] Handling call
0709 12:23:00.038216 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.038446 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.038475 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:00.038551 (+    76us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:00.038598 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.038657 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.038799 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.070392 18024 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:23:00.074232 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2842) took 3113 ms (client timeout 1606 ms). Trace:
W20260709 12:23:00.074779 18024 rpcz_store.cc:269] 0709 12:22:56.960527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.965658 (+  5131us) service_pool.cc:224] Handling call
0709 12:23:00.074109 (+3108451us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:00.076478 18024 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:00.077262 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2846) took 3110 ms. Trace:
W20260709 12:22:56.994529 18022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.017s	sys 0.000s
W20260709 12:23:00.081501 18022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.787s	user 0.021s	sys 0.000s
I20260709 12:23:00.084481 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2835) took 3840 ms. Trace:
I20260709 12:23:00.085002 18022 rpcz_store.cc:276] 0709 12:22:56.267463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.292489 (+ 25026us) service_pool.cc:224] Handling call
0709 12:22:56.292972 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.293634 (+   662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.293669 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.294458 (+   789us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.082212 (+3787754us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:00.082264 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.084064 (+  1800us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.084133 (+    69us) raft_consensus.cc:1685] finished
0709 12:23:00.084148 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.084353 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2442,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":151,"thread_start_us":3787045,"threads_started":1,"wal-append.queue_time_us":403572,"wal-append.run_cpu_time_us":2688,"wal-append.run_wall_time_us":1002711}
I20260709 12:22:59.910881 21908 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:22:59.908360 18023 rpcz_store.cc:269] 0709 12:22:56.961356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.903995 (+2942639us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:59.904148 (+   153us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:59.261052 21910 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:00.087355 21910 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.089252 21817 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:59.934682 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1954) took 3933 ms (client timeout 2025 ms). Trace:
W20260709 12:23:00.090039 18168 rpcz_store.cc:269] 0709 12:22:56.001223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.414044 (+412821us) service_pool.cc:224] Handling call
0709 12:22:59.934573 (+3520529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:59.935086 18173 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
W20260709 12:23:00.091234 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 1943) took 4181 ms (client timeout 2291 ms). Trace:
W20260709 12:23:00.091743 18173 rpcz_store.cc:269] 0709 12:22:55.909381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.265251 (+355870us) service_pool.cc:224] Handling call
0709 12:23:00.091129 (+3825878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:00.093113 21910 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:22:59.950496 21389 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.093618 18023 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:00.094416 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2847) took 3126 ms. Trace:
W20260709 12:22:59.943547 18807 rpcz_store.cc:269] 0709 12:22:56.325057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.339912 (+ 14855us) service_pool.cc:224] Handling call
0709 12:22:59.942772 (+3602860us) spinlock_profiling.cc:243] Waited 91.6 ms on lock 0x7b5400557154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:22:59.942920 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210691840}
I20260709 12:22:59.952117 21830 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:59.957518 18171 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:59.957719 21683 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:23:00.077620 18024 rpcz_store.cc:276] 0709 12:22:56.967167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.075672 (+3108505us) service_pool.cc:224] Handling call
0709 12:23:00.076197 (+   525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.076947 (+   750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.077187 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.130741 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:00.097256 18024 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:00.097355 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:23:00.097995 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2840) took 3138 ms (client timeout 1914 ms). Trace:
W20260709 12:23:00.098496 18015 rpcz_store.cc:269] 0709 12:22:56.959700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.130323 (+170623us) service_pool.cc:224] Handling call
0709 12:23:00.097903 (+2967580us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:59.984534 18138 connection.cc:256] RPC call timeout handler was delayed by -2.4714649259999533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20260709 12:22:59.973646 18065 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.657s	user 0.000s	sys 0.015s
W20260709 12:23:00.099498 18138 connection.cc:256] RPC call timeout handler was delayed by -2.4543649259999256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260709 12:23:00.099946 18138 connection.cc:256] RPC call timeout handler was delayed by -2.4480649259999154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260709 12:23:00.100636 18138 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.556s (SENT)
I20260709 12:23:00.101078 21908 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.991451 18018 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:23:00.101897 21828 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:00.102687 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2845) took 3140 ms. Trace:
W20260709 12:22:57.519634 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.260s	user 0.016s	sys 0.057s
I20260709 12:23:00.103101 18018 rpcz_store.cc:276] 0709 12:22:56.961750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.980604 (+3018854us) service_pool.cc:224] Handling call
0709 12:23:00.102302 (+121698us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:00.103917 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.844s	user 0.018s	sys 0.059s
W20260709 12:22:59.992203 18165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.679s	user 0.023s	sys 0.071s
I20260709 12:22:59.986147 21746 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:00.106215 21869 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:00.106622 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 523) took 4200 ms. Trace:
I20260709 12:22:56.773798 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:00.026609 21709 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:00.110109 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 1941) took 4201 ms. Trace:
I20260709 12:23:00.110695 21905 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.111534 21830 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.111666 21905 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.112169 21830 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:22:59.889088 17989 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.967s (SENT) [suppressed 14 similar messages]
I20260709 12:23:00.034618 21724 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:00.021683 21690 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:22:59.875833 17920 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:00.005432 21857 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:00.035377 18194 debug-util.cc:398] Leaking SignalData structure 0x7b08003e5bc0 after lost signal to thread 18839
W20260709 12:22:59.993793 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59012.8us
I20260709 12:23:00.033655 21904 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:00.119333 18194 debug-util.cc:398] Leaking SignalData structure 0x7b08001cf1a0 after lost signal to thread 18840
I20260709 12:23:00.046236 21704 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:00.129256 21704 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.110669 18165 rpcz_store.cc:276] 0709 12:22:55.908692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.310551 (+401859us) service_pool.cc:224] Handling call
0709 12:22:56.311303 (+   752us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.312079 (+   776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.312135 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.312920 (+   785us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.105543 (+3792623us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:00.105596 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.105698 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.105756 (+    58us) raft_consensus.cc:1685] finished
0709 12:23:00.105767 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.110019 (+  4252us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":257,"thread_start_us":3792077,"threads_started":1,"wal-append.queue_time_us":437678,"wal-append.run_cpu_time_us":2256,"wal-append.run_wall_time_us":1002258}
I20260709 12:23:00.130945 17920 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 4 to 5, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:00.132311 21704 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.005191 18164 rpcz_store.cc:276] 0709 12:22:56.000723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.925690 (+3924967us) service_pool.cc:224] Handling call
0709 12:22:59.933451 (+  7761us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.004275 (+ 70824us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.004315 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:00.004365 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:00.004417 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.004481 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.004631 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:00.047526 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.030691 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:00.134191 21793 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:00.074471 21831 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.071534 18016 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:00.102689 21828 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:00.135573 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2841) took 3175 ms (client timeout 1670 ms). Trace:
W20260709 12:23:00.136039 18016 rpcz_store.cc:269] 0709 12:22:56.960125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.101840 (+141715us) service_pool.cc:224] Handling call
0709 12:23:00.135476 (+3033636us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:00.068756 18095 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:23:00.137008 21831 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.137192 21828 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:00.078663 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 87276.7us
W20260709 12:23:00.139039 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2900) took 4690 ms (client timeout 2596 ms). Trace:
W20260709 12:23:00.139675 18095 rpcz_store.cc:269] 0709 12:22:55.448654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.538722 (+ 90068us) service_pool.cc:224] Handling call
0709 12:23:00.138676 (+4599954us) spinlock_profiling.cc:243] Waited 4.52 s on lock 0x7b540011d854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:00.138875 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10391148928}
I20260709 12:23:00.094762 18023 rpcz_store.cc:276] 0709 12:22:56.967616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.087205 (+3119589us) service_pool.cc:224] Handling call
0709 12:23:00.093222 (+  6017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.094091 (+   869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.094321 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:59.897805 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 2.96877e+06us
W20260709 12:23:00.140800 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.141990 21910 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:00.142539 21910 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.142607 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:00.101856 21908 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.143548 21907 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:00.144213 21907 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.145232 21908 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.098052 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2848) took 3130 ms. Trace:
I20260709 12:23:00.142992 21910 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:00.106859 21869 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:00.111011 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 173811us
I20260709 12:23:00.147877 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:00.147519 21869 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.149290 21869 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:00.107650 18781 connection.cc:256] RPC call timeout handler was delayed by -2.6010073399997964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:23:00.107131 18171 rpcz_store.cc:276] 0709 12:22:55.906241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.914177 (+  7936us) service_pool.cc:224] Handling call
0709 12:22:55.915102 (+   925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.096294 (+4181192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.106212 (+  9918us) spinlock_profiling.cc:243] Waited 4.04 s on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:00.106486 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9295794048}
W20260709 12:22:59.889810 17919 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 292) took 17603 ms (client timeout 15000 ms). Trace:
I20260709 12:23:00.153519 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.114347 21724 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.154855 18806 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 12:23:00.156430 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.157196 18810 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:00.157281 21724 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.106920 21746 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:00.153692 17919 rpcz_store.cc:269] 0709 12:22:42.286068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.686271 (+400203us) service_pool.cc:224] Handling call
0709 12:22:59.889519 (+17203248us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:51.387271 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:51.546502 (+159231us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:51.551214 (+  4712us) write_op.cc:435] Acquired schema lock
0709 12:22:51.557699 (+  6485us) tablet.cc:662] Decoding operations
0709 12:22:51.674772 (+117073us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:51.674844 (+    72us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:51.674887 (+    43us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:22:51.676260 (+  1373us) tablet.cc:701] Row locks acquired
0709 12:22:51.676284 (+    24us) write_op.cc:260] PREPARE: finished
0709 12:22:51.676466 (+   182us) write_op.cc:270] Start()
0709 12:22:51.676611 (+   145us) write_op.cc:276] Timestamp: P: 1783599771676415 usec, L: 0
0709 12:22:51.676638 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:22:51.781125 (+104487us) log.cc:844] Serialized 6356 byte log entry
0709 12:22:53.055086 (+1273961us) op_driver.cc:464] REPLICATION: finished
0709 12:22:53.322500 (+267414us) write_op.cc:301] APPLY: starting
0709 12:22:53.322644 (+   144us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:53.323880 (+  1236us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:53.324024 (+   144us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:53.331203 (+  7179us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:53.331671 (+   468us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=32
0709 12:22:53.331726 (+    55us) write_op.cc:312] APPLY: finished
0709 12:22:53.332153 (+   427us) log.cc:844] Serialized 113 byte log entry
0709 12:22:53.387180 (+ 55027us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:53.434700 (+ 47520us) write_op.cc:454] Released schema lock
0709 12:22:53.463940 (+ 29240us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:53.522567 (+ 58627us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:53.522606 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":325975,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":265844,"apply.run_cpu_time_us":11683,"apply.run_wall_time_us":158799,"num_ops":16,"prepare.queue_time_us":292612,"prepare.run_cpu_time_us":55376,"prepare.run_wall_time_us":968275,"raft.queue_time_us":128737,"raft.run_cpu_time_us":24366,"raft.run_wall_time_us":178553,"replication_time_us":1367458,"spinlock_wait_cycles":57088,"thread_start_us":580640,"threads_started":4,"wal-append.queue_time_us":602732,"wal-append.run_cpu_time_us":26262,"wal-append.run_wall_time_us":1916915}]]}
I20260709 12:23:00.158362 21746 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:00.129558 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 3.35581e+06us
I20260709 12:23:00.132025 21905 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:00.134393 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.151669 18778 connection.cc:256] RPC call timeout handler was delayed by -3.7490717260002384s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.211s for a response
W20260709 12:23:00.134701 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104143us
I20260709 12:23:00.146994 21907 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.110486 21709 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.154492 21683 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:00.146176 18024 rpcz_store.cc:276] 0709 12:22:56.968016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.096579 (+3128563us) service_pool.cc:224] Handling call
0709 12:23:00.097029 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.097744 (+   715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.097995 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:00.030243 21913 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:59.992092 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08000d7c40 after lost signal to thread 16795
W20260709 12:23:00.170756 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08002ed320 after lost signal to thread 17956
I20260709 12:22:59.929525 18167 rpcz_store.cc:276] 0709 12:22:56.002815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.927867 (+3925052us) service_pool.cc:224] Handling call
0709 12:22:59.928246 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.928946 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.929147 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:00.047403 21670 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.163877 18807 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:00.168617 18811 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:00.173480 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800216060 after lost signal to thread 18048
W20260709 12:23:00.176779 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003d4100 after lost signal to thread 18123
W20260709 12:23:00.182298 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800341480 after lost signal to thread 18124
W20260709 12:23:00.183231 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3100 after lost signal to thread 18166
W20260709 12:23:00.183607 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800257a00 after lost signal to thread 18167
W20260709 12:23:00.183967 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08001b7a20 after lost signal to thread 18173
W20260709 12:23:00.184674 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003d6540 after lost signal to thread 18197
I20260709 12:23:00.183789 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:00.177897 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 288846us
W20260709 12:23:00.190747 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800380a00 after lost signal to thread 18778
I20260709 12:23:00.189939 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:00.191423 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800369440 after lost signal to thread 18780
W20260709 12:23:00.191792 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08001c7020 after lost signal to thread 18781
I20260709 12:23:00.190760 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.182987 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:00.174767 18137 connection.cc:256] RPC call timeout handler was delayed by -2.34809511200001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.167s for a response
W20260709 12:23:00.196609 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6080 after lost signal to thread 18807
W20260709 12:23:00.197239 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6f00 after lost signal to thread 18812
W20260709 12:23:00.198185 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800334ce0 after lost signal to thread 18839
W20260709 12:23:00.198513 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08002a4ba0 after lost signal to thread 18840
W20260709 12:23:00.198812 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08002f4ae0 after lost signal to thread 21389
W20260709 12:23:00.194614 18137 connection.cc:256] RPC call timeout handler was delayed by -2.293195112000376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260709 12:23:00.199760 18137 connection.cc:256] RPC call timeout handler was delayed by -2.150995112000146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
W20260709 12:23:00.199355 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08000f2360 after lost signal to thread 21683
W20260709 12:23:00.200306 18137 connection.cc:256] RPC call timeout handler was delayed by -1.2608951120000711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2878s for a response
I20260709 12:23:00.169260 21683 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:00.202740 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800352460 after lost signal to thread 21704
W20260709 12:23:00.203229 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003b6100 after lost signal to thread 21709
W20260709 12:23:00.203572 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080037cf80 after lost signal to thread 21715
W20260709 12:23:00.204768 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080037cf00 after lost signal to thread 21716
W20260709 12:23:00.205184 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003a7ee0 after lost signal to thread 21724
I20260709 12:23:00.205390 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:00.205471 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080037cfe0 after lost signal to thread 21731
I20260709 12:23:00.172761 21913 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.174551 21704 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.180728 21828 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:00.210315 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.181699 21690 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.210951 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080037d060 after lost signal to thread 21746
I20260709 12:23:00.207407 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:00.226605 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080037ce80 after lost signal to thread 21757
I20260709 12:23:00.211617 18809 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260709 12:23:00.210837 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:00.227164 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800381ee0 after lost signal to thread 21767
I20260709 12:23:00.214210 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.214774 21389 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.215632 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2831) took 4055 ms. Trace:
I20260709 12:23:00.218219 21919 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:23:00.216010 21907 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:00.206593 21724 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:00.230057 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:00.219962 21908 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.221000 17989 connection.cc:256] RPC call timeout handler was delayed by -2.8332932220000657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
I20260709 12:23:00.213943 21709 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:00.227566 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080034d0e0 after lost signal to thread 21768
W20260709 12:23:00.251962 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08001bd440 after lost signal to thread 21771
W20260709 12:23:00.252331 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080037da80 after lost signal to thread 21793
W20260709 12:23:00.252607 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003e8320 after lost signal to thread 21808
W20260709 12:23:00.252892 18045 debug-util.cc:398] Leaking SignalData structure 0x7b08003119e0 after lost signal to thread 21810
W20260709 12:23:00.253226 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800345420 after lost signal to thread 21828
W20260709 12:23:00.253497 18045 debug-util.cc:398] Leaking SignalData structure 0x7b080035c4e0 after lost signal to thread 21829
W20260709 12:23:00.253961 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800381340 after lost signal to thread 21865
W20260709 12:23:00.254328 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800367300 after lost signal to thread 21871
W20260709 12:23:00.254650 18045 debug-util.cc:398] Leaking SignalData structure 0x7b0800396760 after lost signal to thread 21877
I20260709 12:23:00.205574 21683 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:00.226492 18812 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:00.224423 18165 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:00.231698 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:00.236459 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.241017 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68390.9us
I20260709 12:23:00.182130 21831 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.056s
W20260709 12:23:00.242321 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53642.4us
I20260709 12:23:00.237478 21724 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:00.239941 21690 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.242813 21704 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:00.243448 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.243844 21908 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.262679 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:00.237779 21907 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.244912 21904 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:00.244232 17920 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 4 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 6 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:00.247676 18167 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:00.240469 17989 connection.cc:256] RPC call timeout handler was delayed by -2.824407865000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
W20260709 12:23:00.271473 17989 connection.cc:256] RPC call timeout handler was delayed by -2.68822597999997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2025s for a response
I20260709 12:23:00.260613 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:00.271852 17989 connection.cc:256] RPC call timeout handler was delayed by -2.3114939800002503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.225s for a response
I20260709 12:23:00.180573 21670 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.250432 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2839) took 3291 ms. Trace:
I20260709 12:23:00.250700 18164 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:00.251164 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45209.6us
I20260709 12:23:00.251324 21919 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:00.223605 21857 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:23:00.279279 21808 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:00.279486 21857 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:00.279950 21857 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:00.280578 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:00.282495 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:00.283119 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.283545 21905 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:00.283895 21817 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:00.273767 18020 rpcz_store.cc:276] 0709 12:22:56.959231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.007820 (+3048589us) service_pool.cc:224] Handling call
0709 12:23:00.008308 (+   488us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:00.040944 (+ 32636us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.040977 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:00.041724 (+   747us) log.cc:844] Serialized 52 byte log entry
0709 12:23:00.164599 (+122875us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:00.164645 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.164719 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.250126 (+ 85407us) raft_consensus.cc:1685] finished
0709 12:23:00.250155 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.250358 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":159,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":104760,"threads_started":1,"wal-append.queue_time_us":155385}
W20260709 12:23:00.255352 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84930.6us
I20260709 12:23:00.285391 18097 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [6.9-8.10]
I20260709 12:23:00.286936 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:00.281004 21857 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.260613 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.260957 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.244154 21830 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:00.272220 17989 connection.cc:256] RPC call timeout handler was delayed by -2.2750041849999434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20260709 12:23:00.277923 21690 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.283217 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.294503 17989 connection.cc:256] RPC call timeout handler was delayed by -2.1630677120001565s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2534s for a response
I20260709 12:23:00.284067 18090 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
W20260709 12:23:00.294993 17989 connection.cc:256] RPC call timeout handler was delayed by -2.1346041850001711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
I20260709 12:23:00.284597 21817 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.296694 21817 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:00.296455 21389 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:00.295961 21914 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:00.297777 21914 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:00.270370 21904 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:00.299052 21914 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.300141 21904 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.270526 21907 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.287036 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.285597 18171 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:00.229282 18017 rpcz_store.cc:276] 0709 12:22:56.160576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.245291 (+ 84715us) service_pool.cc:224] Handling call
0709 12:22:56.250166 (+  4875us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:56.258275 (+  8109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.258315 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:56.259370 (+  1055us) log.cc:844] Serialized 52 byte log entry
0709 12:23:00.104496 (+3845126us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:00.179080 (+ 74584us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.210992 (+ 31912us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.211221 (+   229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.211304 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.211343 (+    39us) raft_consensus.cc:1685] finished
0709 12:23:00.211355 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.215554 (+  4199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7141,"prepare.queue_time_us":75669,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":10954,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":2222,"spinlock_wait_cycles":28881920,"thread_start_us":3949592,"threads_started":3,"wal-append.queue_time_us":340342,"wal-append.run_cpu_time_us":3108,"wal-append.run_wall_time_us":1110543}
I20260709 12:23:00.288995 18168 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:00.240891 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51840.7us
I20260709 12:23:00.283776 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.259512 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 157401us
W20260709 12:23:00.295336 17989 connection.cc:256] RPC call timeout handler was delayed by -1.7423259800002597s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30760000000000004s for a response
I20260709 12:23:00.295922 21909 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:00.294137 21919 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:00.268620 21724 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:00.309332 17989 connection.cc:256] RPC call timeout handler was delayed by -1.6982660000003307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
W20260709 12:23:00.314489 17989 connection.cc:256] RPC call timeout handler was delayed by -1.6379660000002332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260709 12:23:00.315012 17989 connection.cc:256] RPC call timeout handler was delayed by -1.62356600000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
W20260709 12:23:00.315328 17989 connection.cc:256] RPC call timeout handler was delayed by -1.6082660000001852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
I20260709 12:23:00.286803 21704 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:00.322031 21857 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:00.323678 21918 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.315650 17989 connection.cc:256] RPC call timeout handler was delayed by -1.5710713020002913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3332s for a response
I20260709 12:23:00.284013 21905 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:00.316115 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.319305 17920 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 4 to 5, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:00.324512 17989 connection.cc:256] RPC call timeout handler was delayed by -1.422640340000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31110000000000004s for a response
I20260709 12:23:00.320922 21919 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:00.338398 17989 leader_election.cc:336] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.594s (SENT)
I20260709 12:23:00.307196 21904 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:00.349037 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:00.317166 18778 connection.cc:256] RPC call timeout handler was delayed by -3.3811918060000608s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25579999999999997s for a response
I20260709 12:23:00.330579 21905 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.331130 21908 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.331768 21910 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.277925 21670 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.330613 21918 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.335197 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:00.336261 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.336133 17920 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:00.363667 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.351727 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.355546 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.355808 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.338995 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:00.356508 21724 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.357137 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:00.348218 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:00.350800 18778 connection.cc:256] RPC call timeout handler was delayed by -3.1872557410001718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:23:00.351670 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.364125 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:00.367350 18778 connection.cc:256] RPC call timeout handler was delayed by -3.1161557410000569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
I20260709 12:23:00.364984 18814 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:00.367878 18778 connection.cc:256] RPC call timeout handler was delayed by -3.0639557409999725s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20260709 12:23:00.368177 18092 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:00.365525 18811 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
W20260709 12:23:00.368239 18778 connection.cc:256] RPC call timeout handler was delayed by -2.9203918060002252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.307s for a response
W20260709 12:23:00.369544 18778 connection.cc:256] RPC call timeout handler was delayed by -2.8411918060000971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3158s for a response
W20260709 12:23:00.369956 18778 connection.cc:256] RPC call timeout handler was delayed by -2.7034570750001876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
W20260709 12:23:00.370234 18778 connection.cc:256] RPC call timeout handler was delayed by -2.6877557410002737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260709 12:23:00.370512 18778 connection.cc:256] RPC call timeout handler was delayed by -2.4445899659999668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20260709 12:23:00.370785 18778 connection.cc:256] RPC call timeout handler was delayed by -2.3878899659998751s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:23:00.371054 18778 connection.cc:256] RPC call timeout handler was delayed by -2.2471570750003593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2154s for a response
I20260709 12:23:00.375634 18097 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:23:00.383450 18090 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:00.383649 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.386848 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.389626 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.390043 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.391245 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.398125 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.398967 18811 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:00.403192 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.395520 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.406250 18099 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 10 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:23:00.392190 18809 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:00.408188 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.411094 18097 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 12:23:00.413956 18138 connection.cc:256] RPC call timeout handler was delayed by -2.4219649259998732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
W20260709 12:23:00.414641 18138 connection.cc:256] RPC call timeout handler was delayed by -2.2806649260000995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
W20260709 12:23:00.415020 18138 connection.cc:256] RPC call timeout handler was delayed by -2.2275649260000137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21589999999999998s for a response
W20260709 12:23:00.415460 18138 connection.cc:256] RPC call timeout handler was delayed by -2.0895172020004793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260709 12:23:00.415827 18138 connection.cc:256] RPC call timeout handler was delayed by -1.9764649260000624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24380000000000002s for a response
W20260709 12:23:00.416256 18138 connection.cc:256] RPC call timeout handler was delayed by -1.6019785309999861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
W20260709 12:23:00.416676 18138 connection.cc:256] RPC call timeout handler was delayed by -1.4885229350002191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2878s for a response
I20260709 12:23:00.388773 17920 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 5 to 3, term changed from 6 to 5, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from VOTER to NON_VOTER. New cstate: current_term: 5 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:23:00.417097 18138 connection.cc:256] RPC call timeout handler was delayed by -1.3662649259999853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31160000000000004s for a response
I20260709 12:23:00.384450 21389 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:00.434614 21817 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.435989 18098 raft_consensus.cc:2368] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
I20260709 12:23:00.437491 21817 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.427034 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.420168 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.422043 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:00.427817 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:00.441951 18167 raft_consensus.cc:2376] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8. Re-sending same reply.
I20260709 12:23:00.443212 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.433614 21389 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.447660 21389 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.451556 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.452477 18806 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:00.453523 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:00.413666 21793 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:00.455495 21793 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:00.456085 21793 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:00.458302 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 101259us
W20260709 12:23:00.462244 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169480 after lost signal to thread 16792
W20260709 12:23:00.462759 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08001693c0 after lost signal to thread 16795
W20260709 12:23:00.463177 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08001696a0 after lost signal to thread 17910
W20260709 12:23:00.464298 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08001692c0 after lost signal to thread 17953
W20260709 12:23:00.464635 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800166f00 after lost signal to thread 17956
W20260709 12:23:00.465127 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169100 after lost signal to thread 17987
I20260709 12:23:00.465334 21918 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.428602 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:00.466753 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.426604 21522 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:00.430528 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.431227 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.505362 18807 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:00.431620 21910 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.431922 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.439388 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.511615 18170 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:23:00.420168 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:00.385514 21828 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:00.513590 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.513916 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:00.441982 21724 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:00.428121 18138 connection.cc:256] RPC call timeout handler was delayed by -1.1439649260000806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33630000000000004s for a response
W20260709 12:23:00.475281 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.465375 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08001690e0 after lost signal to thread 17988
W20260709 12:23:00.499964 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.467044 18169 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:00.517923 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800331100 after lost signal to thread 17989
W20260709 12:23:00.519536 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169180 after lost signal to thread 18015
I20260709 12:23:00.441007 18810 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:00.510258 18092 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 4: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:00.514530 18811 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:23:00.515010 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.516413 18171 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:00.516803 18097 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:23:00.524382 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.514386 18098 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:00.507345 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 196086us
W20260709 12:23:00.527014 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169520 after lost signal to thread 18016
I20260709 12:23:00.532829 18173 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:00.535854 18166 raft_consensus.cc:606] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:00.536314 18166 raft_consensus.cc:614] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:00.536769 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169a20 after lost signal to thread 18017
W20260709 12:23:00.537209 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169880 after lost signal to thread 18019
W20260709 12:23:00.540096 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.514s (SENT)
I20260709 12:23:00.540063 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.541139 18098 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:00.539755 18168 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:00.457196 21793 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.506186 21522 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:00.541759 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169a60 after lost signal to thread 18022
I20260709 12:23:00.542586 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.550230 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800169780 after lost signal to thread 18023
W20260709 12:23:00.550617 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08001694e0 after lost signal to thread 18024
I20260709 12:23:00.545693 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:00.542881 18809 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:23:00.551039 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.551280 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800166e00 after lost signal to thread 18048
I20260709 12:23:00.550308 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.514057 21828 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:00.542236 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.551888 18812 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:23:00.551891 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 213546us
I20260709 12:23:00.547355 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:00.554138 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800166ea0 after lost signal to thread 18063
W20260709 12:23:00.554458 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800166e80 after lost signal to thread 18064
W20260709 12:23:00.554709 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080030ba60 after lost signal to thread 18065
W20260709 12:23:00.555886 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002dc280 after lost signal to thread 18090
W20260709 12:23:00.556159 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080031df20 after lost signal to thread 18091
W20260709 12:23:00.556643 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080032d300 after lost signal to thread 18104
W20260709 12:23:00.557344 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080029db80 after lost signal to thread 18123
W20260709 12:23:00.557608 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080032e2c0 after lost signal to thread 18124
W20260709 12:23:00.558002 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080029db60 after lost signal to thread 18136
W20260709 12:23:00.562906 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800058cc0 after lost signal to thread 18164
W20260709 12:23:00.563347 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800345e40 after lost signal to thread 18166
W20260709 12:23:00.563609 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6060 after lost signal to thread 18167
W20260709 12:23:00.563911 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800345ea0 after lost signal to thread 18172
I20260709 12:23:00.560985 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:00.532752 21907 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.517441 21913 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.561023 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2838) took 3605 ms. Trace:
I20260709 12:23:00.558599 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:00.566754 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080030b2e0 after lost signal to thread 18173
W20260709 12:23:00.622772 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800300080 after lost signal to thread 18197
I20260709 12:23:00.595165 18098 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:00.594682 18806 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:00.641000 18092 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:00.641479 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003d8c20 after lost signal to thread 18778
I20260709 12:23:00.615024 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:00.641909 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003add20 after lost signal to thread 18780
I20260709 12:23:00.617506 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.640331 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.647809 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002442a0 after lost signal to thread 18781
W20260709 12:23:00.687132 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080033e5e0 after lost signal to thread 18807
W20260709 12:23:00.687541 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003d5fe0 after lost signal to thread 18812
W20260709 12:23:00.687762 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003a1860 after lost signal to thread 18814
W20260709 12:23:00.688361 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080038b3e0 after lost signal to thread 18839
I20260709 12:23:00.648883 21522 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.650756 18095 raft_consensus.cc:2368] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
W20260709 12:23:00.654675 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123529us
I20260709 12:23:00.655335 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:00.651073 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.657859 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:00.692833 18811 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:00.658524 21907 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.666128 21917 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:00.667887 21913 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.669461 18097 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:00.620479 18021 rpcz_store.cc:276] 0709 12:22:56.955745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.985119 (+ 29374us) service_pool.cc:224] Handling call
0709 12:22:56.985586 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.094472 (+3108886us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:00.230090 (+135618us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.530565 (+300475us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:00.530643 (+    78us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:00.530722 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.560394 (+ 29672us) spinlock_profiling.cc:243] Waited 3.11 s on lock 0x7b5400108bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:00.560759 (+   365us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.560935 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1769,"prepare.queue_time_us":84635,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":148138,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":13414,"spinlock_wait_cycles":7149108096,"thread_start_us":393928,"threads_started":3,"wal-append.queue_time_us":156446}
I20260709 12:23:00.642165 21683 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:00.672039 18138 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.943s (SENT)
I20260709 12:23:00.672879 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.678624 18173 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:00.678947 18171 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:23:00.679430 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.680469 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:00.680954 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.680675 21828 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:00.645078 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.689293 18093 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 12:23:00.690886 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003668a0 after lost signal to thread 18840
I20260709 12:23:00.693085 18806 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.682653 21389 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.655241 21932 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.696235 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:00.716166 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.729939 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.593s (SENT)
I20260709 12:23:00.730204 18095 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
W20260709 12:23:00.716593 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.730870 18092 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:00.722450 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.724455 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.732585 18810 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:00.732757 18815 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:00.724946 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.726046 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:00.734016 18811 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7. Re-sending same reply.
I20260709 12:23:00.735167 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:00.735935 18811 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:00.737504 18811 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:23:00.738303 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.739831 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.740512 18098 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:00.742818 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85134.5us
I20260709 12:23:00.743805 21389 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:00.744067 21704 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:00.744632 21389 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:00.745054 21389 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:00.745432 21389 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:00.729521 21679 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:00.746481 21679 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.746866 21679 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:00.747609 18814 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:00.751983 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:00.752781 18166 raft_consensus.cc:606] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
W20260709 12:23:00.754228 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58144us
W20260709 12:23:00.754688 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82701.7us
W20260709 12:23:00.755120 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.755442 18138 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.100s (SENT)
I20260709 12:23:00.755899 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:00.733423 18808 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:23:00.743834 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:00.727342 18812 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:00.755880 21919 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:00.735986 21683 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:00.741498 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.728649 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003bf320 after lost signal to thread 21389
I20260709 12:23:00.760949 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:00.762039 17988 connection.cc:256] RPC call timeout handler was delayed by -2.66742262400021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260709 12:23:00.774462 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080031bac0 after lost signal to thread 21503
W20260709 12:23:00.774644 17988 connection.cc:256] RPC call timeout handler was delayed by -2.5270226240004376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
I20260709 12:23:00.774855 18095 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:00.775063 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003ff740 after lost signal to thread 21683
W20260709 12:23:00.762112 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75509.7us
W20260709 12:23:00.775239 17988 connection.cc:256] RPC call timeout handler was delayed by -2.081113797999933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260709 12:23:00.775490 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800192360 after lost signal to thread 21704
W20260709 12:23:00.775862 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800307b20 after lost signal to thread 21709
I20260709 12:23:00.762607 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:00.779716 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080015b2c0 after lost signal to thread 21715
W20260709 12:23:00.780354 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080035d2a0 after lost signal to thread 21716
W20260709 12:23:00.780731 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800134680 after lost signal to thread 21724
W20260709 12:23:00.781018 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003b36e0 after lost signal to thread 21731
W20260709 12:23:00.781281 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800382bc0 after lost signal to thread 21746
W20260709 12:23:00.781579 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800352a00 after lost signal to thread 21757
W20260709 12:23:00.781810 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800362cc0 after lost signal to thread 21767
W20260709 12:23:00.782100 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002f6ea0 after lost signal to thread 21768
W20260709 12:23:00.782375 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800374f00 after lost signal to thread 21771
W20260709 12:23:00.782670 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800101b40 after lost signal to thread 21793
W20260709 12:23:00.782953 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003f7dc0 after lost signal to thread 21808
W20260709 12:23:00.783234 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002a2cc0 after lost signal to thread 21810
W20260709 12:23:00.783555 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003b5960 after lost signal to thread 21828
W20260709 12:23:00.783818 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003c1600 after lost signal to thread 21829
W20260709 12:23:00.784088 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080032bc00 after lost signal to thread 21831
W20260709 12:23:00.784361 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003da660 after lost signal to thread 21837
W20260709 12:23:00.784785 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002874a0 after lost signal to thread 21865
W20260709 12:23:00.785049 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003d3180 after lost signal to thread 21871
W20260709 12:23:00.785305 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800226e60 after lost signal to thread 21877
W20260709 12:23:00.785710 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60204.4us
W20260709 12:23:00.785964 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.776626 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:00.797349 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.798234 18095 raft_consensus.cc:2376] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6. Re-sending same reply.
I20260709 12:23:00.820729 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:00.824133 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.768699 21913 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:00.833675 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:00.843870 18095 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:00.787822 21919 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:00.819550 21917 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:00.840991 18021 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:00.786491 21670 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.873907 18020 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:00.874230 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.838069 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78284.3us
W20260709 12:23:00.865653 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.870198 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.877110 18017 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:23:00.877853 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:00.876303 18806 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:00.832751 21936 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:00.882064 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:00.853842 21828 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:00.882457 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.849459 21913 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:00.887693 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 157792us
I20260709 12:23:00.891263 18095 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:00.892436 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:00.768885 21932 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:00.901193 16797 debug-util.cc:398] Leaking SignalData structure 0x7b080000fca0 after lost signal to thread 17988
I20260709 12:23:00.901197 21934 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.125s
I20260709 12:23:00.912079 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:23:00.915947 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.917397 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:00.917893 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:23:00.889597 21857 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.567s	user 0.011s	sys 0.004s
I20260709 12:23:00.933218 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:00.892517 21704 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:00.874782 21683 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:23:00.937973 18091 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:00.938894 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:00.950142 18017 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:00.939548 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.945510 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:00.899574 21907 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:00.944135 18097 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:23:00.899623 21913 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:00.966138 21910 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:00.967208 21910 raft_consensus.cc:2764] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:23:00.968256 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:00.930411 21935 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.239s	user 0.002s	sys 0.000s
I20260709 12:23:00.969569 21909 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:00.958289 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:00.959658 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65881us
W20260709 12:23:00.991744 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 3.363s (SENT)
I20260709 12:23:00.975826 21919 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:00.947458 21704 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.993108 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:00.993119 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:00.994738 18021 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:23:01.028718 18097 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:23:01.019533 18020 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:01.022048 21679 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.022974 18166 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:01.021430 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:01.023957 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.024914 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.992149 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:01.052949 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.045593 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.046319 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.047845 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.045262 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.047878 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.049219 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.113677 18024 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:01.048508 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.043464 18166 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:01.100059 18020 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:23:01.116324 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92047.7us
I20260709 12:23:01.048445 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.114204 18016 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:23:01.127722 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.042965 18170 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:01.039552 18092 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:01.132547 18015 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:01.133103 18095 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:23:01.053885 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.134238 18165 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:23:01.030774 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107057us
I20260709 12:23:01.112260 18018 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:01.112205 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:01.054332 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.052158 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:01.137996 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 221647us
W20260709 12:23:01.139473 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.647s (SENT)
W20260709 12:23:01.031726 21945 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.097s	user 0.000s	sys 0.002s
I20260709 12:23:01.029522 18017 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:01.032064 21704 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:01.065487 18171 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:01.151026 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.159843 18167 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:01.053112 21932 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.170069 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.183914 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.156844 21936 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:01.032928 21907 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:01.139263 18164 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:01.163920 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.200845 18024 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:01.048979 21679 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:01.155058 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.208719 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69278.4us
W20260709 12:23:01.209863 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.651s (SENT)
I20260709 12:23:01.190750 18094 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:23:01.196614 18018 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:23:01.211817 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:01.197587 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:01.202271 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.206003 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.213930 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:01.214283 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.214957 18021 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:01.207039 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 214942us
W20260709 12:23:01.208086 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.208148 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.206146 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.216903 18016 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:01.217758 21932 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:01.218292 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.219143 21932 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:01.219568 21932 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:01.220594 21932 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.222787 18022 raft_consensus.cc:2368] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
I20260709 12:23:01.223404 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.224241 18018 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:01.225222 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.225414 18172 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:01.230051 21934 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:01.175101 21933 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 12:23:01.231840 21933 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:01.200531 21936 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.232191 21670 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:01.229491 21389 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.232643 21936 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:01.233198 18019 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:01.234560 21389 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:01.240231 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:01.246012 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.248322 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:01.248478 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.251147 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:01.252503 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.198603 21913 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.268294 21913 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.271317 21936 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.232250 21933 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:01.274194 21936 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.274636 21933 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.246870 18164 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:01.280174 18017 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:01.254065 18020 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:01.254315 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44535.6us
I20260709 12:23:01.259274 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:01.297106 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:01.303256 18096 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:01.311151 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.311724 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.254801 21389 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:01.319519 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.319519 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.314180 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.321134 18094 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:01.276495 21933 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:01.314339 18016 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:01.229391 21679 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:23:01.320520 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:01.330008 18809 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:01.297250 21670 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:01.344962 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.325784 18092 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:01.246538 21522 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:01.321022 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:01.347206 18813 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:01.299218 21932 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:01.356668 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.355360 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58369.6us
W20260709 12:23:01.348439 21905 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.983s	user 0.008s	sys 0.009s
I20260709 12:23:01.353184 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:01.361352 21905 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.296284 21683 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:23:01.331305 21907 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.361732 18780 leader_election.cc:400] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
W20260709 12:23:01.351912 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.154s (SENT)
I20260709 12:23:01.362176 18172 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:01.337723 21914 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:01.370734 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.352638 21936 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.376144 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:01.319893 21389 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:01.374780 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.375276 18780 leader_election.cc:403] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:23:01.333583 21828 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:01.372822 18164 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:01.380692 21959 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.381207 18024 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:01.381323 21959 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.382355 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.377728 21910 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:01.383109 18015 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:01.383186 21910 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.366302 21952 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:01.384460 21952 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:01.385545 21952 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.386200 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.387746 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:01.388461 18094 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:01.366444 21938 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.372395 21944 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.342s	user 0.000s	sys 0.002s
I20260709 12:23:01.327915 21917 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:01.395921 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.377661 21934 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:01.398727 21389 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:01.435837 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.383962 21959 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:01.428464 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66857.3us
I20260709 12:23:01.372126 21960 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.429440 18020 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:01.438668 18016 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:01.438723 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:01.438926 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 179829us
I20260709 12:23:01.437281 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.376327 21914 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.445369 18023 raft_consensus.cc:2376] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:23:01.393230 21670 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:23:01.440716 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88868us
I20260709 12:23:01.452075 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.444547 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.415997 21910 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.419189 21961 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:01.427417 21958 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:23:01.398121 21793 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.453161 18020 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 5.
I20260709 12:23:01.457253 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.446863 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:01.461762 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.467876 18015 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:23:01.468065 18016 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7. Re-sending same reply.
I20260709 12:23:01.440344 21683 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.415259 21936 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:01.480532 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.486359 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.455068 21914 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:01.458163 21670 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.490509 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
W20260709 12:23:01.496593 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.435518 21828 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.463486 21961 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:01.482180 21956 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.221s	user 0.000s	sys 0.006s
I20260709 12:23:01.427604 21938 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.533994 18018 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
W20260709 12:23:01.499799 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 552549us
W20260709 12:23:01.506179 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59484.1us
I20260709 12:23:01.507061 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:01.508755 21936 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.544790 18019 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:01.451655 21947 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:23:01.512894 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:01.438738 21934 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.542033 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:01.551777 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.542138 18806 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:01.552614 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.559309 21817 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.495337 21670 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:01.562175 18091 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:23:01.563696 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:01.583711 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2914) took 1120 ms. Trace:
I20260709 12:23:01.572355 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.574998 18023 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:01.559741 21936 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:01.588059 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.589113 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:01.463928 21960 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.545461 21910 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.604136 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:01.475773 21793 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.605198 18811 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:01.605996 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.606496 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:01.610615 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 68779us
I20260709 12:23:01.612560 18020 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:01.613050 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:01.611595 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 241822us
I20260709 12:23:01.584466 18099 rpcz_store.cc:276] 0709 12:23:00.462800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.502727 (+ 39927us) service_pool.cc:224] Handling call
0709 12:23:00.516651 (+ 13924us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.534639 (+ 17988us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.534674 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.549760 (+ 15086us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.794700 (+244940us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:00.794750 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.794834 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.583314 (+788480us) raft_consensus.cc:1685] finished
0709 12:23:01.583343 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.583595 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2626,"raft.queue_time_us":2880,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7502,"thread_start_us":151076,"threads_started":1,"wal-append.queue_time_us":343428}
I20260709 12:23:01.620780 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:01.624564 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.625078 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.574550 21704 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:01.630147 21949 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.298s	user 0.000s	sys 0.002s
I20260709 12:23:01.655856 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.626061 21965 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.661307 18015 raft_consensus.cc:2376] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8. Re-sending same reply.
I20260709 12:23:01.662223 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:01.662720 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.649609 18017 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:01.596150 21817 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.684737 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:01.646862 21905 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.612049 21910 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:01.705638 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92741.3us
I20260709 12:23:01.698339 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:01.708002 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:01.710867 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96277us
I20260709 12:23:01.698872 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:01.750187 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:01.750931 18015 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:01.648102 21936 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:01.709719 18812 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:01.761710 18809 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:01.762502 18018 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:23:01.754630 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44265.5us
W20260709 12:23:01.762725 18021 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.006s
I20260709 12:23:01.765872 18021 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:01.755154 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:01.766978 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.697239 21952 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.768285 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:01.697549 21962 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:01.779855 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:01.781172 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:01.782272 18815 raft_consensus.cc:2368] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:23:01.783177 18099 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.790239 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.702778 21961 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:01.753983 21971 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.682284 21965 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.802939 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:01.817068 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.811137 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2928) took 1132 ms. Trace:
W20260709 12:23:01.822994 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67993.3us
I20260709 12:23:01.824277 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:01.824798 18094 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:01.823843 18098 rpcz_store.cc:276] 0709 12:23:00.679034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.741179 (+ 62145us) service_pool.cc:224] Handling call
0709 12:23:01.810860 (+1069681us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:01.811053 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1930089344}
W20260709 12:23:01.784013 21964 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.323s	user 0.002s	sys 0.000s
W20260709 12:23:01.839771 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 150100us
I20260709 12:23:01.843904 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.851105 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:01.858042 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.002s	sys 0.009s
W20260709 12:23:01.858680 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.002s	sys 0.009s
W20260709 12:23:01.794178 21967 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:23:01.793783 21969 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
W20260709 12:23:01.795279 21966 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:23:01.807554 21704 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:01.816147 21965 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.862272 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:01.815042 21917 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:01.865994 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.862332 18095 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:01.885699 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.809643 21679 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:01.806157 21828 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.794406 21683 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
I20260709 12:23:01.894893 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.896009 18095 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:23:01.844878 21933 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.512s	user 0.008s	sys 0.006s
I20260709 12:23:01.875162 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:01.859269 18064 connection.cc:256] RPC call timeout handler was delayed by -4.2625320759998431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
I20260709 12:23:01.817337 21817 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:01.881356 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113152us
I20260709 12:23:01.883481 18815 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:01.770673 21936 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.887213 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:01.888792 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:01.900000 21976 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:01.901798 18016 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:01.902204 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.885718 21938 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.919100 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:01.920483 18094 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:01.902212 18064 connection.cc:256] RPC call timeout handler was delayed by -4.1860320759997194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260709 12:23:01.920740 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49776.7us
W20260709 12:23:01.921249 18064 connection.cc:256] RPC call timeout handler was delayed by -4.1455320759996539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20260709 12:23:01.926548 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:01.921753 18064 connection.cc:256] RPC call timeout handler was delayed by -4.114032075999603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260709 12:23:01.928459 18064 connection.cc:256] RPC call timeout handler was delayed by -4.0969320759995753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
W20260709 12:23:01.928980 18064 connection.cc:256] RPC call timeout handler was delayed by -3.9826320759998453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
W20260709 12:23:01.929415 18064 connection.cc:256] RPC call timeout handler was delayed by -3.9295320759997594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20260709 12:23:01.929889 18064 connection.cc:256] RPC call timeout handler was delayed by -3.9295320759997594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20260709 12:23:01.930363 18064 connection.cc:256] RPC call timeout handler was delayed by -3.8845320759996866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
W20260709 12:23:01.930851 18064 connection.cc:256] RPC call timeout handler was delayed by -3.6847320759998183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
I20260709 12:23:01.932039 18811 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:01.932433 18064 connection.cc:256] RPC call timeout handler was delayed by -3.6496320759997616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
W20260709 12:23:01.933629 18064 connection.cc:256] RPC call timeout handler was delayed by -3.6469320759997572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260709 12:23:01.803306 21970 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.113s
I20260709 12:23:01.901297 21679 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:01.904675 21917 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:01.919871 21933 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:01.920413 21976 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.807988 21971 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:01.952229 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:01.931818 21709 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:01.940523 21987 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:01.963187 18098 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:01.950920 21973 raft_consensus.cc:1064] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: attempting to promote NON_VOTER 910f8c8d787e4abbb092413c1f90ebf3 to VOTER
I20260709 12:23:01.966773 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2912) took 1506 ms. Trace:
I20260709 12:23:01.943012 21960 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:01.967917 18096 rpcz_store.cc:276] 0709 12:23:00.460662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.461158 (+   496us) service_pool.cc:224] Handling call
0709 12:23:00.461904 (+   746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:01.303850 (+841946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.964623 (+660773us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:01.966686 (+  2063us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1933903616}
I20260709 12:23:01.971550 18094 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:23:01.974364 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 208704us
I20260709 12:23:01.945711 21976 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.979588 18098 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:01.971765 21987 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.985908 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:01.975386 21947 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:01.972321 21990 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:23:01.997272 21936 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:01.984311 21933 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 12:23:02.007193 21936 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:02.010293 21947 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:02.010689 21933 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:02.011595 21933 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:01.999128 21983 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:01.995172 21389 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.546s	user 0.015s	sys 0.001s
I20260709 12:23:02.013432 21983 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:02.013733 21389 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.013887 21679 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.014734 21983 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.016727 21389 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:02.001351 21960 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:02.018954 21960 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:02.019366 21960 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.017905 21917 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.023042 18015 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:02.024889 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47492us
I20260709 12:23:01.992497 21987 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.027053 18099 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:02.034540 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 183596us
I20260709 12:23:02.037250 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:02.040735 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 65525us
I20260709 12:23:02.032398 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.011438 21947 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:02.049903 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.051967 18015 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:02.057595 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:02.059316 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.060214 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:02.063061 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 199691us
I20260709 12:23:02.064930 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.066812 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.013401 21933 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.050909 21389 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:02.074813 21933 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:02.078387 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.079144 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.081046 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:02.082114 18093 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:02.082386 17988 leader_election.cc:400] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:23:02.082847 17988 leader_election.cc:403] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:23:02.046344 21947 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:02.009341 21973 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.086076 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.089496 18169 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:02.091490 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.092958 18172 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.064147 21936 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:02.097213 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.083874 21389 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:02.102777 21947 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.103201 21389 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.104427 21947 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.105880 21817 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:02.088419 21670 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.086005 21975 raft_consensus.cc:2764] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 12:23:02.110139 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48492.4us
I20260709 12:23:02.114359 21971 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:02.115396 21936 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.116222 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.118125 21988 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:02.118714 21988 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:02.118155 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.124271 21670 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:02.124807 21950 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.854s	user 0.006s	sys 0.001s
I20260709 12:23:02.111536 21817 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:02.126895 21817 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.129076 21952 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:02.132733 21960 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:02.134641 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.137189 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.142827 18099 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:02.146422 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66843.8us
W20260709 12:23:02.134581 21914 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.598s	user 0.011s	sys 0.003s
I20260709 12:23:02.150871 18093 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:02.119122 21988 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.152426 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.152642 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:02.155982 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.156641 18173 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:02.156751 18167 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.158862 21952 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
W20260709 12:23:02.160849 21954 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.866s	user 0.006s	sys 0.000s
I20260709 12:23:02.162674 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2924) took 1484 ms. Trace:
I20260709 12:23:02.163121 18097 rpcz_store.cc:276] 0709 12:23:00.677862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.038331 (+360469us) service_pool.cc:224] Handling call
0709 12:23:01.120484 (+ 82153us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:01.123469 (+  2985us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:01.123524 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:01.158472 (+ 34948us) log.cc:844] Serialized 26 byte log entry
0709 12:23:01.238476 (+ 80004us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:01.238545 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.238656 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:02.129527 (+890871us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:02.162297 (+ 32770us) raft_consensus.cc:1685] finished
0709 12:23:02.162330 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.162598 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2165,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":10937,"spinlock_wait_cycles":5759872,"thread_start_us":36982,"threads_started":1,"wal-append.queue_time_us":122299}
I20260709 12:23:02.165009 21910 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.153570 21914 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.167294 18810 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:02.168569 21914 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:23:02.175473 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.176679 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.182482 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.185088 21869 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:02.185658 21869 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:02.186722 21869 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.187809 21869 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.200698 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.201363 18172 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:02.202696 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.203539 18173 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:02.205147 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.205839 18165 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:23:02.206506 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.211758 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.151645 21965 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:02.245044 18168 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:23:02.221889 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.225885 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.252028 18173 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:02.254760 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 136864us
I20260709 12:23:02.219033 18167 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
W20260709 12:23:02.244035 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.861s (SENT)
I20260709 12:23:02.260299 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.258466 21987 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:02.268594 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.269191 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.271221 18806 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:23:02.272727 21934 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.276964 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.277309 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.277905 18173 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.278046 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.278918 18167 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:02.280993 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:02.282605 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.214771 21898 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.285734 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.286201 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.286742 18166 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:02.287019 18172 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:02.287698 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.287858 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.288508 18172 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:02.213805 21961 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.290120 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.290583 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.291349 18172 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:02.267936 21952 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.294102 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.294168 18167 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:02.244922 21965 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:02.304177 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.307816 18172 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:23:02.309134 18165 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:02.310122 18164 raft_consensus.cc:2368] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10.
I20260709 12:23:02.316874 18173 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:02.324440 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:02.329550 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.310902 21952 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.337441 21952 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:02.338449 18171 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:02.291314 21914 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:02.350070 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:02.352789 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45563.3us
W20260709 12:23:02.362310 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 181810us
I20260709 12:23:02.314626 21965 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.371335 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:02.370404 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.921s (SENT)
I20260709 12:23:02.372177 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.370357 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:02.373162 18092 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.349560 21913 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:02.380929 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.379444 21817 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:02.387616 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.391661 18164 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:02.395964 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.979s (SENT)
I20260709 12:23:02.397370 21817 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:02.397984 21934 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:02.398451 21989 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.427s	user 0.000s	sys 0.003s
I20260709 12:23:02.398726 21817 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.402079 21910 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.402885 21910 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.391505 21913 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.408720 21913 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:02.385342 21917 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:02.398584 21934 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.410707 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2069) took 1039 ms. Trace:
I20260709 12:23:02.411171 18091 rpcz_store.cc:276] 0709 12:23:01.371837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.374441 (+  2604us) service_pool.cc:224] Handling call
0709 12:23:02.410479 (+1036038us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:02.410630 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":421389184}
I20260709 12:23:02.402244 21960 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.421581 18092 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:02.423501 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 336658us
I20260709 12:23:02.428417 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.434317 21984 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:02.436036 21984 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:02.436465 21984 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:02.437507 21984 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.446791 18023 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:02.448474 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.439635 21984 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.444229 21938 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:02.456516 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60597.1us
W20260709 12:23:02.457619 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.428200 21987 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:02.458082 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.459535 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:02.459865 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:02.460314 18017 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.461954 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.463019 18166 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:02.396881 21952 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:23:02.464160 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:02.465155 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.462767 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.489575 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:02.444744 21933 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:02.454458 21704 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:02.499699 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:02.515128 22005 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:23:02.517099 21993 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:02.517668 21993 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:02.517970 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:02.518957 18017 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:02.520011 21983 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:02.520179 21934 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.520694 21983 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:02.521543 21793 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.521754 21983 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.522691 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2934) took 1157 ms. Trace:
I20260709 12:23:02.523144 18095 rpcz_store.cc:276] 0709 12:23:01.365151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.908106 (+542955us) service_pool.cc:224] Handling call
0709 12:23:02.418459 (+510353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:02.418790 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:02.418836 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:02.418910 (+    74us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:02.418985 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.419078 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.522483 (+103405us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54002fbcd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:02.522616 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172302208}
I20260709 12:23:02.518222 21993 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.519591 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.526398 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.537093 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.538743 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.539494 21998 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:02.539975 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.540191 21998 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.542080 21973 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.537757 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.540661 18164 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.465337 21938 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:02.553455 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90805us
I20260709 12:23:02.490088 21970 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.548300 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.504768 21933 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:02.564289 22012 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:23:02.568140 21998 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.502735 21960 raft_consensus.cc:2764] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
W20260709 12:23:02.546427 21976 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.004s
W20260709 12:23:02.580045 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.579960 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:02.572101 22015 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.055s	user 0.003s	sys 0.001s
I20260709 12:23:02.579730 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.583316 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:02.546938 21683 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:02.585615 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.586611 18166 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:02.576351 21973 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:02.593416 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:02.588969 21898 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.537060 21704 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.596177 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.596933 18170 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:02.598140 18173 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:02.602850 21704 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:02.591759 21971 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:02.606464 21971 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.606580 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.606971 21971 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.608100 21996 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:02.610670 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.612249 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.612782 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.585779 21976 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.614061 18169 raft_consensus.cc:2368] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
I20260709 12:23:02.614475 21389 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:02.614995 21389 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:02.615396 21389 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.615741 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.616497 18165 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:02.616726 21997 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:02.617241 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:02.618093 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.619005 21976 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:02.619680 21898 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:02.619840 21947 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:02.620339 21898 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:02.620414 21947 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:02.610196 21938 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.623281 21947 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.577265 21970 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.621690 21898 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.627525 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.628237 21970 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:02.628103 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.629652 18164 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:02.634490 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:02.630419 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52383.4us
I20260709 12:23:02.635506 18092 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:02.633172 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.633528 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.636204 18091 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:02.637434 21936 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.003s
W20260709 12:23:02.639657 22004 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.282s	user 0.003s	sys 0.000s
I20260709 12:23:02.629333 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.637948 21936 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8.
I20260709 12:23:02.642623 21936 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:02.643366 21936 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:02.643841 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:02.634055 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.644964 21975 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:02.646914 21975 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.650296 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.651299 21817 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:02.651930 21817 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:02.652885 21817 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.655856 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.656733 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:02.657258 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.657693 18171 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:02.659703 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.663301 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.666420 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:02.668480 18093 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:23:02.669198 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:02.673600 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.678741 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.678821 18814 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEARNER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:02.679834 18092 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:02.682301 21913 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.682979 21913 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.684726 21993 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.687532 21993 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:02.688269 21997 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:02.685654 21913 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.644438 21936 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.690579 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:02.691427 21913 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:02.697336 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:02.701640 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:02.693688 21988 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.526s	user 0.006s	sys 0.005s
I20260709 12:23:02.672173 21709 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:02.728947 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.729972 18171 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:02.698961 21913 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:02.680717 21970 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:02.699241 21998 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:02.757053 18780 leader_election.cc:400] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
I20260709 12:23:02.757761 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:02.675966 21965 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.703383 21993 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
W20260709 12:23:02.773387 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.784788 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:02.726533 21976 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:02.791839 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 134316us
I20260709 12:23:02.795516 18814 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:02.769308 18099 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:23:02.769738 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54997.5us
W20260709 12:23:02.773998 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.758704 21997 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:02.802966 21997 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.803067 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:02.759666 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62504.5us
I20260709 12:23:02.716022 21934 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.724741 21952 raft_consensus.cc:2764] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 12:23:02.774181 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.806566 18099 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:02.808931 21910 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.809525 21709 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:02.906054 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:02.749239 21522 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:23:02.811281 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.812292 21996 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:02.794874 21971 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:02.854013 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54022.6us
I20260709 12:23:02.770823 21679 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:23:02.921880 18099 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:02.783993 21670 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:02.906947 21961 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.611s	user 0.004s	sys 0.010s
I20260709 12:23:02.750073 21917 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.799229 21913 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.803429 21997 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.856971 18096 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:02.847663 18094 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:02.800570 21683 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:02.954885 21683 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:02.957960 21932 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:02.722466 21828 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
I20260709 12:23:02.941056 21670 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.960139 21828 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:02.960613 21828 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:02.961630 21828 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.962435 21670 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:02.962991 21828 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.964146 21965 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:02.943959 21995 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:23:02.806599 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:02.920594 21975 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:02.976809 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 170290us
I20260709 12:23:02.980237 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:02.969466 22021 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
W20260709 12:23:02.935454 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 236464us
I20260709 12:23:02.729758 21988 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:02.863794 21976 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:02.940363 21917 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:02.948050 22008 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:23:02.948546 22014 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.163s	user 0.001s	sys 0.001s
W20260709 12:23:02.950683 22024 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260709 12:23:02.913372 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.014135 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.991951 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:02.849803 21934 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:02.992895 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.926865 21984 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:23:02.920802 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:02.998713 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:02.978706 21975 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.992831 21910 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:03.013058 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 216092us
I20260709 12:23:03.017756 18095 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:03.000680 21793 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.976256 21670 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:23:03.022630 17989 leader_election.cc:400] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10.
I20260709 12:23:03.040566 17989 leader_election.cc:403] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:03.045472 18171 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:23:03.064461 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 143736us
I20260709 12:23:03.001201 21932 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
W20260709 12:23:03.079559 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.081773 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59249.4us
I20260709 12:23:03.083524 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.035778 21975 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:03.037125 21938 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:03.095559 18806 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:03.084726 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:03.101392 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.018435 21857 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260709 12:23:03.080900 21936 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:03.120806 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.081403 21709 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.126441 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:03.127578 21947 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:03.119832 21932 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:03.118387 21910 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:03.084049 21999 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.124363 21933 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:03.124547 22023 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:23:03.098641 21938 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.044389 21907 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260709 12:23:03.078680 21961 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:03.121798 21389 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.506s	user 0.014s	sys 0.000s
W20260709 12:23:03.153033 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.158403 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 149209us
I20260709 12:23:03.201408 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:03.206405 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:03.207676 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:03.172570 21938 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:03.265086 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 138736us
I20260709 12:23:03.204625 21990 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.203634 21932 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:03.155687 21999 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.111625 21976 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.278275 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:03.285157 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 213355us
I20260709 12:23:03.292814 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:03.271075 21971 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:03.246579 21961 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:03.288193 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80632us
I20260709 12:23:03.221719 21389 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.309691 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:03.315069 18023 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:03.303306 18020 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:03.284983 21857 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:03.322376 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:03.325609 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 204886us
I20260709 12:23:03.291478 21990 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:03.326767 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.291047 21988 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.331682 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:03.332676 18017 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
W20260709 12:23:03.345443 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.347421 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:03.350615 18815 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:03.328501 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:03.368712 21996 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:03.370304 22025 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:03.370826 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65565.1us
I20260709 12:23:03.371476 22009 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:03.372004 22009 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.372218 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:03.372413 22009 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.373088 21986 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:03.374487 22031 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:23:03.375738 21962 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:03.376284 21962 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:03.370051 21817 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:03.370909 22025 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:03.378175 22025 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:03.377256 21962 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.383796 21962 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:03.389846 18093 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:03.392107 21960 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:03.393357 18091 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:23:03.392696 21960 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:03.396461 21960 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.397500 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:03.403007 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:03.403831 18173 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.386901 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:03.416524 21913 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.418740 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:03.419327 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.618s (SENT)
I20260709 12:23:03.422219 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:03.321593 21683 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:03.423276 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.419339 18173 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:03.427881 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80618.3us
I20260709 12:23:03.330356 21933 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.435025 18809 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:03.369535 21996 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:03.444901 21857 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.446002 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:03.360535 21917 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:03.435525 22034 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:23:03.425091 21971 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:03.473114 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 144701us
I20260709 12:23:03.474824 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.424129 22045 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:03.434394 21793 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.443126 21990 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:03.442385 21984 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:03.484357 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:03.437836 21986 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:03.486644 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.681s (SENT)
I20260709 12:23:03.489305 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:03.494490 18170 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:03.476468 21988 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.502982 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.476411 21971 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.492441 21978 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:03.498245 21995 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:23:03.512301 21952 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:03.513069 21952 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:03.515780 21952 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:03.518035 22026 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.347s	user 0.000s	sys 0.002s
W20260709 12:23:03.525038 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.525794 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.449602 21932 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:23:03.440479 21683 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:03.486168 21990 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:03.535694 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60955.3us
I20260709 12:23:03.540566 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.473932 21919 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:23:03.538147 18023 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:03.483011 22043 raft_consensus.cc:2764] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:23:03.543416 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.491011 22045 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:03.551738 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:03.527246 22023 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:23:03.522711 21971 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:03.554059 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69887.2us
W20260709 12:23:03.531915 21997 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.575s	user 0.007s	sys 0.007s
I20260709 12:23:03.527622 21998 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.567868 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:03.573935 18812 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:03.533922 21984 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:03.605628 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52145.6us
I20260709 12:23:03.610347 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.611824 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.487668 21793 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.622064 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:03.621017 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.547096 21990 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:03.549525 21910 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:03.560822 22039 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.570600 21704 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:03.633726 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:03.558928 21932 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.551226 22046 raft_consensus.cc:2764] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 12:23:03.644562 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:03.634382 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 96790.9us
W20260709 12:23:03.649875 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 122970us
I20260709 12:23:03.656688 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:03.631131 21933 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:03.558314 21898 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.083s
I20260709 12:23:03.636448 21947 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:03.636740 22043 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.640424 21999 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:03.619449 21857 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:23:03.561475 22040 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.108s
I20260709 12:23:03.639889 21905 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:23:03.582216 21997 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.649258 21984 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:03.765988 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 106846us
I20260709 12:23:03.636482 21914 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:23:03.578068 21975 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.681121 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:03.676589 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.005s	sys 0.021s
W20260709 12:23:03.770788 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111880us
W20260709 12:23:03.795624 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.005s	sys 0.021s
I20260709 12:23:03.646745 21704 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:03.784780 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 156428us
W20260709 12:23:03.796078 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115068us
I20260709 12:23:03.796991 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.797333 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:03.653285 21910 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:03.646513 22050 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:03.787940 18173 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:03.789376 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:03.661751 21389 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.678408 21947 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:03.775242 22041 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.292s	user 0.000s	sys 0.001s
I20260709 12:23:03.683458 21961 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.768083 21932 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:03.779075 21938 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.508s	user 0.013s	sys 0.001s
I20260709 12:23:03.804996 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.664644 21998 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.130s
I20260709 12:23:03.828882 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:03.829075 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:03.829793 22046 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:23:03.830384 21991 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.154s
I20260709 12:23:03.799268 21984 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:03.838829 22053 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:03.876997 21934 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:03.877161 21997 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.840711 22009 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.824936 21932 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.878737 21932 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:03.879107 21913 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:03.821349 21828 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.856s	user 0.016s	sys 0.002s
I20260709 12:23:03.880026 22009 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.802457 21978 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:03.814039 21910 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.882105 21910 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:03.830853 22042 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.347s	user 0.000s	sys 0.001s
I20260709 12:23:03.886349 21984 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:03.889169 22047 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:23:03.890751 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 94776us
I20260709 12:23:03.778671 22045 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.897341 21975 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.846721 21933 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:03.778235 21952 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:23:03.818404 21919 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:03.677433 21986 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:03.859391 22048 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:03.860661 21683 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.818107 21947 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.809854 22043 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:03.817420 22029 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:04.086696 22043 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:04.088013 22043 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.678843 21995 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.158s
I20260709 12:23:03.824934 21679 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:03.875170 21704 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.827196 21389 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:03.876592 21965 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:03.824412 22049 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.861215 21938 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.881371 21978 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:03.891004 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 799) took 1228 ms. Trace:
I20260709 12:23:04.104295 18811 rpcz_store.cc:276] 0709 12:23:02.662321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.662517 (+   196us) service_pool.cc:224] Handling call
0709 12:23:02.662963 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:02.668305 (+  5342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:02.668344 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:02.668982 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:23:03.321546 (+652564us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b1800074f80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:03.613587 (+292041us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:03.613652 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.613770 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:03.890627 (+276857us) raft_consensus.cc:1685] finished
0709 12:23:03.890651 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.890924 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":199,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":8178,"spinlock_wait_cycles":31022976,"thread_start_us":291713,"threads_started":1,"wal-append.queue_time_us":1092000}
I20260709 12:23:04.109997 21679 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.114375 21389 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:03.645900 22039 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:04.043769 21960 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.647s	user 0.002s	sys 0.009s
W20260709 12:23:04.045621 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 145772us
I20260709 12:23:04.117658 21960 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.118227 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:04.120149 21960 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.120471 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:04.120913 21960 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:04.121395 21960 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:04.121968 21960 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:04.132576 21971 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.136547 22039 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:03.872567 22025 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:04.084945 21988 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.557s	user 0.008s	sys 0.004s
W20260709 12:23:04.072751 22011 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.245s	user 0.035s	sys 0.023s
I20260709 12:23:04.076097 21919 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.096372 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 204700us
I20260709 12:23:04.103236 21996 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:04.199102 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.027s	sys 0.078s
W20260709 12:23:04.199652 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.027s	sys 0.078s
I20260709 12:23:04.200968 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:04.113488 21679 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:04.152734 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.168169 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:04.150211 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:04.216898 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:04.156527 22011 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.641s	user 0.035s	sys 0.023s
I20260709 12:23:04.218043 18171 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
W20260709 12:23:04.218781 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68740.6us
I20260709 12:23:04.219720 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:04.159107 21990 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.506s	user 0.004s	sys 0.007s
I20260709 12:23:04.075347 22050 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:04.226434 18090 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:04.233335 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.235544 18017 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:04.239250 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.240794 18170 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:04.156746 22029 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:04.242671 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:04.217461 21936 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.077s	user 0.016s	sys 0.000s
I20260709 12:23:04.173949 21988 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:04.251068 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.254549 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.254851 18811 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:23:04.255265 18173 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:04.271456 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.256096 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:04.260190 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 431345us
I20260709 12:23:04.265115 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.272418 18094 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:04.272333 18096 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:04.217978 22039 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.266217 18093 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:04.218952 22049 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.272270 18019 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:04.227267 21965 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:23:04.283198 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 542) took 1235 ms. Trace:
I20260709 12:23:04.284602 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.287637 18171 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:04.241008 21971 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:04.244223 21389 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.290130 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.287144 18164 rpcz_store.cc:276] 0709 12:23:03.035241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.038253 (+  3012us) service_pool.cc:224] Handling call
0709 12:23:03.038801 (+   548us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:03.046251 (+  7450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:03.046304 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:03.384080 (+337776us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b2c001268f8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:03.384659 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 12:23:03.528931 (+144272us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:03.528975 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.529058 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:04.260513 (+731455us) raft_consensus.cc:1685] finished
0709 12:23:04.260543 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:04.270227 (+  9684us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12425,"raft.queue_time_us":11983,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":20279,"spinlock_wait_cycles":73133824,"thread_start_us":143717,"threads_started":1,"wal-append.queue_time_us":264247}
I20260709 12:23:04.307317 18019 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:04.238694 22050 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.296685 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:04.253911 21828 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:04.208775 21919 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.258289 21996 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:04.241782 21998 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:04.313571 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:04.266348 21936 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10.
I20260709 12:23:04.324848 18019 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:04.268522 21938 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.257048 21683 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 12:23:04.241564 21990 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.352588 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.357864 18016 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:04.354601 21998 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.363986 21998 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:04.366904 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.304142 22025 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:04.329375 21988 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:04.327239 22050 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:04.294793 22039 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.397904 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.346786 21913 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.407490 18019 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:04.415660 18021 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:04.418244 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.367684 21975 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:04.422047 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.339425 21999 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260709 12:23:04.432984 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.435853 18018 raft_consensus.cc:2368] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:04.437179 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.430398 18019 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:04.386233 21988 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:04.408510 21910 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.001s
I20260709 12:23:04.373145 22059 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:04.286312 22049 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.397079 21936 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:04.392848 21933 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:04.421051 22038 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:23:04.462241 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:04.465447 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 109691us
I20260709 12:23:04.490888 18019 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:23:04.370607 21898 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:23:04.385038 21971 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 12:23:04.456305 21988 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:04.440503 21869 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:04.496166 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:04.497187 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.509886 18171 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:04.510226 18172 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:04.463749 22059 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.478682 21936 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:04.516697 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.524392 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.524760 18164 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:04.518504 18173 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:04.509145 18170 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:04.419488 21913 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.522326 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.537161 18167 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:04.463495 21910 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:04.582748 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.583624 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.584280 18019 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:04.457651 21975 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.612289 18172 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:04.612416 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.613721 18171 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:04.593982 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.494378 21933 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.627187 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.523535 22059 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:04.628404 18016 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:04.590757 21986 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.641155 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.642045 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.643076 18018 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:04.644942 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:04.658322 21990 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:04.531790 21869 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:04.667454 22073 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:23:04.670135 22064 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:23:04.679391 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:04.682720 22062 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:23:04.683913 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:04.684957 22075 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:04.681371 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.667232 22039 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.644868 21936 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.685765 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:04.686570 22066 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:23:04.696208 22056 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:04.687168 21971 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:04.699401 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.700256 21990 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:04.702418 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:04.703881 21990 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:04.704351 21990 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.704845 21990 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:04.710690 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:04.714255 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.689039 18020 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:04.692806 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.734587 18165 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:04.715408 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:04.715467 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:04.695811 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.738587 18164 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:04.757704 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.758977 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:04.762462 18171 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:23:04.771323 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.795444 18167 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:04.713356 18096 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Committing config change with OpId 11.9: config changed from index 3 to 9, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:04.806298 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:04.689576 22039 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:04.807951 22039 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.808333 22039 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:04.810843 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.812300 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:04.688140 22070 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:23:04.691061 21936 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:04.816979 21793 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.694491 21869 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.817991 21919 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:04.818176 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:04.818379 21869 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:04.818948 18170 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:23:04.819046 21869 consensus_queue.cc:1243] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:04.819859 21913 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:04.819947 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.820544 18170 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:23:04.821398 22065 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 12:23:04.822057 22060 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.822642 22060 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:04.822966 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 112701us
I20260709 12:23:04.635426 22076 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:04.821676 18164 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:04.833236 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:04.736176 22061 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
I20260709 12:23:04.827610 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.696872 22056 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:04.828542 18167 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:04.830313 18170 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:04.836277 18095 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:04.826336 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:04.852893 18022 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:04.838017 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:04.837431 18165 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:04.868268 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:04.877537 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 366832us
I20260709 12:23:04.645960 21986 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.802314 21952 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.887734 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.003s	sys 0.012s
I20260709 12:23:04.888769 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:04.975306 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.004s	sys 0.012s
I20260709 12:23:04.975637 18024 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:23:04.870726 21960 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.748s	user 0.005s	sys 0.009s
I20260709 12:23:04.960496 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:04.872113 21983 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:04.959872 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.984938 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 545) took 1197 ms. Trace:
W20260709 12:23:04.985116 18778 connection.cc:256] RPC call timeout handler was delayed by -1.3271546510000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260709 12:23:04.985762 18778 connection.cc:256] RPC call timeout handler was delayed by -1.2902546509999411s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.211s for a response
W20260709 12:23:04.986609 18778 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.598s (SENT)
I20260709 12:23:04.987221 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:04.971392 18019 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:23:04.883585 21793 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:04.981948 18172 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:04.851179 22076 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:04.985572 18166 rpcz_store.cc:276] 0709 12:23:03.787401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.787575 (+   174us) service_pool.cc:224] Handling call
0709 12:23:03.808416 (+ 20841us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:03.871065 (+ 62649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:03.871104 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:03.875246 (+  4142us) log.cc:844] Serialized 26 byte log entry
0709 12:23:04.290183 (+414937us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:04.290258 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.290351 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:04.984613 (+694262us) raft_consensus.cc:1685] finished
0709 12:23:04.984643 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:04.984834 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61455,"raft.queue_time_us":58016,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":13949,"thread_start_us":414440,"threads_started":1,"wal-append.queue_time_us":484323}
W20260709 12:23:04.979494 22050 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.576s	user 0.009s	sys 0.016s
W20260709 12:23:05.011174 18812 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.008s	sys 0.027s
W20260709 12:23:05.012213 18812 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.009s	sys 0.027s
I20260709 12:23:05.010947 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:05.059679 18024 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:04.984391 21952 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.023756 22084 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:04.993257 21960 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:05.026003 22080 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:23:04.998804 21793 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:05.027732 22079 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:23:05.007437 21947 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.906s	user 0.016s	sys 0.000s
I20260709 12:23:05.010329 22060 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.008302 22049 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:05.038269 21996 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.695s	user 0.014s	sys 0.001s
I20260709 12:23:04.953387 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:05.045744 21986 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.048563 22011 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.009985 21932 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.131s	user 0.007s	sys 0.008s
I20260709 12:23:05.053459 21990 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:05.055946 21988 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.533s	user 0.010s	sys 0.001s
W20260709 12:23:05.064668 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 222011us
W20260709 12:23:05.067204 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 243197us
I20260709 12:23:05.076081 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:05.031865 22088 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.090160 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:05.100131 22086 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:05.113222 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:05.114612 18024 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:05.120666 22076 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.008252 22050 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:05.124768 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.990844 22056 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:05.126178 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:05.132956 21960 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.106573 22049 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:05.100175 21988 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.083559 21947 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.098968 22011 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.098587 21932 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:05.138862 22087 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:05.103139 22088 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.140458 21793 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.141834 21977 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Committing config change with OpId 5.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:05.145193 22050 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.146019 18018 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:05.146567 21683 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:05.150872 22060 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:05.193190 18023 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:05.151804 22030 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:05.161621 18024 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:05.166322 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 270107us
W20260709 12:23:05.170495 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 183927us
I20260709 12:23:05.197489 18017 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:05.128067 22086 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.182615 22056 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:05.204243 18811 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:05.207305 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:05.218808 18024 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:05.169139 22049 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.188868 21793 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.157658 22039 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.213665 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.191762 21683 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.216562 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.218747 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.232798 18812 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:05.208308 21828 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.867s	user 0.015s	sys 0.002s
I20260709 12:23:05.229928 18022 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:23:05.226271 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.235337 18020 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:23:05.238623 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.031s (SENT)
I20260709 12:23:05.239454 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:05.241554 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.202728 21898 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:05.247859 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.248878 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:05.251183 21828 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:05.255901 22096 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:05.259094 22030 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:05.259716 22030 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:05.262799 22089 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:05.266263 22095 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:05.268198 22095 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:05.269114 22088 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:05.270673 18090 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:05.271518 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:05.271661 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.069s (SENT)
I20260709 12:23:05.272593 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:05.243552 21683 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:05.272420 18812 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:23:05.200347 22060 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.181435 22087 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:05.288228 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.073s (SENT)
I20260709 12:23:05.289085 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:05.297000 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78327.1us
I20260709 12:23:05.298087 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:05.295883 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.283425 21934 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:05.283701 22050 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:05.271929 22095 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 12:23:05.307533 21932 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.271768 21898 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.310779 21704 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.246022 21793 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:05.349530 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51549.2us
I20260709 12:23:05.271478 22097 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.061s
I20260709 12:23:05.351586 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.295883 21990 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:05.353451 21947 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.355134 22086 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:05.355747 22104 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:23:05.356503 22077 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.475s	user 0.002s	sys 0.000s
I20260709 12:23:05.298840 22030 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:05.313022 22050 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.313637 22011 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:05.372584 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 84375us
W20260709 12:23:05.374928 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.110s (SENT)
W20260709 12:23:05.385444 17987 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.588s (SENT) [suppressed 55 similar messages]
I20260709 12:23:05.366572 21936 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:05.353525 21988 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.367980 22020 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:05.372001 22086 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.373301 22030 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.375640 22050 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:05.393118 21793 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.353525 22099 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:05.373302 21932 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:05.374660 22087 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:05.388267 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:05.402443 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.411831 22086 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:05.419323 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 318964us
I20260709 12:23:05.428397 18138 leader_election.cc:400] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:05.428771 18138 leader_election.cc:403] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:05.420535 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 330543us
I20260709 12:23:05.429621 22060 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.422688 22076 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:05.422224 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.437693 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.438200 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52796.6us
I20260709 12:23:05.388475 21936 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:05.421475 22057 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:05.434783 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:05.441740 18814 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:23:05.430097 22092 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:05.445580 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.460314 18811 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:05.425709 22039 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.464913 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.467430 18807 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:05.472318 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:05.473412 18808 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:05.431120 21932 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:05.476878 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.442920 22049 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.445328 22087 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:05.445112 21936 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:05.423627 22030 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:05.501562 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70898.3us
I20260709 12:23:05.503533 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:05.512257 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72340.8us
I20260709 12:23:05.516800 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:05.519816 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:05.520638 18164 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:05.496639 22059 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.842s	user 0.013s	sys 0.003s
I20260709 12:23:05.492776 22087 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:05.550633 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 175739us
W20260709 12:23:05.552071 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.183s (SENT)
I20260709 12:23:05.557744 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:05.562812 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59449.6us
I20260709 12:23:05.567905 18173 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:23:05.572211 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 137510us
I20260709 12:23:05.501166 21934 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:05.577373 22102 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:23:05.576970 21934 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:05.577744 21522 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:05.582161 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.582926 18169 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:05.584038 18164 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:05.595539 18164 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:05.573614 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:05.553553 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.558s (SENT)
I20260709 12:23:05.602177 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:05.628217 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.629367 18165 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:05.561770 22087 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:05.653707 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.654939 18169 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
W20260709 12:23:05.662596 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77463.7us
I20260709 12:23:05.580453 21970 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:05.512036 21999 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
W20260709 12:23:05.691134 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.687508 18815 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Committing config change with OpId 11.9: config changed from index 3 to 9, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:05.674540 22079 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:05.704282 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.711652 18165 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:05.714043 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 160510us
I20260709 12:23:05.587455 22056 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20260709 12:23:05.547569 22059 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.732729 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:05.734951 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 161469us
I20260709 12:23:05.736122 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:05.673766 21947 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:05.730016 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65690.9us
I20260709 12:23:05.728324 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.688512 21970 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:05.723151 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 3.070s (SENT)
I20260709 12:23:05.745903 22093 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:05.750314 22107 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:05.770269 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:05.771054 18017 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.771171 22030 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.772441 21910 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.772732 21975 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:05.687816 22092 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 12:23:05.754987 18807 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:05.724328 21999 consensus_queue.cc:1243] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260709 12:23:05.740716 22061 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:05.749099 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:05.787321 18095 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:05.791203 18017 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:05.799437 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:05.800449 18165 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:05.804071 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55119.3us
W20260709 12:23:05.806849 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0225689270000657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20260709 12:23:05.625154 21932 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:05.810307 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:05.810551 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:05.811830 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.758199 22112 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:05.756850 22113 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:05.756575 22059 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.779718 22049 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.819759 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.826161 18170 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:05.820226 18098 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:05.816500 18095 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:05.767815 22086 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:05.822907 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:05.775420 21970 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:05.758383 21670 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:23:05.835098 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112012us
W20260709 12:23:05.836221 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 3.158s (SENT)
I20260709 12:23:05.837498 18165 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:05.839412 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:05.840381 18164 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:05.841292 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.840134 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:05.843099 18173 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:05.844058 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:05.844861 18173 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:05.844074 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:05.845707 18164 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:05.848383 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.849669 21793 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:05.850121 21793 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.850488 21793 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.866395 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:05.821249 21999 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:05.813143 21975 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:05.813375 21910 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.824083 21996 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:05.819371 22112 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.813476 22043 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:05.883782 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:05.885916 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49756.6us
I20260709 12:23:05.825400 22113 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:05.823029 22060 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:05.875140 22101 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.813374 22092 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 12:23:05.835690 21670 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:05.861222 21522 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:05.888185 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:05.928010 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2968) took 1679 ms. Trace:
I20260709 12:23:05.940320 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.940804 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.941191 18809 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:05.881018 21999 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.880332 22086 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.835139 22049 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:05.955097 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67068.6us
I20260709 12:23:05.957643 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.818905 22093 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 12:23:05.982357 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
W20260709 12:23:05.961455 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.939014 18090 rpcz_store.cc:276] 0709 12:23:04.248960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.249109 (+   149us) service_pool.cc:224] Handling call
0709 12:23:04.280157 (+ 31048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.273067 (+992910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.927726 (+654659us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54004123d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:05.927929 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2277417216}
W20260709 12:23:05.985430 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.966856 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:05.922744 22050 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.003s
W20260709 12:23:05.928956 21704 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.575s	user 0.002s	sys 0.010s
I20260709 12:23:05.948814 17919 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 9, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 9 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:05.991508 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:05.970542 18017 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:05.970983 22059 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:05.973282 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.971060 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:05.876344 22100 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:05.988561 18814 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:05.955861 22092 consensus_queue.cc:1243] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:23:05.944270 21996 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:06.001838 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:05.994573 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:05.943877 22113 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:05.961032 22098 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:05.968107 21999 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.893301 21910 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:05.893625 22112 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.945924 22043 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.000116 21934 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:05.954324 21522 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:05.956104 22060 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.078138 22049 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.078137 22060 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:06.079533 22121 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:23:06.029917 21991 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.322s	user 0.016s	sys 0.000s
I20260709 12:23:06.011749 22095 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:06.007953 22030 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:06.012580 21947 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:06.086472 18017 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:05.889729 21975 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.087844 18171 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:06.041599 21704 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:05.956286 22101 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:05.984530 22086 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.040030 21971 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:23:06.099051 21978 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Committing config change with OpId 11.9: config changed from index 3 to 9, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:06.100925 22110 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.426s	user 0.000s	sys 0.001s
I20260709 12:23:06.103852 22112 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.083602 21934 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:06.089181 21991 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.381s	user 0.020s	sys 0.001s
I20260709 12:23:06.001600 22050 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.129993 21996 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.131754 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:06.089450 22039 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.150189 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 148455us
I20260709 12:23:06.072721 22098 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:06.151849 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 196111us
I20260709 12:23:06.157785 18017 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:06.126585 21709 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.128849 21947 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:06.163648 18806 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:06.141268 21704 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.130556 22095 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:06.172595 21709 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.127247 21970 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.171026 22039 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.195765 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2885) took 1836 ms. Trace:
I20260709 12:23:06.197811 22101 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.086360 21670 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.199417 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:06.200584 22052 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:06.201525 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.241500 22060 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.201759 18166 pending_rounds.cc:85] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 7
I20260709 12:23:06.247757 18166 pending_rounds.cc:107] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:23:06.248116 18166 raft_consensus.cc:2889] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:06.247820 22060 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.202876 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:06.249178 22039 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.249544 18017 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:06.251542 22112 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:06.209462 21793 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.172675 21947 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:06.215196 21910 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.216441 21977 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:06.225796 22122 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:23:06.226842 22098 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.200613 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.158319 21934 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:06.232018 21971 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.242830 22127 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 12:23:06.198446 21936 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.006s
I20260709 12:23:06.209241 21975 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.247237 18164 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:06.202245 21991 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:06.255074 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2292) took 1490 ms. Trace:
I20260709 12:23:06.253774 22057 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:06.238451 18015 rpcz_store.cc:276] 0709 12:23:04.359138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.237854 (+878716us) service_pool.cc:224] Handling call
0709 12:23:05.273219 (+ 35365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.303756 (+ 30537us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:05.303793 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.317771 (+ 13978us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.470313 (+152542us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:05.470358 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.470432 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.100212 (+629780us) spinlock_profiling.cc:243] Waited 9.44 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:06.195301 (+ 95089us) raft_consensus.cc:1685] finished
0709 12:23:06.195331 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.195668 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20045,"raft.queue_time_us":37918,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":27140,"spinlock_wait_cycles":21706624,"thread_start_us":96639,"threads_started":1,"wal-append.queue_time_us":140351}
I20260709 12:23:06.229801 21704 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:06.242451 22126 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:23:06.254659 22043 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.256822 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:06.268129 18171 rpcz_store.cc:276] 0709 12:23:04.765056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.827257 (+ 62201us) service_pool.cc:224] Handling call
0709 12:23:06.254861 (+1427604us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:06.254969 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2655834496}
I20260709 12:23:06.257328 18167 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:06.265303 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:06.301869 18020 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:06.260998 21683 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.969s	user 0.013s	sys 0.004s
W20260709 12:23:06.303934 18812 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.021s	sys 0.019s
I20260709 12:23:06.255632 21977 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:06.249202 21970 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:06.255259 21793 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.286477 22057 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:06.281332 22050 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:06.308445 18812 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.021s	sys 0.019s
I20260709 12:23:06.302740 21991 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:06.258286 21971 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.315872 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:06.307991 21869 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:06.261703 21936 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.315219 22060 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.321237 21991 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:06.309985 21683 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.328102 21869 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.329916 22078 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:06.343670 22057 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.345176 22057 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.339567 22086 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.342893 22078 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.323007 22095 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.358958 22113 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.361776 22113 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.357182 22078 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:06.362715 22087 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.004s
I20260709 12:23:06.363302 22087 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.358388 22076 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:06.356796 22132 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.070s	user 0.000s	sys 0.002s
I20260709 12:23:06.366122 22076 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:06.412643 17919 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 6, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 6 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:06.434587 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2873) took 2203 ms (client timeout 1789 ms). Trace:
W20260709 12:23:06.435341 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2902) took 1788 ms (client timeout 1987 ms). Trace:
W20260709 12:23:06.435416 18021 rpcz_store.cc:269] 0709 12:23:04.231067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.242370 (+ 11303us) service_pool.cc:224] Handling call
0709 12:23:06.433947 (+2191577us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54001fca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:06.434289 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":280432384}
W20260709 12:23:06.436101 18023 rpcz_store.cc:269] 0709 12:23:04.646401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.699000 (+ 52599us) service_pool.cc:224] Handling call
0709 12:23:06.434683 (+1735683us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:06.435064 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1127639424}
I20260709 12:23:06.423523 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:06.438778 18015 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:06.440871 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:06.427320 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2214) took 2071 ms (client timeout 1625 ms). Trace:
W20260709 12:23:06.442096 18016 rpcz_store.cc:269] 0709 12:23:04.355459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.371247 (+ 15788us) service_pool.cc:224] Handling call
0709 12:23:06.426452 (+2055205us) spinlock_profiling.cc:243] Waited 80.9 ms on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:06.426899 (+   447us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186007808}
W20260709 12:23:06.444324 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2889) took 1960 ms (client timeout 1872 ms). Trace:
I20260709 12:23:06.430035 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:06.445480 18017 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:06.445057 18018 rpcz_store.cc:269] 0709 12:23:04.484189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.644496 (+160307us) service_pool.cc:224] Handling call
0709 12:23:06.443635 (+1799139us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b540010a9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:06.444027 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1146843520}
I20260709 12:23:06.447342 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:06.448351 18017 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:06.448523 18018 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:06.446170 18020 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:06.450418 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2222) took 1985 ms. Trace:
I20260709 12:23:06.451360 18024 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:06.453177 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2895) took 1919 ms. Trace:
I20260709 12:23:06.452769 17919 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: config changed from index 3 to 5, term changed from 5 to 9, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to <none>, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 9 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:06.454536 18022 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:06.451510 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:06.456945 18022 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:06.432505 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:06.453647 18024 rpcz_store.cc:276] 0709 12:23:04.533537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.450147 (+1916610us) service_pool.cc:224] Handling call
0709 12:23:06.450779 (+   632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.452783 (+  2004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.453092 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.458134 18023 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:06.458647 18019 raft_consensus.cc:2368] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
I20260709 12:23:06.459098 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2894) took 1928 ms. Trace:
I20260709 12:23:06.459738 18024 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:06.445350 18016 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:06.460472 18019 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:06.461030 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2219) took 2039 ms. Trace:
I20260709 12:23:06.462110 18021 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:06.468106 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2221) took 2020 ms. Trace:
I20260709 12:23:06.468828 18021 rpcz_store.cc:276] 0709 12:23:04.447584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.445801 (+1998217us) service_pool.cc:224] Handling call
0709 12:23:06.461633 (+ 15832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.467721 (+  6088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.468010 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.463315 17919 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 6, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:06.461253 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2897) took 1868 ms. Trace:
I20260709 12:23:06.449496 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2893) took 1918 ms. Trace:
I20260709 12:23:06.469789 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2225) took 1891 ms. Trace:
I20260709 12:23:06.441558 18015 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:06.471354 17919 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:06.474097 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2886) took 2081 ms. Trace:
I20260709 12:23:06.471602 18024 rpcz_store.cc:276] 0709 12:23:04.594730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.458667 (+1863937us) service_pool.cc:224] Handling call
0709 12:23:06.459307 (+   640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.460663 (+  1356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.461184 (+   521us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.474980 17919 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 7, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.478066 18024 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:06.479130 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2899) took 1845 ms. Trace:
I20260709 12:23:06.479576 18024 rpcz_store.cc:276] 0709 12:23:04.634100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.476695 (+1842595us) service_pool.cc:224] Handling call
0709 12:23:06.477334 (+   639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.478700 (+  1366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.479061 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.471768 18018 rpcz_store.cc:276] 0709 12:23:04.530815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.447192 (+1916377us) service_pool.cc:224] Handling call
0709 12:23:06.448160 (+   968us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.449082 (+   922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.449377 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13184}
I20260709 12:23:06.483690 17919 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 8, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 8 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:06.461477 18016 rpcz_store.cc:276] 0709 12:23:04.421342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.443836 (+2022494us) service_pool.cc:224] Handling call
0709 12:23:06.444551 (+   715us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.460653 (+ 16102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.460929 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.486289 18018 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:06.487166 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2228) took 1827 ms. Trace:
I20260709 12:23:06.487982 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 500) took 1285 ms. Trace:
I20260709 12:23:06.487619 18018 rpcz_store.cc:276] 0709 12:23:04.659743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.485463 (+1825720us) service_pool.cc:224] Handling call
0709 12:23:06.486042 (+   579us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.486821 (+   779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.487095 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.494370 18016 rpcz_store.cc:276] 0709 12:23:05.202734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.486707 (+1283973us) service_pool.cc:224] Handling call
0709 12:23:06.487189 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.487572 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.487601 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:06.487674 (+    73us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:06.487736 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.487795 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.487918 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.472219 18022 rpcz_store.cc:276] 0709 12:23:04.578113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.455636 (+1877523us) service_pool.cc:224] Handling call
0709 12:23:06.456356 (+   720us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.457569 (+  1213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.469703 (+ 12134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.450937 18017 rpcz_store.cc:276] 0709 12:23:04.464879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.447084 (+1982205us) service_pool.cc:224] Handling call
0709 12:23:06.447646 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.450045 (+  2399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.450339 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.471689 18021 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:06.459600 18023 rpcz_store.cc:276] 0709 12:23:04.531119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.448862 (+1917743us) service_pool.cc:224] Handling call
0709 12:23:06.449494 (+   632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.458776 (+  9282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.459005 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.502312 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2226) took 1882 ms. Trace:
I20260709 12:23:06.502916 18021 rpcz_store.cc:276] 0709 12:23:04.620195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.470446 (+1850251us) service_pool.cc:224] Handling call
0709 12:23:06.471184 (+   738us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.501926 (+ 30742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.502228 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.503641 18023 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:06.474778 18015 rpcz_store.cc:276] 0709 12:23:04.392337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.440336 (+2047999us) service_pool.cc:224] Handling call
0709 12:23:06.441137 (+   801us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.473485 (+ 32348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.473968 (+   483us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.461210 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2898) took 1841 ms. Trace:
I20260709 12:23:06.504587 18019 rpcz_store.cc:276] 0709 12:23:04.619544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.459542 (+1839998us) service_pool.cc:224] Handling call
0709 12:23:06.460108 (+   566us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.460933 (+   825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.461130 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.505201 18021 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:06.506670 18019 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:06.490100 18024 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:06.509774 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.511557 18092 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:23:06.512528 17919 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 7, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 7 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:06.509912 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.514901 17919 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 11. New cstate: current_term: 11 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:06.515939 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.517241 18090 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:06.512938 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.518663 18092 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:06.510193 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2227) took 1868 ms. Trace:
I20260709 12:23:06.521251 18024 rpcz_store.cc:276] 0709 12:23:04.641286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.480646 (+1839360us) service_pool.cc:224] Handling call
0709 12:23:06.481405 (+   759us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.509627 (+ 28222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.510034 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.523063 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.520135 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.527756 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.517150 17919 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 7, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:06.530258 17919 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:06.535442 18095 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:06.550791 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:06.535087 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.552193 18164 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:06.552644 18094 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:06.538681 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.551270 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.552374 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.540542 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.553644 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.553728 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.540886 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.557032 18166 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:06.555742 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.558785 18170 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 12:23:06.559579 18165 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:06.560681 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.553959 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.555763 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:06.554764 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.555518 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:06.565873 18167 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:06.566639 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:06.568709 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.569537 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:06.570174 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.572379 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.568722 18169 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:06.574347 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2016) took 2079 ms (client timeout 1778 ms). Trace:
I20260709 12:23:06.565696 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:06.576915 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.579504 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.579582 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:06.584228 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:06.587042 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:06.588831 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:06.590584 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:06.592296 18811 raft_consensus.cc:2368] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:06.592837 21793 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:06.594916 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:06.596093 21960 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:06.598354 21919 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.599006 22011 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:06.367213 22076 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.530287 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.533484 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.617704 22087 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.663625 21793 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.674894 21793 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:06.674376 21960 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.675299 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2974) took 1963 ms. Trace:
I20260709 12:23:06.677577 21988 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.678628 21988 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.680308 18815 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:06.615365 21932 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260709 12:23:06.681355 18092 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:06.682307 21869 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:06.683029 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.683934 18092 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:06.601038 21683 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:06.641129 18020 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:23:06.650028 18168 rpcz_store.cc:269] 0709 12:23:04.495262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.495479 (+   217us) service_pool.cc:224] Handling call
0709 12:23:06.555416 (+2059937us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592f8c 00007f6c13592ef6 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:06.574243 (+ 18827us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7501056}
I20260709 12:23:06.651002 18165 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:06.602880 22058 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:06.684434 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:06.687863 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:06.685658 18165 raft_consensus.cc:614] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:06.689169 21965 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.690187 21965 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.691622 22058 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.692121 22058 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:06.686196 18098 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:06.685376 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.702973 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:06.690897 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:06.619333 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45689.6us
I20260709 12:23:06.716830 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:06.723044 18094 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:06.727010 18809 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:23:06.692960 21965 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.684298 21919 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:06.729490 18020 raft_consensus.cc:2376] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6. Re-sending same reply.
I20260709 12:23:06.738586 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:06.745719 21991 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.746652 22095 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:06.748083 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.747037 17919 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 9, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 9 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.755059 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:06.753225 18016 raft_consensus.cc:2376] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10. Re-sending same reply.
I20260709 12:23:06.761160 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:06.761528 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 509) took 1035 ms. Trace:
I20260709 12:23:06.769579 22095 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.771510 18023 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Committing config change with OpId 11.9: config changed from index 3 to 9, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:06.712646 22011 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.778050 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:06.782287 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2975) took 2062 ms. Trace:
I20260709 12:23:06.782912 18096 rpcz_store.cc:276] 0709 12:23:04.712340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.712492 (+   152us) service_pool.cc:224] Handling call
0709 12:23:04.712859 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.713126 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:05.184215 (+471089us) log.cc:844] Serialized 14 byte log entry
0709 12:23:06.724622 (+1540407us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:06.724762 (+   140us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:06.732017 (+  7255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.748409 (+ 16392us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.774745 (+ 26336us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1267214,"raft.queue_time_us":1394347,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":36892,"thread_start_us":282432,"threads_started":2,"wal-append.queue_time_us":190302,"wal-append.run_cpu_time_us":2946,"wal-append.run_wall_time_us":1300272}
I20260709 12:23:06.785948 18813 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:06.787324 21936 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:06.789441 22140 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 12:23:06.342118 21977 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.790212 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.762400 18017 rpcz_store.cc:276] 0709 12:23:05.727103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.502373 (+775270us) service_pool.cc:224] Handling call
0709 12:23:06.502868 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.503285 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:06.687280 (+183995us) log.cc:844] Serialized 14 byte log entry
0709 12:23:06.757327 (+ 70047us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:06.760362 (+  3035us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:06.760490 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.761153 (+   663us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.761402 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":113219,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":84,"raft.queue_time_us":7862,"thread_start_us":188643,"threads_started":2}
I20260709 12:23:06.774334 22090 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:06.801025 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:06.777168 21960 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.805377 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.811124 22150 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.812367 22150 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.813958 22093 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:06.814155 17919 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 7, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 7 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:23:06.816054 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 232197us
I20260709 12:23:06.820127 22113 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:06.821791 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67662.2us
I20260709 12:23:06.822957 21988 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.823319 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:06.817447 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:06.824587 18171 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:06.805555 18815 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:23:06.816742 17919 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:06.831518 18171 raft_consensus.cc:614] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:06.830513 22090 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:06.838002 22090 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:06.844177 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111404us
I20260709 12:23:06.836439 21973 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.856097 18090 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:06.844333 21990 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:06.866226 21990 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:06.867820 21990 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.849794 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:06.774398 22092 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260709 12:23:06.870168 18096 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:23:06.845757 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:06.813247 21793 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:06.858264 18809 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:06.883201 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 501) took 1667 ms. Trace:
I20260709 12:23:06.848578 22095 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.883910 18018 rpcz_store.cc:276] 0709 12:23:05.203069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.494932 (+1291863us) service_pool.cc:224] Handling call
0709 12:23:06.495609 (+   677us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.496432 (+   823us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.496475 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:06.497571 (+  1096us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.667740 (+170169us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:06.667818 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.667946 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.869737 (+201791us) raft_consensus.cc:1685] finished
0709 12:23:06.870021 (+   284us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.870221 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99681,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":299,"thread_start_us":156892,"threads_started":1,"wal-append.queue_time_us":234240}
I20260709 12:23:06.893110 17919 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 7, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.848030 22150 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.926090 18092 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:06.868412 22011 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:06.925551 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:06.927701 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110445us
I20260709 12:23:06.934554 18099 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:06.929988 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:06.939121 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:06.915892 22078 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.553s	user 0.015s	sys 0.002s
I20260709 12:23:06.956087 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:06.966621 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143466us
I20260709 12:23:06.967039 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:06.971774 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 280) took 28890 ms (client timeout 15000 ms). Trace:
W20260709 12:23:06.960938 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.974303 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:06.961107 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.982815 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.984066 18099 raft_consensus.cc:2376] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9. Re-sending same reply.
I20260709 12:23:06.898025 21973 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:06.992134 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.993955 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 148510us
I20260709 12:23:06.967188 22055 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:06.961485 21984 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:06.961140 21970 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:06.986122 21990 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:06.971086 21971 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.020391 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.018137 22025 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:06.980221 17920 rpcz_store.cc:269] 0709 12:22:38.083312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.090067 (+  6755us) service_pool.cc:224] Handling call
0709 12:22:38.646129 (+556062us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16621090 00007f6c166bff3a 00007f6c1c475e9c 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:23:06.954030 (+28307901us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:00.878026 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:00.981849 (+103823us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:00.981951 (+   102us) write_op.cc:435] Acquired schema lock
0709 12:23:00.981981 (+    30us) tablet.cc:662] Decoding operations
0709 12:23:01.031993 (+ 50012us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:01.056673 (+ 24680us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:01.056715 (+    42us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:23:01.057454 (+   739us) tablet.cc:701] Row locks acquired
0709 12:23:01.057495 (+    41us) write_op.cc:260] PREPARE: finished
0709 12:23:01.057693 (+   198us) write_op.cc:270] Start()
0709 12:23:01.057889 (+   196us) write_op.cc:276] Timestamp: P: 1783599781057621 usec, L: 0
0709 12:23:01.057914 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:23:01.058755 (+   841us) log.cc:844] Serialized 2516 byte log entry
0709 12:23:01.261365 (+202610us) op_driver.cc:464] REPLICATION: finished
0709 12:23:01.422784 (+161419us) write_op.cc:301] APPLY: starting
0709 12:23:01.422887 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:01.624797 (+201910us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:01.624948 (+   151us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:02.016056 (+391108us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:02.016386 (+   330us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=12,mrs_lookups=0
0709 12:23:02.016458 (+    72us) write_op.cc:312] APPLY: finished
0709 12:23:02.016894 (+   436us) log.cc:844] Serialized 65 byte log entry
0709 12:23:02.017154 (+   260us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:02.017286 (+   132us) write_op.cc:454] Released schema lock
0709 12:23:02.017350 (+    64us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:02.038575 (+ 21225us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:02.044933 (+  6358us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":726560,"reported_tablets":46,"spinlock_wait_cycles":75862784,"child_traces":[["op",{"apply.queue_time_us":161084,"apply.run_cpu_time_us":21477,"apply.run_wall_time_us":595086,"cfile_cache_hit":22,"cfile_cache_hit_bytes":23023,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6220,"cfile_init":2,"lbm_read_time_us":82316,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":10,"num_ops":6,"prepare.queue_time_us":321312,"prepare.run_cpu_time_us":20608,"prepare.run_wall_time_us":350389,"raft.queue_time_us":32012,"raft.run_cpu_time_us":27981,"raft.run_wall_time_us":161651,"replication_time_us":203195,"thread_start_us":520467,"threads_started":4,"wal-append.queue_time_us":163137,"wal-append.run_cpu_time_us":11681,"wal-append.run_wall_time_us":1795884}]]}
W20260709 12:23:07.028126 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.029575 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.030506 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:07.008561 21984 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.993510 17919 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:07.019518 21793 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.003087 21960 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.053133 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:06.989539 22078 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.066962 18812 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:07.034538 22011 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.116263 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.094787 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.077988 21960 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.075477 21977 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:07.098047 21828 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:07.102926 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 148746us
I20260709 12:23:07.103919 21973 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.104975 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:07.070175 22152 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.111088 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:07.025350 21988 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:07.052106 21984 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:07.122900 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:07.137146 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46871.3us
I20260709 12:23:07.121568 17919 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:07.079277 21704 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.777s	user 0.017s	sys 0.001s
I20260709 12:23:07.050861 22055 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:07.153998 21971 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:07.056437 21919 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.067399 22045 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:07.076195 21999 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.003s	user 0.016s	sys 0.005s
I20260709 12:23:07.071846 22059 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:07.070473 21670 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.079787 22025 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:07.154290 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 92150.8us
I20260709 12:23:07.065727 22151 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.182476 22151 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.186210 22091 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:23:07.188586 22091 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:07.189314 22091 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:07.142028 21828 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:07.175954 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.166354 17919 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:07.180492 18092 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:07.190194 22045 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.194078 21999 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.167840 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.205188 21898 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:07.205739 21934 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.941s	user 0.019s	sys 0.001s
W20260709 12:23:07.205765 21947 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.866s	user 0.015s	sys 0.005s
I20260709 12:23:07.206595 22030 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:07.207998 21828 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.141286 22011 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:07.210590 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.210443 22094 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:23:07.210522 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123125us
I20260709 12:23:07.172122 21704 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.171445 21988 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.212198 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:07.219640 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2973) took 2438 ms. Trace:
I20260709 12:23:07.220201 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 821) took 1693 ms. Trace:
I20260709 12:23:07.201200 22152 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.212945 17919 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 6 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:07.238454 18090 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:07.241118 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.244766 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.247779 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.248628 18096 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:23:07.244134 18814 rpcz_store.cc:276] 0709 12:23:05.478943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.507953 (+ 29010us) service_pool.cc:224] Handling call
0709 12:23:05.984751 (+476798us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.000722 (+ 15971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.154926 (+1154204us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b5400494150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:07.171627 (+ 16701us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":999534848}
W20260709 12:23:07.254022 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.365676 18164 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:07.370074 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.377875 22152 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:07.381635 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 144752us
I20260709 12:23:07.226320 22151 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.398720 17919 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:07.400298 21709 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.401306 17919 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:07.246934 21960 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.419776 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.420842 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.432204 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.380610 21932 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.402793 18780 leader_election.cc:400] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
I20260709 12:23:07.429260 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:07.377992 21704 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.438299 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:07.439312 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 300199us
I20260709 12:23:07.426748 21709 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:07.442112 21936 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.442677 21709 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.442951 21936 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.445262 21709 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:07.446903 21978 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.193s	user 0.017s	sys 0.004s
I20260709 12:23:07.392206 22092 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.459288 22092 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.243651 18097 rpcz_store.cc:276] 0709 12:23:04.711691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.712861 (+  1170us) service_pool.cc:224] Handling call
0709 12:23:04.713225 (+   364us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:06.755884 (+2042659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.760674 (+  4790us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:06.790092 (+ 29418us) log.cc:844] Serialized 78 byte log entry
0709 12:23:06.980704 (+190612us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:07.028477 (+ 47773us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.035332 (+  6855us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.049103 (+ 13771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.065253 (+ 16150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.077640 (+ 12387us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:07.136958 (+ 59318us) raft_consensus.cc:1685] finished
0709 12:23:07.136993 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.149780 (+ 12787us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1870778,"raft.queue_time_us":2014187,"raft.run_cpu_time_us":1024,"raft.run_wall_time_us":107308,"spinlock_wait_cycles":9887488,"thread_start_us":287755,"threads_started":3,"wal-append.queue_time_us":281354}
I20260709 12:23:07.424544 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.436913 18780 leader_election.cc:403] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:07.383345 22025 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:07.430537 22058 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.738s	user 0.015s	sys 0.006s
I20260709 12:23:07.227838 22055 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.468727 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:07.243893 21934 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:07.480139 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49595.8us
I20260709 12:23:07.246846 21947 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.487872 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:07.489500 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47762.9us
I20260709 12:23:07.490741 22030 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:07.490551 21932 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.491513 22030 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.495608 21793 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.496460 21793 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:07.498493 22158 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:23:07.502180 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 277036us
I20260709 12:23:07.506038 22030 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.506994 22030 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
I20260709 12:23:07.507230 22090 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:07.508131 22090 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.470407 21936 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.501181 18092 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:07.493425 21932 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.477679 21960 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:07.515575 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:23:07.427515 22093 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20260709 12:23:07.525290 18809 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:07.507051 21988 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.507443 21933 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.531343 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.536860 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.538506 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.541793 22011 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.533696 18097 raft_consensus.cc:2376] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10. Re-sending same reply.
I20260709 12:23:07.547631 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.485332 22058 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.518469 21960 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.502086 22094 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 12:23:07.551261 18096 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 4.
I20260709 12:23:07.564934 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.561597 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.556525 18092 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:07.566802 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:07.504508 22151 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:23:07.533108 22025 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:07.550388 22078 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.559192 21793 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.562301 21960 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.579731 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:07.565680 22092 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.570107 22025 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.520712 22091 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.588887 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.591444 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.591398 18096 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:07.600564 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:07.602232 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.602936 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:07.604477 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.592478 22025 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.606825 21934 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.607982 21522 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.610925 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:07.574749 21709 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:07.621209 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:07.596385 22090 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.624141 22052 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.624908 22052 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.630084 18166 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:07.631265 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.635706 18096 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:07.638036 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.638566 22058 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.595124 21999 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.595283 21988 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
W20260709 12:23:07.656827 18023 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.830s	user 0.017s	sys 0.000s
W20260709 12:23:07.646064 21869 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.957s	user 0.006s	sys 0.013s
I20260709 12:23:07.657904 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.713162 18170 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:23:07.705680 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47614us
I20260709 12:23:07.639372 22045 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.651578 22113 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:07.724244 22166 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:23:07.730082 22149 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 12:23:07.731518 22149 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:07.731956 22149 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:07.733328 18170 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:07.734011 21910 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.734489 22052 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.734736 21910 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:07.735335 22057 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.363s	user 0.017s	sys 0.004s
I20260709 12:23:07.734011 22112 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:07.713399 22153 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:07.736891 22153 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:07.655566 22091 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:07.742165 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.742981 18096 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:07.749969 18170 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:23:07.752447 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 141612us
I20260709 12:23:07.754427 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:07.753937 18814 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:07.756042 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.733069 22149 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:07.740115 21984 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.544s	user 0.013s	sys 0.000s
I20260709 12:23:07.652094 21709 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:07.771422 22167 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.140s	user 0.002s	sys 0.000s
I20260709 12:23:07.768162 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:07.765736 22141 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.132s	user 0.010s	sys 0.023s
I20260709 12:23:07.651628 21995 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.213s
W20260709 12:23:07.775417 22170 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:23:07.815317 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:07.716197 21869 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:07.768224 21990 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.732s	user 0.013s	sys 0.000s
I20260709 12:23:07.761127 21947 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.827852 18096 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:07.823006 18099 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:07.739842 21988 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:23:07.836210 22052 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:06.675760 18093 rpcz_store.cc:276] 0709 12:23:04.712034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.713645 (+  1611us) service_pool.cc:224] Handling call
0709 12:23:04.714036 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.714255 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:04.714619 (+   364us) log.cc:844] Serialized 14 byte log entry
0709 12:23:06.634119 (+1919500us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:06.663814 (+ 29695us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:06.674705 (+ 10891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.674836 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.675082 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1551925,"prepare.queue_time_us":440,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":1658109,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":236,"thread_start_us":366782,"threads_started":2,"wal-append.queue_time_us":438242,"wal-append.run_cpu_time_us":2324,"wal-append.run_wall_time_us":1111719}
I20260709 12:23:07.839846 21869 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:07.841908 22149 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:07.845170 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:07.845970 21934 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:23:07.846823 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92539.8us
W20260709 12:23:07.873745 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.828111 22151 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.776001 22153 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.873939 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.779337 21984 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.866384 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 631) took 1382 ms. Trace:
W20260709 12:23:07.867087 18811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 819) took 2402 ms (client timeout 2407 ms). Trace:
I20260709 12:23:07.870067 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.871534 21991 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:07.821476 21670 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.618s	user 0.016s	sys 0.000s
W20260709 12:23:07.828325 21977 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.683s	user 0.011s	sys 0.002s
I20260709 12:23:07.783141 22112 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:07.757578 22113 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.817276 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:07.856050 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.910645 18812 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:07.910851 18170 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:07.880223 22164 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.681s	user 0.001s	sys 0.000s
I20260709 12:23:07.892149 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.886835 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:07.892804 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 641) took 1404 ms. Trace:
I20260709 12:23:07.913303 18093 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:07.891760 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.895268 21934 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:23:07.895653 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:07.896034 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51007.8us
I20260709 12:23:07.890400 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:07.891556 21709 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.902719 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:07.902531 22151 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.902943 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:07.899721 18811 rpcz_store.cc:269] 0709 12:23:05.464439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.464613 (+   174us) service_pool.cc:224] Handling call
0709 12:23:07.866820 (+2402207us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54005f5854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:07.867005 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2579354880}
I20260709 12:23:07.903738 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:07.903863 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2965) took 3694 ms. Trace:
I20260709 12:23:07.875878 21913 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
W20260709 12:23:07.907694 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.909663 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:07.897145 18090 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:07.911396 18167 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:07.912441 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:07.899186 18096 rpcz_store.cc:276] 0709 12:23:06.484549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.754752 (+1270203us) service_pool.cc:224] Handling call
0709 12:23:07.866293 (+111541us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:07.914784 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137203us
I20260709 12:23:07.916514 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.913385 18098 rpcz_store.cc:276] 0709 12:23:06.488517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.557896 (+ 69379us) service_pool.cc:224] Handling call
0709 12:23:06.576614 (+ 18718us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:07.892460 (+1315846us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b5400214b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:07.892652 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87690368}
I20260709 12:23:07.876935 21793 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:23:07.882365 22076 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:07.919217 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.912184 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:07.921334 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:07.924965 18017 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:07.876757 21933 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:07.922430 21988 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:07.926494 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.926862 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.927529 21984 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:07.928239 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:07.930730 18806 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:07.931089 18018 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:07.935523 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.926494 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:07.938441 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 749) took 1001 ms. Trace:
I20260709 12:23:07.938688 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.939709 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:07.940379 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.938898 18806 rpcz_store.cc:276] 0709 12:23:06.933352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.905118 (+971766us) service_pool.cc:224] Handling call
0709 12:23:07.934237 (+ 29119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.942289 18814 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:07.943532 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.945951 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:07.947613 18815 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:07.922149 18091 rpcz_store.cc:276] 0709 12:23:04.209214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.209526 (+   312us) service_pool.cc:224] Handling call
0709 12:23:04.209982 (+   456us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:06.748222 (+2538240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.752634 (+  4412us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:06.802885 (+ 50251us) log.cc:844] Serialized 52 byte log entry
0709 12:23:06.953072 (+150187us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:06.953116 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.903189 (+950073us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54001afc54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:07.903245 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.903299 (+    54us) raft_consensus.cc:1685] finished
0709 12:23:07.903314 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.903655 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2147873,"raft.queue_time_us":2243897,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":26577,"spinlock_wait_cycles":379890048,"thread_start_us":360567,"threads_started":2,"wal-append.queue_time_us":197356}
I20260709 12:23:07.928930 21934 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:07.950433 18173 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:07.933445 21913 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:07.941228 22107 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:07.949923 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:07.952428 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.941985 21709 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.945245 21936 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:07.957123 18090 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:07.977365 21936 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:07.978566 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.979966 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.954689 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:07.957690 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:07.955729 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:07.951682 21988 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:07.959928 22148 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:23:07.958459 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:07.988595 21965 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:07.962651 22154 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.882s	user 0.018s	sys 0.013s
I20260709 12:23:07.989108 21965 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:07.990950 21933 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:07.990679 21965 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.969745 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:07.971467 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.992409 21933 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:07.993307 18092 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:07.993858 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2108) took 1236 ms. Trace:
I20260709 12:23:07.993505 21988 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.994251 18092 rpcz_store.cc:276] 0709 12:23:06.757723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.971047 (+1213324us) service_pool.cc:224] Handling call
0709 12:23:07.993757 (+ 22710us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.976246 21910 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:07.995497 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.967228 21934 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.996830 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:07.974434 22107 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.984794 18815 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:07.989001 18173 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.986306 21709 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:07.994000 21933 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:07.996722 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:07.992326 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2906) took 2177 ms. Trace:
I20260709 12:23:08.003384 18169 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:08.000797 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46027.4us
I20260709 12:23:07.986197 18170 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:07.980644 18090 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:08.014719 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.015512 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2116) took 1192 ms. Trace:
I20260709 12:23:07.942229 22025 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.001744 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.015812 18090 rpcz_store.cc:276] 0709 12:23:06.823652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.014368 (+1190716us) service_pool.cc:224] Handling call
0709 12:23:08.015457 (+  1089us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.020977 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.021970 18095 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:08.022594 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 634) took 1537 ms. Trace:
I20260709 12:23:08.019778 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:07.978037 21936 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:08.001731 21977 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:08.026749 21913 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:08.026974 21977 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.004067 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:08.007663 21709 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:08.006426 18023 rpcz_store.cc:276] 0709 12:23:05.815049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.504990 (+689941us) service_pool.cc:224] Handling call
0709 12:23:06.505925 (+   935us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:06.640793 (+134868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.640837 (+    44us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:06.679604 (+ 38767us) log.cc:844] Serialized 608 byte log entry
0709 12:23:06.713676 (+ 34072us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:06.730954 (+ 17278us) log.cc:844] Serialized 14 byte log entry
0709 12:23:06.754615 (+ 23661us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.072834 (+318219us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b18004367e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:07.714437 (+641603us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.720659 (+  6222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.727185 (+  6526us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.961451 (+234266us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:07.991937 (+ 30486us) raft_consensus.cc:1685] finished
0709 12:23:07.991969 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.992226 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29797,"prepare.queue_time_us":47980,"prepare.run_cpu_time_us":182,"prepare.run_wall_time_us":1184,"raft.queue_time_us":15406,"raft.run_cpu_time_us":861,"raft.run_wall_time_us":80246,"spinlock_wait_cycles":22373504,"thread_start_us":33480,"threads_started":1,"wal-append.queue_time_us":400679}
W20260709 12:23:07.989125 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43794.4us
I20260709 12:23:08.004256 21970 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.008695 18098 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:08.033407 22055 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:08.034268 22055 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:08.034910 22055 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.035630 22148 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:08.036396 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:08.009471 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 643) took 1495 ms. Trace:
I20260709 12:23:08.005604 21910 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:08.025202 18092 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:23:08.022938 18095 rpcz_store.cc:276] 0709 12:23:06.485675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.556518 (+ 70843us) service_pool.cc:224] Handling call
0709 12:23:07.891619 (+1335101us) spinlock_profiling.cc:243] Waited 75.3 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592f8c 00007f6c13592ef6 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:08.022524 (+130905us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":173169792}
I20260709 12:23:08.031065 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:08.030717 21995 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:08.043509 21995 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.043763 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:08.044054 21995 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:08.044431 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2043) took 1496 ms (client timeout 1749 ms). Trace:
I20260709 12:23:08.045253 21971 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:08.033576 21970 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.045804 21971 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.046394 21971 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.047900 21970 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:08.049703 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:08.050438 18167 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.034403 21976 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:08.051029 21976 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:08.038003 18094 rpcz_store.cc:276] 0709 12:23:06.514469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.571936 (+ 57467us) service_pool.cc:224] Handling call
0709 12:23:08.009142 (+1437206us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b54002963d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:08.009372 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71623552}
I20260709 12:23:08.052143 21976 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.053411 21976 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.040274 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:08.040839 18096 raft_consensus.cc:2368] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:23:08.021237 21965 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.057587 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2127) took 1038 ms. Trace:
I20260709 12:23:07.985265 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:23:08.044906 18164 rpcz_store.cc:269] 0709 12:23:06.548211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.555019 (+  6808us) service_pool.cc:224] Handling call
0709 12:23:08.044114 (+1489095us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:08.044352 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1616046976}
I20260709 12:23:08.058239 18092 rpcz_store.cc:276] 0709 12:23:07.019608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.020613 (+1001005us) service_pool.cc:224] Handling call
0709 12:23:08.057502 (+ 36889us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.062067 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:08.037612 21977 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:08.064679 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:08.065438 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107839us
I20260709 12:23:08.029695 21990 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:08.054281 21913 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:08.041205 22148 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:08.073729 18137 leader_election.cc:400] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:08.074169 18137 leader_election.cc:403] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:08.076493 22150 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:08.077153 22150 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:08.079726 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:08.001600 22112 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.067116 21793 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:08.073119 21913 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:08.086673 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 924) took 2885 ms. Trace:
I20260709 12:23:08.086659 21793 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:08.087702 22148 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.088557 22113 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:08.088356 21793 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.092090 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 822) took 1887 ms. Trace:
I20260709 12:23:08.087324 18813 rpcz_store.cc:276] 0709 12:23:05.201132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.203002 (+  1870us) service_pool.cc:224] Handling call
0709 12:23:05.203382 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.786740 (+1583358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.083039 (+1296299us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400557154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:08.086573 (+  3534us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3226923776}
I20260709 12:23:08.091586 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:08.086849 21913 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:08.102685 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 50) took 1346 ms. Trace:
I20260709 12:23:08.099406 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:08.092636 18810 rpcz_store.cc:276] 0709 12:23:06.204193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.208396 (+  4203us) service_pool.cc:224] Handling call
0709 12:23:06.208737 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.586507 (+377770us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.586593 (+    86us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:06.586679 (+    86us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:06.586747 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.083400 (+1496653us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54005e01d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:08.083455 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.091806 (+  8351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":866791168}
I20260709 12:23:08.091729 21828 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:08.065604 21910 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.103260 18171 rpcz_store.cc:276] 0709 12:23:06.752257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.752693 (+   436us) service_pool.cc:224] Handling call
0709 12:23:08.098431 (+1345738us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:08.098613 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":143839616}
I20260709 12:23:08.124936 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.135468 22045 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:23:08.137115 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.147878 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.145807 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66717us
W20260709 12:23:08.146621 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 73047.2us
I20260709 12:23:08.156530 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.078317 22150 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.160197 18807 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:23:08.174647 18815 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:08.186399 22113 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.187359 22146 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.552s	user 0.000s	sys 0.022s
I20260709 12:23:08.187238 22113 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.196060 21952 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:08.196417 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.197386 18815 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:08.200151 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 753) took 1093 ms. Trace:
I20260709 12:23:08.200609 18812 rpcz_store.cc:276] 0709 12:23:07.106600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.391735 (+285135us) service_pool.cc:224] Handling call
0709 12:23:08.199906 (+808171us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b54006153d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:08.200069 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1077805952}
I20260709 12:23:08.151374 22151 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:08.228039 18017 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:08.201508 18093 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:08.083161 21990 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.228751 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:08.212606 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.156963 22030 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:08.245640 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 174882us
I20260709 12:23:08.246711 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:08.240075 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:08.249364 18018 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:08.199235 22153 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.272307 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:08.274420 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70948.3us
I20260709 12:23:08.275663 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:08.278537 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:08.279536 18024 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:08.203888 21947 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:08.212076 21984 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:08.286756 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:08.287663 18812 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:08.288517 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:08.233851 21952 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:08.288947 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 206544us
I20260709 12:23:08.248807 22030 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.284427 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:08.301929 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.302898 18093 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:08.296936 18021 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:08.307073 18015 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:08.309971 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a07f730a06794e2cb530edca57871763"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:08.278627 22151 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:08.281921 22011 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:23:08.320768 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74198.8us
I20260709 12:23:08.323328 18169 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:08.291425 22057 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:08.347896 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 748) took 1456 ms. Trace:
I20260709 12:23:08.349346 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 2988) took 1424 ms. Trace:
I20260709 12:23:08.354526 18096 rpcz_store.cc:276] 0709 12:23:06.924860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.936558 (+1011698us) service_pool.cc:224] Handling call
0709 12:23:08.348938 (+412380us) spinlock_profiling.cc:243] Waited 99.4 ms on lock 0x7b540011dd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:08.349226 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228723968}
I20260709 12:23:08.302368 22030 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:08.351176 18809 rpcz_store.cc:276] 0709 12:23:06.891479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.897190 (+  5711us) service_pool.cc:224] Handling call
0709 12:23:06.926577 (+ 29387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.527608 (+601031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.347538 (+819930us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b540009bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:08.347791 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1376364928}
I20260709 12:23:08.357471 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.342842 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:08.349493 18167 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:08.282601 22153 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.331570 21999 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:08.331288 22025 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:08.373255 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.321936 21709 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:08.368991 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.335101 22107 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:08.337013 22151 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:08.394322 22178 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:23:08.395677 22181 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.208s	user 0.002s	sys 0.000s
I20260709 12:23:08.396193 22153 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:08.396308 22184 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:08.396689 22180 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:23:08.397099 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.386112 18091 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:23:08.397503 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54778.2us
I20260709 12:23:08.337736 22011 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:08.399199 18093 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:08.355337 22058 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.350924 22045 consensus_queue.cc:1243] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:23:08.321676 21947 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.367002 22090 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.436561 22090 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.335251 21984 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.414508 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:08.413300 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:08.444442 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:08.397680 22179 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:23:08.430413 18096 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:08.400439 21709 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.435501 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2102) took 1758 ms. Trace:
W20260709 12:23:08.437927 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 135209us
I20260709 12:23:08.440061 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:08.441728 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.450569 18091 rpcz_store.cc:276] 0709 12:23:06.676625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.434305 (+1757680us) service_pool.cc:224] Handling call
0709 12:23:08.434754 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.435061 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.435091 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.435144 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:08.435185 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.435250 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.435401 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.363523 22057 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.350534 21952 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.383051 21828 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 12:23:08.454715 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2101) took 1770 ms. Trace:
I20260709 12:23:08.424710 22011 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:08.454174 21709 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:08.462541 18096 rpcz_store.cc:276] 0709 12:23:06.676234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.391311 (+1715077us) service_pool.cc:224] Handling call
0709 12:23:08.413213 (+ 21902us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.446468 (+ 33255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.446738 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.472529 21996 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:08.470072 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 201209us
W20260709 12:23:08.473402 22182 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:23:08.473968 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2111) took 1715 ms. Trace:
I20260709 12:23:08.474363 22056 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:08.474421 18096 rpcz_store.cc:276] 0709 12:23:06.759029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.472827 (+1713798us) service_pool.cc:224] Handling call
0709 12:23:08.473251 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.473548 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.473574 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.473620 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:08.473665 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.473725 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.473882 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.456195 22152 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:08.477698 22152 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:08.478919 22152 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.479660 21947 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:08.480142 22152 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:08.480319 21952 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:08.482497 17989 leader_election.cc:336] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.778s (SENT)
I20260709 12:23:08.483098 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:08.483269 22057 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:08.485740 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.509042 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 94645.6us
I20260709 12:23:08.510346 18136 leader_election.cc:400] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:23:08.510879 18136 leader_election.cc:403] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:23:08.459728 22184 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:23:08.525724 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100518us
W20260709 12:23:08.531788 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58045.8us
I20260709 12:23:08.451898 22058 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:08.535522 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:08.481668 22113 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:08.536715 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:08.538419 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2984) took 1783 ms. Trace:
I20260709 12:23:08.539076 18091 rpcz_store.cc:276] 0709 12:23:06.755079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.465246 (+1710167us) service_pool.cc:224] Handling call
0709 12:23:08.516584 (+ 51338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.531760 (+ 15176us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.531800 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.537674 (+  5874us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:08.537731 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.537811 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.538340 (+   529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.507659 21996 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:08.530514 22149 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.015s	sys 0.000s
I20260709 12:23:08.538791 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:08.550238 18173 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.549s	user 0.025s	sys 0.001s
I20260709 12:23:08.551017 18173 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:08.554244 22184 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:08.554682 22185 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:08.555437 22185 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:08.555905 22185 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:08.557026 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.554762 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:08.558609 22184 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:08.560034 22184 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:08.559842 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:08.560890 18812 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:08.562332 22186 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:08.562881 22186 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:08.563971 22186 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:08.565882 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.565809 22186 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:08.567555 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:08.534658 21999 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:08.571025 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 647) took 2054 ms. Trace:
I20260709 12:23:08.570254 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:08.579466 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:08.580245 18173 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:08.568512 21973 raft_consensus.cc:2764] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:23:08.580053 18095 rpcz_store.cc:276] 0709 12:23:06.516249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.042408 (+1526159us) service_pool.cc:224] Handling call
0709 12:23:08.042923 (+   515us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:08.060679 (+ 17756us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.060717 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:08.069508 (+  8791us) log.cc:844] Serialized 52 byte log entry
0709 12:23:08.289595 (+220087us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:08.518375 (+228780us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.540935 (+ 22560us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.554141 (+ 13206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.554239 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.554302 (+    63us) raft_consensus.cc:1685] finished
0709 12:23:08.554316 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.570802 (+ 16486us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46061,"prepare.queue_time_us":279478,"raft.queue_time_us":60135,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":27160,"thread_start_us":399371,"threads_started":2,"wal-append.queue_time_us":311764}
I20260709 12:23:08.572973 22113 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:08.576061 21965 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:08.623420 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 823) took 2417 ms. Trace:
I20260709 12:23:08.631860 18093 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:08.578647 21990 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:08.636828 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126728us
I20260709 12:23:08.638247 18095 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:08.524057 22151 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.583979 22149 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:08.582172 21934 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:08.635617 18807 rpcz_store.cc:276] 0709 12:23:06.206234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.207305 (+  1071us) service_pool.cc:224] Handling call
0709 12:23:06.207755 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.161034 (+1953279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.581851 (+420817us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b540070f8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:08.623300 (+ 41449us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4488648064}
W20260709 12:23:08.647707 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.651142 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2986) took 1805 ms. Trace:
I20260709 12:23:08.651273 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2118) took 1807 ms. Trace:
W20260709 12:23:08.676972 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.676870 18093 rpcz_store.cc:276] 0709 12:23:06.831271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.584245 (+1752974us) service_pool.cc:224] Handling call
0709 12:23:08.631532 (+ 47287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.636643 (+  5111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.636956 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.676952 18095 rpcz_store.cc:276] 0709 12:23:06.841175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.611034 (+1769859us) service_pool.cc:224] Handling call
0709 12:23:08.637855 (+ 26821us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.647953 (+ 10098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.648211 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:08.658118 18170 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.639s	user 0.009s	sys 0.013s
I20260709 12:23:08.679615 18170 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:08.681185 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:08.669260 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 101817us
W20260709 12:23:08.645459 22076 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.711s	user 0.012s	sys 0.003s
I20260709 12:23:08.698354 18172 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:08.703539 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:08.704770 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:08.644043 22196 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:23:08.652545 22149 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:08.682734 21960 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.084s	user 0.017s	sys 0.003s
I20260709 12:23:08.673216 22153 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:08.682331 22113 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:08.711293 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:08.689424 21990 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.682652 21670 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:08.704474 21999 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.714177 18093 pending_rounds.cc:85] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:23:08.703652 22094 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.728767 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:08.702395 22197 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:23:08.712033 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2990) took 1717 ms. Trace:
I20260709 12:23:08.731034 22113 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.741705 21960 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:08.752104 22091 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.000s	user 0.009s	sys 0.009s
I20260709 12:23:08.745263 21984 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:08.756611 22192 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:23:08.759197 22189 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:08.762619 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:08.850749 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88310.4us
I20260709 12:23:08.852612 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:08.817584 22185 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.723511 22149 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.820554 18095 rpcz_store.cc:276] 0709 12:23:06.994252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.700627 (+1706375us) service_pool.cc:224] Handling call
0709 12:23:08.701104 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.701330 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.701376 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.711607 (+ 10231us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:08.711669 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.711760 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.711946 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.813144 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2983) took 2047 ms. Trace:
I20260709 12:23:08.677253 21996 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.824208 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:08.833576 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:08.810290 22151 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:23:08.881287 18018 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.611s	user 0.015s	sys 0.000s
I20260709 12:23:08.821399 18093 pending_rounds.cc:107] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.4
W20260709 12:23:08.844250 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 361763us
I20260709 12:23:08.882272 18093 raft_consensus.cc:2889] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:08.883072 18018 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:23:08.854141 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 209987us
W20260709 12:23:08.824138 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127339us
I20260709 12:23:08.836727 22113 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:08.865418 18097 raft_consensus.cc:2368] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8.
I20260709 12:23:08.872443 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.781796 21934 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:08.810989 21990 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:08.862649 18094 rpcz_store.cc:276] 0709 12:23:06.736352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.436309 (+1699957us) service_pool.cc:224] Handling call
0709 12:23:08.436765 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.437082 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:08.529402 (+ 92320us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.782680 (+253278us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.782774 (+    94us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:08.782884 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.783005 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.783235 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":184563,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":136,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":8351,"thread_start_us":300237,"threads_started":2}
I20260709 12:23:08.851825 22184 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:08.891224 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:08.862970 22052 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:08.830660 22094 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:08.898341 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.862483 22091 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:08.904965 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:08.895061 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 410696us
I20260709 12:23:08.912837 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:08.915017 18099 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:08.886446 22058 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:08.909921 18164 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:08.913580 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.909376 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 382576us
I20260709 12:23:08.923848 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:08.892231 21934 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:08.850294 22189 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:08.932359 18099 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:08.908422 22052 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.942725 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2991) took 1810 ms. Trace:
I20260709 12:23:08.913892 22204 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:08.918009 22091 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.943238 18099 rpcz_store.cc:276] 0709 12:23:07.126038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.916267 (+1790229us) service_pool.cc:224] Handling call
0709 12:23:08.932148 (+ 15881us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.935591 (+  3443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.936705 (+  1114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.927448 21793 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:08.929612 22153 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8.
I20260709 12:23:08.956722 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2115) took 2122 ms. Trace:
I20260709 12:23:08.944793 22090 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:08.959223 21793 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:08.944227 22052 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:08.978770 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.951263 22091 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:08.964030 22203 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.323s	user 0.002s	sys 0.000s
W20260709 12:23:09.003088 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 99172.3us
I20260709 12:23:09.004741 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:08.986280 18099 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260709 12:23:09.006404 18099 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 2
I20260709 12:23:08.989063 21990 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:08.990756 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:08.991304 22208 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:23:08.992024 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:09.017277 18099 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:23:09.017864 18099 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:08.998220 21988 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:09.002492 22209 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
W20260709 12:23:08.999835 21971 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.953s	user 0.015s	sys 0.009s
I20260709 12:23:09.081979 21971 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.000186 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2985) took 2206 ms. Trace:
W20260709 12:23:08.965081 21913 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.862s	user 0.014s	sys 0.003s
I20260709 12:23:09.084579 21971 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.084976 22133 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.999421 21960 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:08.982595 21793 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:09.021525 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 150448us
I20260709 12:23:09.025213 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:09.085700 22133 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.086866 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61736.6us
I20260709 12:23:09.087661 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:09.027859 21910 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.088706 22133 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.089162 21910 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:09.020200 21988 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:09.089787 21910 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:09.090984 21988 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.091158 21910 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.092137 22185 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.030524 18090 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:09.092806 21988 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:09.030932 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:09.094362 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.094660 22058 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:09.031783 22184 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.095202 22058 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:09.084044 18096 rpcz_store.cc:276] 0709 12:23:06.794266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.475799 (+1681533us) service_pool.cc:224] Handling call
0709 12:23:08.476212 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.545060 (+ 68848us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.545098 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.625919 (+ 80821us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.756937 (+131018us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:08.863629 (+106692us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:08.863706 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.863817 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.999754 (+135937us) raft_consensus.cc:1685] finished
0709 12:23:08.999782 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.000089 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68346,"raft.queue_time_us":68610,"raft.run_cpu_time_us":1827,"raft.run_wall_time_us":209609,"spinlock_wait_cycles":42271104,"thread_start_us":166715,"threads_started":1,"wal-append.queue_time_us":313628}
I20260709 12:23:09.096513 22058 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.097626 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:09.098387 18096 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:09.095165 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:09.099911 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:09.100152 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:09.034565 22206 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.211s	user 0.003s	sys 0.000s
I20260709 12:23:09.101434 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:08.984081 21933 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:08.978323 22189 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:09.008957 22055 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.973s	user 0.022s	sys 0.004s
W20260709 12:23:09.039906 21936 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.982s	user 0.010s	sys 0.005s
I20260709 12:23:09.040243 18092 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:09.040752 21947 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:09.041739 22220 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:23:09.043076 21995 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.998s	user 0.016s	sys 0.002s
W20260709 12:23:09.044490 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2039) took 2504 ms (client timeout 2282 ms). Trace:
I20260709 12:23:09.115357 21947 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.115664 18172 rpcz_store.cc:269] 0709 12:23:06.546261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.578832 (+ 32571us) service_pool.cc:224] Handling call
0709 12:23:09.044176 (+2465344us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b5400032ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:09.044337 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4662089216}
I20260709 12:23:09.117470 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.048643 22094 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.059684 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:09.124372 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 636) took 2638 ms. Trace:
I20260709 12:23:09.124723 18015 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:09.067219 21934 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.068863 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2137) took 1558 ms. Trace:
I20260709 12:23:09.070166 22090 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:09.126243 21936 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.144727 22056 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.670s	user 0.009s	sys 0.005s
I20260709 12:23:09.144950 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:09.145247 22056 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:09.075404 18810 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:23:09.145910 22094 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:09.146347 22094 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:09.075949 22207 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:23:09.069422 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:09.079188 22195 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.265s	user 0.000s	sys 0.002s
I20260709 12:23:08.973322 22204 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.147493 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:09.146764 22094 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:09.080775 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.150372 21971 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:09.150869 21971 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.151247 21971 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:09.152494 18018 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:09.155004 18016 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:09.077543 22091 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:09.079197 22201 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.389s	user 0.006s	sys 0.000s
I20260709 12:23:09.001798 22153 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.084321 21913 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.029727 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 630) took 2545 ms (client timeout 3070 ms). Trace:
I20260709 12:23:09.088726 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:09.162972 18097 rpcz_store.cc:269] 0709 12:23:06.484001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.604186 (+1120185us) service_pool.cc:224] Handling call
0709 12:23:09.029430 (+1425244us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540035edd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:09.029644 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2897823488}
W20260709 12:23:09.163909 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75279us
W20260709 12:23:09.095137 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64332.4us
W20260709 12:23:09.028569 22152 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.548s	user 0.014s	sys 0.000s
I20260709 12:23:09.110874 18164 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:09.114316 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2998) took 1561 ms. Trace:
I20260709 12:23:09.166515 18092 rpcz_store.cc:276] 0709 12:23:07.552689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.039609 (+1486920us) service_pool.cc:224] Handling call
0709 12:23:09.040030 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.113779 (+ 73749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.114196 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.167719 18165 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
W20260709 12:23:09.119418 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 195651us
W20260709 12:23:09.108196 21991 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.203s	user 0.017s	sys 0.006s
W20260709 12:23:09.109539 21709 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.579s	user 0.002s	sys 0.011s
I20260709 12:23:09.124894 18090 rpcz_store.cc:276] 0709 12:23:06.486460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.024192 (+1537732us) service_pool.cc:224] Handling call
0709 12:23:09.030252 (+1006060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.092828 (+ 62576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.124099 (+ 31271us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540035edd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:09.124288 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2312751488}
I20260709 12:23:09.174881 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.130637 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:09.128769 22186 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.558s	user 0.015s	sys 0.000s
I20260709 12:23:09.127591 21995 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.138108 18099 rpcz_store.cc:276] 0709 12:23:07.510641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.951945 (+1441304us) service_pool.cc:224] Handling call
0709 12:23:08.980028 (+ 28083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.068535 (+ 88507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.068792 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.143930 22205 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:09.042048 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:08.961356 18091 rpcz_store.cc:276] 0709 12:23:06.823202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.540392 (+1717190us) service_pool.cc:224] Handling call
0709 12:23:08.542952 (+  2560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.543208 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:08.810475 (+267267us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.945263 (+134788us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.945345 (+    82us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:08.945432 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.945541 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.945734 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":129994,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":5998,"raft.queue_time_us":1868,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":9550,"thread_start_us":325848,"threads_started":2}
W20260709 12:23:09.170571 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.201210 18097 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:09.215787 18097 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:09.162427 22076 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:09.171423 21970 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.168145 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:09.193382 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.130347 22184 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:09.203207 21709 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.200958 18094 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:09.217885 18173 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:09.218022 18090 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:09.226054 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2147) took 1572 ms. Trace:
I20260709 12:23:09.220098 18172 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:09.118625 22055 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.226609 18090 rpcz_store.cc:276] 0709 12:23:07.661644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.195166 (+1533522us) service_pool.cc:224] Handling call
0709 12:23:09.209669 (+ 14503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.225697 (+ 16028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.225982 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.227398 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58297us
I20260709 12:23:09.173666 21933 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:09.232774 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:09.239197 18170 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:09.240464 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.242265 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.197294 22223 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:09.244793 18807 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:23:09.226606 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.009s	sys 0.021s
I20260709 12:23:09.192231 22216 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:09.200662 22045 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:09.249588 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.010s	sys 0.021s
I20260709 12:23:09.250669 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3003) took 1102 ms. Trace:
I20260709 12:23:09.254679 18090 rpcz_store.cc:276] 0709 12:23:08.137082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.231680 (+1094598us) service_pool.cc:224] Handling call
0709 12:23:09.238820 (+  7140us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.239055 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.239082 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.239133 (+    51us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:09.239179 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.239238 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.239386 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.255439 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:09.256016 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 676668us
I20260709 12:23:09.256418 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:09.257565 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.258134 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.258711 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.263053 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:09.263993 18169 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:09.263393 18171 raft_consensus.cc:606] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:09.163210 22090 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.291137 18092 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:09.289749 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47655.8us
I20260709 12:23:09.291836 18092 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:09.293563 18167 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.025s	sys 0.023s
W20260709 12:23:09.294005 18167 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.025s	sys 0.023s
I20260709 12:23:09.219498 21934 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:09.193064 21965 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:23:09.224728 22189 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:09.224470 21984 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:09.302532 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:09.231818 21977 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.193217 22153 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.303850 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:09.312721 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 270802us
W20260709 12:23:09.254974 22198 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.011s	sys 0.035s
I20260709 12:23:09.221002 22205 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.288908 22186 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:09.225215 22076 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.305910 21933 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:09.314734 18092 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.332741 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:09.320812 21976 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.267s	user 0.009s	sys 0.009s
I20260709 12:23:09.246701 21793 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.337734 22198 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.012s	sys 0.042s
I20260709 12:23:09.293355 22045 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.352151 18092 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:09.257745 21970 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.356032 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 640) took 2868 ms. Trace:
I20260709 12:23:09.291462 22216 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.370422 18023 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:09.335246 22148 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:09.331504 21933 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:09.335981 22186 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:09.361791 18098 rpcz_store.cc:276] 0709 12:23:06.488105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.034647 (+1546542us) service_pool.cc:224] Handling call
0709 12:23:08.035143 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.054747 (+ 19604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.054787 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.073334 (+ 18547us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.519391 (+446057us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:08.940036 (+420645us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.940376 (+   340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.940484 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.078317 (+137833us) spinlock_profiling.cc:243] Waited 48.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:09.332026 (+253709us) raft_consensus.cc:1685] finished
0709 12:23:09.355759 (+ 23733us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.355949 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9640,"prepare.queue_time_us":132781,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":54,"raft.queue_time_us":27674,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":12369,"spinlock_wait_cycles":110660992,"thread_start_us":812585,"threads_started":2,"wal-append.queue_time_us":495041}
I20260709 12:23:09.321606 21977 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.404048 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 171406us
W20260709 12:23:09.409988 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44204.6us
I20260709 12:23:09.413982 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:09.416181 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3002) took 1272 ms. Trace:
W20260709 12:23:09.417917 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.422665 22224 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.120s	user 0.001s	sys 0.001s
W20260709 12:23:09.424893 22011 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.916s	user 0.015s	sys 0.003s
I20260709 12:23:09.398842 21933 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:09.429886 22204 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.434305 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 522) took 1191 ms. Trace:
I20260709 12:23:09.434900 18016 rpcz_store.cc:276] 0709 12:23:08.242932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.243088 (+   156us) service_pool.cc:224] Handling call
0709 12:23:08.265608 (+ 22520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.162276 (+896668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.433565 (+271289us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b54003bb0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:09.433776 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2044679040}
I20260709 12:23:09.435688 18173 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:09.424928 22153 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.436812 22216 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.438493 22205 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:09.439945 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 107621us
I20260709 12:23:09.443958 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:09.450868 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.450711 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:09.455128 18170 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:23:09.455600 18170 pending_rounds.cc:85] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 7
I20260709 12:23:09.455830 18170 pending_rounds.cc:107] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:23:09.456072 18170 raft_consensus.cc:2889] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.460321 18173 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:23:09.416790 18099 rpcz_store.cc:276] 0709 12:23:08.136682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.231584 (+1094902us) service_pool.cc:224] Handling call
0709 12:23:09.264323 (+ 32739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.264596 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:09.294075 (+ 29479us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.408232 (+114157us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.408313 (+    81us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:09.408374 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.408436 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.408665 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33708,"prepare.queue_time_us":32200,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":5073,"raft.queue_time_us":135,"thread_start_us":113410,"threads_started":1}
I20260709 12:23:09.399526 21936 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.460772 18173 pending_rounds.cc:85] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 8
I20260709 12:23:09.472852 18173 pending_rounds.cc:107] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:23:09.473557 18173 raft_consensus.cc:2889] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.478045 18092 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:23:09.478822 18092 pending_rounds.cc:85] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 5
I20260709 12:23:09.486541 18098 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:09.487215 18098 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:09.425494 22011 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.488066 18092 pending_rounds.cc:107] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:23:09.488684 18092 raft_consensus.cc:2889] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:09.459098 22201 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:23:09.452696 22216 raft_consensus.cc:2764] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 12:23:09.524232 22225 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:23:09.526067 22091 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:09.527931 22230 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:23:09.530267 22055 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.542593 21996 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:09.449009 21984 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:09.503991 22226 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:09.550297 21984 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.552745 21984 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:09.555033 22113 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.648s	user 0.014s	sys 0.000s
I20260709 12:23:09.555603 22113 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.558455 22045 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:09.559255 22045 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:09.559991 18022 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:09.562695 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 2987) took 2671 ms. Trace:
I20260709 12:23:09.563143 18093 rpcz_store.cc:276] 0709 12:23:06.894360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.678511 (+1784151us) service_pool.cc:224] Handling call
0709 12:23:08.701803 (+ 23292us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:08.908631 (+206828us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.916472 (+  7841us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:08.919492 (+  3020us) log.cc:844] Serialized 52 byte log entry
0709 12:23:09.246782 (+327290us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:09.246844 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.246939 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.562422 (+315483us) raft_consensus.cc:1685] finished
0709 12:23:09.562450 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.562619 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5914,"raft.queue_time_us":6123,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":12833,"thread_start_us":273156,"threads_started":1,"wal-append.queue_time_us":380941}
W20260709 12:23:09.521245 22151 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.635s	user 0.011s	sys 0.001s
W20260709 12:23:09.500001 22030 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.110s	user 0.018s	sys 0.000s
W20260709 12:23:09.571923 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121361us
W20260709 12:23:09.574115 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 165278us
I20260709 12:23:09.452322 21965 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260709 12:23:09.586520 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:09.590173 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 158642us
I20260709 12:23:09.596000 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:09.599215 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:09.570592 21995 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.602401 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:09.601012 18098 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.585575 21913 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:09.599745 22149 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.707s	user 0.015s	sys 0.001s
I20260709 12:23:09.616055 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:09.577508 22113 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:09.618366 18098 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:09.618963 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2066) took 1096 ms. Trace:
I20260709 12:23:09.619801 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:09.620548 18169 rpcz_store.cc:276] 0709 12:23:08.515307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.521655 (+  6348us) service_pool.cc:224] Handling call
0709 12:23:08.522021 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.264454 (+742433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.611251 (+346797us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54003020d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:09.611490 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1705789312}
I20260709 12:23:09.623647 18094 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:09.600584 22196 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:09.584035 21971 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.631047 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.575505 22148 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:09.584121 22030 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.617619 21965 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:09.624236 22090 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:09.639935 21965 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:09.646026 21913 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:09.647842 22235 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
W20260709 12:23:09.649941 22234 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:23:09.659312 21971 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.660576 21793 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.633293 22239 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:09.667621 22094 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.519s	user 0.013s	sys 0.000s
I20260709 12:23:09.602545 22186 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:09.669771 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53795.5us
I20260709 12:23:09.625253 22149 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.643772 22076 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.683741 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:09.676426 21984 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:09.699546 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.695953 22052 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.690s	user 0.014s	sys 0.000s
I20260709 12:23:09.710078 18090 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:09.715639 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.720481 21965 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.710974 22245 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:09.694310 22239 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:09.729861 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:09.725960 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:09.734884 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.735733 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:09.714501 21984 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:09.728368 22184 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.738338 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.686721 22094 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.704285 21933 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.774987 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.775920 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:09.752889 21988 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.660s	user 0.012s	sys 0.002s
W20260709 12:23:09.752239 22240 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:23:09.787463 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2048) took 2200 ms (client timeout 1778 ms). Trace:
I20260709 12:23:09.786705 22226 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:09.789880 18098 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:23:09.755347 21977 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.798496 18098 pending_rounds.cc:85] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:09.788093 22223 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:09.753413 22245 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:09.756152 22094 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:09.749557 22151 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:09.749274 21793 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:09.717401 22052 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.798861 18098 pending_rounds.cc:107] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:09.751076 22196 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:09.805379 18098 raft_consensus.cc:2889] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.805464 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.782232 21965 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:09.793974 18165 rpcz_store.cc:269] 0709 12:23:07.587331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.587466 (+   135us) service_pool.cc:224] Handling call
0709 12:23:09.787222 (+2199756us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400030a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:09.787372 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2814228736}
I20260709 12:23:09.802372 18096 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:23:09.807322 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.727046 22148 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.809332 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 577) took 1284 ms. Trace:
I20260709 12:23:09.810309 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.809202 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.758518 22184 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.797955 22113 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:09.813109 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.813616 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.814034 18169 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:23:09.816319 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.786230 21996 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:23:09.822245 18168 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:09.804240 22030 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:09.804554 22226 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:09.827875 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:09.826350 22189 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.516s	user 0.016s	sys 0.003s
I20260709 12:23:09.819777 22237 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:09.831935 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.753208 22239 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.840691 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.819211 22113 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:09.842835 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.804265 22223 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.829972 22196 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:09.848384 18095 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:23:09.854254 18095 pending_rounds.cc:85] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:09.854604 18095 pending_rounds.cc:107] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:23:09.854867 18095 raft_consensus.cc:2889] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.848819 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.856079 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:09.845310 18097 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:09.863344 18165 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:09.864394 18097 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:09.865418 18169 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:09.870574 22250 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:09.844935 22226 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.871893 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.873332 21709 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.874468 22226 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.876637 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 51) took 1603 ms. Trace:
I20260709 12:23:09.877380 18171 rpcz_store.cc:276] 0709 12:23:08.272924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.283931 (+ 11007us) service_pool.cc:224] Handling call
0709 12:23:09.876376 (+1592445us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:09.876529 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2190359296}
I20260709 12:23:09.848436 22113 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:09.883419 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47585us
I20260709 12:23:09.817092 21984 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:09.888310 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53873.3us
I20260709 12:23:09.853549 22196 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:09.897889 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.901669 18169 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:09.908959 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.887358 22237 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.918334 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.905375 22196 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:09.914264 22226 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:09.938094 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:09.928184 21996 consensus_queue.cc:1243] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:09.861601 22245 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:09.940587 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:09.921540 22149 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.954841 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.936138 22226 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.936645 21933 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:09.957160 21996 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:09.959116 22090 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:23:10.024497 22227 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.599s	user 0.005s	sys 0.000s
I20260709 12:23:10.024530 21996 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:10.025833 18165 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:23:10.027318 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.960203 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.962491 22237 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:10.027942 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:10.028628 18171 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:10.028779 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.029191 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.029999 21828 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:10.030786 21828 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.037693 21709 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.038722 21709 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:09.970638 22251 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:23:09.970935 22196 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.045595 22196 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:10.045805 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2075) took 1117 ms. Trace:
W20260709 12:23:09.986488 22249 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.132s	user 0.001s	sys 0.001s
I20260709 12:23:09.999409 22223 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:10.005613 22184 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:10.046439 18172 rpcz_store.cc:276] 0709 12:23:08.928053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.117085 (+189032us) service_pool.cc:224] Handling call
0709 12:23:10.045539 (+928454us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b54000433d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:10.045727 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226433408}
I20260709 12:23:10.010857 22052 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.011080 22011 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:10.015373 18169 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:10.048324 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:09.961691 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:10.050436 18169 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:10.023655 22226 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:10.050487 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88922.4us
I20260709 12:23:10.056134 22237 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:10.056772 22237 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:10.057114 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 579) took 1524 ms. Trace:
I20260709 12:23:10.026700 21996 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.061981 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2130) took 2838 ms. Trace:
I20260709 12:23:10.062708 18095 rpcz_store.cc:276] 0709 12:23:07.226241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.928201 (+1701960us) service_pool.cc:224] Handling call
0709 12:23:08.943699 (+ 15498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.864254 (+920555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.061497 (+197243us) spinlock_profiling.cc:243] Waited 898 ms on lock 0x7b54002972d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:10.061853 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066262912}
I20260709 12:23:10.050230 18016 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:23:10.065248 18016 raft_consensus.cc:606] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:10.065807 18016 raft_consensus.cc:614] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:10.030002 18171 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:10.049471 18172 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:10.074229 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:10.057703 18173 rpcz_store.cc:276] 0709 12:23:08.532887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.436570 (+903683us) service_pool.cc:224] Handling call
0709 12:23:09.437003 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.486420 (+ 49417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.056716 (+570296us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5400042ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:10.057012 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36258560}
I20260709 12:23:10.077474 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:10.083737 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:10.087024 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:10.088629 18097 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:23:10.089331 18097 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:10.086251 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:10.092499 18810 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13.
I20260709 12:23:10.074761 22196 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:23:10.057847 22237 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.089903 18097 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:10.107487 18097 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:10.069080 22052 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.108417 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 207446us
I20260709 12:23:10.075760 21828 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:10.057648 21933 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:23:10.130374 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2145) took 2536 ms. Trace:
I20260709 12:23:10.137094 18097 rpcz_store.cc:276] 0709 12:23:07.595058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.164410 (+1569352us) service_pool.cc:224] Handling call
0709 12:23:09.164823 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.825189 (+660366us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b4c00195740. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14983a20 00007f6c14994a29 00007f6c1495a644 00007f6c14981f2d 00007f6c14981b93 00007f6c166b218b 00007f6c166b2cb5 00007f6c166adf30 00007f6c166af7de 00007f6c166b9b49 00007f6c166ba416
0709 12:23:10.129985 (+304796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.130266 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44073344}
I20260709 12:23:10.133235 18171 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:10.058696 21936 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:23:10.101423 21793 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:10.147277 18164 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:10.105055 21996 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.074604 22223 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 12:23:10.160606 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:10.171639 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2068) took 1653 ms. Trace:
W20260709 12:23:10.216286 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82349.1us
I20260709 12:23:10.175735 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:10.149938 21934 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.843s	user 0.012s	sys 0.006s
I20260709 12:23:10.217913 18168 rpcz_store.cc:276] 0709 12:23:08.517784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.895742 (+377958us) service_pool.cc:224] Handling call
0709 12:23:08.896243 (+   501us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:09.829752 (+933509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.171193 (+341441us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:10.171543 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2124745600}
I20260709 12:23:10.218753 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:10.232097 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:10.160483 21709 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.170526 21936 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.239880 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d440327a63754b889f7edd91e1373942"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:10.137236 22052 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:10.245609 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.252059 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.252506 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:10.251272 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75655.5us
I20260709 12:23:10.253779 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:10.259510 18173 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13.
I20260709 12:23:10.260947 18164 raft_consensus.cc:606] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:10.261610 18164 raft_consensus.cc:614] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:10.269904 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 262443us
W20260709 12:23:10.273726 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.251669 21793 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:10.275482 18169 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:10.264318 22258 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:10.279503 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:10.279852 17989 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13.
I20260709 12:23:10.283092 17989 leader_election.cc:403] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:23:10.283232 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:10.272128 21984 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:10.297744 18807 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:10.275120 21913 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.599s	user 0.016s	sys 0.000s
I20260709 12:23:10.299597 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:10.301074 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:10.272663 21947 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.311708 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.315320 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.315709 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.316991 18017 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.317412 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:10.320209 18808 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:10.320741 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.321467 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:10.322103 22223 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:10.322690 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.322976 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.323464 18812 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:10.323580 18021 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:10.323884 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.303901 22226 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.325052 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.326519 22226 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:10.290053 22258 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.328634 22258 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:10.313874 18019 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:10.301854 21984 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:10.331332 21984 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:10.331526 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:10.270247 21934 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:23:10.333142 21934 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:10.301051 22260 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:10.323601 18171 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.335172 22260 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:10.335572 22260 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:10.336537 22260 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.339061 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:10.339679 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 523) took 2096 ms. Trace:
I20260709 12:23:10.339824 18164 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:10.340140 18022 rpcz_store.cc:276] 0709 12:23:08.243440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.243852 (+   412us) service_pool.cc:224] Handling call
0709 12:23:08.272438 (+ 28586us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.560450 (+1288012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.339345 (+778895us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:10.339589 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2960594176}
I20260709 12:23:10.341049 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.341348 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:10.341645 21970 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.264179 21936 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.347960 21936 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:10.348829 21936 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:10.349236 21936 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:10.351428 18024 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:10.353428 22011 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:10.353940 22011 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:10.319181 21793 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:23:10.355023 22255 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:10.356338 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:10.360088 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41710 (request call id 1) took 1167 ms. Trace:
I20260709 12:23:10.360610 18170 rpcz_store.cc:276] 0709 12:23:09.193189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.497871 (+304682us) service_pool.cc:224] Handling call
0709 12:23:10.359679 (+861808us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54000438d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bd735 00007f6c166b59fa 00007f6c1b67d4c9 00007f6c14bf9c54 00007f6c14bf9be2 00007f6c14bf9b11 00007f6c14bf9a8c 00007f6c14bf8d42 00007f6c149f248c 00007f6c149f18c6
0709 12:23:10.359961 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":712619904}
W20260709 12:23:10.362171 21965 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.555s	user 0.014s	sys 0.002s
I20260709 12:23:10.316541 18016 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 6.
I20260709 12:23:10.363916 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:10.364320 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3014) took 1202 ms. Trace:
I20260709 12:23:10.364761 18090 rpcz_store.cc:276] 0709 12:23:09.161710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.420093 (+258383us) service_pool.cc:224] Handling call
0709 12:23:09.420505 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.713560 (+293055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.364068 (+650508us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:10.364257 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665075456}
I20260709 12:23:10.365244 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:10.300789 22254 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:23:10.366379 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.367092 18017 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:10.300799 22239 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:23:10.370678 22052 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.371799 21977 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:10.371944 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.372769 18017 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:10.338132 18020 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:10.379812 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.380174 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.380657 18097 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:10.381135 18092 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:10.379987 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.381919 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.382467 18098 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:10.382759 18170 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:10.386364 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.394541 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.395457 18092 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:10.380667 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.404017 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:10.415022 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:10.314361 21913 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:10.374105 22260 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.418027 18172 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:10.446131 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.447052 18021 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.448518 18172 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:10.429540 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.429455 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.430395 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.378338 21988 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:10.450467 18091 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:10.450414 22052 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:10.431227 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.450398 18094 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.453454 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:10.454636 22094 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:10.455210 22094 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:10.455631 22094 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:10.463667 22052 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:10.464270 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:10.434690 18090 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:10.435854 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.466275 18099 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:10.466440 22052 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:10.466986 22052 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.436110 18016 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:10.428470 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.467511 22052 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:10.468237 18022 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.440711 18015 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:10.438055 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:10.469848 18164 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:10.444074 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.470961 18019 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:10.445626 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.472062 18018 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:10.472417 22232 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.038s	user 0.004s	sys 0.000s
I20260709 12:23:10.341717 22113 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.474673 22113 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:10.475095 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.475534 18023 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:10.475775 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.476406 18094 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:10.467109 21984 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.418869 22239 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.478430 22239 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.479931 21984 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:10.450628 21988 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.481314 22239 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.478978 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.483948 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:10.466046 21970 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:10.482749 21988 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:10.478431 18021 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:10.487033 18094 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:23:10.483975 18019 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:10.491128 18097 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:10.442616 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.500371 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.437544 21936 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.501533 18018 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.501988 21936 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:10.502297 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.502481 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.503096 18019 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:10.503705 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.504366 18022 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:10.486459 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.916s (SENT) [suppressed 32 similar messages]
I20260709 12:23:10.506347 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.507110 18016 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:10.508093 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.508208 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:10.508613 18094 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:10.508735 18016 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:10.509186 21970 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:10.509747 21970 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:10.501253 22052 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.509786 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.510605 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.510872 21970 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.512791 22030 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:10.513248 22030 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:10.514232 22030 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.515451 22030 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:10.515650 22052 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.516436 18098 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:10.500437 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.518638 18021 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:10.518685 22094 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.520730 18171 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:10.522059 18019 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
W20260709 12:23:10.522717 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.517897 18099 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:10.528944 21934 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13.
W20260709 12:23:10.534451 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:10.535219 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.536331 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:10.542887 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.542924 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.544946 18091 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:10.542909 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.545876 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:10.546711 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.552896 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.553912 18095 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:10.556249 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.557271 18094 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.560886 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:10.565781 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.566749 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.567868 18099 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:10.569000 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.558806 22090 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.570155 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.570429 22090 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.570724 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.572078 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.573030 22090 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:10.573750 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.574919 22186 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:10.578711 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:10.579898 21965 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:10.580986 22148 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:10.581056 21965 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:10.581771 21965 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:10.583034 21965 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.584389 21965 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:10.589097 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:10.589339 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.593425 18095 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:10.589993 18099 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.590648 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.594347 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.596041 18021 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:10.598650 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.599694 18021 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:10.601322 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93223.5us
I20260709 12:23:10.606350 18094 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:10.606019 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.617286 18023 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:10.611809 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.618906 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.619520 18095 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:23:10.609995 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 369998us
I20260709 12:23:10.624884 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.625929 18094 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:10.631016 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 352911us
I20260709 12:23:10.607448 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:10.635747 18090 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:10.641077 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.644905 18090 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.602814 22228 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:10.651734 18097 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:10.653121 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:10.665263 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57917.4us
I20260709 12:23:10.665701 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.650116 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.658978 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.661576 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.666951 18021 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:10.667997 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:10.668920 18090 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:10.667623 18023 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.675362 21828 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:10.675885 21828 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.676226 21828 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:10.650987 21988 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.678283 21988 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.679492 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.686224 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:10.687193 18099 raft_consensus.cc:2368] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
I20260709 12:23:10.685356 18091 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:10.700613 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:10.706537 18094 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:10.685355 18023 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:10.709542 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.710152 18095 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 5.
I20260709 12:23:10.721966 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.717502 18018 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:10.728096 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.721439 22045 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:10.743798 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.714277 22148 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.745680 22045 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:10.749172 22094 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:10.708968 22186 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:23:10.756424 18017 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:10.757683 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.791229 18018 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.725127 22258 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:10.765035 18022 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:10.767335 22233 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.337s	user 0.004s	sys 0.002s
I20260709 12:23:10.768913 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.776399 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3011) took 1703 ms. Trace:
I20260709 12:23:10.793138 18096 rpcz_store.cc:276] 0709 12:23:09.072732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.099556 (+ 26824us) service_pool.cc:224] Handling call
0709 12:23:10.776010 (+1676454us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:10.776295 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1611927424}
I20260709 12:23:10.777666 18098 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:23:10.794613 18098 pending_rounds.cc:85] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:23:10.794951 18098 pending_rounds.cc:107] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:23:10.795265 18098 raft_consensus.cc:2889] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:10.780607 22011 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.782814 22052 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:10.783934 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.787518 18020 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:10.799149 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.801745 18022 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:10.790405 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 490938us
I20260709 12:23:10.806232 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.807472 18020 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:10.818881 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.817339 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.832863 18021 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.833209 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.834584 18018 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.838425 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.839254 18020 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.843152 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.843503 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.844298 18022 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:10.848433 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:10.802191 22052 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:10.853763 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.858733 18015 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:10.853928 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:10.859421 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.854403 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.888007 18021 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.791733 22148 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.892949 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.894304 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.895325 18018 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.895750 18016 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:10.897125 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.897984 18016 raft_consensus.cc:2376] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13. Re-sending same reply.
I20260709 12:23:10.916929 18020 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:10.920365 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.924050 18021 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:10.888093 22052 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.888283 22272 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.891856 21977 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.934242 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 233952us
I20260709 12:23:10.943476 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.944796 18016 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:10.946420 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:10.947624 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.951036 18016 raft_consensus.cc:2376] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8. Re-sending same reply.
I20260709 12:23:10.954314 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.961513 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:10.971580 18097 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:23:10.977178 18097 pending_rounds.cc:85] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 5
I20260709 12:23:10.977481 18097 pending_rounds.cc:107] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:23:10.977734 18097 raft_consensus.cc:2889] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:10.979868 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:10.981839 18778 leader_election.cc:400] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9.
W20260709 12:23:10.977747 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 341185us
I20260709 12:23:10.975167 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:10.984865 18020 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:10.986662 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.976536 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:10.966424 21934 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.927508 22030 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:23:10.999385 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:11.001160 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.001690 17988 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13.
I20260709 12:23:10.921149 22045 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:10.984978 21936 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:10.957906 22272 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.021860 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:11.023151 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 102468us
I20260709 12:23:10.970511 21977 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.000978 22273 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:11.079342 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.084858 18016 raft_consensus.cc:2376] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9. Re-sending same reply.
W20260709 12:23:11.080020 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78650.8us
I20260709 12:23:11.003712 21934 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.018409 21936 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:11.026388 22285 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.813315 18166 rpcz_store.cc:276] 0709 12:23:08.531862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.353371 (+821509us) service_pool.cc:224] Handling call
0709 12:23:09.396914 (+ 43543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.409303 (+ 12389us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:09.681743 (+272440us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.806394 (+124651us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.808045 (+  1651us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.808105 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:09.808165 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.808812 (+   647us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.809265 (+   453us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39318,"prepare.queue_time_us":86093,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":111,"raft.queue_time_us":3629,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":3033,"spinlock_wait_cycles":7364096,"thread_start_us":275261,"threads_started":2}
I20260709 12:23:11.103070 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.104976 18015 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:23:11.105962 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92759.4us
I20260709 12:23:11.107586 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.111531 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:11.112730 18019 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:11.079280 22186 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:11.114965 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.123157 18016 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 5: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:11.108565 18021 raft_consensus.cc:2376] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7. Re-sending same reply.
I20260709 12:23:11.091009 22152 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:11.123226 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.157171 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 157924us
I20260709 12:23:11.219980 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.102995 21936 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:11.199055 22277 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:23:11.224349 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.095207 22094 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:11.164866 22275 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.181s	user 0.000s	sys 0.002s
I20260709 12:23:11.114580 22267 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.225350 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:11.199990 22148 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.194018 22288 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.234364 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:11.199692 21934 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:11.240206 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:11.202641 22229 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.768s	user 0.004s	sys 0.002s
I20260709 12:23:11.122592 22186 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:11.237438 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.130749 22293 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:11.121897 22285 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.249154 22045 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:11.243144 22148 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:11.250752 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.130893 21970 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.127s
I20260709 12:23:11.228916 22289 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:11.161520 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:11.173053 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.258533 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.264021 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.271117 18024 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:23:11.129566 18015 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:11.238072 22267 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.251562 22094 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:11.261019 21828 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.584s	user 0.014s	sys 0.001s
I20260709 12:23:11.259598 22294 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:11.159608 22228 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.172463 22292 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:11.299183 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 256586us
I20260709 12:23:11.299793 18015 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:11.268646 22148 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:11.297161 21828 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.316512 22296 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:11.317149 22276 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:11.317636 22276 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:11.318431 21971 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:11.318899 21971 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:11.320118 22260 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.874s	user 0.015s	sys 0.000s
I20260709 12:23:11.319977 21971 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.320868 22260 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:11.260488 22186 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.304154 18016 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:23:11.325469 18165 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:11.326534 22282 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:11.328172 22282 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:11.283227 22258 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.359262 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:11.360324 18090 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:11.365617 18019 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
W20260709 12:23:11.370004 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.398335 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.272673 21933 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:23:11.263276 22152 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.406625 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2065) took 2853 ms. Trace:
I20260709 12:23:11.321470 22260 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.297475 21709 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:11.428836 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:11.427735 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 420729us
I20260709 12:23:11.243739 22288 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.432821 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:11.419287 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 160873us
W20260709 12:23:11.420017 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 295181us
I20260709 12:23:11.322104 21971 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:11.421350 18167 rpcz_store.cc:276] 0709 12:23:08.514850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.520930 (+  6080us) service_pool.cc:224] Handling call
0709 12:23:08.521311 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.533757 (+ 12446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.533839 (+    82us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.541280 (+  7441us) log.cc:844] Serialized 26 byte log entry
0709 12:23:09.296121 (+754841us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:09.296162 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.296239 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.031446 (+1735207us) spinlock_profiling.cc:243] Waited 8.76 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:11.324870 (+293424us) raft_consensus.cc:1685] finished
0709 12:23:11.324900 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.355716 (+ 30816us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":70570,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":31879,"spinlock_wait_cycles":20144000,"thread_start_us":701183,"threads_started":1,"wal-append.queue_time_us":774305}
I20260709 12:23:11.318465 22285 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:11.418335 22293 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:11.354156 21977 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:11.324891 22272 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.434881 18166 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:11.452427 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.284368 22289 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.469941 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:11.449858 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:11.444808 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:11.463440 18090 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:11.427568 22186 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:11.477609 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.481438 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:11.323259 22228 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.488528 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.492820 22285 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:11.493412 22285 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.493887 22285 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:11.494418 22052 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.528s	user 0.014s	sys 0.001s
I20260709 12:23:11.495036 22052 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.496310 22311 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:11.497875 22052 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:11.491876 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.500162 22152 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:11.500933 22152 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.500243 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.529551 18098 pending_rounds.cc:85] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:23:11.530051 18098 pending_rounds.cc:107] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:23:11.530550 18098 raft_consensus.cc:2889] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:11.537132 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.356323 22292 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.528276 22094 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:11.546682 18095 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:11.548676 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.486512 21828 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.490079 22300 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:11.483578 22303 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:11.553556 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.501513 22152 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.556969 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.555961 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:11.558897 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:11.559407 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.487069 22272 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 12:23:11.580614 18096 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 5.
I20260709 12:23:11.582398 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.585888 18095 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:11.487337 22301 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 12:23:11.583513 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.590075 18094 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:23:11.542827 22299 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:11.604487 18090 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:11.604667 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:11.547161 21933 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:23:11.552083 21709 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:11.607456 18091 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:11.609634 22305 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:23:11.605480 18096 raft_consensus.cc:2376] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14. Re-sending same reply.
I20260709 12:23:11.590982 21828 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:11.548982 22312 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:11.613605 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.615882 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.616268 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:11.616873 18171 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:11.617324 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:11.557510 22309 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:23:11.627342 18093 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:11.605163 22313 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.630218 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:11.635905 18167 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:11.625674 18099 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:11.624773 18164 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:11.629000 18094 raft_consensus.cc:2376] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8. Re-sending same reply.
I20260709 12:23:11.629355 18095 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:11.634979 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:11.713719 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78532.7us
I20260709 12:23:11.716615 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:11.625299 22304 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.717499 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:11.609722 22321 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:11.723055 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:11.725029 18093 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:11.726776 18171 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
W20260709 12:23:11.633924 22307 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.169s	user 0.000s	sys 0.002s
W20260709 12:23:11.753546 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 197658us
I20260709 12:23:11.754428 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:11.754750 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:11.756686 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.758369 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:11.740413 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.761477 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44976.3us
I20260709 12:23:11.714594 22300 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:11.764011 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:11.760596 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.724282 22298 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.776330 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.777339 18094 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:11.738920 22305 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:23:11.782027 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:11.782825 18171 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:11.752758 22321 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:11.785295 22321 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.632244 22276 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.791290 22276 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:11.793968 22300 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.797053 22300 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.735564 22304 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.800132 22304 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.761396 22260 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.801040 22282 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:11.803031 22292 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:11.804289 22267 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.722633 22094 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.806289 22228 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.807217 22298 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.808311 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:11.774860 22297 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.345s	user 0.000s	sys 0.002s
I20260709 12:23:11.809315 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2242) took 1039 ms. Trace:
I20260709 12:23:11.810190 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:11.811079 18094 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:11.809695 18022 rpcz_store.cc:276] 0709 12:23:10.770060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.921794 (+151734us) service_pool.cc:224] Handling call
0709 12:23:11.808109 (+886315us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:11.809238 (+  1129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35096960}
I20260709 12:23:11.734768 22306 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.827093 18093 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:11.828769 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:11.829631 18093 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:11.842090 18020 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:11.845124 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:11.848929 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.855295 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:11.856585 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 425762us
I20260709 12:23:11.783890 22301 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:23:11.857968 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.864007 18019 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:23:11.867390 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.868522 22300 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:11.873665 22300 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.622778 22293 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.876246 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.734520 22313 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.847404 22267 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.858952 22292 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:11.894750 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.902166 18171 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
W20260709 12:23:11.902689 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.866039 22304 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:11.895044 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:11.900983 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.907493 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.907981 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:11.908771 18019 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:11.878999 22327 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:11.847806 22228 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:23:11.878772 22282 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:11.889439 22301 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:11.903609 21970 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.004s	sys 0.006s
I20260709 12:23:11.925921 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:11.921805 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.895777 22090 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.938966 22328 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.938066 18016 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:11.934767 22309 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:23:11.925073 22317 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.858230 22306 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.954774 22090 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.988664 18168 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:11.994019 22300 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.995397 22282 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.909642 22267 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.995469 22196 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:11.952914 22328 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:11.998461 22196 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:11.999506 22196 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.997670 22317 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.000831 22196 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:11.997577 21933 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.952500 22094 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.003032 22301 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.004881 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:12.006268 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:11.940357 21970 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.004s	sys 0.007s
W20260709 12:23:12.029371 22326 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:23:11.966539 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:12.053552 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2250) took 1280 ms. Trace:
I20260709 12:23:12.054061 18018 rpcz_store.cc:276] 0709 12:23:10.772934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.922010 (+149076us) service_pool.cc:224] Handling call
0709 12:23:11.966256 (+1044246us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:12.053461 (+ 87205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34755968}
I20260709 12:23:11.974488 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2163) took 1430 ms. Trace:
I20260709 12:23:11.943524 22288 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:11.947875 22296 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:23:12.058364 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:12.060364 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:11.932680 22285 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.084584 22285 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:11.933138 22318 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 12:23:11.910645 22292 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:11.997674 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2317) took 1972 ms. Trace:
I20260709 12:23:12.088670 18165 rpcz_store.cc:276] 0709 12:23:10.025710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.029743 (+  4033us) service_pool.cc:224] Handling call
0709 12:23:11.325153 (+1295410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.325947 (+   794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.997432 (+671485us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400382f50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:11.997596 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2978156416}
I20260709 12:23:12.007433 18096 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:23:12.090169 22011 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:12.009066 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.092057 18808 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:12.010169 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.093590 18807 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
I20260709 12:23:12.103516 22285 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:12.104725 22285 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.110682 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:12.112440 22152 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.114959 18811 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:12.116219 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:12.116485 22260 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:12.117517 22030 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:12.013411 22309 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.016769 22256 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.777s	user 0.009s	sys 0.000s
W20260709 12:23:12.017630 22290 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.847s	user 0.000s	sys 0.008s
I20260709 12:23:12.019322 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:12.020243 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3036) took 1518 ms. Trace:
I20260709 12:23:12.020885 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 2929) took 1646 ms. Trace:
W20260709 12:23:12.022208 21936 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.783s	user 0.016s	sys 0.001s
I20260709 12:23:11.923583 18093 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:12.113462 21977 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:12.037214 18020 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:12.037389 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:11.957886 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:12.040906 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:12.041728 22270 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:12.043612 22237 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:12.050556 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:12.058645 22296 consensus_queue.cc:1243] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:11.865178 21709 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.057391 18097 rpcz_store.cc:276] 0709 12:23:10.543868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.695600 (+151732us) service_pool.cc:224] Handling call
0709 12:23:10.695981 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.985276 (+289295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.974102 (+988826us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b5400215054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:11.974388 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":632623360}
W20260709 12:23:11.982416 22323 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:23:12.072475 22298 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.076610 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:12.080973 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.081563 21984 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.121387 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 540) took 1010 ms. Trace:
I20260709 12:23:12.158193 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:12.132084 22330 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:12.160279 22330 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.127074 22309 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.160652 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:12.132611 22270 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:12.161456 22270 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:12.159103 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:12.126715 21977 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:12.161669 22303 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:12.162228 22303 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:12.122399 21936 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.162714 22303 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:12.162911 21977 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.164584 21977 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.164898 21936 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.133263 22237 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.135020 21934 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.143131 18098 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:12.168637 22237 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.168853 18098 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: [7.3-7.3]
I20260709 12:23:12.135098 18090 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:12.170763 22328 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.173344 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:12.143436 18099 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:12.174646 22306 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.143940 22273 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:23:12.176090 22317 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:12.176309 22223 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:12.178509 22294 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:12.178944 22294 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:12.179836 22294 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.154495 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.148371 21984 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.132915 18091 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:12.182592 18098 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:12.182785 22267 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.183416 18015 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:12.183694 21984 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.165712 21936 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:12.154733 22285 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.160742 22330 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:12.155687 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:12.118002 22030 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.186004 22030 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:12.155229 18095 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:23:12.187707 22309 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:12.189980 22313 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.163139 21933 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.190989 22294 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:12.191382 18022 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:12.158423 18021 rpcz_store.cc:276] 0709 12:23:11.111195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.263667 (+152472us) service_pool.cc:224] Handling call
0709 12:23:12.019122 (+755455us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:12.121271 (+102149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8567936}
I20260709 12:23:12.121568 18092 rpcz_store.cc:276] 0709 12:23:10.501426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.547032 (+ 45606us) service_pool.cc:224] Handling call
0709 12:23:10.547438 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.562233 (+ 14795us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:10.948293 (+386060us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.156739 (+208446us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.984271 (+827532us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:11.984959 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.985216 (+   257us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:11.985271 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.985385 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.007691 (+ 22306us) raft_consensus.cc:1685] finished
0709 12:23:12.007819 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.020121 (+ 12302us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44149,"prepare.queue_time_us":469230,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":113449,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":10202,"spinlock_wait_cycles":4001152,"thread_start_us":536485,"threads_started":2,"wal-append.queue_time_us":307878}
I20260709 12:23:12.197744 18092 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:23:12.201305 18022 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:12.156829 22292 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.202487 21988 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.166971 21934 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.227208 22299 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:12.228971 22237 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:12.230134 21936 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:12.230561 21936 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:12.230930 21971 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.766s	user 0.009s	sys 0.010s
I20260709 12:23:12.113456 18807 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:23:12.231753 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:12.231575 21936 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.203503 22090 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.232617 22223 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:23:12.202194 18092 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:12.234176 22322 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.234629 22322 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.234990 22322 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.234979 22223 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:12.234985 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:12.236378 22272 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:12.237250 22272 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:12.237135 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:12.237655 22272 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:12.238061 18092 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:12.238713 22272 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.239239 18092 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:12.240061 22272 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.146618 22260 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.242060 22312 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.530s	user 0.007s	sys 0.010s
I20260709 12:23:12.242404 22223 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:12.242197 22260 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.242990 22223 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:12.154992 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.158742 22289 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.127530 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.243894 22273 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:12.243971 18811 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:12.244393 22273 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:12.142419 18097 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:12.245364 22273 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.245914 21936 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:12.246912 22273 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:12.117784 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:12.147840 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:12.248490 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100759us
I20260709 12:23:12.206660 22309 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:12.250361 21984 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:12.250216 22309 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.250618 22258 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:23:12.243873 22186 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.756s	user 0.009s	sys 0.009s
I20260709 12:23:12.254333 22223 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.256106 22223 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.210374 22030 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.003341 22094 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.213179 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.219543 18015 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:12.228507 21934 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.242592 22312 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:12.122043 18017 rpcz_store.cc:276] 0709 12:23:10.374203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.374603 (+   400us) service_pool.cc:224] Handling call
0709 12:23:12.020567 (+1645964us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:12.020802 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843313024}
I20260709 12:23:12.272696 18812 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:12.250918 21984 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:12.273568 22030 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.273857 21984 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:12.257759 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84502us
I20260709 12:23:12.258338 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:12.274770 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:12.266695 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:12.274852 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:12.212502 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 254762us
W20260709 12:23:12.277664 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 159979us
I20260709 12:23:12.279088 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:12.279726 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:12.276813 18811 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:12.209981 21933 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.253388 22258 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:23:12.291000 18165 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:12.293737 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:12.206969 21828 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.008s
I20260709 12:23:12.253576 22309 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:12.306532 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48334us
I20260709 12:23:12.308795 18096 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:12.314307 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:12.307267 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.313081 18018 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:12.320065 18165 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:12.323939 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:12.325167 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.326207 18168 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:12.329926 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.331485 18168 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:12.303349 22186 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:12.305089 21977 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:12.284762 22313 raft_consensus.cc:2764] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:23:12.307276 22030 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:12.312618 22237 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:12.327909 21828 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:12.327848 21971 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:12.401216 21971 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:12.402822 21971 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.352051 22289 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.406486 22289 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.408126 22260 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:12.409096 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.303790 22303 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.335916 22276 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:23:12.334514 22317 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 4, Time since last communication: 0.104s
I20260709 12:23:12.361843 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:12.363498 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.363794 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:12.511816 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:12.477231 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.360790 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:12.352562 22030 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.338047 22186 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:12.566607 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:12.583880 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.409603 22289 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.423945 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.423547 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.431020 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 593) took 1002 ms. Trace:
W20260709 12:23:12.777423 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 463197us
I20260709 12:23:12.778558 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:12.777722 18173 rpcz_store.cc:276] 0709 12:23:11.428592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.429386 (+   794us) service_pool.cc:224] Handling call
0709 12:23:11.449136 (+ 19750us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.634801 (+185665us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.634836 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.635439 (+   603us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.890310 (+254871us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:11.890361 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.098894 (+208533us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400383454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:12.098953 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.430495 (+331542us) raft_consensus.cc:1685] finished
0709 12:23:12.430531 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.430882 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87090,"raft.queue_time_us":109125,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":13762,"spinlock_wait_cycles":357569408,"thread_start_us":167706,"threads_started":1,"wal-append.queue_time_us":592679}
I20260709 12:23:12.370981 18166 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:12.341144 21977 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:12.241720 22270 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:12.655797 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.794885 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2326) took 1195 ms. Trace:
I20260709 12:23:12.576434 22186 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:12.422889 22094 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:12.667974 21828 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.406525 22237 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:12.441128 22322 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:12.709154 18170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.007s	sys 0.001s
W20260709 12:23:12.752007 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.025s	sys 0.030s
W20260709 12:23:12.824309 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.025s	sys 0.031s
I20260709 12:23:12.824532 18021 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:12.826931 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.827980 18172 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:12.824084 18170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.007s	sys 0.002s
I20260709 12:23:12.829471 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:12.754158 18018 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:23:12.735247 22325 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.014s	sys 0.040s
I20260709 12:23:12.391371 22309 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:12.410470 22313 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:12.384163 22267 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.564539 22196 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.724555 22293 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.001s	sys 0.007s
W20260709 12:23:12.667933 18024 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.064s	user 0.013s	sys 0.019s
I20260709 12:23:12.409381 22260 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:12.757850 18094 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:12.759085 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:12.758932 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 478314us
W20260709 12:23:12.774333 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 413605us
I20260709 12:23:12.777478 18164 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:12.858330 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:12.709161 22308 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.004s	sys 0.014s
W20260709 12:23:12.854163 18024 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.250s	user 0.017s	sys 0.021s
W20260709 12:23:12.788206 18019 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.873s	user 0.011s	sys 0.005s
I20260709 12:23:12.859803 18019 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:12.859994 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2253) took 2086 ms. Trace:
I20260709 12:23:12.860404 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2263) took 1034 ms. Trace:
I20260709 12:23:12.860885 18019 rpcz_store.cc:276] 0709 12:23:11.826219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.907675 (+ 81456us) service_pool.cc:224] Handling call
0709 12:23:12.860323 (+952648us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:12.737062 21709 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.004s	sys 0.004s
W20260709 12:23:12.792793 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 226743us
W20260709 12:23:12.823072 22282 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.005s	sys 0.010s
I20260709 12:23:12.795385 18166 rpcz_store.cc:276] 0709 12:23:11.599975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.600158 (+   183us) service_pool.cc:224] Handling call
0709 12:23:12.794623 (+1194465us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:12.794807 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391250304}
I20260709 12:23:12.884891 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.885670 18166 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:12.797504 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:12.806481 18064 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.004s	sys 0.008s
W20260709 12:23:12.887148 18064 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.005s	sys 0.008s
W20260709 12:23:12.887552 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 676998us
W20260709 12:23:12.806867 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.008s	sys 0.004s
I20260709 12:23:12.813982 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:12.815393 18093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.003s	sys 0.012s
W20260709 12:23:12.779153 22011 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.688s	user 0.019s	sys 0.000s
W20260709 12:23:12.890369 18093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.004s	sys 0.012s
W20260709 12:23:12.709719 18091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.003s	sys 0.000s
W20260709 12:23:12.891096 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77266.5us
I20260709 12:23:12.891336 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:12.793962 21977 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:12.862672 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:12.862884 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:12.893553 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96139.7us
I20260709 12:23:12.864430 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:12.871017 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:12.841946 21996 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.013s	sys 0.005s
I20260709 12:23:12.567938 22030 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:12.875192 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:12.835237 22325 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.014s	sys 0.041s
W20260709 12:23:12.933573 22294 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.742s	user 0.017s	sys 0.000s
W20260709 12:23:12.890090 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.008s	sys 0.004s
W20260709 12:23:12.798949 22272 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.558s	user 0.016s	sys 0.000s
I20260709 12:23:12.930984 22260 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:23:12.891222 18091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.005s	sys 0.000s
I20260709 12:23:12.790875 22306 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:12.892225 18019 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:12.847460 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.891969 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:12.862253 22308 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.922s	user 0.004s	sys 0.015s
W20260709 12:23:12.864121 22293 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.002s	sys 0.007s
I20260709 12:23:12.835480 22309 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.914601 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:12.987613 18018 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:23:12.914683 22317 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
W20260709 12:23:12.992491 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.992956 18090 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:23:12.838004 21933 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:12.929450 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:12.873740 21709 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.004s	sys 0.004s
W20260709 12:23:12.884047 22282 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.006s	sys 0.010s
I20260709 12:23:12.936165 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:12.846511 22090 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.016s	sys 0.049s
I20260709 12:23:12.824061 21828 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.884675 22313 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:12.931393 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:12.950011 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79139.9us
W20260709 12:23:12.947320 22332 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.001s
I20260709 12:23:12.854663 22267 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:12.930435 22276 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:12.953505 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89144.3us
I20260709 12:23:13.028702 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.860420 18024 rpcz_store.cc:276] 0709 12:23:10.792585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.284890 (+492305us) service_pool.cc:224] Handling call
0709 12:23:11.285367 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.298134 (+ 12767us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.298176 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.322903 (+ 24727us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.491134 (+168231us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:11.603530 (+112396us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.859591 (+1256061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.859690 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.859740 (+    50us) raft_consensus.cc:1685] finished
0709 12:23:12.859752 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.859926 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71169,"prepare.queue_time_us":361616,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":27808,"raft.run_cpu_time_us":388,"raft.run_wall_time_us":40474,"thread_start_us":1389375,"threads_started":3,"wal-append.queue_time_us":273859,"wal-append.run_cpu_time_us":3219,"wal-append.run_wall_time_us":1078889}
I20260709 12:23:12.979019 18015 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
W20260709 12:23:13.027809 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97325.9us
I20260709 12:23:12.883066 22196 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:12.950925 21996 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.013s	sys 0.005s
I20260709 12:23:12.916296 18023 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
W20260709 12:23:12.961530 21936 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.715s	user 0.013s	sys 0.010s
I20260709 12:23:12.923173 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.993410 18090 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
W20260709 12:23:12.948796 22301 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.015s	sys 0.022s
I20260709 12:23:13.006008 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:13.062970 18090 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:23:13.063339 18090 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:13.063046 18164 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:13.004283 18165 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:23:13.064311 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:12.966899 22296 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:13.029170 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:13.031383 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:13.063124 22301 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.015s	sys 0.024s
I20260709 12:23:12.983580 22325 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:12.988261 22237 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.065126 18167 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:13.036463 22303 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:13.003543 22090 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.018s	sys 0.049s
I20260709 12:23:12.999526 22258 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:12.996872 22308 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:13.071462 22308 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.003218 22294 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:13.074309 22294 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:12.978077 22289 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:13.078661 22289 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.079437 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.043735 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:13.037387 22267 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:12.972276 22306 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:12.986629 22272 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:13.082479 18171 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:13.082808 22272 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.017050 21933 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.082515 22306 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:13.086788 21933 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.088532 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2186) took 1181 ms. Trace:
W20260709 12:23:13.089277 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60200.3us
I20260709 12:23:13.090358 18778 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:13.092284 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:12.991137 22309 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:13.003183 22270 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.062039 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2180) took 1166 ms. Trace:
I20260709 12:23:13.088531 22313 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.095031 22313 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.094574 18091 rpcz_store.cc:276] 0709 12:23:11.895156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.182937 (+287781us) service_pool.cc:224] Handling call
0709 12:23:12.183335 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.183763 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.183790 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.205480 (+ 21690us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.962496 (+757016us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:12.962559 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.962659 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.061581 (+ 98922us) raft_consensus.cc:1685] finished
0709 12:23:13.061621 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.061919 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":872,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":65,"thread_start_us":756394,"threads_started":1,"wal-append.queue_time_us":853220}
I20260709 12:23:13.097498 22313 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.047571 22321 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.098044 22258 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:13.003959 22239 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:13.099617 22308 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.077795 22294 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.100994 22355 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:23:13.101941 22289 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.102427 22260 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.103400 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:13.103974 22330 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.918s	user 0.019s	sys 0.005s
I20260709 12:23:13.089071 18095 rpcz_store.cc:276] 0709 12:23:11.908520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.203630 (+295110us) service_pool.cc:224] Handling call
0709 12:23:12.204111 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.204904 (+   793us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.204938 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.207668 (+  2730us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.692447 (+484779us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:12.692507 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.692616 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.088223 (+395607us) raft_consensus.cc:1685] finished
0709 12:23:13.088257 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.088455 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":169,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":104,"thread_start_us":484173,"threads_started":1,"wal-append.queue_time_us":778373}
I20260709 12:23:13.106251 22196 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:13.067065 22336 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:23:13.207496 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:13.134397 22239 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:13.003444 21709 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:13.098135 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:13.136006 18019 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:13.065084 18164 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:13.137522 22313 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:13.209337 18095 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:13.137614 22322 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:13.207193 22330 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.156347 22334 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:13.197587 22294 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.120734 22321 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.200969 22312 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:13.257210 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.372985 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.158984 21828 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:13.294735 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.376801 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:13.380553 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:13.361416 21984 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.057s	user 0.014s	sys 0.002s
I20260709 12:23:13.303238 22306 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.309693 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:13.393400 18097 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:13.394845 18097 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:13.395155 18168 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:13.395593 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3049) took 1007 ms. Trace:
I20260709 12:23:13.396034 18097 rpcz_store.cc:276] 0709 12:23:12.388229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.394267 (+1006038us) service_pool.cc:224] Handling call
0709 12:23:13.394621 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.395284 (+   663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.395495 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:13.310910 22339 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:23:13.195044 22289 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:13.314085 22223 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.057s	user 0.016s	sys 0.004s
W20260709 12:23:13.314615 22304 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.402s	user 0.019s	sys 0.005s
W20260709 12:23:13.315301 22273 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.068s	user 0.008s	sys 0.008s
W20260709 12:23:13.316375 22338 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:13.316918 21977 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.216821 22258 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.101s
I20260709 12:23:13.239748 22335 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.330968 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2245) took 2559 ms (client timeout 1916 ms). Trace:
I20260709 12:23:13.363266 18171 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:13.366858 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 591) took 1945 ms. Trace:
W20260709 12:23:13.446370 18023 rpcz_store.cc:269] 0709 12:23:10.771194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.818487 (+ 47293us) service_pool.cc:224] Handling call
0709 12:23:13.330682 (+2512195us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:13.330838 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4817870208}
I20260709 12:23:13.397850 22299 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.310062 22310 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:13.208031 22352 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:13.384495 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:13.386343 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:13.358423 22282 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:13.446908 18170 rpcz_store.cc:276] 0709 12:23:11.421174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.421555 (+   381us) service_pool.cc:224] Handling call
0709 12:23:11.423269 (+  1714us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.429143 (+  5874us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:11.449080 (+ 19937us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.120519 (+671439us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b1800441b80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:23:12.829764 (+709245us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.289214 (+459450us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b2c001268f8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:13.289946 (+   732us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.290157 (+   211us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:13.290209 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.290326 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.366445 (+ 76119us) raft_consensus.cc:1685] finished
0709 12:23:13.366483 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.366757 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30323,"prepare.queue_time_us":20971,"prepare.run_cpu_time_us":178,"prepare.run_wall_time_us":22565,"raft.queue_time_us":143913,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":46998,"spinlock_wait_cycles":12932864,"thread_start_us":708943,"threads_started":1,"wal-append.queue_time_us":1817105}
I20260709 12:23:13.387847 18808 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:13.368587 22333 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:13.413906 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 496247us
I20260709 12:23:13.422935 22304 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:13.389055 22322 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.475761 18166 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:13.397447 21934 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:13.413349 22030 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.413894 22196 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.387847 22300 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:13.427163 18099 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.015s	sys 0.016s
I20260709 12:23:13.370365 22309 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:13.480754 18099 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.016s	sys 0.017s
I20260709 12:23:13.426888 18097 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:13.483785 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3051) took 1069 ms. Trace:
I20260709 12:23:13.484364 18097 rpcz_store.cc:276] 0709 12:23:12.414008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.416413 (+1002405us) service_pool.cc:224] Handling call
0709 12:23:13.423385 (+  6972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.483315 (+ 59930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.483685 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.465461 18015 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:23:13.466643 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82276.1us
I20260709 12:23:13.390293 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:13.391333 22312 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.472658 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:13.394649 21984 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.388585 22294 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:13.474676 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.018s	sys 0.019s
W20260709 12:23:13.497859 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.018s	sys 0.019s
I20260709 12:23:13.491480 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:13.492857 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.473320 22273 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:13.464548 22352 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:13.502314 18097 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:13.418810 22289 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.507422 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2179) took 1612 ms. Trace:
W20260709 12:23:13.499598 18065 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.001s	sys 0.021s
W20260709 12:23:13.508611 18065 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.001s	sys 0.021s
I20260709 12:23:13.426558 21709 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:13.508636 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.503659 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 409885us
W20260709 12:23:13.488525 22341 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.273s	user 0.001s	sys 0.001s
I20260709 12:23:13.508059 18099 rpcz_store.cc:276] 0709 12:23:11.894875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.175085 (+280210us) service_pool.cc:224] Handling call
0709 12:23:12.195101 (+ 20016us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:23:12.195348 (+   247us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.196098 (+   750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.196134 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.196656 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.693706 (+497050us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:13.490684 (+796978us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.497608 (+  6924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.497748 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.497840 (+    92us) raft_consensus.cc:1685] finished
0709 12:23:13.506678 (+  8838us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.506949 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10119,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":29757824,"thread_start_us":1167353,"threads_started":2,"wal-append.queue_time_us":792623}
I20260709 12:23:13.512583 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3052) took 1098 ms. Trace:
I20260709 12:23:13.513151 18097 rpcz_store.cc:276] 0709 12:23:12.414394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.486273 (+1071879us) service_pool.cc:224] Handling call
0709 12:23:13.502028 (+ 15755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.512268 (+ 10240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.512512 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.478947 22282 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:13.516249 18811 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:13.469957 22299 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.524646 18091 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:13.503345 22273 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.464953 22335 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.503780 22223 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:13.590893 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.594823 22282 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.529572 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3045) took 1926 ms. Trace:
I20260709 12:23:13.544589 22270 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:23:13.553769 22359 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:23:13.618347 21977 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.479703 22333 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.518818 22350 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.102s
I20260709 12:23:13.632230 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2181) took 1734 ms. Trace:
I20260709 12:23:13.556164 22309 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.556991 22272 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.557960 22030 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.559958 22330 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.522819 21996 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:13.561079 22363 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.531119 22304 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:13.559989 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2178) took 1665 ms. Trace:
I20260709 12:23:13.692994 18090 rpcz_store.cc:276] 0709 12:23:11.894603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.173617 (+279014us) service_pool.cc:224] Handling call
0709 12:23:12.182007 (+  8390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.063978 (+881971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.559606 (+495628us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:13.559894 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1806528896}
I20260709 12:23:13.511973 22352 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:13.544929 22273 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.579770 22296 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:13.583689 22348 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.596s	user 0.003s	sys 0.000s
I20260709 12:23:13.600059 18093 rpcz_store.cc:276] 0709 12:23:11.602815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.123036 (+520221us) service_pool.cc:224] Handling call
0709 12:23:12.123930 (+   894us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.124533 (+   603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.124581 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.125343 (+   762us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.890840 (+765497us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:12.890903 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.890984 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.529208 (+638224us) raft_consensus.cc:1685] finished
0709 12:23:13.529244 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.529480 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2998,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":131,"thread_start_us":764982,"threads_started":1,"wal-append.queue_time_us":1016247}
I20260709 12:23:13.514637 21828 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.555183 22342 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:23:13.493985 22196 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:13.554585 22343 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:23:13.494231 22300 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.481259 22322 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.632853 18098 rpcz_store.cc:276] 0709 12:23:11.897521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.183482 (+285961us) service_pool.cc:224] Handling call
0709 12:23:12.183852 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.184244 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.184279 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.184856 (+   577us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.953695 (+768839us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:12.953756 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.953867 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.631937 (+678070us) raft_consensus.cc:1685] finished
0709 12:23:13.631966 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.632146 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":527,"raft.run_cpu_time_us":45,"raft.run_wall_time_us":46,"thread_start_us":767220,"threads_started":1,"wal-append.queue_time_us":880631}
I20260709 12:23:13.645676 22094 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:13.494691 21934 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.526585 22090 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 12:23:13.523962 21709 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:13.735698 22364 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.716674 22306 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.748621 22299 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:13.748484 22364 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:13.700719 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.020s	sys 0.046s
W20260709 12:23:13.750743 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.020s	sys 0.046s
W20260709 12:23:13.751757 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 720443us
I20260709 12:23:13.704996 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3044) took 2102 ms. Trace:
I20260709 12:23:13.723181 18807 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:13.754076 18094 rpcz_store.cc:276] 0709 12:23:11.602418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.851924 (+249506us) service_pool.cc:224] Handling call
0709 12:23:11.852317 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.854814 (+1002497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.704600 (+849786us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:13.704895 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1780761472}
I20260709 12:23:13.724212 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:13.699700 21996 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:13.708050 22296 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:13.740761 18019 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
W20260709 12:23:13.709008 22325 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.028s	sys 0.068s
I20260709 12:23:13.753587 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:13.756512 18810 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:13.697474 22304 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.762476 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:13.791157 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.795276 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.762742 22325 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:13.796579 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.797605 18807 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:13.697768 22363 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.802246 18807 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:23:13.734512 22094 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:13.714504 21828 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:13.770278 22317 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:13.826143 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63805.7us
W20260709 12:23:13.800071 22186 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.948s	user 0.011s	sys 0.015s
I20260709 12:23:13.842694 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.858805 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.849403 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2190) took 1633 ms. Trace:
I20260709 12:23:13.849561 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:13.828368 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 339421us
I20260709 12:23:13.864276 18092 rpcz_store.cc:276] 0709 12:23:12.216191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.240034 (+ 23843us) service_pool.cc:224] Handling call
0709 12:23:12.240387 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.241422 (+  1035us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.241455 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.242091 (+   636us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.715438 (+473347us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:12.715511 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.715633 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.582671 (+867038us) spinlock_profiling.cc:243] Waited 7.85 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:13.832911 (+250240us) raft_consensus.cc:1685] finished
0709 12:23:13.832947 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.849286 (+ 16339us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":289,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":135,"spinlock_wait_cycles":18049920,"thread_start_us":472771,"threads_started":1,"wal-append.queue_time_us":641590}
I20260709 12:23:13.771656 22364 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.797001 22299 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.820307 22272 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.870229 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.880463 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:13.750887 22258 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.100s
I20260709 12:23:13.820847 22365 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.915658 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66234.1us
I20260709 12:23:13.831957 21984 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:13.913689 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.917316 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:13.942255 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:13.943656 18809 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:13.943763 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:13.944015 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.944839 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:23:13.917904 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.006s	sys 0.011s
W20260709 12:23:13.946089 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.008s	sys 0.011s
W20260709 12:23:13.947096 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.922914 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:13.943194 18023 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:13.956941 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:13.959199 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:13.960009 18020 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:13.971907 18018 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:13.971746 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:13.974107 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:13.975870 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:13.906546 22359 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.900295 22312 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:13.832717 22306 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:14.039280 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:13.859486 22317 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:13.992836 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50705.4us
I20260709 12:23:13.847239 22370 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.995587 18815 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:13.996556 18807 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:23:13.998116 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:13.998886 22276 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.951s	user 0.020s	sys 0.007s
W20260709 12:23:13.987247 22334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.029s	sys 0.019s
I20260709 12:23:13.871027 22186 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.000684 18020 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:23:14.002239 22239 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:14.014707 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2333) took 1653 ms (client timeout 2125 ms). Trace:
W20260709 12:23:14.108963 18168 rpcz_store.cc:269] 0709 12:23:12.361185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.361409 (+   224us) service_pool.cc:224] Handling call
0709 12:23:14.014407 (+1652998us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:14.014614 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1424903552}
I20260709 12:23:14.110788 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.112566 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:14.053810 22300 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.018661 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:14.019418 22372 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:13.971989 22270 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.021552 21828 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.022126 22321 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.022295 22237 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.022804 22223 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:14.017506 22360 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.551s	user 0.006s	sys 0.004s
W20260709 12:23:14.113735 22334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.029s	sys 0.020s
I20260709 12:23:14.024343 22335 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.023931 22325 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.024818 22293 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:14.013793 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:14.217108 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 203475us
I20260709 12:23:14.218324 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:14.219955 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:14.025889 22260 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.824s	user 0.013s	sys 0.014s
I20260709 12:23:14.043334 21996 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.029038 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 777) took 1096 ms. Trace:
I20260709 12:23:14.029791 22363 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.032600 22294 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.034687 22322 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.035656 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:13.952216 22365 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.036024 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:13.942252 22299 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.019445 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3056) took 1305 ms. Trace:
I20260709 12:23:14.216246 22293 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:13.899160 22362 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.042433 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:13.911322 22333 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:14.049504 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.925s	user 0.010s	sys 0.031s
W20260709 12:23:13.998931 22366 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:14.064853 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:14.073724 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:14.266872 17919 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 277) took 13421 ms (client timeout 15000 ms). Trace:
I20260709 12:23:14.268263 18171 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:14.236203 22260 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:14.087872 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:14.093863 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.051s	sys 0.080s
W20260709 12:23:14.276595 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.052s	sys 0.080s
W20260709 12:23:14.277091 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 756435us
I20260709 12:23:14.269519 18170 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:14.277990 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:14.279526 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:13.942101 22272 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.069808 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:14.044149 22289 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.048688 22317 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:14.090583 21971 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.335s	user 0.062s	sys 0.109s
I20260709 12:23:14.106423 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.994943 21984 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
I20260709 12:23:14.039887 22359 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:14.120056 18065 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.004s	sys 0.057s
I20260709 12:23:14.049037 22312 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 12:23:14.023356 21709 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.248214 22308 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:23:14.134971 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:14.083030 22350 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:23:14.333552 18810 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:14.137698 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:14.334491 18807 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:14.139490 18096 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:14.138574 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.994002 22094 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.221002 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:14.126422 22300 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.337078 22300 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.339730 22300 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.027714 22304 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.092522 22370 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.118741 22276 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:14.231997 18811 rpcz_store.cc:276] 0709 12:23:12.932602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.932815 (+   213us) service_pool.cc:224] Handling call
0709 12:23:12.933223 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.524410 (+591187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.028652 (+504242us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b540059fe54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:14.028940 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1340163456}
I20260709 12:23:14.136075 21828 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.344411 21828 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.344934 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:14.346086 18812 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:14.162855 22309 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.003s	sys 0.006s
W20260709 12:23:14.243366 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 268798us
I20260709 12:23:14.346784 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:14.142413 22331 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.016s	sys 0.010s
I20260709 12:23:14.140206 22310 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:14.348250 22331 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.018s	sys 0.010s
I20260709 12:23:14.348429 22310 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:14.158249 22196 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.249094 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:14.250736 18091 rpcz_store.cc:276] 0709 12:23:12.713674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.097770 (+384096us) service_pool.cc:224] Handling call
0709 12:23:14.019192 (+921422us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b540035edd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:14.019369 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":617812736}
I20260709 12:23:14.304023 22289 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:14.338120 22384 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:23:14.231252 22335 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:14.138293 22334 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:14.139986 22237 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.146363 22223 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:14.042951 22367 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
W20260709 12:23:14.271564 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.148s	user 0.010s	sys 0.035s
I20260709 12:23:14.244212 22381 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:14.132167 22270 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.248224 22322 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:14.289278 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:14.244457 22363 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.301891 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:14.307535 18065 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.004s	sys 0.057s
I20260709 12:23:14.272706 22321 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:14.311929 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 198632us
W20260709 12:23:14.268265 17919 rpcz_store.cc:269] 0709 12:23:00.887881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.888220 (+   339us) service_pool.cc:224] Handling call
0709 12:23:14.253871 (+13365651us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:07.633400 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:07.638082 (+  4682us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:07.638144 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:23:07.638165 (+    21us) tablet.cc:662] Decoding operations
0709 12:23:07.882723 (+244558us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:07.882911 (+   188us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:07.882951 (+    40us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:23:07.884746 (+  1795us) tablet.cc:701] Row locks acquired
0709 12:23:07.884809 (+    63us) write_op.cc:260] PREPARE: finished
0709 12:23:07.885014 (+   205us) write_op.cc:270] Start()
0709 12:23:07.885197 (+   183us) write_op.cc:276] Timestamp: P: 1783599787884941 usec, L: 0
0709 12:23:07.885238 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:23:07.949267 (+ 64029us) log.cc:844] Serialized 8212 byte log entry
0709 12:23:08.709923 (+760656us) op_driver.cc:464] REPLICATION: finished
0709 12:23:09.077653 (+367730us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1a0edef0
0709 12:23:09.514077 (+436424us) write_op.cc:301] APPLY: starting
0709 12:23:09.514243 (+   166us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:09.958227 (+443984us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:09.958275 (+    48us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:09.972427 (+ 14152us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:09.973102 (+   675us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=42,mrs_lookups=0
0709 12:23:09.973202 (+   100us) write_op.cc:312] APPLY: finished
0709 12:23:09.974058 (+   856us) log.cc:844] Serialized 188 byte log entry
0709 12:23:10.271883 (+297825us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:10.309250 (+ 37367us) write_op.cc:454] Released schema lock
0709 12:23:10.339634 (+ 30384us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:10.340228 (+   594us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:10.340272 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1129285,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":742638,"apply.run_cpu_time_us":75856,"apply.run_wall_time_us":836029,"cfile_cache_hit":86,"cfile_cache_hit_bytes":92033,"num_ops":21,"prepare.queue_time_us":179065,"prepare.run_cpu_time_us":10898,"prepare.run_wall_time_us":457971,"raft.queue_time_us":192676,"raft.run_cpu_time_us":56431,"raft.run_wall_time_us":700775,"replication_time_us":782571,"spinlock_wait_cycles":184286976,"thread_start_us":1147282,"threads_started":5,"wal-append.queue_time_us":678500,"wal-append.run_cpu_time_us":33992,"wal-append.run_wall_time_us":2357234}]]}
I20260709 12:23:14.322665 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:14.133901 18806 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20260709 12:23:14.299718 21971 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.544s	user 0.062s	sys 0.109s
I20260709 12:23:14.350437 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:13.939687 22330 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:14.367686 22094 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.368610 22304 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:14.346989 22309 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.005s	sys 0.014s
W20260709 12:23:14.370059 22382 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:23:14.371833 22312 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:14.351269 22392 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:14.373904 22350 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:14.375944 22272 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.379136 22365 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.380126 22196 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:14.381301 22299 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.381788 22186 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.383168 22359 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.316581 22294 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:23:14.382707 22310 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.386253 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:14.386943 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.387151 21934 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.272687 22362 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.387735 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2105) took 1552 ms. Trace:
I20260709 12:23:14.387879 22293 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.147368 22372 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:14.401064 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:14.411852 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 146998us
W20260709 12:23:14.665582 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 376384us
W20260709 12:23:14.574059 22282 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.014s	sys 0.042s
I20260709 12:23:14.390069 22289 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.547909 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3047) took 2160 ms. Trace:
I20260709 12:23:14.550536 18811 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:14.551296 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.692378 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.551553 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:14.554069 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 331430us
I20260709 12:23:14.550196 18815 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:14.560719 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.565259 22391 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.565179 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:14.564925 17918 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 10 to 12, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 12 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:14.715065 18806 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:14.566076 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.559520 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:14.686242 22282 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.019s	sys 0.043s
W20260709 12:23:14.716401 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 156947us
I20260709 12:23:14.718001 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:14.718782 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:14.541913 22381 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:14.610625 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.549196 22380 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:14.615221 18015 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.015s	sys 0.050s
I20260709 12:23:14.543084 22393 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:14.622300 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3048) took 2234 ms. Trace:
I20260709 12:23:14.548782 22321 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:14.555912 22396 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:14.558144 22273 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.847s	user 0.015s	sys 0.001s
I20260709 12:23:14.640815 18170 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:14.638170 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 332621us
I20260709 12:23:14.560742 22304 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:14.752745 22321 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:14.614542 22350 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:14.763725 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47158.9us
I20260709 12:23:14.554344 22363 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.570336 22335 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.654708 18169 rpcz_store.cc:276] 0709 12:23:12.835425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.091649 (+256224us) service_pool.cc:224] Handling call
0709 12:23:13.092176 (+   527us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:13.700975 (+608799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.701014 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:13.711051 (+ 10037us) log.cc:844] Serialized 52 byte log entry
0709 12:23:14.118866 (+407815us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:14.118914 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.119000 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.387434 (+268434us) raft_consensus.cc:1685] finished
0709 12:23:14.387458 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.387653 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":102855,"raft.queue_time_us":300361,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":24316,"thread_start_us":841252,"threads_started":2,"wal-append.queue_time_us":646270}
I20260709 12:23:14.405135 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:14.674535 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.785020 18094 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:14.675963 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.786460 18807 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:14.677239 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.786585 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.669507 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:14.787815 18094 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:14.788972 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.632167 22365 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:14.689913 18096 rpcz_store.cc:276] 0709 12:23:12.387480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.392955 (+  5475us) service_pool.cc:224] Handling call
0709 12:23:12.412951 (+ 19996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.335042 (+1922091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.547527 (+212485us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400410fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:14.547818 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3970080000}
I20260709 12:23:14.700950 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.702884 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:14.698014 18164 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:14.694406 18172 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:14.706668 18097 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:23:14.708662 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.801227 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:14.603820 18166 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:14.789757 18094 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:23:14.723301 18015 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.016s	sys 0.050s
I20260709 12:23:14.725638 18809 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:14.691401 22293 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:14.686419 22186 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:14.728045 18095 rpcz_store.cc:276] 0709 12:23:12.387867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.383241 (+995374us) service_pool.cc:224] Handling call
0709 12:23:13.383719 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.637322 (+253603us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:14.093365 (+456043us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.271458 (+178093us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.271538 (+    80us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:14.271622 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.388858 (+117236us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b1800477100. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:23:14.621980 (+233122us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540005fad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:14.622048 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.622218 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1809,"prepare.queue_time_us":115287,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":18898,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":20052,"spinlock_wait_cycles":585176832,"thread_start_us":605633,"threads_started":2,"wal-append.queue_time_us":456232}
I20260709 12:23:14.731917 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:14.820853 18173 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.734818 22310 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.735590 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:14.742547 22387 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.338s	user 0.000s	sys 0.000s
I20260709 12:23:14.744477 22355 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:14.695127 22289 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.703133 21828 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:14.748759 22352 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.043s	user 0.015s	sys 0.003s
I20260709 12:23:14.849754 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.850627 18093 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:23:14.751123 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54909.7us
I20260709 12:23:14.750439 18806 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:14.755604 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:14.690130 22294 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:14.759251 22273 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.759123 22359 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.006s
W20260709 12:23:14.856746 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 101289us
I20260709 12:23:14.765183 22223 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:14.766527 22372 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.867841 22407 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:14.868530 22407 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:14.753255 22361 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:23:14.763334 22396 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.874101 22368 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:14.874702 22368 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:14.876413 22317 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.559s	user 0.011s	sys 0.000s
I20260709 12:23:14.876982 22317 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.845988 22408 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.774480 22237 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.776439 22362 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.754508 22380 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.722782 22381 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:14.727655 22258 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:14.791584 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.883335 18090 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:14.792384 22335 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:14.794195 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124852us
I20260709 12:23:14.888190 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:14.720346 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.824842 18815 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:14.850642 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.659032 22334 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.863142 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:14.867046 22411 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:14.890887 18096 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:14.856760 22270 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:14.903633 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.856113 22294 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:14.724889 22391 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.936929 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:14.907285 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44250.7us
I20260709 12:23:14.909540 18090 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:14.879086 21934 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:14.944350 18171 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:14.944460 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:14.828205 22186 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:14.945866 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 227140us
I20260709 12:23:14.965289 18173 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:14.967628 18167 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:14.886726 22258 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.793315 22350 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.975971 18168 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:23:14.976261 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.878304 22408 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:14.980878 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92808.5us
I20260709 12:23:14.943140 21934 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.909948 22237 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:14.827870 22304 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:14.974431 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:14.983853 22296 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.187s	user 0.017s	sys 0.000s
I20260709 12:23:14.978009 22258 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:15.023537 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49253.1us
W20260709 12:23:15.001366 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.008075 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:14.865569 22352 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.011057 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.047818 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:15.046259 18092 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:23:14.869498 22407 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:14.990412 22396 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:15.055755 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.057447 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:15.058902 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 120117us
I20260709 12:23:14.899103 22400 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
I20260709 12:23:15.008807 21934 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.066707 18096 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:15.114198 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.086452 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.108816 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:15.121845 18096 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
W20260709 12:23:15.121758 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.05196e+06us
I20260709 12:23:15.072641 18808 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:15.127082 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:15.129254 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:15.130414 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:15.131601 18173 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:15.135298 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:15.136104 18806 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:15.133635 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.141589 18020 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:23:15.062309 22304 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:15.142061 18015 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:15.110242 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2110) took 2223 ms. Trace:
I20260709 12:23:15.062037 22350 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:15.113119 22394 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.488s	user 0.003s	sys 0.000s
I20260709 12:23:15.185751 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:15.163218 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 218845us
I20260709 12:23:15.186839 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:15.187808 18814 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:15.161621 18165 rpcz_store.cc:276] 0709 12:23:12.887352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.337563 (+450211us) service_pool.cc:224] Handling call
0709 12:23:13.345995 (+  8432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.346372 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:13.895878 (+549506us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.748529 (+852651us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b18002ab1a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca557 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:15.109254 (+360725us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b18002ab1a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:23:15.109635 (+   381us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:15.109717 (+    82us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:15.109842 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.109949 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.110148 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26444,"prepare.queue_time_us":906679,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":11683,"raft.queue_time_us":837489,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":154,"spinlock_wait_cycles":13324928,"thread_start_us":856721,"threads_started":2,"wal-append.queue_time_us":493609}
I20260709 12:23:15.056690 22355 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:15.163008 18018 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:15.057996 22380 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:15.114533 22306 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.185791 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:15.209908 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.014078 22413 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.132s
I20260709 12:23:15.210690 18018 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.187041 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:15.100701 22389 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.377s	user 0.007s	sys 0.007s
I20260709 12:23:15.212726 18166 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
W20260709 12:23:15.095333 22412 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:23:15.189184 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:15.216138 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:15.132941 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:15.218935 18172 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:15.112421 22308 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:15.111331 22369 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:15.199855 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.200098 18015 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:15.236810 18020 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:15.202040 18017 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:15.238263 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.238718 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.239267 18020 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:15.201848 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:15.092617 22405 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:23:15.204602 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.253010 18016 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:15.207494 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:15.253623 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:15.091670 22411 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:15.208343 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.097803 22408 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:15.096911 21934 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.088028 22273 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:15.087388 22335 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:15.075001 22424 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:15.158461 22414 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.242331 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.240969 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.310750 18023 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:15.235966 22355 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:15.026110 22294 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:15.254262 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.254264 18164 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:15.179603 22396 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.158766 21984 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:15.258664 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:15.188416 22363 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:15.048650 22296 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.090142 22270 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.134s
I20260709 12:23:15.095423 22317 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:15.106128 22237 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.117s
W20260709 12:23:15.272660 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.362186 22296 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:15.364154 22406 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.644s	user 0.000s	sys 0.004s
I20260709 12:23:15.273969 18022 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:23:15.365782 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.366468 18136 connection.cc:256] RPC call timeout handler was delayed by -1.0874230350000289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18380000000000002s for a response
I20260709 12:23:15.366586 22355 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.262850 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.179126 22372 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.085s
I20260709 12:23:15.369019 22372 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:15.369444 22372 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:15.222929 22308 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:15.369896 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:15.370651 22308 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.371029 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:15.371867 22308 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.306886 18166 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:15.373708 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.374164 22363 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:15.304097 22424 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.374755 22363 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:15.375068 22424 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:15.375891 22363 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.377641 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.378054 22380 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:15.378487 22380 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.253780 21709 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:15.378944 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:15.379236 21709 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:15.328296 18017 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:15.381875 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:15.383249 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:15.384076 21984 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.386770 21984 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:15.326758 18018 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:15.388588 22422 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:15.388648 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 549) took 1471 ms. Trace:
I20260709 12:23:15.325546 18015 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:15.330834 21934 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:15.389778 18024 rpcz_store.cc:276] 0709 12:23:13.917235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.917495 (+   260us) service_pool.cc:224] Handling call
0709 12:23:13.917895 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.977790 (+ 59895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.977905 (+   115us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.998657 (+ 20752us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.220576 (+221919us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:14.220622 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.220697 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:15.099595 (+878898us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:15.388198 (+288603us) raft_consensus.cc:1685] finished
0709 12:23:15.388226 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.388567 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4290,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":8940,"spinlock_wait_cycles":40692992,"thread_start_us":221435,"threads_started":1,"wal-append.queue_time_us":629139}
W20260709 12:23:15.342499 22321 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.006s
I20260709 12:23:15.358822 22375 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:15.367235 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:15.370527 22372 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.407977 22372 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:15.390369 22396 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.374504 18093 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:15.411056 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:15.378459 18097 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:23:15.411783 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 420801us
I20260709 12:23:15.379458 22186 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:15.234668 22306 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:15.245850 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91955.9us
I20260709 12:23:15.381712 21709 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:15.236402 22369 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:15.413803 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46675.5us
W20260709 12:23:15.418377 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.420310 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2117) took 1314 ms. Trace:
I20260709 12:23:15.447100 18171 rpcz_store.cc:276] 0709 12:23:14.106043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.702505 (+596462us) service_pool.cc:224] Handling call
0709 12:23:15.420033 (+717528us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540002add4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:15.420225 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338450176}
I20260709 12:23:15.424368 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.448886 18024 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:15.425088 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:15.426704 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:15.428853 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:15.428390 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 263856us
I20260709 12:23:15.378917 22380 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:15.451083 18806 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:15.452963 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.455869 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.464177 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.465354 18024 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:15.477979 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.479133 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:15.477665 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.491840 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:15.491984 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48880.2us
I20260709 12:23:15.492926 18094 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:15.494568 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.503219 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.498148 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.505899 18015 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:15.506587 18099 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:15.508682 18018 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:15.517848 18024 raft_consensus.cc:2368] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
W20260709 12:23:15.519441 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.510689 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.449725 22369 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:15.464838 21709 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:15.522807 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.516688 22413 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:15.521157 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.423772 22321 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.532209 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.504242 22375 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:15.529261 18017 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:15.502924 22186 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:15.535265 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:15.322947 22414 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.451184 22258 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.539965 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.540220 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.549860 18016 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:15.523629 21828 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:15.546761 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:15.554210 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.554965 18017 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:15.526542 22293 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.701s	user 0.015s	sys 0.000s
I20260709 12:23:15.523801 22296 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:15.561664 18807 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:15.561105 18023 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:15.538154 18018 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:15.542127 18024 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:15.563450 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.552919 18016 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 12:23:15.564309 18016 pending_rounds.cc:85] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 6
I20260709 12:23:15.564589 18016 pending_rounds.cc:107] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:23:15.564843 18016 raft_consensus.cc:2889] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:15.548784 18097 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:15.528652 21709 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.558141 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:15.570811 18806 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.568403 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:15.574280 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.575548 18022 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:23:15.576486 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:15.578383 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:15.552623 22413 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:15.580435 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:15.587109 18017 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:15.589335 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:15.553966 22396 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:15.561079 22296 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.595814 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:15.597445 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.577948 21709 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:15.570649 21828 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:15.578346 22402 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:15.593608 21977 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:15.584069 22289 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.719s	user 0.015s	sys 0.000s
I20260709 12:23:15.605232 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:15.608542 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.609293 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:15.564438 22428 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:15.575469 22352 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:15.617574 18806 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:15.586611 22334 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:15.632611 18097 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:15.592192 22413 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:15.629640 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.629966 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.605422 22296 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.638113 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:15.659284 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.068s (SENT) [suppressed 24 similar messages]
I20260709 12:23:15.607400 22331 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.607043 22237 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:15.661187 18090 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:15.586911 22309 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:23:15.619663 22402 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.602360 22270 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:23:15.671115 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:15.674118 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44274.3us
I20260709 12:23:15.675762 18015 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:15.648780 22322 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:15.631731 22365 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:15.659375 22413 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:15.679280 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.632275 22428 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:15.678158 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 146072us
I20260709 12:23:15.689004 18138 leader_election.cc:400] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:15.694494 18138 leader_election.cc:403] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:23:15.691721 18096 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:15.664117 22237 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:15.697021 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.675980 22402 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.696784 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.711134 21971 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:15.711628 21971 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.694949 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.717199 22407 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:15.717734 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.718041 22407 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:15.718129 22380 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:15.684101 21934 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:15.680609 22300 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:15.678428 22293 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:15.649498 22289 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:15.683216 22322 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.712431 21971 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.727737 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:15.694538 22306 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:15.707904 21977 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.699409 22237 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:15.724408 21934 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.697329 22428 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.688076 22334 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:23:15.696165 22443 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:15.742946 22293 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:15.726976 22300 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.760260 18091 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:15.761276 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.735731 22322 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.765715 22374 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:15.746416 21934 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.766579 22293 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:15.767972 22378 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:15.769086 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:15.741247 22186 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:15.736943 22432 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:15.770901 22300 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:15.773571 22435 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:23:15.807575 22432 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:15.812095 22434 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.223s	user 0.002s	sys 0.000s
W20260709 12:23:15.787237 22431 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260709 12:23:15.719676 22407 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:15.708909 21828 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:15.771327 22321 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:15.773622 22374 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.817858 22432 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:15.762637 22309 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 12:23:15.793666 22186 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:15.827899 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1927719489999618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
I20260709 12:23:15.820531 22429 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:15.886593 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.891814 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5006822699997429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20260709 12:23:15.890079 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:15.893137 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4619822699996803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
W20260709 12:23:15.893512 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4520822699996643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260709 12:23:15.893877 18098 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:15.807744 22414 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:15.893803 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4115822699995988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260709 12:23:15.914414 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2081822699997247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
W20260709 12:23:15.914783 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1307822699995995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20260709 12:23:15.915037 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0443822699994598s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
W20260709 12:23:15.868597 22448 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:23:15.891868 22374 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.917114 22186 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.911347 22452 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.938553 22312 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.288s	user 0.005s	sys 0.016s
W20260709 12:23:15.937666 22435 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:23:15.912890 22429 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:15.988917 22308 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.617s	user 0.014s	sys 0.000s
W20260709 12:23:15.950613 22449 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.196s	user 0.005s	sys 0.005s
W20260709 12:23:15.998754 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 108275us
I20260709 12:23:15.946403 22380 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:16.017123 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.000010 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 965) took 2306 ms. Trace:
I20260709 12:23:16.000236 18098 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:23:16.020732 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:16.021787 18098 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:23:16.018115 18808 rpcz_store.cc:276] 0709 12:23:13.719225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.719427 (+   202us) service_pool.cc:224] Handling call
0709 12:23:13.719909 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.124972 (+1405063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.999404 (+874432us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54006f2cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:15.999889 (+   485us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3110340480}
W20260709 12:23:16.025178 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.17203e+06us
I20260709 12:23:16.027261 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:16.030956 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.028048 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:15.984084 22312 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.341s	user 0.006s	sys 0.017s
W20260709 12:23:16.011114 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85487.6us
I20260709 12:23:16.039575 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.050073 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:15.997987 22429 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.076741 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.078164 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:16.087034 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 611) took 1837 ms. Trace:
I20260709 12:23:16.087908 18169 rpcz_store.cc:276] 0709 12:23:14.249271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.783512 (+534241us) service_pool.cc:224] Handling call
0709 12:23:14.784057 (+   545us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.116484 (+332427us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:15.116833 (+   349us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.453636 (+336803us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.456764 (+  3128us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.486522 (+ 29758us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:15.486601 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.486730 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.086585 (+599855us) raft_consensus.cc:1685] finished
0709 12:23:16.086626 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.086908 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":331734,"prepare.queue_time_us":1122,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":333467,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":8519,"thread_start_us":335634,"threads_started":1,"wal-append.queue_time_us":440003}
I20260709 12:23:16.024176 22452 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:16.091720 22444 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.303s	user 0.001s	sys 0.001s
I20260709 12:23:16.093524 22452 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.090305 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.071503 22352 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:16.104146 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:23:16.110100 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 396263us
I20260709 12:23:16.106933 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:16.114742 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:16.084282 22289 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.116379 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:16.111625 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:16.121248 22325 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:16.127439 22308 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.128042 22308 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:16.128465 22308 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:16.134984 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.141258 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:16.141345 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:16.125511 21977 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:16.142434 18171 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:16.152115 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.153223 18098 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:23:16.158172 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:16.138995 22414 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:16.149147 22304 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.962s	user 0.017s	sys 0.000s
I20260709 12:23:16.138234 22313 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:16.172036 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.175997 22454 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260709 12:23:16.178149 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.179363 22453 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:16.209136 18092 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260709 12:23:16.205616 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.206962 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:16.170235 22442 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.258s	user 0.000s	sys 0.001s
W20260709 12:23:16.200709 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.165894 22239 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:16.208801 18808 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:16.203648 22258 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.267464 22294 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:16.269959 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.270951 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.232391 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.272212 18020 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:16.170158 22450 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.185s	user 0.002s	sys 0.002s
I20260709 12:23:16.234364 18024 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:16.165287 22414 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.253736 18022 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:16.274652 22414 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.275292 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2193) took 1604 ms. Trace:
I20260709 12:23:16.275792 18091 rpcz_store.cc:276] 0709 12:23:14.670689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.689456 (+ 18767us) service_pool.cc:224] Handling call
0709 12:23:16.275054 (+1585598us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:16.275214 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2365488000}
I20260709 12:23:16.280153 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:16.281075 18098 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:16.283489 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:16.185946 22441 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:23:16.260706 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:16.206176 22408 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:16.185094 22313 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.262630 22370 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:16.263499 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2101) took 3435 ms. Trace:
I20260709 12:23:16.318185 18167 rpcz_store.cc:276] 0709 12:23:12.828091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.066970 (+238879us) service_pool.cc:224] Handling call
0709 12:23:13.067421 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.975438 (+1908017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.263153 (+1287715us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:16.263415 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4369555840}
I20260709 12:23:16.262681 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.320266 18093 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:16.265265 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 987) took 1130 ms. Trace:
W20260709 12:23:16.283489 22428 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.001s
I20260709 12:23:16.230786 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.328644 18017 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:16.328933 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.277393 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:16.285372 18172 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.049s	sys 0.057s
W20260709 12:23:16.295116 22446 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.291s	user 0.000s	sys 0.002s
I20260709 12:23:16.304487 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:16.309933 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:16.314085 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 634821us
I20260709 12:23:16.314354 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 989) took 1178 ms. Trace:
W20260709 12:23:16.315354 22372 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.907s	user 0.014s	sys 0.007s
I20260709 12:23:16.315630 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2977) took 1109 ms. Trace:
I20260709 12:23:16.320761 18807 rpcz_store.cc:276] 0709 12:23:15.135224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.136661 (+  1437us) service_pool.cc:224] Handling call
0709 12:23:16.264986 (+1128325us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54006f2054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:16.265151 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":862068224}
I20260709 12:23:16.270525 22368 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.325403 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.335043 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:16.327335 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:16.265358 21934 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:16.330276 18172 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.050s	sys 0.057s
W20260709 12:23:16.259262 22459 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:23:16.343248 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.351408 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 614) took 1200 ms. Trace:
I20260709 12:23:16.334796 18811 rpcz_store.cc:276] 0709 12:23:15.135931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.136992 (+  1061us) service_pool.cc:224] Handling call
0709 12:23:15.137307 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.142743 (+  5436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.142796 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.178201 (+ 35405us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.585542 (+407341us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:15.682781 (+ 97239us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.683080 (+   299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.683173 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.313954 (+630781us) raft_consensus.cc:1685] finished
0709 12:23:16.313994 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.314245 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4996,"prepare.queue_time_us":465711,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":24768,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":31263,"thread_start_us":422931,"threads_started":2,"wal-append.queue_time_us":722936}
I20260709 12:23:16.353613 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.304395 22352 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:23:16.345588 21971 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.595s	user 0.014s	sys 0.001s
I20260709 12:23:16.341425 22370 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:16.383607 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.384788 18024 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:16.372816 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 2974) took 1169 ms. Trace:
I20260709 12:23:16.367715 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.392735 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.375574 18811 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:16.394016 18093 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:16.395828 18018 raft_consensus.cc:2368] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:16.338580 22443 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:16.407114 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.408571 18022 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:16.343190 22333 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:16.401453 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 373489us
I20260709 12:23:16.411317 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.368494 22439 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.422290 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.372481 22258 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.389110 18021 rpcz_store.cc:276] 0709 12:23:15.203217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.203551 (+   334us) service_pool.cc:224] Handling call
0709 12:23:15.248433 (+ 44882us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.570775 (+322342us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:15.626136 (+ 55361us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.334709 (+708573us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:16.356118 (+ 21409us) log.cc:844] Serialized 26 byte log entry
0709 12:23:16.356343 (+   225us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:16.356391 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.356492 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.372505 (+ 16013us) raft_consensus.cc:1685] finished
0709 12:23:16.372535 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.372726 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":583537,"prepare.queue_time_us":85525,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":5068,"raft.queue_time_us":631571,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":14733,"spinlock_wait_cycles":19200,"thread_start_us":394157,"threads_started":2,"wal-append.queue_time_us":352877}
I20260709 12:23:16.363049 22352 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:16.372089 22368 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.321929 22408 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.340840 22428 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.436184 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.427757 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.443528 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 615) took 1292 ms. Trace:
I20260709 12:23:16.448050 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.450160 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2218) took 1259 ms. Trace:
I20260709 12:23:16.415273 22321 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:16.450582 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.444497 18165 rpcz_store.cc:276] 0709 12:23:15.151493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.249619 (+ 98126us) service_pool.cc:224] Handling call
0709 12:23:15.250124 (+   505us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:15.259936 (+  9812us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.259988 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:15.330349 (+ 70361us) log.cc:844] Serialized 52 byte log entry
0709 12:23:15.582381 (+252032us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:15.582447 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.582552 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.132389 (+549837us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:16.433198 (+300809us) raft_consensus.cc:1685] finished
0709 12:23:16.433240 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.443402 (+ 10162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6015,"raft.queue_time_us":129,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":16457,"spinlock_wait_cycles":23441792,"thread_start_us":138220,"threads_started":1,"wal-append.queue_time_us":541377}
I20260709 12:23:16.461844 18090 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:16.435497 22362 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.414887 22270 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:16.451043 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 86401.9us
I20260709 12:23:16.438210 22352 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:16.473973 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:16.476255 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.429217 22258 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:16.449016 22399 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:16.441478 22439 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.482807 18021 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:16.449307 22299 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:16.398003 21971 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.404165 22370 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.477806 22362 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:16.485728 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48031.9us
I20260709 12:23:16.424742 22304 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
W20260709 12:23:16.498199 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.511050 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:16.466491 22296 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.799s	user 0.018s	sys 0.000s
I20260709 12:23:16.425978 22333 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.480484 22270 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:16.509598 18098 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:16.492336 22258 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:16.508384 22237 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.753s	user 0.001s	sys 0.015s
I20260709 12:23:16.465586 22475 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:16.494745 22400 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:16.507771 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.485903 22471 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:16.511790 21709 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.884s	user 0.015s	sys 0.002s
W20260709 12:23:16.529585 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.08294e+06us
I20260709 12:23:16.500730 22439 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:16.535753 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2112) took 2462 ms (client timeout 1828 ms). Trace:
W20260709 12:23:16.548393 18168 rpcz_store.cc:269] 0709 12:23:14.095205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.112225 (+ 17020us) service_pool.cc:224] Handling call
0709 12:23:16.535392 (+2423167us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:16.535626 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1519379712}
W20260709 12:23:16.518430 22381 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.623s	user 0.011s	sys 0.010s
I20260709 12:23:16.529688 22304 consensus_queue.cc:1243] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:16.511431 22362 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.537626 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:16.546241 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.563129 18022 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
W20260709 12:23:16.521737 22413 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.801s	user 0.003s	sys 0.015s
I20260709 12:23:16.555533 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.542299 22299 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:16.514850 22470 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
W20260709 12:23:16.557344 22350 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.359s	user 0.015s	sys 0.005s
I20260709 12:23:16.572698 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.558898 22471 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.577039 22469 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:23:16.577744 22445 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.838s	user 0.001s	sys 0.003s
I20260709 12:23:16.511948 22443 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:16.581996 22433 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.894s	user 0.005s	sys 0.001s
I20260709 12:23:16.582731 22408 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:16.584769 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:16.659083 22299 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:16.521636 22296 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.537918 22237 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:16.586813 22402 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.876s	user 0.018s	sys 0.000s
I20260709 12:23:16.619367 22370 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:16.620656 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.621358 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:16.635749 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.636169 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.564988 22381 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:16.643237 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157994us
W20260709 12:23:16.581459 22429 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.003s
W20260709 12:23:16.646723 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 311769us
I20260709 12:23:16.679443 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:16.677968 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:16.565984 22270 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.551275 21709 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.574419 22400 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.575906 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:16.695622 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:16.697134 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123355us
I20260709 12:23:16.667575 18811 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:16.658792 22424 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.275s	user 0.017s	sys 0.003s
I20260709 12:23:16.647332 21934 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.671463 18023 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
W20260709 12:23:16.685670 22322 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.898s	user 0.015s	sys 0.003s
I20260709 12:23:16.662256 22395 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.599648 22413 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:16.647276 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74751us
I20260709 12:23:16.714073 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:16.664350 22313 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:16.700856 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.703545 18809 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:16.697154 18017 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:16.703560 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:16.699162 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:16.707105 18096 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:16.707764 18815 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:16.711382 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:16.723742 18806 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:16.716948 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:16.684557 22428 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.727964 18023 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:16.728309 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:16.729379 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:16.729734 22378 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:16.734681 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:16.735854 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.726504 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 610) took 2653 ms. Trace:
I20260709 12:23:16.739145 18170 rpcz_store.cc:276] 0709 12:23:14.073013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.763530 (+690517us) service_pool.cc:224] Handling call
0709 12:23:14.763979 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.118115 (+354136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.118167 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.118898 (+   731us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.534158 (+415260us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:15.534225 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.534330 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.577026 (+1042696us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:16.712900 (+135874us) raft_consensus.cc:1685] finished
0709 12:23:16.712941 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.726375 (+ 13434us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":353635,"raft.queue_time_us":353860,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":8628608,"thread_start_us":414674,"threads_started":1,"wal-append.queue_time_us":559538}
W20260709 12:23:16.711853 22473 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 12:23:16.744128 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:16.756592 18023 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:16.745481 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:16.753562 18099 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:16.735186 22296 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.701088 22400 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:16.766294 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.771934 18098 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:16.761334 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2219) took 1187 ms. Trace:
I20260709 12:23:16.781872 18095 rpcz_store.cc:276] 0709 12:23:15.574236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.759958 (+1185722us) service_pool.cc:224] Handling call
0709 12:23:16.760518 (+   560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.760832 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.760897 (+    65us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:16.760966 (+    69us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:16.761012 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.761070 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.761234 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:16.774197 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 111419us
I20260709 12:23:16.784703 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:16.774948 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3068) took 1409 ms. Trace:
I20260709 12:23:16.706703 22461 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:16.695779 22299 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:23:16.787580 18099 rpcz_store.cc:276] 0709 12:23:15.365658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.736015 (+1370357us) service_pool.cc:224] Handling call
0709 12:23:16.753049 (+ 17034us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.774511 (+ 21462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.774854 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:16.788691 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2220) took 1207 ms. Trace:
I20260709 12:23:16.675535 22402 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:16.792794 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47426.3us
I20260709 12:23:16.797644 22486 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:16.727538 22469 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.737872 22223 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:16.712313 22372 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 12:23:16.729408 22481 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:16.820634 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 133762us
I20260709 12:23:16.805577 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:16.692288 22429 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:16.793143 22186 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.786s	user 0.010s	sys 0.006s
I20260709 12:23:16.803973 18095 rpcz_store.cc:276] 0709 12:23:15.581363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.786172 (+1204809us) service_pool.cc:224] Handling call
0709 12:23:16.786666 (+   494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.786962 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.787003 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:16.787071 (+    68us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:16.787135 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.787219 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.788607 (+  1388us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:16.802168 22492 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:16.703647 22478 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.111s
I20260709 12:23:16.744716 22378 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:16.743456 21971 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:16.709983 22424 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.831840 18022 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:16.792017 22400 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.719157 22322 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.840873 22196 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:16.847596 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3063) took 1483 ms. Trace:
I20260709 12:23:16.882107 18096 rpcz_store.cc:276] 0709 12:23:15.364382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.703158 (+338776us) service_pool.cc:224] Handling call
0709 12:23:15.709116 (+  5958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.722922 (+1013806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.847182 (+124260us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54001afc54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:16.847498 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2294670080}
I20260709 12:23:16.849692 22392 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:16.851217 22391 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:16.847627 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 139473us
I20260709 12:23:16.859077 22333 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:16.864653 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59145.8us
I20260709 12:23:16.900027 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:16.834868 22374 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.866s	user 0.017s	sys 0.000s
I20260709 12:23:16.724220 22395 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:16.824398 22223 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.879885 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:16.860997 22481 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.809247 22310 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:16.887791 22294 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.872031 22378 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:16.885073 22300 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.056s	user 0.011s	sys 0.005s
I20260709 12:23:16.865579 22186 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.887342 22392 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:16.896034 22304 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.893322 22296 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:16.907083 22369 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.361s	user 0.005s	sys 0.014s
I20260709 12:23:16.828564 22469 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.905128 22471 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.892563 22485 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:16.917658 22432 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.989s	user 0.013s	sys 0.005s
I20260709 12:23:16.918059 22299 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:23:16.917009 22481 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:16.919530 22310 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:16.974475 22372 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.917614 22492 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:16.969703 22495 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:17.030819 22310 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:16.969753 22477 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.305s	user 0.001s	sys 0.001s
I20260709 12:23:16.906777 22374 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:16.908540 22475 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 12:23:16.908416 22333 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:17.021703 22485 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:17.031316 22476 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.318s	user 0.000s	sys 0.002s
I20260709 12:23:17.032831 22492 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:17.035214 22495 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:17.024397 22381 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.019531 22369 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.039284 22413 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.058471 22395 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:17.051036 22487 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.236s	user 0.002s	sys 0.000s
I20260709 12:23:17.026988 22432 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.050061 22485 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:17.054674 22482 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.156s
I20260709 12:23:17.055246 22495 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.066932 22492 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.069576 22499 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
W20260709 12:23:16.867571 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124519us
I20260709 12:23:16.859333 18015 raft_consensus.cc:2368] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:23:16.662978 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:16.337371 18019 rpcz_store.cc:276] 0709 12:23:15.206399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.207147 (+   748us) service_pool.cc:224] Handling call
0709 12:23:15.207503 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.253703 (+ 46200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.253740 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.264451 (+ 10711us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.599041 (+334590us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:15.726133 (+127092us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.315078 (+588945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.315223 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.315300 (+    77us) raft_consensus.cc:1685] finished
0709 12:23:16.315322 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.315534 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":617019,"prepare.queue_time_us":68370,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":84,"raft.queue_time_us":666629,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":14796,"thread_start_us":279324,"threads_started":2,"wal-append.queue_time_us":766708}
I20260709 12:23:16.453269 18097 rpcz_store.cc:276] 0709 12:23:15.190350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.633671 (+443321us) service_pool.cc:224] Handling call
0709 12:23:15.670462 (+ 36791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.670716 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:15.692037 (+ 21321us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.308109 (+616072us) spinlock_profiling.cc:243] Waited 4.64 ms on lock 0x7b1800105280. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:23:16.449658 (+141549us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:16.449728 (+    70us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:16.449785 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.449898 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.450074 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21096,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":14315,"raft.queue_time_us":2511,"spinlock_wait_cycles":10667776,"thread_start_us":141162,"threads_started":1}
I20260709 12:23:16.357182 18172 rpcz_store.cc:276] 0709 12:23:15.151191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.248551 (+ 97360us) service_pool.cc:224] Handling call
0709 12:23:15.248975 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.249247 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:15.272030 (+ 22783us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.757204 (+485174us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b18003c1a80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:23:16.350759 (+593555us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:16.350867 (+   108us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:16.350952 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.351067 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.351305 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22380,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":3614,"raft.queue_time_us":7601,"spinlock_wait_cycles":10122624,"thread_start_us":592954,"threads_started":1}
W20260709 12:23:16.993140 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.967612 21709 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:16.915453 22223 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:17.196211 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:17.359143 18094 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:17.307145 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:17.121601 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 336756us
I20260709 12:23:16.978626 22304 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.364403 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:17.338016 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:17.367053 22490 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.367524 22365 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:17.368404 22490 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.051007 22392 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.181524 18023 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:17.127604 18016 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:17.169399 18015 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:23:17.382845 22392 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:16.933751 22300 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.180495 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:17.205295 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2213) took 2198 ms (client timeout 1768 ms). Trace:
I20260709 12:23:17.199326 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:17.386363 18093 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:17.388120 22414 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.113s	user 0.015s	sys 0.003s
I20260709 12:23:17.185707 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3059) took 1822 ms. Trace:
I20260709 12:23:17.389992 18090 rpcz_store.cc:276] 0709 12:23:15.363282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.662443 (+299161us) service_pool.cc:224] Handling call
0709 12:23:15.678268 (+ 15825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.462469 (+784201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.184743 (+722274us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b5400280354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:17.185356 (+   613us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1801466240}
I20260709 12:23:16.983619 22237 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.392534 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:17.394209 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.273679 18811 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:23:17.397107 22424 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:16.928952 22294 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.399757 22471 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.198248 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:17.402349 22475 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:17.385113 18092 rpcz_store.cc:269] 0709 12:23:15.006459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.057106 (+ 50647us) service_pool.cc:224] Handling call
0709 12:23:17.204545 (+2147439us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540035e8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:17.204938 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2626629504}
I20260709 12:23:17.403604 22475 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.404482 22322 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.405115 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:17.407546 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:17.410727 22475 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.261215 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:17.071000 22429 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:17.291509 17918 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 178) took 33422 ms (client timeout 15000 ms). Trace:
I20260709 12:23:17.037559 22372 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.377910 22374 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:17.073522 22500 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:23:17.378937 22196 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:17.203886 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:17.218997 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3066) took 1853 ms. Trace:
I20260709 12:23:17.305191 18809 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:17.055197 22402 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.402532 18097 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:17.347823 22308 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.219s	user 0.010s	sys 0.006s
I20260709 12:23:17.374994 22483 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.365746 18812 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:17.413291 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 152557us
I20260709 12:23:17.414002 22313 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:17.125322 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:17.418531 22294 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.418977 21709 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.419191 22498 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.410389 18092 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:23:17.419844 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 55548.1us
I20260709 12:23:17.420950 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:17.073676 22482 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:17.365789 22258 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.828s	user 0.014s	sys 0.000s
I20260709 12:23:17.368923 22365 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.353384 22469 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.160095 22504 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:16.983014 22296 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.479303 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:17.446586 18024 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:17.434065 18098 rpcz_store.cc:276] 0709 12:23:15.365161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.518562 (+1153401us) service_pool.cc:224] Handling call
0709 12:23:16.519400 (+   838us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.781746 (+262346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.218283 (+436537us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54004119d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:17.218742 (+   459us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580035456}
I20260709 12:23:17.389133 22414 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.078490 22492 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:17.401181 22471 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.419878 22381 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.462801 22196 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:17.489388 18095 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.007s	sys 0.015s
I20260709 12:23:17.488911 21709 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.500954 22482 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:17.501979 22294 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:17.503031 22294 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.506098 22402 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:17.506879 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.507771 22369 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.508939 22381 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.510150 22486 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:17.510824 22486 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:17.510957 22372 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.512120 22186 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.493153 22258 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.513938 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2222) took 1930 ms. Trace:
I20260709 12:23:17.515565 18810 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:17.506258 22304 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.495275 18091 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:17.506129 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 381474us
W20260709 12:23:17.520460 18095 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.007s	sys 0.016s
I20260709 12:23:17.508965 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:17.492139 22365 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.514591 18096 rpcz_store.cc:276] 0709 12:23:15.583545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.883605 (+1300060us) service_pool.cc:224] Handling call
0709 12:23:16.887806 (+  4201us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.888093 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.888124 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:16.888685 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.186951 (+298266us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:17.187093 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.187285 (+   192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.513287 (+326002us) raft_consensus.cc:1685] finished
0709 12:23:17.513328 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.513640 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":297419,"threads_started":1,"wal-append.queue_time_us":84769}
I20260709 12:23:17.532117 18090 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:17.504626 22504 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:17.548722 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.531631 22490 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.479475 22308 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:17.559470 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64067.2us
I20260709 12:23:17.559649 22296 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.560537 21709 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:17.415171 17918 rpcz_store.cc:269] 0709 12:22:43.869037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.873226 (+  4189us) service_pool.cc:224] Handling call
0709 12:23:17.291062 (+33417836us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:15.055677 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:15.096206 (+ 40529us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:15.096248 (+    42us) write_op.cc:435] Acquired schema lock
0709 12:23:15.096269 (+    21us) tablet.cc:662] Decoding operations
0709 12:23:15.096676 (+   407us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:15.096749 (+    73us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:15.096776 (+    27us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:23:15.097012 (+   236us) tablet.cc:701] Row locks acquired
0709 12:23:15.097032 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:23:15.097270 (+   238us) write_op.cc:270] Start()
0709 12:23:15.097431 (+   161us) write_op.cc:276] Timestamp: P: 1783599795097195 usec, L: 0
0709 12:23:15.097462 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:23:15.098520 (+  1058us) log.cc:844] Serialized 533 byte log entry
0709 12:23:16.311105 (+1212585us) op_driver.cc:464] REPLICATION: finished
0709 12:23:16.462087 (+150982us) write_op.cc:301] APPLY: starting
0709 12:23:16.472399 (+ 10312us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:16.605509 (+133110us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:16.605690 (+   181us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:16.607133 (+  1443us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:16.607482 (+   349us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=2,mrs_lookups=0
0709 12:23:16.607543 (+    61us) write_op.cc:312] APPLY: finished
0709 12:23:16.607831 (+   288us) log.cc:844] Serialized 25 byte log entry
0709 12:23:16.609889 (+  2058us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:16.610251 (+   362us) write_op.cc:454] Released schema lock
0709 12:23:16.610321 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:16.612940 (+  2619us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:16.612995 (+    55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":596187,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":115778,"apply.run_cpu_time_us":7007,"apply.run_wall_time_us":157841,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8313,"num_ops":1,"prepare.queue_time_us":243461,"prepare.run_cpu_time_us":17202,"prepare.run_wall_time_us":416185,"raft.queue_time_us":344097,"raft.run_cpu_time_us":4622,"raft.run_wall_time_us":159286,"replication_time_us":1186390,"thread_start_us":742140,"threads_started":4,"wal-append.queue_time_us":437299}]]}
I20260709 12:23:17.512683 22486 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.497877 22498 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:17.593289 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57897.5us
I20260709 12:23:17.549971 22258 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.554845 22372 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:17.553659 22482 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.610450 22482 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.639088 18091 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:17.554517 22483 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.643369 18098 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:17.645460 18093 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:17.653065 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:17.655812 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 693) took 1911 ms. Trace:
I20260709 12:23:17.656366 18098 rpcz_store.cc:276] 0709 12:23:15.739894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.537080 (+1797186us) service_pool.cc:224] Handling call
0709 12:23:17.607999 (+ 70919us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.651285 (+ 43286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.651556 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:17.662684 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:17.666883 22471 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.667989 22471 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:17.640232 22480 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:17.665378 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:17.690253 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53975.3us
I20260709 12:23:17.657634 22381 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.622763 22429 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.169s
W20260709 12:23:17.701967 22331 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.993s	user 0.024s	sys 0.001s
W20260709 12:23:17.702432 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67537.6us
I20260709 12:23:17.702735 22331 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:17.703848 21709 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:17.707064 22486 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:17.682687 22480 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:17.712054 22484 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:17.713121 22484 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.712039 22414 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.717801 18023 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:17.713140 22308 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.703918 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:17.729146 18022 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:17.708412 22379 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:17.724787 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2277) took 1398 ms. Trace:
W20260709 12:23:17.733319 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.682049 22372 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.731163 18016 rpcz_store.cc:276] 0709 12:23:16.323759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.323922 (+   163us) service_pool.cc:224] Handling call
0709 12:23:17.721420 (+1397498us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:17.721688 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1832371584}
I20260709 12:23:17.716219 22294 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.730523 22379 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:17.737497 22368 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.281s	user 0.008s	sys 0.016s
I20260709 12:23:17.748005 18097 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:17.748533 18098 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:23:17.763422 22414 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:23:17.764438 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.751303 22504 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:17.780902 22485 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.709s	user 0.014s	sys 0.000s
I20260709 12:23:17.735788 21709 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:17.798380 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.782583 18093 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:17.786935 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:17.749749 22239 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.464s	user 0.026s	sys 0.004s
I20260709 12:23:17.807063 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 707) took 1547 ms. Trace:
I20260709 12:23:17.810880 18093 rpcz_store.cc:276] 0709 12:23:16.260160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.672586 (+1412426us) service_pool.cc:224] Handling call
0709 12:23:17.739870 (+ 67284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.806735 (+ 66865us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.806991 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:17.816676 18091 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:17.817286 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:17.818111 18097 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.819171 18094 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:17.782706 22498 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.787868 22483 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.722970 22480 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.828877 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:17.821508 22439 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.194s	user 0.011s	sys 0.011s
W20260709 12:23:17.831903 22481 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.799s	user 0.011s	sys 0.012s
I20260709 12:23:17.832285 18813 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:23:17.833678 22282 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:17.833886 21709 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:17.827612 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 111762us
I20260709 12:23:17.837396 18098 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:17.781428 22379 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:17.839197 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 135444us
W20260709 12:23:17.840642 22495 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.766s	user 0.011s	sys 0.005s
I20260709 12:23:17.840950 22090 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.841506 22372 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.838606 22282 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.843165 22375 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:17.844297 21709 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:17.844846 18098 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 12:23:17.845747 22484 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:17.846374 22480 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:17.847937 22478 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:17.848410 22325 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.850291 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 690) took 2111 ms. Trace:
I20260709 12:23:17.847973 22481 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.851617 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:17.855199 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.855564 18093 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:23:17.857352 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.858186 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.859800 22300 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:17.862217 22439 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:17.850299 18172 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:17.863004 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.878162 18098 pending_rounds.cc:85] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 3
I20260709 12:23:17.814755 22485 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.860167 18169 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:17.905972 18094 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:17.876191 22282 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:17.896641 18092 rpcz_store.cc:276] 0709 12:23:15.738811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.485541 (+1746730us) service_pool.cc:224] Handling call
0709 12:23:17.501111 (+ 15570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.502006 (+   895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.502051 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.502987 (+   936us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.637341 (+134354us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:17.637409 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.637499 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.849927 (+212428us) raft_consensus.cc:1685] finished
0709 12:23:17.849960 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.850164 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6828,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":161,"thread_start_us":122569,"threads_started":1,"wal-append.queue_time_us":227665}
I20260709 12:23:17.910705 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:17.912814 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:17.878032 22090 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.893555 22375 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.913769 18171 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.916654 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2221) took 2334 ms. Trace:
I20260709 12:23:17.903318 18098 pending_rounds.cc:107] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:23:17.925168 18098 raft_consensus.cc:2889] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:17.923529 18808 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:17.894137 22478 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:17.931900 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:17.937371 18168 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:17.878758 22495 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:17.950637 18806 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:17.917083 22378 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.949s	user 0.016s	sys 0.003s
I20260709 12:23:17.958423 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:17.971323 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77552.5us
I20260709 12:23:17.973654 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.926137 18095 rpcz_store.cc:276] 0709 12:23:15.582134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.860649 (+1278515us) service_pool.cc:224] Handling call
0709 12:23:16.861114 (+   465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.861636 (+   522us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:17.549431 (+687795us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.678122 (+128691us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.782105 (+103983us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.782399 (+   294us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:17.782457 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.790294 (+  7837us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.916351 (+126057us) raft_consensus.cc:1685] finished
0709 12:23:17.916377 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.916573 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2806,"prepare.queue_time_us":987752,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":7169,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":6753,"thread_start_us":719765,"threads_started":2,"wal-append.queue_time_us":222017}
I20260709 12:23:17.976461 18021 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:17.980482 18097 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:17.983040 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:17.984563 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.976315 22443 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s	user 0.024s	sys 0.008s
I20260709 12:23:17.985248 18092 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:17.991641 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.985795 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:17.914008 22423 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:23:18.002976 18093 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:17.942203 22325 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:17.994706 22443 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.024s	sys 0.009s
I20260709 12:23:18.008529 18814 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 11. Ops: []
W20260709 12:23:18.014099 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 112795us
W20260709 12:23:18.019744 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100790us
I20260709 12:23:18.021220 18092 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:23:17.996833 22484 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:17.974285 22378 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:18.010030 22362 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.435s	user 0.007s	sys 0.020s
I20260709 12:23:17.999157 22495 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:18.008664 22310 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.885s	user 0.008s	sys 0.013s
I20260709 12:23:18.008273 22488 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.008667 22461 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.183s	user 0.014s	sys 0.010s
I20260709 12:23:18.032635 18097 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:23:18.042563 18098 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:23:18.044818 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.029183 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.048063 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:18.052742 22400 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.163s	user 0.009s	sys 0.012s
I20260709 12:23:18.054292 22270 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:18.057761 18810 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:18.062886 22439 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.059983 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:18.066362 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.066527 18809 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.066802 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.065506 22481 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.079604 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:18.080658 18092 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:18.082648 22310 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:18.084306 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2287) took 1529 ms (client timeout 1794 ms). Trace:
W20260709 12:23:18.084923 18024 rpcz_store.cc:269] 0709 12:23:16.554991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.555199 (+   208us) service_pool.cc:224] Handling call
0709 12:23:18.083923 (+1528724us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:18.084212 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1944712064}
I20260709 12:23:18.090416 18172 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:18.043274 18098 pending_rounds.cc:85] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:18.091740 18098 pending_rounds.cc:107] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:23:18.092083 18098 raft_consensus.cc:2889] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:18.024713 22484 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:18.106283 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56972.3us
I20260709 12:23:18.059955 22378 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:18.053427 22400 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.148270 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.105204 22484 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:18.110387 22310 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:18.170527 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.111732 22461 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:18.182456 18019 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:18.065470 17922 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 7, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.228984 18168 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:18.229610 18815 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260709 12:23:18.237540 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.239511 22381 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:18.239794 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.241092 22381 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:18.229269 17922 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:18.241518 22381 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:18.184541 22400 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.185123 22439 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.186200 18015 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:18.186940 21934 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:18.187737 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 863) took 1488 ms. Trace:
I20260709 12:23:18.094669 22488 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:18.189086 22482 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.578s	user 0.004s	sys 0.012s
I20260709 12:23:18.088376 22362 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.189781 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.189803 18024 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:18.181527 22507 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.625s	user 0.003s	sys 0.000s
I20260709 12:23:18.111032 22478 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:18.229007 22512 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.544s	user 0.000s	sys 0.002s
I20260709 12:23:18.241765 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.191172 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:18.243356 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68059us
W20260709 12:23:18.193092 17989 connection.cc:256] RPC call timeout handler was delayed by -2.5327211030003127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260709 12:23:18.269038 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.195371 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.202371 22485 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:18.270496 18023 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
W20260709 12:23:18.190352 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.271525 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:18.272444 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.272285 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:18.273046 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.273134 18023 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
W20260709 12:23:18.204371 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 861) took 1509 ms (client timeout 1725 ms). Trace:
I20260709 12:23:18.208851 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:18.274108 22352 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.774s	user 0.006s	sys 0.022s
I20260709 12:23:18.274468 18097 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:18.209460 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3001) took 1054 ms. Trace:
I20260709 12:23:18.274638 22352 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.209880 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:18.277045 18807 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:18.212020 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 706) took 2004 ms. Trace:
I20260709 12:23:18.278079 18091 rpcz_store.cc:276] 0709 12:23:16.207551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.655095 (+1447544us) service_pool.cc:224] Handling call
0709 12:23:17.701391 (+ 46296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.817252 (+115861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.211744 (+394492us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:18.211943 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211801472}
I20260709 12:23:18.279151 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 860) took 1585 ms. Trace:
I20260709 12:23:18.212852 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3007) took 1051 ms. Trace:
I20260709 12:23:18.280519 18022 rpcz_store.cc:276] 0709 12:23:17.161785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.555061 (+393276us) service_pool.cc:224] Handling call
0709 12:23:17.723812 (+168751us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.729913 (+  6101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.212652 (+482739us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54001cfa50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:18.212784 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373155072}
I20260709 12:23:18.211534 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:18.219096 22313 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.756s	user 0.014s	sys 0.002s
I20260709 12:23:18.282456 22313 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:18.194451 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.269517 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:18.216162 22090 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:18.224534 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:18.226487 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:18.220611 22282 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.241961 22423 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:23:18.190639 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.286665 18811 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:18.286777 18165 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:18.288203 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.244252 17922 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 1 to 8, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 8 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.245401 22381 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.204608 22432 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:18.266671 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 686) took 2529 ms. Trace:
I20260709 12:23:18.279987 22325 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:18.267987 18164 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:18.291498 22381 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.247258 22439 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.292753 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.290599 18173 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:18.159442 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:18.247200 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.298113 18812 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:18.298473 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 155420us
I20260709 12:23:18.299450 22370 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:18.299630 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:18.299991 22370 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.300209 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.249002 21934 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.300462 22370 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:18.303985 21934 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.249712 18814 rpcz_store.cc:276] 0709 12:23:16.699185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.700011 (+   826us) service_pool.cc:224] Handling call
0709 12:23:16.738422 (+ 38411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.031724 (+1293302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.187342 (+155618us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54004b46d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:18.187632 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2722912000}
I20260709 12:23:18.250403 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:18.309653 17922 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.310833 18020 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:18.311020 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:18.275384 18021 rpcz_store.cc:276] 0709 12:23:17.155399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.155644 (+   245us) service_pool.cc:224] Handling call
0709 12:23:17.156433 (+   789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.977035 (+820602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.209123 (+232088us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:18.209351 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1793118464}
I20260709 12:23:18.289592 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:18.213622 22486 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.005s
I20260709 12:23:18.293865 18164 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:18.282960 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:18.261631 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.357601 18090 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:18.312192 17922 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 8, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 8 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.358314 18024 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:18.313050 22223 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.958s	user 0.017s	sys 0.003s
I20260709 12:23:18.265087 22478 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:18.321370 22439 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.248570 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.360121 17922 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 8, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 8 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:23:18.361208 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.361411 18015 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:18.359660 22223 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.246318 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.362203 18164 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:18.362653 18018 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:18.363166 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.363960 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3004) took 1207 ms. Trace:
I20260709 12:23:18.285154 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.365258 18018 rpcz_store.cc:276] 0709 12:23:17.156593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.160696 (+  4103us) service_pool.cc:224] Handling call
0709 12:23:17.161315 (+   619us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.363081 (+1201766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.363711 (+   630us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540004c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:18.363904 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2761844992}
W20260709 12:23:18.274179 18808 rpcz_store.cc:269] 0709 12:23:16.695105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.695263 (+   158us) service_pool.cc:224] Handling call
0709 12:23:18.204007 (+1508744us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540070f8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:18.204255 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2445308672}
I20260709 12:23:18.366902 18016 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:18.367084 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.368099 18806 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:18.368208 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.369938 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.247669 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 977) took 4250 ms. Trace:
I20260709 12:23:18.347640 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:18.345419 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.330606 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:18.371517 18814 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:18.371876 18095 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.372159 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.253218 22488 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.253696 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:18.252920 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3002) took 1097 ms. Trace:
I20260709 12:23:18.255954 22362 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.374465 18019 rpcz_store.cc:276] 0709 12:23:17.155848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.169555 (+ 13707us) service_pool.cc:224] Handling call
0709 12:23:18.086418 (+916863us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.227392 (+140974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.252673 (+ 25281us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b5400251050. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:18.252836 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2106917760}
I20260709 12:23:18.378710 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.375119 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.374467 18098 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:23:18.374466 18165 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:18.381202 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.381739 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.383255 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.252437 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:18.291127 18099 rpcz_store.cc:276] 0709 12:23:15.736760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.915073 (+1178313us) service_pool.cc:224] Handling call
0709 12:23:17.187242 (+272169us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.188234 (+   992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.188273 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.189169 (+   896us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.343061 (+153892us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:17.343109 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.343206 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.180539 (+837333us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:18.266017 (+ 85478us) raft_consensus.cc:1685] finished
0709 12:23:18.266043 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.266458 (+   415us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155006,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":359,"spinlock_wait_cycles":43314432,"thread_start_us":153266,"threads_started":1,"wal-append.queue_time_us":260295}
I20260709 12:23:18.365684 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.351220 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.318081 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.324889 18809 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:18.384999 18022 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.363807 18167 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:18.377342 18813 rpcz_store.cc:276] 0709 12:23:13.997350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.997483 (+   133us) service_pool.cc:224] Handling call
0709 12:23:13.999543 (+  2060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.840032 (+3840489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.247304 (+407272us) spinlock_profiling.cc:243] Waited 3.82 s on lock 0x7b5400616054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:18.247565 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8787021056}
I20260709 12:23:18.382180 18173 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:18.258018 18017 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:18.279712 18807 rpcz_store.cc:276] 0709 12:23:16.693743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.694008 (+   265us) service_pool.cc:224] Handling call
0709 12:23:16.694492 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.277740 (+1583248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.278723 (+   983us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54005e01d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:18.279065 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3638332800}
I20260709 12:23:18.364398 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:18.384132 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112785us
I20260709 12:23:18.380012 18019 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.291760 22282 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:18.388661 18092 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:18.388556 22496 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:18.387758 22325 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:23:18.373476 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.244131 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.395607 18811 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:18.386976 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.370421 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.397346 18016 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
W20260709 12:23:18.312242 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2289) took 1677 ms (client timeout 2226 ms). Trace:
W20260709 12:23:18.252122 22516 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
I20260709 12:23:18.402037 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:18.398311 18020 rpcz_store.cc:269] 0709 12:23:16.635037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.635247 (+   210us) service_pool.cc:224] Handling call
0709 12:23:18.311980 (+1676733us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54001095d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:18.312177 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3769845120}
W20260709 12:23:18.404501 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105035us
I20260709 12:23:18.406876 18172 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:18.398200 22485 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:18.410529 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.404675 18171 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:18.412225 22432 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:18.413028 22432 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:18.415587 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.416685 18172 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:18.383711 22352 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.415776 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3008) took 1253 ms. Trace:
I20260709 12:23:18.420764 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.419862 18017 rpcz_store.cc:276] 0709 12:23:17.162965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.179652 (+ 16687us) service_pool.cc:224] Handling call
0709 12:23:18.415537 (+1235885us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b54001fc554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:18.415707 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2007383296}
I20260709 12:23:18.423629 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.425966 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.426357 18023 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.427629 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.429243 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.431625 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 695) took 2691 ms. Trace:
I20260709 12:23:18.414273 22496 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:18.432487 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 163090us
I20260709 12:23:18.433130 22496 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.432183 18096 rpcz_store.cc:276] 0709 12:23:15.740466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.598901 (+1858435us) service_pool.cc:224] Handling call
0709 12:23:17.670245 (+ 71344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:17.720076 (+ 49831us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.720126 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:17.721136 (+  1010us) log.cc:844] Serialized 52 byte log entry
0709 12:23:17.813006 (+ 91870us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:17.927081 (+114075us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.928395 (+  1314us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.928507 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.251254 (+322747us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:18.431190 (+179936us) raft_consensus.cc:1685] finished
0709 12:23:18.431340 (+   150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.431544 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":368,"prepare.queue_time_us":39265,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":11599,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":12466,"spinlock_wait_cycles":12534656,"thread_start_us":171456,"threads_started":2,"wal-append.queue_time_us":178077}
I20260709 12:23:18.436550 22432 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.439113 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.440248 18173 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:18.440387 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.418836 22300 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.441444 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.442265 18173 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:18.427057 18169 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:18.441303 22300 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.454262 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.455310 18164 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:18.454876 17922 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 11, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 11 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } }
W20260709 12:23:18.461141 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74348.2us
I20260709 12:23:18.453760 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.467448 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.470618 18016 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:18.473924 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.478066 18169 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:18.482384 18021 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:18.456763 22381 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:18.477296 17922 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.489106 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.491245 18168 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:18.496884 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:18.503497 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 56012.2us
W20260709 12:23:18.503576 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.475553 22400 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:18.469336 22484 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.503261 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:18.474993 22485 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:18.510540 18813 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:18.512528 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:18.534009 18024 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:18.481809 22423 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:18.512656 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:18.476555 22490 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.007s	sys 0.016s
I20260709 12:23:18.547405 18167 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.520364 18168 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:18.523011 18016 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:23:18.536557 18021 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:18.544104 18812 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:18.543184 18172 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:18.531774 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69484.4us
W20260709 12:23:18.544306 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59105.4us
I20260709 12:23:18.545233 18810 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:18.532006 18807 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9. Re-sending same reply.
I20260709 12:23:18.557214 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:18.551432 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:18.566900 22484 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.554418 22270 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:18.568635 18808 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:18.573094 18169 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:18.573978 18024 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:18.583369 22439 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:18.584896 22439 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:18.565249 17922 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 4 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:18.591274 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57628.1us
W20260709 12:23:18.548852 22490 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.008s	sys 0.017s
I20260709 12:23:18.599409 22443 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 12:23:18.600672 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:18.606004 22379 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:18.606846 22379 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.607297 22379 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:18.608167 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.608898 18171 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:18.609638 22486 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:18.610246 22486 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.610802 22486 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:18.613750 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.615024 18165 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:18.616465 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.618505 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.619448 18168 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.621420 22282 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:18.623449 22282 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:18.624758 18172 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:18.549705 22423 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.635397 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.632272 18023 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
W20260709 12:23:18.642403 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.642959 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 150039us
I20260709 12:23:18.644130 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.644688 22300 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:18.558945 22483 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.091s
I20260709 12:23:18.650918 22483 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:18.651473 22483 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:18.588860 22294 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:18.654383 22223 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.654932 22294 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:18.655606 22294 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.655984 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.658319 22485 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:18.662129 22480 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.762s	user 0.007s	sys 0.006s
I20260709 12:23:18.561355 22400 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:18.664575 22400 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.667523 22423 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:18.656106 22294 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.677908 18015 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:18.681949 18171 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:18.698921 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.700654 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.689916 18022 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:18.664440 21934 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:18.705324 21934 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.705971 21934 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.595752 22439 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:18.714479 22439 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:18.716043 22439 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.723124 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:18.691232 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98931.6us
I20260709 12:23:18.704149 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:18.702499 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.728081 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:18.727939 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 158584us
I20260709 12:23:18.729511 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:18.734218 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.731135 18169 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:18.733742 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.718659 22439 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.624214 22282 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:18.747253 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.750319 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:18.750540 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:18.750748 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:18.751632 18096 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:18.751643 18099 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:18.750598 18015 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:18.753558 18813 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:18.653013 22483 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.743875 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.760872 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.742285 18171 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:18.761111 18019 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:18.755365 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.756225 18090 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:18.767361 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:18.768383 18812 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:18.750420 22300 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:18.736476 22443 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:18.769726 22443 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.770216 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:18.769755 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:18.772532 17922 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 8, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 8 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:18.776438 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.751456 22370 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.782846 18064 leader_election.cc:400] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:18.783315 18098 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:18.750368 22381 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:23:18.783335 18064 leader_election.cc:403] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:18.788272 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.789222 18023 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:18.790138 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:18.791069 18097 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:18.801600 18095 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:18.804925 22300 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:18.805668 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.806069 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.806923 18023 raft_consensus.cc:2368] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:18.807545 18016 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:18.807875 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.808053 22223 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:18.808776 22223 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.809762 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.810472 18015 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.807019 22300 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.799926 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:18.821695 18168 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:18.770242 22443 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:18.770970 22480 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:18.840858 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.829047 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.828187 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.827739 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.846642 18171 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:18.784583 22370 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:18.799378 22375 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.806s	user 0.012s	sys 0.002s
I20260709 12:23:18.843089 18169 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:18.860194 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:18.872839 18167 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:18.876993 18015 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:18.879038 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:18.827564 22223 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:18.890199 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.844449 22362 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:18.893337 22480 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.897101 22293 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 12:23:18.877012 22282 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.898098 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:18.898666 22478 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.005s
I20260709 12:23:18.904706 18019 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:18.906001 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.908381 22483 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:18.909320 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 109539us
I20260709 12:23:18.910115 18168 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:18.910444 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.914333 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.915022 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.919394 18097 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:18.901214 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:18.927479 18096 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:18.932271 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.935149 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.936141 18168 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:18.937111 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:18.866909 17922 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 5 to 3, leader changed from <none> to ae9641d5c3184572a102d51a75f670ee (127.16.102.3), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from VOTER to NON_VOTER, now has a pending config: VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1), VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2), VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3), VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 9 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: NON_VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: true } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:18.938253 18168 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:18.939144 17922 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.941989 21934 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.941300 17922 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 5, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 5 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.944355 18024 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:23:18.945364 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:18.934193 18018 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:23:18.955658 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:18.956336 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:18.959369 22484 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:18.959686 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:18.967844 18172 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:18.968403 18172 raft_consensus.cc:614] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:18.958736 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:18.961015 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:18.889638 22294 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.961948 18167 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:18.969962 22484 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:18.881232 22375 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:18.972743 22463 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.098s	sys 0.137s
I20260709 12:23:18.913344 22362 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:18.928362 22480 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:18.970546 22352 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:18.991689 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111399us
I20260709 12:23:18.967984 22282 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:18.994203 17922 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 12 to 13, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 13 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:18.985024 22463 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.098s	sys 0.137s
I20260709 12:23:18.955662 22478 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:23:19.035326 22463 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.982257 22293 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:19.037149 22094 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:19.035563 22484 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:19.038336 18815 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.024s	sys 0.052s
W20260709 12:23:19.043406 18815 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.025s	sys 0.052s
W20260709 12:23:19.040164 18806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.028s	sys 0.062s
W20260709 12:23:19.045063 18806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.028s	sys 0.063s
I20260709 12:23:18.988646 22488 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:19.042907 18166 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:18.995891 22352 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.042778 22094 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:19.046612 17922 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 8, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 8 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:19.060029 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.974615 22486 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:19.103933 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.048626 22090 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:19.123993 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a07f730a06794e2cb530edca57871763"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:19.124675 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 625) took 1267 ms. Trace:
I20260709 12:23:19.124893 18164 raft_consensus.cc:606] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:19.125192 18166 rpcz_store.cc:276] 0709 12:23:17.857024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.858032 (+  1008us) service_pool.cc:224] Handling call
0709 12:23:17.858528 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.067303 (+1208775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.124281 (+ 56978us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:19.124549 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2719337344}
W20260709 12:23:19.126520 22517 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:23:19.127677 21934 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:19.131189 22480 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:19.049438 22443 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.136543 22443 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:19.133589 22362 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:19.069609 22379 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.138686 22094 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:19.139348 22379 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:19.140508 22094 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:19.142943 22294 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:19.143759 22294 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:19.118374 22293 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:23:19.082991 22352 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:19.044852 22463 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.219457 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:19.224536 18166 raft_consensus.cc:606] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:19.225111 18166 raft_consensus.cc:614] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:19.315379 18173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.016s	sys 0.007s
W20260709 12:23:19.316030 18173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.016s	sys 0.007s
W20260709 12:23:19.332558 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.320676 22270 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.013s	sys 0.009s
W20260709 12:23:19.320992 22496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.009s	sys 0.014s
W20260709 12:23:19.323530 18170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.009s	sys 0.014s
W20260709 12:23:19.339102 18170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.010s	sys 0.014s
W20260709 12:23:19.338781 22496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.009s	sys 0.014s
W20260709 12:23:19.336501 22270 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.014s	sys 0.009s
I20260709 12:23:19.323125 22090 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:19.355640 22270 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:19.328369 22375 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:19.363708 18024 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:23:19.368327 18024 raft_consensus.cc:606] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:19.368854 18024 raft_consensus.cc:614] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:19.326035 22432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.008s	sys 0.032s
I20260709 12:23:19.363999 22270 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:19.373338 22423 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.703s	user 0.014s	sys 0.003s
I20260709 12:23:19.357722 22294 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:19.373052 22432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.008s	sys 0.032s
I20260709 12:23:19.367038 22486 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:19.380811 22439 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.626s	user 0.016s	sys 0.000s
W20260709 12:23:19.391029 18808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.000s	sys 0.026s
W20260709 12:23:19.395395 18808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.000s	sys 0.027s
W20260709 12:23:19.399304 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.010s	sys 0.010s
W20260709 12:23:19.399885 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.010s	sys 0.010s
W20260709 12:23:19.401876 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s	user 0.028s	sys 0.038s
W20260709 12:23:19.402601 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.891s	user 0.028s	sys 0.038s
W20260709 12:23:19.403479 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.858s	user 0.016s	sys 0.033s
I20260709 12:23:19.379022 22423 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:19.403915 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.016s	sys 0.033s
W20260709 12:23:19.397727 22400 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.015s	sys 0.015s
W20260709 12:23:19.405665 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.006s	sys 0.019s
W20260709 12:23:19.452078 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.007s	sys 0.019s
W20260709 12:23:19.470104 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.456553 22400 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.015s	sys 0.015s
I20260709 12:23:19.475390 22496 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:19.457875 22490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.042s	sys 0.065s
I20260709 12:23:19.475996 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:19.472998 22554 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:19.394501 22294 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.469362 22432 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.480875 18165 raft_consensus.cc:606] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
W20260709 12:23:19.483969 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 361299us
W20260709 12:23:19.485031 18095 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.003s	sys 0.010s
W20260709 12:23:19.492367 18095 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.003s	sys 0.010s
W20260709 12:23:19.481626 22490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.043s	sys 0.065s
I20260709 12:23:19.477541 22463 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:19.503212 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 731529us
W20260709 12:23:19.488246 22325 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.007s	sys 0.006s
W20260709 12:23:19.483786 22381 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.011s	sys 0.008s
I20260709 12:23:19.477078 22496 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.486238 22379 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.494266 22480 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.509725 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:19.517514 18064 leader_election.cc:400] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:23:19.518850 18064 leader_election.cc:403] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260709 12:23:19.400144 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 574433us
W20260709 12:23:19.513571 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.019s	sys 0.030s
W20260709 12:23:19.522464 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.019s	sys 0.030s
W20260709 12:23:19.523057 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 671692us
W20260709 12:23:19.508944 22381 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.012s	sys 0.008s
I20260709 12:23:19.523976 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:19.504096 22423 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:19.532655 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53660.8us
W20260709 12:23:19.506404 22325 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.007s	sys 0.006s
W20260709 12:23:19.515451 22485 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.017s	sys 0.034s
I20260709 12:23:19.510682 22548 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:23:19.556064 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:19.555272 22485 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.018s	sys 0.035s
I20260709 12:23:19.513247 22432 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:19.515810 22379 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.550797 22463 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
W20260709 12:23:19.574612 18062 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.019s	sys 0.026s
W20260709 12:23:19.575126 18062 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.019s	sys 0.027s
I20260709 12:23:19.537451 22480 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:19.589392 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74782.8us
I20260709 12:23:19.585718 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:19.586283 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:19.562585 22536 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:19.587252 18166 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:19.598686 18172 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:19.601999 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49855.3us
I20260709 12:23:19.603247 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.600493 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:19.607889 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:19.608969 18096 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:19.603330 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:19.601511 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.012s	sys 0.004s
W20260709 12:23:19.625439 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.013s	sys 0.005s
W20260709 12:23:19.627683 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 668953us
I20260709 12:23:19.634277 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:19.635651 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:19.642585 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:19.588466 22423 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.652020 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:19.656910 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56573.1us
I20260709 12:23:19.658418 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:19.581775 22488 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:19.655670 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:19.663259 18167 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:19.663743 18167 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:19.666203 18092 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:19.668190 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:19.669093 18097 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:19.672838 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:19.673457 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.612430 22375 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:23:19.604962 22548 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:23:19.601984 22325 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
W20260709 12:23:19.687516 22542 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:23:19.688624 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:19.682204 22362 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:19.692487 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.660147 22536 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:19.687150 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 445636us
I20260709 12:23:19.704700 18096 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:19.712918 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.722134 18095 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:19.722410 18090 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:19.714915 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 173230us
I20260709 12:23:19.665148 22545 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:19.727172 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:19.719808 18092 raft_consensus.cc:2368] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:19.723958 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:19.729154 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.647584 22484 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:23:19.733474 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.728077 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:19.742512 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.697767 22439 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:19.694770 22443 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.661196 22423 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.676347 22490 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:23:19.758533 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:19.759009 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.768675 18094 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:19.746805 18167 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:19.759688 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65892.3us
I20260709 12:23:19.770918 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:19.696439 22544 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:19.720003 22294 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:19.775791 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.787031 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:19.788004 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54278.3us
I20260709 12:23:19.755642 22381 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:19.748708 22557 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:19.790263 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:19.754175 22545 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:19.772274 22439 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:19.773329 22375 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:19.798848 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 195686us
I20260709 12:23:19.691908 22293 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:23:19.800616 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122010us
I20260709 12:23:19.797521 18138 leader_election.cc:400] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:19.773881 22496 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:19.801098 18090 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:19.803200 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:19.804112 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:19.805150 22294 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:19.805879 22537 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.229s	user 0.002s	sys 0.000s
W20260709 12:23:19.807232 22534 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 12:23:19.808135 22536 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:19.813125 22557 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.814059 22379 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:19.881762 22543 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:19.814352 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:19.792627 22531 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.349s	user 0.004s	sys 0.004s
I20260709 12:23:19.885527 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 804) took 1526 ms. Trace:
I20260709 12:23:19.886139 18806 rpcz_store.cc:276] 0709 12:23:18.358850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.369196 (+ 10346us) service_pool.cc:224] Handling call
0709 12:23:18.369617 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.377469 (+  7852us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.377509 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.378184 (+   675us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.045573 (+667389us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:19.045631 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.050611 (+  4980us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:19.513424 (+462813us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:19.885077 (+371653us) raft_consensus.cc:1685] finished
0709 12:23:19.885105 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.885424 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1644,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":123,"spinlock_wait_cycles":50601728,"thread_start_us":666923,"threads_started":1,"wal-append.queue_time_us":900123}
I20260709 12:23:19.817762 18167 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:19.818163 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1005) took 1052 ms. Trace:
I20260709 12:23:19.857179 18138 leader_election.cc:403] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:23:19.860200 18096 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:19.821491 18094 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:19.863878 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:19.889216 18808 rpcz_store.cc:276] 0709 12:23:18.766033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.766244 (+   211us) service_pool.cc:224] Handling call
0709 12:23:18.770658 (+  4414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.785361 (+ 14703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.785397 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.808633 (+ 23236us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.395955 (+587322us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:19.451195 (+ 55240us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.451544 (+   349us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.451655 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:19.817631 (+365976us) raft_consensus.cc:1685] finished
0709 12:23:19.817669 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.818074 (+   405us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20643,"prepare.queue_time_us":60813,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":62,"raft.queue_time_us":18974,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":11001,"thread_start_us":586827,"threads_started":1,"wal-append.queue_time_us":751298}
I20260709 12:23:19.893457 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2141) took 1655 ms. Trace:
I20260709 12:23:19.895493 18098 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:19.857712 18813 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:19.775357 22325 consensus_queue.cc:1243] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:23:19.880404 22533 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:23:19.881356 22560 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:19.882427 22543 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.799463 22443 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:19.920281 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 575) took 1726 ms. Trace:
I20260709 12:23:19.920832 18020 rpcz_store.cc:276] 0709 12:23:18.193412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.409086 (+215674us) service_pool.cc:224] Handling call
0709 12:23:19.919927 (+1510841us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:19.920196 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2332592384}
I20260709 12:23:19.855762 22294 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:19.864583 22463 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:19.956234 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 171765us
I20260709 12:23:19.960673 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:19.819517 22439 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:19.865237 22293 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:19.963089 18024 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:19.863322 22375 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:19.899786 22480 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:19.962795 22532 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.381s	user 0.000s	sys 0.001s
W20260709 12:23:19.877301 18124 heartbeater.cc:646] Failed to heartbeat to 127.16.102.62:42293 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.102.62:42293 timed out after 15.000s (SENT)
W20260709 12:23:19.919355 22094 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.009s
I20260709 12:23:19.891183 22484 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:19.960848 22432 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:19.981215 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.891556 22282 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.853s	user 0.016s	sys 0.002s
I20260709 12:23:19.982010 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:19.858500 22223 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 12:23:19.984638 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:19.984809 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121744us
I20260709 12:23:19.916733 22560 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:19.927491 22362 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:19.920575 22543 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:19.993017 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.792217 22488 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:19.962430 22294 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:19.964305 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:20.011114 18019 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:19.961527 22463 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:19.979053 22529 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.699s	user 0.003s	sys 0.002s
I20260709 12:23:19.998615 18023 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:19.919648 22548 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260709 12:23:19.926949 22325 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.066s
I20260709 12:23:20.000370 18021 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:20.004238 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:19.980685 22563 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
W20260709 12:23:20.020792 18780 connection.cc:256] RPC call timeout handler was delayed by -1.1570976630000587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20260709 12:23:19.900292 22559 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.136s
W20260709 12:23:20.016002 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 160593us
I20260709 12:23:19.921988 22553 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.110s
I20260709 12:23:20.012128 22463 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:20.008667 22094 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260709 12:23:20.023548 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:23:20.023999 22535 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.375s	user 0.001s	sys 0.001s
I20260709 12:23:20.044075 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 569) took 1852 ms. Trace:
W20260709 12:23:20.044217 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 156477us
I20260709 12:23:20.022621 22362 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:20.031224 22539 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
W20260709 12:23:20.021517 22486 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:23:19.999310 22484 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:20.056842 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55705.2us
W20260709 12:23:20.057441 22551 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.319s	user 0.000s	sys 0.001s
I20260709 12:23:20.058034 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.063634 18019 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:23:20.045933 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.042383 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:23:20.039017 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 242814us
I20260709 12:23:20.044600 18017 rpcz_store.cc:276] 0709 12:23:18.191502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.430123 (+238621us) service_pool.cc:224] Handling call
0709 12:23:18.430696 (+   573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.493218 (+ 62522us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:18.493262 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.510976 (+ 17714us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.403252 (+892276us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:19.403308 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.403418 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.043788 (+640370us) raft_consensus.cc:1685] finished
0709 12:23:20.043811 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.043995 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48475,"raft.queue_time_us":65348,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":27664,"thread_start_us":891746,"threads_started":1,"wal-append.queue_time_us":1231335}
I20260709 12:23:20.065758 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2140) took 1828 ms. Trace:
I20260709 12:23:20.004374 22375 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.085955 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:20.050779 22480 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:20.104423 22556 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.355s	user 0.000s	sys 0.002s
I20260709 12:23:20.077440 18173 rpcz_store.cc:276] 0709 12:23:18.237586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.474015 (+236429us) service_pool.cc:224] Handling call
0709 12:23:18.501482 (+ 27467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.509389 (+  7907us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.509429 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.510945 (+  1516us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.316528 (+805583us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:19.316582 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.316681 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.065408 (+748727us) raft_consensus.cc:1685] finished
0709 12:23:20.065444 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.065680 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1201,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":805096,"threads_started":1,"wal-append.queue_time_us":1042491}
W20260709 12:23:20.108490 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80860.4us
I20260709 12:23:20.049890 22432 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:20.133369 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75515.1us
I20260709 12:23:20.134363 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:20.085748 22423 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:20.131345 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:20.084789 22282 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:20.133535 22432 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:20.140570 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 353734us
I20260709 12:23:20.037801 22545 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.137468 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.104882 22270 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:20.153128 22541 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.319s	user 0.000s	sys 0.002s
I20260709 12:23:20.153945 22567 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:20.154546 22567 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:20.157773 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:20.158663 18169 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.151132 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.134590 22572 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:20.103927 22559 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:23:20.173695 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:20.175613 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:20.176153 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:20.147111 22432 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:20.140403 22282 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.127674 22480 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.190215 18172 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:20.190611 18171 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:23:20.180996 18165 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
W20260709 12:23:20.178372 22352 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.970s	user 0.018s	sys 0.001s
I20260709 12:23:20.206025 22575 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:20.142817 22443 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:20.179524 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:20.203524 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 237032us
I20260709 12:23:20.204557 22580 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:20.219017 22423 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:20.219365 22561 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:20.219732 22423 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:20.219911 22561 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.220906 22423 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.222139 22576 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:23:20.215797 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78443.7us
I20260709 12:23:20.175645 22572 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.216311 22484 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:20.194525 22375 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:20.220929 22561 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.217604 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:20.230772 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:20.231796 22562 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.367s	user 0.002s	sys 0.000s
I20260709 12:23:20.231400 22282 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.232977 22282 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:20.233062 22546 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:20.233899 22546 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:20.234882 22546 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.237053 22544 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:23:20.209247 22550 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:23:20.237797 22544 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:20.183586 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:20.216259 22299 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.549s	user 0.022s	sys 0.048s
I20260709 12:23:20.085227 22293 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.222013 22463 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.239471 22544 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:20.252180 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68717us
I20260709 12:23:20.253434 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:20.257472 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:20.247179 22483 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.301s	user 0.015s	sys 0.004s
W20260709 12:23:20.278072 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52394.5us
I20260709 12:23:20.278280 22571 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:23:20.249100 22299 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.582s	user 0.022s	sys 0.048s
I20260709 12:23:20.272774 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:20.284041 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 125066us
I20260709 12:23:20.278887 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.289839 18092 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:23:20.210947 22578 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:20.291229 18017 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:20.291443 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:20.291733 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:20.298728 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:20.300433 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:20.209426 22552 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.105s
W20260709 12:23:20.314929 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.366184 18090 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:20.367421 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.273753 22559 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:20.365454 18019 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260709 12:23:20.366173 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66470.4us
I20260709 12:23:20.135730 22560 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:20.366407 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66895.5us
I20260709 12:23:20.210532 22352 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.179690 22362 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.258011 22557 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:20.278921 22561 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.285959 22478 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:20.359977 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75682.1us
I20260709 12:23:20.356889 22580 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:20.382227 18099 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:23:20.382558 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78924.8us
I20260709 12:23:20.296758 22578 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:20.303934 22484 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:19.894312 18170 rpcz_store.cc:276] 0709 12:23:18.382531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.481551 (+ 99020us) service_pool.cc:224] Handling call
0709 12:23:18.511076 (+ 29525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.511431 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:18.511887 (+   456us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.782037 (+1270150us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.892574 (+110537us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:19.892785 (+   211us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:19.892930 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.893125 (+   195us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.893355 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119794,"prepare.queue_time_us":998,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":319881,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":7909,"thread_start_us":1266933,"threads_started":3,"wal-append.queue_time_us":1034015}
I20260709 12:23:20.369149 22090 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:20.396179 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:20.396931 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 245863us
I20260709 12:23:20.389990 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.390866 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.398037 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.391400 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.398702 18017 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:20.396638 22090 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:20.399242 22568 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.146s	user 0.000s	sys 0.002s
I20260709 12:23:20.391753 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.399597 22090 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.399829 22538 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.400337 18023 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:23:20.400487 22538 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.403244 22538 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:20.404516 22586 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.140s	user 0.001s	sys 0.002s
I20260709 12:23:20.374939 22379 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:20.392856 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:20.409893 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.373452 21934 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.412886 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:20.398741 18024 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:20.383097 22478 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.381366 22564 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:20.397634 22484 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:20.435653 18064 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.504s (SENT)
I20260709 12:23:20.398108 22578 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.443826 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:20.459597 22484 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:20.460527 18017 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:20.442862 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:20.461973 18022 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:20.447511 18020 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 12:23:20.410045 22581 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:20.464289 22581 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.465561 22581 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.467660 22581 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:20.409982 22580 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.456609 22557 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:20.478068 22370 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:20.480583 22370 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:20.474304 22583 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:20.481940 22583 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:20.481768 22370 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.482630 22583 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.483913 22583 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.479004 22582 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:23:20.418231 21934 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.442508 22478 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.490692 22563 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:20.490998 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:20.492066 18015 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:20.493415 22585 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.205s
I20260709 12:23:20.494729 22480 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:20.491267 22563 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.381295 22567 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.521653 22567 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.523104 22580 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.442838 22564 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.526750 22564 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:20.553014 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 166234us
I20260709 12:23:20.569411 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:20.565919 22602 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:20.573213 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137618us
W20260709 12:23:20.574225 18064 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.547s (SENT)
I20260709 12:23:20.548890 22370 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:20.583915 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 216580us
I20260709 12:23:20.586004 22559 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:20.481185 22325 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.092s
I20260709 12:23:20.598737 18022 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:20.607906 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.613080 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:20.614262 18815 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:20.616055 22325 consensus_queue.cc:1243] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:23:20.616818 22325 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:20.617213 22325 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:20.617966 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.575822 22590 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:20.618304 22325 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.587467 22596 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:20.619907 22325 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.620354 22596 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.617553 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:20.623345 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 209537us
I20260709 12:23:20.619061 22590 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.570091 22352 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:20.627358 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 250068us
I20260709 12:23:20.627082 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:20.638782 22575 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.638680 22536 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:20.644626 22575 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.646589 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:20.647506 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:20.648491 18813 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:20.649694 18166 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:20.650633 22575 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:20.650944 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:20.651562 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:20.652429 18808 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:20.648515 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:20.654129 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:20.655750 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:20.656638 18170 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:20.588209 22602 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.580966 22293 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.646885 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.651772 22423 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.572268 22571 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:20.585729 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:20.578953 22463 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:20.676262 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52046.8us
I20260709 12:23:20.677975 18808 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:20.684026 18815 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:20.686444 18167 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:20.686900 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.642370 22400 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:20.625917 22496 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:20.691799 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 117620us
W20260709 12:23:20.695801 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110229us
I20260709 12:23:20.668552 22575 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.701074 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:20.700469 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.789s (SENT) [suppressed 33 similar messages]
I20260709 12:23:20.741763 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:20.705065 22594 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:20.707545 22375 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:20.685410 22379 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:20.711232 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.711912 18173 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:20.552160 22563 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:20.696717 22294 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.664s	user 0.014s	sys 0.013s
I20260709 12:23:20.587589 22559 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.755909 22379 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.690097 22423 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.698172 22352 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:23:20.736588 22270 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:20.761523 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.748088 22583 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:20.736481 22536 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:20.759980 22294 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.790465 22592 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:23:20.767760 22585 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:23:20.783102 22582 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.793984 22571 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.789206 22090 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.738215 22463 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:23:20.813937 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:20.750079 22575 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:20.825721 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 138932us
I20260709 12:23:20.826813 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.800254 22580 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:20.798979 22583 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.802805 22552 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:20.834895 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134471us
W20260709 12:23:20.822867 22439 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.832s	user 0.015s	sys 0.001s
I20260709 12:23:20.839942 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:20.843942 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82537.7us
I20260709 12:23:20.808442 22596 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:20.835009 22580 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.827994 22553 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.855865 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:20.856190 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.838706 22552 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.848552 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.862313 22580 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.841526 22590 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:20.874169 22375 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:20.844203 22439 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.868417 22553 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.879123 22553 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:20.884414 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 174032us
I20260709 12:23:20.870489 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.847172 22595 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:20.904790 18136 leader_election.cc:400] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:20.808585 22582 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.913174 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:20.872390 22546 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:20.910930 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9801765740003248s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2788s for a response
I20260709 12:23:20.911456 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.870182 22552 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:20.916326 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60312.1us
I20260709 12:23:20.914302 18015 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:20.920486 18022 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:23:20.921401 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8964765740001894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28809999999999997s for a response
W20260709 12:23:20.923949 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7589685369998733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2642s for a response
I20260709 12:23:20.924049 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:20.924484 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6803660390000914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260709 12:23:20.907863 22300 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:20.905481 22590 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:20.931815 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6641660390000652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
I20260709 12:23:20.894743 22609 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:20.938160 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:20.943650 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5921660389999488s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1605s for a response
I20260709 12:23:20.919838 22546 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:20.943210 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:20.944128 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5399660389998644s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20260709 12:23:20.938612 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:20.923038 22439 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:20.945902 18064 connection.cc:256] RPC call timeout handler was delayed by -1.432866039000146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
W20260709 12:23:20.952692 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0368660389999604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
W20260709 12:23:20.953156 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0243959340000401s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20260709 12:23:20.954660 22599 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:23:20.957113 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:20.946488 22608 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:20.950505 22300 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:20.925289 22593 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.152s
W20260709 12:23:20.953523 22607 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:23:20.913228 22611 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:20.917932 22595 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:20.943205 22553 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:20.965991 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:20.871155 22536 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:20.959542 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.935273 22602 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:20.973623 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:20.919519 22583 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:20.937521 22362 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:23:20.975399 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:20.975908 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.085162 18098 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:21.085811 22565 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:21.086637 22565 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:21.087280 21934 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:20.978556 22608 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:21.087729 22565 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.088850 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:21.072046 22478 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.581s	user 0.010s	sys 0.011s
I20260709 12:23:20.983848 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.113787 18093 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:20.983318 18096 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:20.980496 18094 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:20.965446 22609 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:20.992174 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.118506 18016 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:20.982345 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:21.101806 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 128285us
I20260709 12:23:21.119246 18090 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:21.026227 22282 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.793s	user 0.017s	sys 0.009s
I20260709 12:23:21.038132 22548 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:20.986749 22575 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.112666 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:21.094822 18810 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:21.129797 18808 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:20.977056 22484 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.009s
I20260709 12:23:21.119608 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:20.968387 22094 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:21.081859 22578 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.622s	user 0.005s	sys 0.016s
W20260709 12:23:21.089058 22432 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.879s	user 0.018s	sys 0.004s
W20260709 12:23:21.123452 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 157088us
I20260709 12:23:20.975054 22590 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:20.982609 22423 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.026793 22090 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.112322 18015 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:21.156183 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.159198 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.159425 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.159937 18015 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10. Re-sending same reply.
I20260709 12:23:21.132414 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.161379 18809 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:23:20.972808 22370 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.144s
I20260709 12:23:20.968482 22439 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.155s
W20260709 12:23:21.142087 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.065013 22580 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:20.984824 22582 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.018528 22300 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.110s
I20260709 12:23:21.120868 22553 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:21.152558 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:20.975754 22294 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.126174 22548 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:21.164966 18099 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: [7.3-8.4]
I20260709 12:23:21.153595 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.165954 18096 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.980013 22546 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.144426 22486 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:21.123204 22478 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.171047 18098 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:21.159806 22565 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:21.167737 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.176327 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.177735 18093 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:21.148322 22282 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:23:21.183292 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:21.199630 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 227842us
W20260709 12:23:21.173550 22561 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.788s	user 0.013s	sys 0.011s
I20260709 12:23:21.208098 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.211584 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:21.156742 22432 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.218915 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.219906 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.178397 22486 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.220592 22432 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.204427 22300 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:21.221419 22300 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.221611 18021 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:21.221863 22300 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:21.222158 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:21.223004 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54585.2us
I20260709 12:23:21.223733 22615 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.223841 22282 consensus_queue.cc:1243] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:21.220631 22486 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:21.224407 22615 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.225003 22548 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.037254 22595 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.226241 22548 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.226297 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.149459 22578 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.182561 22553 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:21.228776 22619 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:23:21.229209 22553 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.166774 22608 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:23:21.231208 22626 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.019s
W20260709 12:23:21.181233 22581 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.713s	user 0.017s	sys 0.000s
I20260709 12:23:21.173918 22585 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:21.204083 22543 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.178s	user 0.016s	sys 0.011s
I20260709 12:23:21.244971 22375 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:21.244727 22543 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.246505 18097 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:21.217919 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.218205 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.218076 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.203992 22370 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:21.220678 18091 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:21.248478 22613 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:21.254827 18016 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:21.256397 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.257364 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:21.216058 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.270658 18169 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:21.239984 18018 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:23:21.240353 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:21.242386 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.219949 22478 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.243431 22585 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.258395 18165 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:23:21.259690 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.214903 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.265027 22370 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.265740 22625 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:21.247689 22543 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.275018 18015 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:21.276885 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.147568 22484 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.297051 18096 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:21.203325 22379 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:21.168984 22622 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.102s
I20260709 12:23:21.316516 18096 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:23:21.321674 18018 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:21.325346 18091 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:21.326273 18093 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
W20260709 12:23:21.326871 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86634.8us
W20260709 12:23:21.326299 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 91247.4us
I20260709 12:23:21.309391 22370 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:21.330379 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 373356us
I20260709 12:23:21.311462 22375 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:21.306905 22625 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:21.343410 18138 leader_election.cc:400] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:21.336073 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.346487 18016 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: [6.3-8.4]
I20260709 12:23:21.350210 18015 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:21.352327 18138 leader_election.cc:403] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:21.354157 18018 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:21.372025 22575 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.374293 22615 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.374984 22585 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:21.381671 18015 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:21.336071 22627 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:21.374390 22293 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.005s
I20260709 12:23:21.387149 22581 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:21.390225 22554 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:23:21.390556 22325 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.770s	user 0.009s	sys 0.007s
I20260709 12:23:21.390693 22554 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:21.391587 18170 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:23:21.391711 22554 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.393128 22554 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:21.393729 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.385870 22294 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 12:23:21.395146 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.396085 18170 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:21.400853 22484 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:21.401432 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:21.402871 22625 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.393460 22300 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.403908 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.404423 18015 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:21.325054 22379 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.402917 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.405369 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.416854 18172 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:21.417439 18166 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:23:21.386549 22564 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.859s	user 0.018s	sys 0.001s
I20260709 12:23:21.379580 22375 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:21.431738 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97734.8us
W20260709 12:23:21.433673 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.434722 22294 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:21.430197 22547 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:21.437851 22547 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:21.443967 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.404492 22625 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:21.455394 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.439116 22547 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.462842 22379 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.463199 22486 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:21.468261 22552 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.534s	user 0.013s	sys 0.000s
I20260709 12:23:21.468741 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:21.468925 22552 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.472718 22300 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.473242 22483 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.473906 22483 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.476811 22483 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.478210 22613 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:21.479990 22560 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:21.504618 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.504449 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.505390 18022 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:21.502584 22486 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:21.510525 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.504441 22484 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.511758 18022 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:21.511649 22484 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.512488 18015 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:21.513556 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.514009 18016 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:21.534929 18018 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:21.439677 22564 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.537304 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:21.538192 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 112449us
W20260709 12:23:21.542594 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.409221 22627 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.537096 18168 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:21.548916 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:21.539222 22615 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:21.561103 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.562264 18016 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:21.559199 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.576886 22564 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.577680 18098 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:21.582691 22637 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:21.583554 22627 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.585071 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:21.586040 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.586539 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:21.586803 18809 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:21.587621 22400 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:21.588188 22400 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:21.588613 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:21.542032 22375 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.598618 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.605208 22645 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:21.608132 22625 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:21.611234 22563 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.589798 22627 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:21.625046 18173 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:21.626474 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.571044 22636 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:21.567808 22370 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.642300 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.642810 18166 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:23:21.635974 22636 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.643908 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 587) took 1368 ms. Trace:
I20260709 12:23:21.636298 22581 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:21.637939 22616 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.627863 22627 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:21.652927 18019 rpcz_store.cc:276] 0709 12:23:20.275531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.275671 (+   140us) service_pool.cc:224] Handling call
0709 12:23:20.276200 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.371843 (+ 95643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.643539 (+1271696us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54001d0954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:21.643799 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49914240}
I20260709 12:23:21.663425 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.671715 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.646376 22637 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:21.657955 22636 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:21.678510 22576 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:21.680178 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.682300 22645 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.685190 22370 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.662462 22627 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.686134 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2310) took 1409 ms. Trace:
I20260709 12:23:21.676647 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.689381 18021 rpcz_store.cc:276] 0709 12:23:20.276336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.276522 (+   186us) service_pool.cc:224] Handling call
0709 12:23:20.278269 (+  1747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.222193 (+943924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.685839 (+463646us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:21.686050 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2168984960}
I20260709 12:23:21.665493 22643 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:21.686100 22484 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.694063 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:21.687978 22545 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.009s	sys 0.009s
I20260709 12:23:21.682323 22564 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:21.700057 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.703014 22484 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:21.704396 22484 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.704963 22484 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:21.700917 22644 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.096s	user 0.000s	sys 0.002s
W20260709 12:23:21.699717 22545 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.009s	sys 0.010s
I20260709 12:23:21.702003 22370 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:21.712244 22564 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:21.702450 22594 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
W20260709 12:23:21.733950 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.747731 18021 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:23:21.752393 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.749558 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.752543 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.720522 22571 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:23:21.753755 18016 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:21.753976 18019 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:21.722074 22559 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:23:21.755728 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:21.715221 22572 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.756578 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.760093 18098 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.729496 22621 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:23:21.762938 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:21.764012 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.748060 22594 consensus_queue.cc:1243] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:21.768149 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.775954 18090 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:21.746181 22375 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:23:21.782145 18098 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:21.752170 22613 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:23:21.767362 22293 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:21.733522 22637 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.786471 22636 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.761773 22548 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.027s
W20260709 12:23:21.801939 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52035.6us
I20260709 12:23:21.803023 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.767148 22554 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 12:23:21.805874 18017 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:21.796979 22293 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.798070 22581 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:21.810400 22370 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:21.821228 22400 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:21.822068 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 808) took 1452 ms (client timeout 1908 ms). Trace:
W20260709 12:23:21.822638 18806 rpcz_store.cc:269] 0709 12:23:20.370016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.370745 (+   729us) service_pool.cc:224] Handling call
0709 12:23:21.821729 (+1450984us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:21.821980 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2132616576}
I20260709 12:23:21.811928 22293 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.816715 22554 consensus_queue.cc:1243] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:23:21.833598 18098 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
W20260709 12:23:21.834509 22650 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
W20260709 12:23:21.838689 22646 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:21.811338 22581 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:21.785586 22564 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:21.877786 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52325.2us
I20260709 12:23:21.880400 18164 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:21.885098 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.886168 18808 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 6.
I20260709 12:23:21.889266 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.870877 22548 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.892297 22400 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.895085 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:21.911388 22548 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.912207 18813 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:21.890141 22652 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:21.913287 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.914156 22548 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:21.898350 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:21.899008 22636 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:21.903086 18091 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:21.916242 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.899425 18807 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:23:21.901641 22579 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.634s	user 0.001s	sys 0.006s
I20260709 12:23:21.903048 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.903748 18808 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:21.931033 18808 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:21.905887 22580 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:21.881512 22598 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:21.907227 18016 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:21.875365 22554 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.907415 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.910215 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:21.937355 18809 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:21.918507 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 883) took 1271 ms. Trace:
W20260709 12:23:21.918584 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115662us
I20260709 12:23:21.938642 18812 rpcz_store.cc:276] 0709 12:23:20.647019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.647183 (+   164us) service_pool.cc:224] Handling call
0709 12:23:21.918276 (+1271093us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540034fdd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:21.918424 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1196509952}
I20260709 12:23:21.940267 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.921456 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.922513 18091 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:21.909045 22627 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.889581 22585 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.945626 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:21.884368 22563 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.955225 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.956372 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.943346 18808 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.957151 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.954098 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.957054 18091 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:21.961020 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.961474 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.961935 18016 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:21.933192 22553 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.963346 22553 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.964818 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:21.966934 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:21.968398 18808 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:21.897506 22484 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:21.971736 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.948437 22605 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:21.974961 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:21.976904 18098 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:21.985085 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:21.984120 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:21.986114 18016 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:21.986992 18090 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:21.952979 22645 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:21.989188 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:21.957332 22643 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:21.992332 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:21.993589 18808 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:23:21.995278 18808 pending_rounds.cc:85] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 6
I20260709 12:23:21.995725 18808 pending_rounds.cc:107] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:23:21.996116 18808 raft_consensus.cc:2889] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:21.970407 22595 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:22.014321 22595 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:22.015014 22595 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.015498 22595 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:21.959678 22293 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.907120 22581 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.018833 22581 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.020243 22293 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:21.985948 22553 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.993698 22557 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:21.985183 22605 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.022439 22557 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:22.022630 22605 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:22.023671 22557 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.958163 22585 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.028864 22585 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:21.967028 22580 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.062s
I20260709 12:23:22.032307 22484 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.036211 18094 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:22.042951 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 881) took 1396 ms. Trace:
I20260709 12:23:22.002921 18814 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:21.967509 22563 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.004343 22618 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:21.987207 22548 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:21.975243 22576 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:21.975718 22485 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:21.952365 22554 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:21.970610 22545 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:23:22.046043 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:22.077951 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72257.4us
I20260709 12:23:22.079073 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:22.021349 22293 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:22.021365 18091 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:22.054143 18094 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:22.056130 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2260) took 2021 ms. Trace:
I20260709 12:23:22.043679 18810 rpcz_store.cc:276] 0709 12:23:20.646079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.646242 (+   163us) service_pool.cc:224] Handling call
0709 12:23:22.042637 (+1396395us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:22.042840 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":704097664}
I20260709 12:23:22.061195 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:22.083375 18095 rpcz_store.cc:276] 0709 12:23:20.034355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.039417 (+  5062us) service_pool.cc:224] Handling call
0709 12:23:20.039863 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:20.040155 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.040188 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:20.043572 (+  3384us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.142163 (+ 98591us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:20.142205 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.142286 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.878774 (+1736488us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:22.002168 (+123394us) raft_consensus.cc:1685] finished
0709 12:23:22.002210 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.055995 (+ 53785us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17415424,"thread_start_us":36630,"threads_started":1,"wal-append.queue_time_us":126757}
I20260709 12:23:21.987383 22583 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:21.952117 22598 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.075807 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 648) took 1410 ms. Trace:
I20260709 12:23:22.091398 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.063405 22576 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:22.090211 18167 rpcz_store.cc:276] 0709 12:23:20.649080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.650674 (+  1594us) service_pool.cc:224] Handling call
0709 12:23:22.059162 (+1408488us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b5400050ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:22.059377 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79844608}
W20260709 12:23:22.093691 22558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.824s	user 0.012s	sys 0.012s
I20260709 12:23:22.125806 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.080786 22548 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:22.076140 22400 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.062407 22485 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.090948 22575 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:22.135223 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.088390 22583 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.088032 22643 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:22.131043 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 141895us
I20260709 12:23:22.131601 18094 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:22.128167 22558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.859s	user 0.013s	sys 0.013s
I20260709 12:23:22.095490 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:22.135341 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.140122 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:22.130604 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 146591us
I20260709 12:23:22.136013 18091 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:22.090464 22636 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:22.138092 22583 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.123292 22576 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:22.095129 22483 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:23:22.141737 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.174122 22538 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:22.143950 22598 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.144482 22554 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.145227 22563 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.135881 22485 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:22.176501 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.177641 22560 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:22.140892 22643 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:22.154568 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:22.159915 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.179497 22643 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.180323 22663 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.181025 22663 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.136099 22400 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.183532 22663 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:22.044464 22325 consensus_queue.cc:1175] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "768723648a9444fda53659e6a8cc3091" responder_term: 10 status { last_received { term: 7 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.5-9.5]" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:23:22.184661 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.185531 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:22.160463 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65106.9us
I20260709 12:23:22.186510 18810 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:22.168491 22605 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.174757 22538 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.187147 22538 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:22.151252 22548 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.187350 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:22.190042 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.190752 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:22.183422 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.192313 18094 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:22.191767 22605 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.246462 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:22.258450 18810 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:22.193491 22663 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:22.246513 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.262903 18094 raft_consensus.cc:2368] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10.
I20260709 12:23:22.264079 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.250506 22563 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:22.265476 22563 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.265893 22563 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.266985 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.249928 22560 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:22.268725 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.269385 22560 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.271879 22560 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.255179 18091 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:22.248660 22400 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.262477 22663 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.252948 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:22.276775 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.996s (SENT)
I20260709 12:23:22.265005 18094 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:22.283023 18810 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 7.
I20260709 12:23:22.283435 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:22.284155 18016 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:22.193579 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:22.289188 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.290241 18095 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:22.285013 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:22.292451 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 105850us
I20260709 12:23:22.294898 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.295799 18095 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:22.296978 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.297502 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.298416 18091 raft_consensus.cc:2376] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9. Re-sending same reply.
I20260709 12:23:22.303742 18016 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:22.268496 22595 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.273690 22618 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.279341 22560 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.310549 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.311936 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.318940 18091 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:22.311826 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:22.274995 22575 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:22.321806 18020 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.316617 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.320495 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.325173 18091 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:22.327641 18091 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:23:22.329752 18091 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:22.335350 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:22.321066 22622 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:22.337086 22583 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.330253 22618 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.339121 22597 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:22.343529 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2249) took 2656 ms (client timeout 1841 ms). Trace:
I20260709 12:23:22.343962 18092 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260709 12:23:22.344358 18097 rpcz_store.cc:269] 0709 12:23:19.686975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.703824 (+ 16849us) service_pool.cc:224] Handling call
0709 12:23:22.343101 (+2639277us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540035e8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:22.343386 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3428334464}
I20260709 12:23:22.367534 22618 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:22.367867 22663 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.291512 22598 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:22.412607 18091 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:22.305426 22611 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:22.456151 22552 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.318568 22554 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.459501 22554 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.463709 22595 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.464046 22583 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.339665 22597 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:22.465397 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150613us
I20260709 12:23:22.467700 18094 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:23:22.468467 18094 pending_rounds.cc:85] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:23:22.469034 18094 pending_rounds.cc:107] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:23:22.469627 18094 raft_consensus.cc:2889] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:22.271925 22576 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.503549 18095 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:22.472071 22598 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.477386 22622 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:22.356658 22560 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.511157 22667 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260709 12:23:22.510751 22611 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:22.524586 21934 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:22.525739 22223 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.526746 22223 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.526988 22597 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.528393 22597 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:22.514003 22605 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:23:22.530016 22575 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.509481 22598 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.538486 18020 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:22.477649 22609 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.052s
I20260709 12:23:22.551476 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:22.525301 21934 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:22.573048 22576 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.545897 22554 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.590806 22223 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:22.593807 22668 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:23:22.525301 22611 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:22.618906 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116380us
W20260709 12:23:22.630429 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2382) took 1451 ms (client timeout 1558 ms). Trace:
W20260709 12:23:22.631109 18168 rpcz_store.cc:269] 0709 12:23:21.179565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.179745 (+   180us) service_pool.cc:224] Handling call
0709 12:23:22.630117 (+1450372us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:22.630325 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547377408}
I20260709 12:23:22.610129 22622 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.628033 22538 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.656996 18024 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:22.647934 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:22.658131 22538 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.658170 22676 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:22.671499 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:22.672307 18022 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:22.681385 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.646328 22560 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.683692 18022 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:22.688297 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 652) took 1876 ms. Trace:
I20260709 12:23:22.688959 18164 rpcz_store.cc:276] 0709 12:23:20.811623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.811878 (+   255us) service_pool.cc:224] Handling call
0709 12:23:22.688115 (+1876237us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:22.688226 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2450036608}
I20260709 12:23:22.695839 22627 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.699560 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:22.700555 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:22.688367 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.707257 18810 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:22.625355 22485 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.712167 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.742549 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.743289 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.743542 18094 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:22.744027 18091 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:22.759064 18091 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:22.734163 22597 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.764394 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2280) took 1025 ms. Trace:
I20260709 12:23:22.769565 18099 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:23:22.767633 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.770282 18099 pending_rounds.cc:85] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:22.771394 18099 pending_rounds.cc:107] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:23:22.771661 18099 raft_consensus.cc:2889] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:22.770190 18096 rpcz_store.cc:276] 0709 12:23:21.738819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.739002 (+   183us) service_pool.cc:224] Handling call
0709 12:23:21.750187 (+ 11185us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.750472 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:21.762721 (+ 12249us) log.cc:844] Serialized 14 byte log entry
0709 12:23:22.701467 (+938746us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b180042c280. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:23:22.763367 (+ 61900us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:22.763430 (+    63us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:22.764037 (+   607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.764149 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.764316 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11476,"prepare.queue_time_us":12069,"prepare.run_cpu_time_us":1090,"prepare.run_wall_time_us":5275,"raft.queue_time_us":109,"spinlock_wait_cycles":4161280,"thread_start_us":61616,"threads_started":1}
I20260709 12:23:22.752914 22552 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.774605 18023 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:22.727522 22563 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.776110 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.781098 18094 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:22.785748 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.786587 18022 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 7.
I20260709 12:23:22.783010 22486 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.808213 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.789203 22560 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:22.814739 22560 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.790241 22486 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.820540 22486 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.784323 22552 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:22.817524 22560 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:22.824359 18020 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:22.824779 22563 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:22.825836 22552 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:22.775059 22597 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:22.829360 22669 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.303s	user 0.000s	sys 0.001s
I20260709 12:23:22.836889 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3100) took 1699 ms. Trace:
I20260709 12:23:22.854959 18092 rpcz_store.cc:276] 0709 12:23:21.137189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.137367 (+   178us) service_pool.cc:224] Handling call
0709 12:23:22.343525 (+1206158us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.344685 (+  1160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.836609 (+491924us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540035e8d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:22.836801 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2730247680}
I20260709 12:23:22.842482 22582 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:22.856370 18016 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:22.856447 22582 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:22.842957 22676 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.857668 22582 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.843976 21934 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.858275 22676 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.858767 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.859516 21934 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.859555 18017 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:22.861236 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:22.848675 22647 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.117s	user 0.008s	sys 0.000s
I20260709 12:23:22.851733 22547 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:22.853660 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.865335 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:22.866318 22554 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:22.866910 22554 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:22.867900 22554 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.877692 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:22.894735 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.893061 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:22.897763 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:22.898955 18809 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:22.901742 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:22.905426 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.984s (SENT)
I20260709 12:23:22.906255 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:22.907434 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:22.908167 18022 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:22.911895 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:22.922326 18814 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:22.919641 18137 connection.cc:256] RPC call timeout handler was delayed by -1.466355506000582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20750000000000002s for a response
W20260709 12:23:22.906129 22615 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:23:22.894570 22611 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.907274 22563 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:22.909183 22552 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:22.923106 22325 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:22.923691 22545 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.908394 22486 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:23:22.963958 18022 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:22.954694 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.921402 22560 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:23:22.935621 22548 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:22.967981 18021 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:22.947485 22563 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.960515 22595 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.999198 22595 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.000222 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.948386 22611 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.002357 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:22.994194 18096 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:23.003237 18095 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:22.981588 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.971060 22545 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:22.965750 22325 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.988152 22613 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:22.986575 22563 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:22.990489 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:22.992002 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:23.013365 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84491us
I20260709 12:23:22.992866 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:22.994076 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2287) took 1159 ms. Trace:
I20260709 12:23:23.014088 18098 rpcz_store.cc:276] 0709 12:23:21.834887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.876183 (+ 41296us) service_pool.cc:224] Handling call
0709 12:23:21.891180 (+ 14997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.985628 (+ 94448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.993770 (+1008142us) spinlock_profiling.cc:243] Waited 85.4 ms on lock 0x7b5400214b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:22.994010 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":196486528}
I20260709 12:23:22.994985 18091 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:22.972457 22538 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.005436 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:23.017026 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:23.010046 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:23.018189 18095 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:23.016407 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2217b6329873485ead60419dc5acbff0"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:23.019232 18167 raft_consensus.cc:606] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:22.988225 22548 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.032699 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.033366 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:23.033690 18164 raft_consensus.cc:2368] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:23:23.015858 18099 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 6: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:23.019676 18167 raft_consensus.cc:614] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:23.019968 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.038071 18096 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:23:23.019666 18017 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:23.041266 22552 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:23.042654 22552 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.043237 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.044005 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.044727 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:23.045789 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.047377 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.010588 22325 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.051540 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.053570 18171 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:23.009575 22613 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:23.060461 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 182841us
I20260709 12:23:23.055568 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.058557 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.062455 18091 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:23:23.059552 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.064291 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:23.065037 18096 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:23.067585 18164 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:23.060038 18167 raft_consensus.cc:2368] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:23.055845 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.070468 18168 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
W20260709 12:23:23.071642 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.171s (SENT)
I20260709 12:23:23.074047 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:23.072332 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.075485 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:23.076828 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.090917 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.091835 18164 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:23.097467 18022 raft_consensus.cc:2368] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:23.079428 22548 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:23.079699 22545 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:23.080452 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:23.082641 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.083182 22485 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:23.083491 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.058027 22595 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:23.100221 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4148555880001368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260709 12:23:23.112082 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4139555880001353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260709 12:23:23.112920 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3869555880000917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260709 12:23:23.113466 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3842555880000873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260709 12:23:23.113914 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3158555879999767s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
W20260709 12:23:23.114348 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2294555880002918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
W20260709 12:23:23.114781 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2267555880002874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
W20260709 12:23:23.115203 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1718555880001986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
W20260709 12:23:23.115501 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1583555880001768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
I20260709 12:23:23.115818 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:23.116853 18099 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 7: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:23.117274 18095 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:23.119598 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:23.115916 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1088555880000968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
I20260709 12:23:23.038070 22538 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.084004 22661 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.083822 22554 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:23.135489 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0692555880000327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
W20260709 12:23:23.143049 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0629555880000225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
I20260709 12:23:23.144050 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:23.152354 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.075668 22611 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:23.152393 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.100466 22578 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:23.160513 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:23.164196 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.164767 18096 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
W20260709 12:23:23.141381 22689 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:23:23.164220 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.181888 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:23.167560 22370 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:23.183990 22543 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:23.184979 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:23.185701 18168 raft_consensus.cc:606] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:23.186084 18168 raft_consensus.cc:614] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:23.186185 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:23.186916 18097 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.187384 18168 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:23.180160 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:23.193423 18173 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:23.139171 22595 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:23.170271 18099 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
W20260709 12:23:23.196957 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 81275.1us
I20260709 12:23:23.198825 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:23.153710 22696 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:23.200919 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:23.205037 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:23.212705 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.214072 18022 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:23.221148 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126610us
I20260709 12:23:23.168560 22643 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:23.223706 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.180193 22554 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:23.188992 22597 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:23.236212 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:23.156452 22661 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.244711 22706 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:23.217232 22627 raft_consensus.cc:2764] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:23:23.247458 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.135313 22613 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.204622 22595 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.238018 22552 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:23.295270 22704 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 12:23:23.304783 22552 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:23.305532 22704 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:23.307641 22704 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:23.228837 22643 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:23.314392 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 115644us
I20260709 12:23:23.310257 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.270233 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.222291 22698 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.251011 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.316707 18022 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 8: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:23.296473 22690 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 12:23:23.252884 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.245687 22696 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:23:23.295475 18099 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:23:23.265077 22597 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:23:23.319247 18023 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:23.259589 22370 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:23.318246 18019 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:23.315471 22661 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:23.226814 22543 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:23.321106 22643 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:23.310331 22595 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:23.310945 22554 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.333338 18019 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:60162
I20260709 12:23:23.412137 18019 raft_consensus.cc:606] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:23.365736 18020 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:23.375352 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.416692 18172 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:23.397984 18090 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:23.333703 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:23.421387 18022 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:23.363041 18164 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:23.367344 18024 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:23.398319 18021 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.7-9.7]
W20260709 12:23:23.391902 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.328086 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.428687 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:23.433959 18023 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:23.438350 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.316896 22704 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:23.448293 22676 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:23.358570 22563 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.448885 22704 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.442830 18022 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:23.453745 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:23.454088 18169 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:23.349272 22597 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:23.454969 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.455363 22597 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:23.455757 18171 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:23.456851 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.456553 22597 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.445699 18020 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:23.445894 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:23.523355 18093 raft_consensus.cc:2368] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10.
I20260709 12:23:23.523938 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 728) took 1262 ms. Trace:
I20260709 12:23:23.524290 18093 rpcz_store.cc:276] 0709 12:23:22.261560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.445544 (+1183984us) service_pool.cc:224] Handling call
0709 12:23:23.523859 (+ 78315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.447376 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.526093 18164 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:23.376739 22572 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:23.411424 22595 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:23:23.451171 22563 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:23.368746 22538 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:23.343225 22325 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.441319 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.458230 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:23.536228 18022 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:23.457759 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.464195 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:23.414908 22486 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:23.457500 18171 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:23.356245 22706 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.411625 22694 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:23:23.414529 22710 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:23:23.511700 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:23.439790 18023 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:23:23.409600 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:23.520115 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.393801 22485 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:23.526646 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:23.539134 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.541075 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:23.542639 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 243654us
I20260709 12:23:23.519075 22543 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.584492 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.521196 22548 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:23.657142 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 409456us
I20260709 12:23:23.803593 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:23.542809 22704 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.637311 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.586306 22597 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:23.609385 22706 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.536672 22552 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.583949 22486 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:23.640174 22708 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:23:23.765332 18167 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:23.583940 22546 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:23.688639 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.023s	sys 0.037s
I20260709 12:23:23.620208 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.522541 22370 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:23.608285 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.843415 18022 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
W20260709 12:23:23.760068 22557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.593s	user 0.009s	sys 0.035s
W20260709 12:23:23.790004 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 278388us
I20260709 12:23:23.787292 22613 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.850689 18173 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:23.791277 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 381780us
I20260709 12:23:23.792949 18017 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:23:23.809208 18169 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:23.810149 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.810201 22720 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.804667 22583 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:23.805088 22485 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.811275 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:23.806221 22543 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.813143 22483 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:23.814751 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:23.816992 22618 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:23.881752 18814 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:23.882314 22618 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.823082 22423 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:23.886132 22706 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:23.886240 22423 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.888840 22423 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:23.827392 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.892987 18168 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:23.894335 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.895720 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:23.896371 18168 raft_consensus.cc:606] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:23.896762 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.897656 18093 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:23.832691 22325 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:23.833246 22582 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.834388 22627 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:23.902424 22627 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:23.903705 22627 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.804833 22554 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:23.906078 22554 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.906585 22554 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:23.907171 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:23.781625 22712 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:23.910259 22712 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.912357 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.913065 22712 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:23.913410 18165 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:23:23.847237 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.327075 22698 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.915155 18164 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:23.916549 18164 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:23.917001 22698 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:23.852104 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:23.855479 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:23.918179 18809 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:23.918361 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:23.920024 18169 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:23.920674 22611 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:23.921166 22611 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.921548 22611 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:23.921685 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.922546 18170 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:23.923933 18170 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 12:23:23.924265 18170 pending_rounds.cc:85] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 6
I20260709 12:23:23.924511 18170 pending_rounds.cc:107] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:23:23.924754 18170 raft_consensus.cc:2889] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:23.925973 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 111521us
I20260709 12:23:23.927748 22370 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.928587 18809 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:23.802603 22439 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.960892 22439 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:23.859076 18806 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:23.860188 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.997236 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:23.942509 18171 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:23.877682 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
W20260709 12:23:23.843323 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.028s	sys 0.043s
I20260709 12:23:23.929941 22693 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:23.883589 22643 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.881407 22483 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:23.849993 22557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.683s	user 0.010s	sys 0.035s
I20260709 12:23:23.936740 18164 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:24.005092 18808 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260709 12:23:24.005541 18808 pending_rounds.cc:85] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 9
I20260709 12:23:23.852757 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:24.005770 18808 pending_rounds.cc:107] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 12:23:24.006130 18808 raft_consensus.cc:2889] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:23.942451 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:23.955006 18165 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:23.959317 22486 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:23.965956 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:24.013628 18092 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:23.854673 22667 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:23.973809 22543 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:23.977283 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:24.022468 18809 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:23.858443 22583 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.859349 22581 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.123s
I20260709 12:23:24.051635 22583 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:23.982790 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 103725us
I20260709 12:23:24.054050 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:23.986629 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 606) took 1707 ms. Trace:
I20260709 12:23:24.054823 18015 rpcz_store.cc:276] 0709 12:23:22.279282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.279415 (+   133us) service_pool.cc:224] Handling call
0709 12:23:22.306104 (+ 26689us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.321493 (+ 15389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.321535 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.322223 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.335092 (+1012869us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b18002aed40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:23.789147 (+454055us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:23.789248 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.789396 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.986255 (+196859us) raft_consensus.cc:1685] finished
0709 12:23:23.986281 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.986544 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144135,"raft.queue_time_us":24222,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":7327104,"thread_start_us":453672,"threads_started":1,"wal-append.queue_time_us":1649559}
I20260709 12:23:23.986958 22293 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:24.056957 22293 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.057464 22293 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:23.986717 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 134716us
I20260709 12:23:24.063467 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:23.863663 22710 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.125s
I20260709 12:23:23.989012 22595 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.066852 22595 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:23.842546 22713 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.989692 22545 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:23.862181 22720 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.070319 22713 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.072549 22720 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:24.072889 22713 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:23.999552 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 925529us
I20260709 12:23:24.070341 22545 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.073849 22704 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:24.074512 22704 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.002324 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:24.004163 18167 pending_rounds.cc:85] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 6
I20260709 12:23:24.017750 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:24.076125 18167 pending_rounds.cc:107] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:23:24.076472 18093 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:24.076581 18167 raft_consensus.cc:2889] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.019992 18168 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:24.025346 22618 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:24.024994 22667 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.081801 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:24.082746 18809 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:24.026404 22439 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:24.090734 22581 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.091468 22581 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.028723 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:24.084131 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:24.093278 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64692.3us
W20260709 12:23:24.094108 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.115s (SENT)
I20260709 12:23:24.094656 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.029089 18172 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:24.032413 18165 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260709 12:23:24.096091 18165 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 3
I20260709 12:23:24.096355 18165 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 12:23:24.096570 18165 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.032822 22485 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:24.042832 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.012180 22693 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.095577 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:24.120728 18092 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:24.122521 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 319047us
W20260709 12:23:24.126237 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62880.1us
I20260709 12:23:24.004732 22483 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:24.128815 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.147214 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:24.169687 18168 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:23:24.170462 18168 pending_rounds.cc:85] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 5
I20260709 12:23:24.170749 18168 pending_rounds.cc:107] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:23:24.171030 18168 raft_consensus.cc:2889] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:24.168547 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44879.2us
I20260709 12:23:24.171706 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.172557 18064 leader_election.cc:400] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:24.172914 18064 leader_election.cc:403] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:23:24.173954 18166 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:23:24.140416 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:24.175356 18018 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:24.137568 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.184525 18171 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:24.185129 18171 raft_consensus.cc:614] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:24.185562 22545 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:24.186275 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2406) took 1139 ms. Trace:
I20260709 12:23:24.186822 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:24.186719 18166 rpcz_store.cc:276] 0709 12:23:23.046865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.047041 (+   176us) service_pool.cc:224] Handling call
0709 12:23:24.186026 (+1138985us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:24.186200 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2587842304}
I20260709 12:23:24.187690 18022 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:24.188241 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:24.189620 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:24.193257 22486 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:24.194110 22691 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:24.195089 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:24.203233 22643 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:24.204818 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:24.136104 22439 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:24.208185 22486 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.118359 22548 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:24.209185 22486 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.209596 22611 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.212291 22611 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:24.126362 22713 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:24.212880 18017 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:24.213268 22713 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.213728 22713 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:24.214927 18015 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:24.217250 22704 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:24.209575 22548 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:24.219802 22548 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.224807 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:24.226082 18022 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:23:24.226894 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.230278 22725 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:24.231829 22725 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:24.232304 22725 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:24.234445 18093 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:24.234556 22483 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.212332 22486 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:24.235941 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.226922 22293 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.239356 22293 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:24.168475 22710 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 12:23:24.244094 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.245910 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.248418 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.262698 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.121783 22598 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.120024 22618 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:24.293311 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:24.291227 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:24.296537 18165 raft_consensus.cc:606] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:24.297137 18165 raft_consensus.cc:614] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:24.301456 18023 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:24.147470 22698 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:24.305490 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 133107us
I20260709 12:23:24.169415 22595 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:24.225613 22693 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:24.235869 22483 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:24.313349 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.338639 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:24.320463 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:24.340914 18023 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:24.341107 18064 leader_election.cc:400] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:24.341147 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:24.341614 18064 leader_election.cc:403] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:23:24.343693 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.345572 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.300717 22723 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.333431 18165 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:24.352965 18165 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:24.347347 22720 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:24.312506 22698 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:24.305469 22710 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:24.323457 22643 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:24.387444 18171 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:24.388459 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2185) took 1005 ms. Trace:
I20260709 12:23:24.391016 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:24.325158 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:24.350756 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:24.366374 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:24.367712 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:24.388901 18171 rpcz_store.cc:276] 0709 12:23:23.382984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.355515 (+972531us) service_pool.cc:224] Handling call
0709 12:23:24.387142 (+ 31627us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.388083 (+   941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.388383 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.326678 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.398825 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.400144 18093 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:24.403782 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.350571 22423 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:23:24.376327 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 378656us
I20260709 12:23:24.306186 22598 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.347729 22543 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:24.350883 22638 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:24.350241 22726 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.414649 18168 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:23:24.374893 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.381992 22698 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:24.416713 22552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.010s	sys 0.010s
I20260709 12:23:24.233686 22725 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.449796 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2178) took 1068 ms. Trace:
W20260709 12:23:24.444540 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 348206us
I20260709 12:23:24.443279 18093 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:23:24.451009 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.119s (SENT)
I20260709 12:23:24.439842 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:24.356899 22712 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 12:23:24.423564 18018 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:24.440714 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:24.412014 22693 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:24.417781 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50219.9us
W20260709 12:23:24.439993 22672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.004s	sys 0.016s
W20260709 12:23:24.453445 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 378325us
I20260709 12:23:24.469159 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:24.448901 22552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.011s	sys 0.010s
W20260709 12:23:24.443930 22613 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.566s	user 0.012s	sys 0.001s
W20260709 12:23:24.458073 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 131546us
I20260709 12:23:24.481060 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:24.482604 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.459430 22595 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:24.460567 22709 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:23:24.443703 22546 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.000s
I20260709 12:23:24.434785 22611 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:24.491427 18090 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.008s	sys 0.006s
W20260709 12:23:24.491940 18090 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.008s	sys 0.006s
I20260709 12:23:24.491173 18169 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:23:24.492707 18169 pending_rounds.cc:85] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 8
I20260709 12:23:24.492987 18169 pending_rounds.cc:107] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:23:24.493221 18169 raft_consensus.cc:2889] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.454278 18093 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:24.465811 18095 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:24.494544 18095 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:24.495661 18095 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.496910 22627 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:24.497584 22627 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:24.479537 22485 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:24.498682 22627 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.464049 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:24.358481 22723 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:24.502871 22723 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:24.463932 22554 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.004s
I20260709 12:23:24.421046 22370 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:23:24.503702 22554 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.504632 18809 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:24.506239 18092 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:24.507541 22554 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:24.458462 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.971s (SENT)
I20260709 12:23:24.455830 22560 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:24.463439 22557 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:24.463083 22693 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.497008 18095 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:24.500188 22627 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:24.510761 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52344.7us
I20260709 12:23:24.477864 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:24.571612 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93857.7us
I20260709 12:23:24.447757 22582 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260709 12:23:24.470724 22672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.004s	sys 0.017s
W20260709 12:23:24.512537 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 163959us
W20260709 12:23:24.516322 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.511698 18094 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:24.574157 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59278.1us
I20260709 12:23:24.477936 22581 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:24.584863 22731 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:23:24.607435 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.591246 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:24.611344 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.611069 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:24.592336 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:24.445874 22726 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.612661 18171 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:24.563438 22694 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:24.585721 22583 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.533s	user 0.014s	sys 0.000s
W20260709 12:23:24.630007 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.570506 22693 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.509680 22725 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:24.574082 22293 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.603092 22552 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.635304 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3286418600005163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2705s for a response
I20260709 12:23:24.652436 22557 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:23:24.491595 22545 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:24.486018 22595 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.651877 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:24.635090 22618 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:24.484885 22736 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.180s
I20260709 12:23:24.657794 18095 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:23:24.658624 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.591293 22546 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:24.490618 22486 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:24.668208 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.663812 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:24.613879 22744 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:24.602486 22572 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:23:24.615216 22485 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:24.675689 18095 pending_rounds.cc:85] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:23:24.710377 18095 pending_rounds.cc:107] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:23:24.710786 18095 raft_consensus.cc:2889] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.637346 22667 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:24.637785 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 173871us
I20260709 12:23:24.670032 22691 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.651582 22543 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:24.613620 22615 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
W20260709 12:23:24.707149 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.707715 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.370s (SENT)
I20260709 12:23:24.708279 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.514016 22737 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.161s
I20260709 12:23:24.652590 22554 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.672629 22545 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.630571 22713 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.653833 22583 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.662885 22293 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.662036 22751 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
I20260709 12:23:24.667660 22552 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.670065 22595 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:24.450385 18164 rpcz_store.cc:276] 0709 12:23:23.380914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.005685 (+624771us) service_pool.cc:224] Handling call
0709 12:23:24.054343 (+ 48658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.054685 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:24.055163 (+   478us) log.cc:844] Serialized 14 byte log entry
0709 12:23:24.447122 (+391959us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.447188 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:24.447249 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.447337 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.449709 (+  2372us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5235,"prepare.queue_time_us":9586,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":6164,"thread_start_us":386228,"threads_started":1,"wal-append.queue_time_us":372148}
I20260709 12:23:24.963054 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:24.933391 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 667) took 1554 ms. Trace:
W20260709 12:23:24.864115 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.714323 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55771.4us
I20260709 12:23:24.734743 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2171) took 1356 ms. Trace:
I20260709 12:23:24.711014 22750 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:24.675441 22581 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.710389 22744 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.968274 22581 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:24.897569 22754 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:23:24.786428 22485 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.906773 22726 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:24.964190 22545 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:24.971065 22741 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.543s	user 0.001s	sys 0.001s
I20260709 12:23:24.971469 22581 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:24.673758 22618 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.972280 22618 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:24.905529 22698 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.972834 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:24.973989 18092 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:24.975265 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 501636us
I20260709 12:23:24.975404 22698 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:24.899004 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 415550us
I20260709 12:23:24.967011 22554 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:24.714766 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51149.4us
I20260709 12:23:24.980162 22757 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:24.980247 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:24.980861 22757 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:24.968688 22744 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:24.909264 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:24.682638 22486 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:24.718194 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.910056 22760 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 12:23:24.994722 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:24.992816 18170 rpcz_store.cc:276] 0709 12:23:23.378217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.935377 (+557160us) service_pool.cc:224] Handling call
0709 12:23:23.935809 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.936071 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.936096 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.943793 (+  7697us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.441529 (+497736us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:24.441571 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.441644 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:24.734384 (+292740us) raft_consensus.cc:1685] finished
0709 12:23:24.734416 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.734645 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":497272,"threads_started":1,"wal-append.queue_time_us":543120}
I20260709 12:23:24.999536 18091 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:25.002794 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.005465 18173 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:24.824769 22752 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:23:24.819691 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:24.881646 22712 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.943753 22739 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.469s	user 0.002s	sys 0.000s
W20260709 12:23:25.008165 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 300474us
I20260709 12:23:24.964387 18169 rpcz_store.cc:276] 0709 12:23:23.375517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.920855 (+545338us) service_pool.cc:224] Handling call
0709 12:23:23.921136 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.493680 (+572544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.929495 (+435815us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:24.929785 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1310008704}
I20260709 12:23:24.963550 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:24.967564 18098 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:25.028025 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.029004 18808 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:25.024895 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:24.826218 22745 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:23:24.820364 22743 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:25.033428 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.034466 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.035713 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:25.038975 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 386521us
I20260709 12:23:25.042631 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:25.006990 22486 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:25.028838 22597 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:25.059576 22597 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.060078 22597 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:25.061134 22572 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:25.004305 22755 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:23:25.062201 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.036477 22743 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:25.066047 22743 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.038115 22557 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:25.066759 22557 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:25.067763 22557 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.071513 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.073465 22757 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.029579 22726 raft_consensus.cc:2764] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:24.718478 22546 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.033413 22712 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.029107 22704 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:25.003460 22693 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:25.042832 22763 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:23:25.078708 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:25.087107 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2177) took 1706 ms. Trace:
I20260709 12:23:25.087616 18167 rpcz_store.cc:276] 0709 12:23:23.380618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.002982 (+622364us) service_pool.cc:224] Handling call
0709 12:23:24.003930 (+   948us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.112423 (+108493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.112459 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.113030 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.454922 (+341892us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:24.454979 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.455067 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.086868 (+631801us) raft_consensus.cc:1685] finished
0709 12:23:25.086893 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.087037 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":269777,"threads_started":1,"wal-append.queue_time_us":367233}
I20260709 12:23:25.091385 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:25.092129 22728 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.667s	user 0.001s	sys 0.001s
I20260709 12:23:25.077762 22767 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.095323 18098 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:25.095592 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92875.5us
I20260709 12:23:25.096947 18064 leader_election.cc:400] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:23:25.097380 18064 leader_election.cc:403] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:23:25.099031 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:25.107111 22693 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:25.109509 22554 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:25.006809 22691 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.111433 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:25.112104 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2315) took 1086 ms. Trace:
I20260709 12:23:25.112534 18097 rpcz_store.cc:276] 0709 12:23:24.026066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.415293 (+389227us) service_pool.cc:224] Handling call
0709 12:23:24.415751 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.475129 (+ 59378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.475183 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.486474 (+ 11291us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.899469 (+412995us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:24.899543 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.899689 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.111856 (+212167us) raft_consensus.cc:1685] finished
0709 12:23:25.111880 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.112026 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108315,"raft.queue_time_us":54050,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":9232,"thread_start_us":313165,"threads_started":1,"wal-append.queue_time_us":335218}
I20260709 12:23:25.114928 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.115733 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:25.116357 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2189) took 1480 ms. Trace:
I20260709 12:23:25.116667 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.116839 18173 rpcz_store.cc:276] 0709 12:23:23.636211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.852420 (+216209us) service_pool.cc:224] Handling call
0709 12:23:25.116113 (+1263693us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b540007c454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:25.116282 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2297644928}
I20260709 12:23:25.081048 22764 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.124742 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.144827 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48095.1us
I20260709 12:23:25.131575 18092 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:25.145802 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.083900 22755 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:25.134920 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.150588 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.153703 18097 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:23:25.155573 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.146593 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:25.159835 18097 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:23:25.160542 18097 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 5
I20260709 12:23:25.160928 18097 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:23:25.161303 18097 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:25.161778 18097 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:23:25.162181 18097 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:25.155455 18098 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:23:25.166335 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134486us
I20260709 12:23:25.168994 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.167603 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:25.147959 22726 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.154394 22751 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:25.160248 22720 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:23:25.182607 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 219150us
I20260709 12:23:25.187148 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:25.193430 18092 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.094318 22767 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.195820 18064 leader_election.cc:400] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:25.178725 22704 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:25.199378 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.203425 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:25.208114 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.210083 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:25.155099 22764 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.226888 18015 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:25.196671 22611 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:25.239660 18092 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:25.222847 22615 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:23:25.332248 22713 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.332417 22720 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:23:25.338711 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 106099us
I20260709 12:23:25.220234 22751 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:25.181571 22761 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:23:25.364934 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:25.366585 18810 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:25.367911 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 180903us
I20260709 12:23:25.369443 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.373807 22767 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:25.375150 22732 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.949s	user 0.004s	sys 0.001s
I20260709 12:23:25.376641 22293 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.365700 22543 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:25.359597 22546 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.378407 22750 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:25.379432 22750 consensus_queue.cc:1243] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:23:25.380029 22750 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:25.380407 22750 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:25.383340 22712 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.383898 22764 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.223289 22753 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:25.386327 22753 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:25.387445 22753 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.392745 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:25.396034 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:25.351307 22583 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.364426 22736 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:25.359889 22770 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.410516 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.377767 22543 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:25.422886 22543 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.424324 22543 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.426661 22743 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.429214 22755 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:25.405231 22485 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:25.431327 22751 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.431957 22751 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.352711 22611 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.421259 18815 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:25.434258 18809 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:25.401085 22764 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:25.440470 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:25.414774 22709 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.419024 22720 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.418591 22572 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:25.456261 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:25.456692 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.401461 22691 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.401319 22769 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:25.460763 18810 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:25.462860 22772 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:23:25.464056 22743 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.446816 22764 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:25.487558 22725 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.827s	user 0.010s	sys 0.005s
I20260709 12:23:25.487922 22764 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:25.441571 22485 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:23:25.381507 22750 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.466593 22483 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:25.468307 22538 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:25.458498 22709 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.408439 22439 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:23:25.454152 22755 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.481020 22769 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:25.455492 22774 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:23:25.486754 18173 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:25.493881 22538 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:25.473682 22720 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.510335 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.493986 22750 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.495638 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.512562 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.498231 18023 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:25.513600 18016 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:23:25.495787 22735 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.072s	user 0.001s	sys 0.002s
W20260709 12:23:25.508273 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 325109us
I20260709 12:23:25.509729 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.493861 18167 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:25.507401 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.491844 22485 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:25.522806 22485 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:25.515251 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.515388 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.519114 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.519883 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.652726 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.520941 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:25.653911 18016 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.646909 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.500959 22439 consensus_queue.cc:1243] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:25.516144 22597 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.648761 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.663046 18021 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:25.529911 22743 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:25.649539 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.649573 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.507131 22538 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:25.651058 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.669989 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.670454 18024 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:25.662902 22725 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.672116 22720 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.671989 22725 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:25.665135 22545 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.694s	user 0.018s	sys 0.000s
I20260709 12:23:25.673523 22545 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.675493 22725 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.670383 18814 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:25.676551 22598 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:25.677227 22744 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.669s	user 0.014s	sys 0.005s
I20260709 12:23:25.677783 22744 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.668473 18173 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:25.682562 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:25.685896 18094 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 731) took 2696 ms (client timeout 1897 ms). Trace:
I20260709 12:23:25.698956 18024 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10. Ops: [4.3-9.4]
I20260709 12:23:25.700112 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.713349 18814 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:25.713188 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.700172 18093 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:25.702220 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.704447 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.718976 18173 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:25.704838 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.706514 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.687382 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2300) took 1744 ms. Trace:
I20260709 12:23:25.721609 18099 rpcz_store.cc:276] 0709 12:23:23.942520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.942659 (+   139us) service_pool.cc:224] Handling call
0709 12:23:23.960425 (+ 17766us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.990404 (+ 29979us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.990438 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.991033 (+   595us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.396992 (+405959us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:24.397036 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.435979 (+ 38943us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:24.436046 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.495051 (+1059005us) spinlock_profiling.cc:243] Waited 65.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:25.687150 (+192099us) raft_consensus.cc:1685] finished
0709 12:23:25.687178 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.687311 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":218194688,"thread_start_us":405522,"threads_started":1,"wal-append.queue_time_us":432648}
I20260709 12:23:25.689780 18806 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:25.651088 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.528892 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.571111 22486 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:25.736502 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 54069.8us
W20260709 12:23:25.711736 18094 rpcz_store.cc:269] 0709 12:23:22.988955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.991653 (+  2698us) service_pool.cc:224] Handling call
0709 12:23:25.685611 (+2693958us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:25.685771 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3444535680}
I20260709 12:23:25.737351 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:25.517926 18169 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:25.657259 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.739539 18172 raft_consensus.cc:2376] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10. Re-sending same reply.
I20260709 12:23:25.658735 18170 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:23:25.665699 18171 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:25.664602 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3139) took 1029 ms. Trace:
I20260709 12:23:25.741312 18164 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:25.729108 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.743027 18165 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:25.744158 18165 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:25.731019 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.745141 18167 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:25.732462 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:25.745973 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.740773 18090 rpcz_store.cc:276] 0709 12:23:24.634785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.121128 (+486343us) service_pool.cc:224] Handling call
0709 12:23:25.121573 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.122348 (+   775us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.122384 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.123016 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.437590 (+314574us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:25.437640 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.437748 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.664342 (+226594us) raft_consensus.cc:1685] finished
0709 12:23:25.664369 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.664532 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":314103,"threads_started":1,"wal-append.queue_time_us":367909}
I20260709 12:23:25.711726 22750 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.646613 22618 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.674s	user 0.011s	sys 0.008s
I20260709 12:23:25.738672 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.752974 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:25.750888 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.734581 18016 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:25.757311 18810 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:25.757859 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:25.667912 22595 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.698s	user 0.007s	sys 0.011s
I20260709 12:23:25.753036 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.683992 22757 raft_consensus.cc:993] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:23:25.754467 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:25.680433 22726 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.501s	user 0.006s	sys 0.007s
I20260709 12:23:25.765071 18814 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:25.765264 22726 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9.
I20260709 12:23:25.766543 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.766845 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:25.767304 18172 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:25.754905 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.767606 22618 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.768144 18170 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:25.768216 22713 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:25.770017 18018 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:25.770165 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.756246 22712 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:25.771229 22712 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.755920 22720 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:25.771656 22712 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:25.771967 22720 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:25.773118 22720 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.752398 22546 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.774660 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.762645 22750 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.776548 22764 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.779037 22764 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.751622 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.761000 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:25.755234 22598 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.691152 22744 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.696684 22751 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.702775 22709 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.764549 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.687628 22545 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.758775 18094 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:25.799413 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.800087 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:25.800783 18094 raft_consensus.cc:2376] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9. Re-sending same reply.
I20260709 12:23:25.767565 18093 raft_consensus.cc:2376] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11. Re-sending same reply.
I20260709 12:23:25.801779 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:25.802657 18094 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20260709 12:23:25.802587 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:25.691547 22766 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.630s	user 0.001s	sys 0.000s
I20260709 12:23:25.779309 22486 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.806937 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.808115 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.779408 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.808998 18164 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:25.809513 18020 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:23:25.779663 18015 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:25.739893 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.811439 18169 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:25.759492 18173 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:25.814209 18094 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:23:25.814894 18094 pending_rounds.cc:85] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:25.791821 22439 raft_consensus.cc:1081] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:23:25.766880 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3150) took 1111 ms. Trace:
I20260709 12:23:25.782737 18171 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:25.818013 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.818617 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.761592 22423 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:23:25.699335 22770 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:25.701956 22597 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.736380 22725 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.792430 22726 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.750506 22769 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.774518 22546 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.829560 22726 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.799628 22598 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:25.831346 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.788s (SENT) [suppressed 35 similar messages]
I20260709 12:23:25.832444 22726 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.832947 22598 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:25.833287 22769 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.834326 22725 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:25.836007 22769 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.800967 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.836865 22725 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.837715 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.838577 18165 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:25.841262 22546 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:25.754058 22485 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.816670 22618 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.843873 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.803234 18093 raft_consensus.cc:2376] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11. Re-sending same reply.
I20260709 12:23:25.805912 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.815378 18094 pending_rounds.cc:107] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:25.848196 18094 raft_consensus.cc:2889] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:25.807879 18807 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:25.851634 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.848323 18166 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:25.854334 18094 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:25.759114 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.816815 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:25.819036 18172 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:25.857283 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98246.4us
I20260709 12:23:25.784512 18170 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:25.678021 22538 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.793298 22750 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:25.782629 22595 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.817970 18097 rpcz_store.cc:276] 0709 12:23:24.655886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.168070 (+512184us) service_pool.cc:224] Handling call
0709 12:23:25.168547 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.173773 (+  5226us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:25.415579 (+241806us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.702502 (+286923us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.702643 (+   141us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.703271 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.703441 (+   170us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:25.703628 (+   187us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.703948 (+   320us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.704027 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.766651 (+ 62624us) raft_consensus.cc:1685] finished
0709 12:23:25.766671 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.766813 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53079,"prepare.queue_time_us":424014,"prepare.run_cpu_time_us":6864,"prepare.run_wall_time_us":120061,"raft.queue_time_us":49790,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":235,"spinlock_wait_cycles":17054464,"thread_start_us":370675,"threads_started":2,"wal-append.queue_time_us":255921}
I20260709 12:23:25.827689 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.857474 18172 pending_rounds.cc:85] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 6
I20260709 12:23:25.865325 18172 pending_rounds.cc:107] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:23:25.865667 18172 raft_consensus.cc:2889] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:25.819505 18171 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:25.858616 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.862705 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.866992 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.867588 18166 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:25.843652 22730 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 1.419s	user 0.005s	sys 0.005s
I20260709 12:23:25.868688 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.873529 18017 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:25.874287 18098 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: [7.8-9.9]
I20260709 12:23:25.870492 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:25.876545 18810 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:23:25.871493 22572 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:25.882561 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.853035 22618 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:25.860567 22726 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.864799 22486 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:25.892859 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:25.894621 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.899402 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:25.904920 18810 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:25.903997 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:25.898031 22538 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.902047 22598 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 12:23:25.918224 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.919003 18024 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:25.920112 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:25.922145 18021 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:25.894447 22618 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.902592 22557 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:25.899178 22572 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:25.927682 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.928436 18165 raft_consensus.cc:2376] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12. Re-sending same reply.
I20260709 12:23:25.923703 18166 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:25.929450 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:25.930012 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:25.930220 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:25.930387 18166 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:25.930958 18021 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:25.925449 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:25.934852 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:25.931020 22720 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.896271 22726 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:25.946460 18165 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:25.946029 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:25.949446 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:25.954815 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:25.957724 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:25.932816 22486 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:25.950233 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:25.932515 22581 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:25.946408 22423 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:23:25.941910 22598 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:25.944419 22725 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:25.945379 22572 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:25.944718 22750 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:23:25.963022 22751 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:25.945748 22560 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:23:25.962857 22543 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:25.963022 22439 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.979866 22725 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:26.001137 22794 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:23:25.962659 22713 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:25.975828 22744 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:23:25.972239 22781 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:25.980053 22750 consensus_queue.cc:1243] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:25.978762 22486 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:27.085554 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002c5d80 after lost signal to thread 16795
I20260709 12:23:27.134433 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 616) took 1935 ms. Trace:
I20260709 12:23:28.169611 22486 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.169850 18022 rpcz_store.cc:276] 0709 12:23:25.199283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.200460 (+  1177us) service_pool.cc:224] Handling call
0709 12:23:25.219920 (+ 19460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.459159 (+239239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.459195 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:25.459242 (+    47us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.459287 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.133994 (+1674707us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540022f954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:27.134083 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.134322 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":549565696}
W20260709 12:23:28.172907 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080034b7e0 after lost signal to thread 17956
I20260709 12:23:26.039088 22439 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.189136 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:27.129223 22572 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:26.050508 22796 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:23:26.070878 22781 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.206449 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2211) took 2394 ms. Trace:
I20260709 12:23:28.207921 22781 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:28.209501 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2210) took 2403 ms. Trace:
I20260709 12:23:25.998826 22751 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:26.034437 22618 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:26.079520 22793 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:23:27.660957 22774 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:27.673520 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.724s	user 0.014s	sys 0.007s
I20260709 12:23:27.675516 22816 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.678514 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2337) took 2183 ms. Trace:
I20260709 12:23:27.725154 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2317) took 3697 ms. Trace:
I20260709 12:23:27.753563 22611 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:26.063105 22712 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:27.795989 22790 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.916s	user 0.032s	sys 0.034s
W20260709 12:23:27.799162 22797 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.823s	user 0.000s	sys 0.009s
I20260709 12:23:27.799551 22485 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:27.808876 22483 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:27.815239 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2207) took 2013 ms. Trace:
I20260709 12:23:28.219645 18095 rpcz_store.cc:276] 0709 12:23:24.027118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.452508 (+425390us) service_pool.cc:224] Handling call
0709 12:23:24.452921 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.711344 (+258423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.724295 (+3012951us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b5400259754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:27.724786 (+   491us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29270784}
I20260709 12:23:27.822399 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2209) took 2014 ms. Trace:
I20260709 12:23:27.822953 22737 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:27.824878 22820 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:27.825268 22753 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.836128 22819 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:27.839911 18165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.829s	user 0.009s	sys 0.009s
W20260709 12:23:28.242671 18165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.232s	user 0.010s	sys 0.009s
I20260709 12:23:28.238468 18167 rpcz_store.cc:276] 0709 12:23:25.801956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.802111 (+   155us) service_pool.cc:224] Handling call
0709 12:23:25.802592 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.803165 (+   573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.803208 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.811172 (+  7964us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.944955 (+133783us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.945007 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.945111 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.814395 (+1869284us) raft_consensus.cc:1685] finished
0709 12:23:27.814457 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.814860 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3424,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":3652,"thread_start_us":115702,"threads_started":1,"wal-append.queue_time_us":129196,"wal-append.run_cpu_time_us":2943,"wal-append.run_wall_time_us":1127194}
I20260709 12:23:27.842340 22821 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:27.845393 22822 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.210430 18173 rpcz_store.cc:276] 0709 12:23:25.806227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.816974 (+ 10747us) service_pool.cc:224] Handling call
0709 12:23:25.817223 (+   249us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.822169 (+  4946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.822210 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.822769 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.962639 (+139870us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:25.977147 (+ 14508us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.978972 (+  1825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.979139 (+   167us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.770252 (+1791113us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b48000cd158. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:28.208576 (+438324us) raft_consensus.cc:1685] finished
0709 12:23:28.208808 (+   232us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.209186 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13105,"prepare.queue_time_us":120523,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":9457,"raft.queue_time_us":8242,"raft.run_cpu_time_us":39003,"raft.run_wall_time_us":1961479,"spinlock_wait_cycles":4704512,"thread_start_us":108532,"threads_started":1,"wal-append.queue_time_us":100018}
I20260709 12:23:28.246134 22822 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.246541 22820 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:28.247066 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2433) took 2274 ms (client timeout 2012 ms). Trace:
W20260709 12:23:28.247707 18173 rpcz_store.cc:269] 0709 12:23:25.972722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.246853 (+2274131us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:28.246982 (+   129us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:27.869266 22709 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:28.238198 22483 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:28.249166 22709 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.238735 18164 rpcz_store.cc:276] 0709 12:23:25.805902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.809964 (+  4062us) service_pool.cc:224] Handling call
0709 12:23:25.822600 (+ 12636us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.824070 (+  1470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.824109 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.824689 (+   580us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.925609 (+100920us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:26.037909 (+112300us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.039341 (+  1432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.163293 (+1123952us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.800176 (+636883us) spinlock_profiling.cc:243] Waited 7.49 ms on lock 0x7b18002551c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:27.817527 (+ 17351us) raft_consensus.cc:1685] finished
0709 12:23:27.818052 (+   525us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.820295 (+  2243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1601,"prepare.queue_time_us":49744,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":8700,"raft.queue_time_us":2023,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":8635,"spinlock_wait_cycles":28470912,"thread_start_us":144264,"threads_started":2,"wal-append.queue_time_us":186935}
I20260709 12:23:28.250643 22483 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.251228 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.252257 22709 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:27.898383 22818 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:28.252719 22821 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:27.904330 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:27.910889 18016 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:27.931216 22546 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:27.942600 18093 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.915s	user 0.004s	sys 0.020s
I20260709 12:23:27.944761 22560 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.947526 22552 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:27.950395 22423 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.969s	user 0.015s	sys 0.028s
I20260709 12:23:27.947455 22827 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:27.974226 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44816.4us
W20260709 12:23:27.977989 18172 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.006s	user 0.005s	sys 0.027s
W20260709 12:23:27.982998 18138 connection.cc:256] RPC call timeout handler was delayed by -1.9936200919996736s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260709 12:23:28.260308 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2242019529999197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
W20260709 12:23:27.983546 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 83884.8us
W20260709 12:23:28.260970 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.997059 22583 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 1.988s
I20260709 12:23:26.026058 22726 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:28.001780 18778 connection.cc:256] RPC call timeout handler was delayed by -2.7440458780001791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
I20260709 12:23:28.008708 22831 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.253468 22818 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.011015 22836 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.013073 22734 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 3.586s	user 0.015s	sys 0.009s
W20260709 12:23:28.274924 18172 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.303s	user 0.008s	sys 0.027s
I20260709 12:23:28.005244 22826 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.018895 22835 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:28.017138 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.032727 18810 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:28.035836 22704 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:28.038985 22824 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:28.045140 18814 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:23:28.063838 22839 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:28.081753 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1046) took 3001 ms (client timeout 2522 ms). Trace:
W20260709 12:23:28.082852 18780 connection.cc:256] RPC call timeout handler was delayed by -1.8288975259997642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
W20260709 12:23:28.307394 18780 connection.cc:256] RPC call timeout handler was delayed by -1.7856975259996943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260709 12:23:28.086290 22842 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:28.099152 22838 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:23:28.095600 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 824) took 3186 ms (client timeout 2099 ms). Trace:
I20260709 12:23:28.110100 22840 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:28.113721 22538 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 2.199s	user 0.022s	sys 0.018s
W20260709 12:23:28.119745 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55226.9us
W20260709 12:23:28.120776 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.133s	user 0.019s	sys 0.038s
W20260709 12:23:28.123700 18171 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.173s	user 0.004s	sys 0.030s
W20260709 12:23:28.123770 22791 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 2.177s	user 0.000s	sys 0.012s
I20260709 12:23:28.134444 22846 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.977150 22581 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.322474 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3151) took 3666 ms. Trace:
I20260709 12:23:28.322256 22846 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.323860 22581 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.323001 18092 rpcz_store.cc:276] 0709 12:23:24.656162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.475958 (+819796us) service_pool.cc:224] Handling call
0709 12:23:25.498847 (+ 22889us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.499138 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.499165 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.506092 (+  6927us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.787945 (+281853us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:25.787988 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.788080 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.794441 (+2006361us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:28.321726 (+527285us) raft_consensus.cc:1685] finished
0709 12:23:28.321755 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.322216 (+   461us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5030272,"thread_start_us":129793,"threads_started":1,"wal-append.queue_time_us":336527}
I20260709 12:23:28.324544 18099 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:23:28.325178 22846 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.325306 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:28.143115 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2437) took 2113 ms (client timeout 1595 ms). Trace:
I20260709 12:23:28.146663 22595 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:28.162412 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.342042 22835 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.344148 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:26.126325 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3138) took 1491 ms. Trace:
I20260709 12:23:28.177686 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.178710 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.337945 22821 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.346202 18094 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:28.342093 18168 rpcz_store.cc:269] 0709 12:23:26.029902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.142664 (+2112762us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:28.142876 (+   212us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:28.179492 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.348630 18168 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:28.349422 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2432) took 2377 ms. Trace:
I20260709 12:23:28.349478 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.345736 18091 rpcz_store.cc:276] 0709 12:23:24.634450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.635038 (+   588us) service_pool.cc:224] Handling call
0709 12:23:24.635475 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.005707 (+370232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.125884 (+1120177us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54002977d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:26.126214 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":836478336}
I20260709 12:23:28.350368 22486 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.030694 22543 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:28.351087 18091 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45032
I20260709 12:23:28.351858 22543 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.351256 22486 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.352661 22820 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.192116 22439 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.338912 22840 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.207430 18169 rpcz_store.cc:276] 0709 12:23:25.812017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.814494 (+  2477us) service_pool.cc:224] Handling call
0709 12:23:25.814922 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.826994 (+ 12072us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.827051 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:28.132234 (+2305183us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b1800465280. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:23:28.133389 (+  1155us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b2c001268f8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:28.134481 (+  1092us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.150459 (+ 15978us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:28.151429 (+   970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.157833 (+  6404us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.166425 (+  8592us) raft_consensus.cc:1685] finished
0709 12:23:28.166457 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.206166 (+ 39709us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8417,"raft.queue_time_us":8600,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":3232,"spinlock_wait_cycles":37772032,"thread_start_us":14278,"threads_started":1,"wal-append.queue_time_us":16319}
I20260709 12:23:28.214984 22751 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:27.661178 17988 connection.cc:256] RPC call timeout handler was delayed by -1.1780783200001679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
W20260709 12:23:28.358047 17988 connection.cc:256] RPC call timeout handler was delayed by -1.0709783200004495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1791s for a response
I20260709 12:23:28.357944 22835 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:28.359405 22817 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.360370 22817 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.231326 22833 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:25.996124 22598 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:28.346210 18019 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:28.363806 22598 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.231866 22816 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.365620 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.367579 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.368059 22816 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.219139 18023 rpcz_store.cc:276] 0709 12:23:25.494485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.494676 (+   191us) service_pool.cc:224] Handling call
0709 12:23:25.495072 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.513521 (+ 18449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.677042 (+2163521us) spinlock_profiling.cc:243] Waited 2.91 ms on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:27.677688 (+   646us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6697856}
I20260709 12:23:28.233017 18097 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:28.368705 18019 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:28.234171 22774 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.369467 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:28.370193 22726 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:28.319792 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 681) took 2364 ms. Trace:
I20260709 12:23:28.371068 22726 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.371672 22774 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.372097 22840 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:28.235633 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.287s	user 0.017s	sys 0.007s
I20260709 12:23:28.373950 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3157) took 3351 ms. Trace:
I20260709 12:23:28.253299 18164 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:28.374388 18090 rpcz_store.cc:276] 0709 12:23:25.022075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.821618 (+799543us) service_pool.cc:224] Handling call
0709 12:23:25.844582 (+ 22964us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.845003 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.845054 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.845584 (+   530us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.900356 (+ 54772us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:25.901921 (+  1565us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.902195 (+   274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.902317 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.797684 (+1895367us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:27.930469 (+132785us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:28.373354 (+442885us) raft_consensus.cc:1685] finished
0709 12:23:28.373380 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.373703 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":998,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":1910,"raft.run_cpu_time_us":16650,"raft.run_wall_time_us":2080259,"spinlock_wait_cycles":15003008,"thread_start_us":54178,"threads_started":1,"wal-append.queue_time_us":94160}
I20260709 12:23:28.376466 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2436) took 2402 ms. Trace:
W20260709 12:23:28.376971 18780 leader_election.cc:336] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.849s (SENT)
I20260709 12:23:27.915583 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2336) took 2425 ms. Trace:
I20260709 12:23:27.847088 22823 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.377640 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.378067 22823 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.378805 22765 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:28.378993 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2335) took 2895 ms. Trace:
I20260709 12:23:28.379477 18015 rpcz_store.cc:276] 0709 12:23:25.483633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.484922 (+  1289us) service_pool.cc:224] Handling call
0709 12:23:25.485474 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.810667 (+325193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.378458 (+2567791us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400251a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:28.378796 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":676038016}
W20260709 12:23:28.380658 18780 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.897s (SENT)
I20260709 12:23:28.381037 22823 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.252152 22483 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:26.039774 22725 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.351127 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:25.975908 22720 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.274372 22850 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.386497 22850 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.377651 18018 rpcz_store.cc:276] 0709 12:23:25.490360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.490526 (+   166us) service_pool.cc:224] Handling call
0709 12:23:25.490923 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.770448 (+279525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.885581 (+2115133us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54003485d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:27.915304 (+ 29723us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641382656}
I20260709 12:23:28.388144 22720 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.246184 22737 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.283619 22560 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.284919 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:28.291735 22827 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.292950 22818 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.277963 22704 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:28.405614 22720 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:28.407032 22704 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.408712 22704 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:28.296314 22712 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:28.305909 18778 connection.cc:256] RPC call timeout handler was delayed by -2.6717237830002887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260709 12:23:28.276670 22423 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.297s	user 0.016s	sys 0.029s
W20260709 12:23:28.307027 18806 rpcz_store.cc:269] 0709 12:23:25.080259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.082436 (+  2177us) service_pool.cc:224] Handling call
0709 12:23:28.080835 (+2998399us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54005f5854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:28.081305 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1356415232}
W20260709 12:23:28.297544 17989 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.858s (SENT)
I20260709 12:23:28.413136 22850 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.327337 22854 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.399338 22820 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 12:23:28.414227 22854 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.328023 22842 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.416051 22737 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.416831 18021 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:28.140902 22841 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.418324 22841 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.420992 22841 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:28.421218 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080033ff00 after lost signal to thread 18048
I20260709 12:23:28.411821 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.422835 18808 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 5.
W20260709 12:23:28.423156 17953 debug-util.cc:398] Leaking SignalData structure 0x7b0800058dc0 after lost signal to thread 18123
I20260709 12:23:28.413561 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:28.424835 17953 debug-util.cc:398] Leaking SignalData structure 0x7b0800331f60 after lost signal to thread 18197
W20260709 12:23:28.425551 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128100us
I20260709 12:23:28.332297 22552 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:28.332919 22781 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.274739 22546 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.333899 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:28.313596 18809 rpcz_store.cc:269] 0709 12:23:24.908709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.908892 (+   183us) service_pool.cc:224] Handling call
0709 12:23:28.095027 (+3186135us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b54005ad554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:28.095327 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":945393536}
I20260709 12:23:28.334574 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:28.334952 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.335939 22826 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.338333 18173 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:28.338718 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:28.338867 22836 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.339973 22831 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.340991 22824 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.277884 22822 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.345299 18024 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:28.370334 18023 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.329092 22839 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.245536 22819 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.247841 22753 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:28.246984 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2435) took 2273 ms (client timeout 1749 ms). Trace:
I20260709 12:23:28.376895 18164 rpcz_store.cc:276] 0709 12:23:25.973612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.250573 (+2276961us) service_pool.cc:224] Handling call
0709 12:23:28.376365 (+125792us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:28.261590 18093 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.234s	user 0.004s	sys 0.020s
I20260709 12:23:28.357885 18168 rpcz_store.cc:276] 0709 12:23:25.972356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.347532 (+2375176us) service_pool.cc:224] Handling call
0709 12:23:28.348170 (+   638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:28.349065 (+   895us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.349326 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:28.373584 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 617) took 2458 ms. Trace:
W20260709 12:23:28.454205 18167 rpcz_store.cc:269] 0709 12:23:25.973318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.246708 (+2273390us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:28.246888 (+   180us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:28.398175 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.419s	user 0.021s	sys 0.038s
I20260709 12:23:28.398792 18016 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:28.400899 18171 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.450s	user 0.005s	sys 0.031s
I20260709 12:23:28.404134 22827 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.405182 18018 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 12:23:28.411803 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 3.222s (SENT)
I20260709 12:23:28.397502 22560 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:28.417486 22842 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.432058 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:28.437954 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2217) took 2465 ms. Trace:
W20260709 12:23:28.443476 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.441596 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:28.442109 17989 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.931s (SENT)
I20260709 12:23:28.453246 18814 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
W20260709 12:23:28.457546 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 160505us
I20260709 12:23:28.450414 18810 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:28.442510 22841 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:28.371111 18165 rpcz_store.cc:276] 0709 12:23:25.952681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.952812 (+   131us) service_pool.cc:224] Handling call
0709 12:23:25.953152 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.970527 (+ 17375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.970568 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.971191 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.243740 (+2272549us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:28.244544 (+   804us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.278476 (+ 33932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.305886 (+ 27410us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.305951 (+    65us) raft_consensus.cc:1685] finished
0709 12:23:28.305969 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.316479 (+ 10510us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16870,"prepare.queue_time_us":283,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":18450,"raft.run_cpu_time_us":494,"raft.run_wall_time_us":4284,"thread_start_us":2263655,"threads_started":2,"wal-append.queue_time_us":140842,"wal-append.run_cpu_time_us":2427,"wal-append.run_wall_time_us":1002430}
I20260709 12:23:28.463804 18173 rpcz_store.cc:276] 0709 12:23:25.967987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.337190 (+2369203us) service_pool.cc:224] Handling call
0709 12:23:28.337854 (+   664us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:28.433241 (+ 95387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.433545 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:28.467268 18091 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:28.469201 18091 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:28.455693 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 2.48664e+06us
I20260709 12:23:28.464586 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.474574 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:28.475927 18169 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:28.478955 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 676) took 2560 ms. Trace:
W20260709 12:23:28.481226 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08003c2a40 after lost signal to thread 18839
I20260709 12:23:28.436424 22751 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.461123 22822 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:28.488586 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52633.9us
I20260709 12:23:28.489396 22737 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:28.490634 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78967.9us
I20260709 12:23:28.487727 22842 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:28.479519 18171 rpcz_store.cc:276] 0709 12:23:25.918903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.921033 (+  2130us) service_pool.cc:224] Handling call
0709 12:23:25.921436 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.926482 (+  5046us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.926552 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.936712 (+ 10160us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.478361 (+2541649us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:28.478478 (+   117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.478579 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.478636 (+    57us) raft_consensus.cc:1685] finished
0709 12:23:28.478649 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.478881 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4539,"raft.queue_time_us":5732,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":10801,"thread_start_us":2505779,"threads_started":1,"wal-append.queue_time_us":165579,"wal-append.run_cpu_time_us":2791,"wal-append.run_wall_time_us":1006443}
I20260709 12:23:28.499684 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.501021 18807 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:28.502890 22817 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.505630 22824 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.444394 22781 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.507228 18815 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.507593 22751 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.508628 22781 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:28.510155 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79505.7us
I20260709 12:23:28.511409 22546 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.511911 22831 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:28.512118 22546 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:28.512689 22546 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:28.513290 22546 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:28.513067 22842 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.515833 22842 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.516459 22842 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.517077 22842 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.518323 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.454726 18017 rpcz_store.cc:276] 0709 12:23:25.915453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.916716 (+  1263us) service_pool.cc:224] Handling call
0709 12:23:25.917046 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.917295 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:25.948162 (+ 30867us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.373000 (+2424838us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:28.373118 (+   118us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:28.373240 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.373360 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.373509 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6145,"prepare.queue_time_us":104970,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":21149,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":5290,"thread_start_us":2435485,"threads_started":2,"wal-append.queue_time_us":148607,"wal-append.run_cpu_time_us":1599,"wal-append.run_wall_time_us":1010035}
I20260709 12:23:28.519181 18095 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:28.519243 18815 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:28.473865 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41710
I20260709 12:23:28.520790 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.521062 22819 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.459836 22543 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:28.522089 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.522760 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:28.523571 22543 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.524564 18096 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:28.524453 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.525707 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.521227 18092 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:28.526535 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.530839 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.531879 22861 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.532233 18807 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:23:28.527877 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:28.533106 18092 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.521114 18167 raft_consensus.cc:606] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Received request to transfer leadership
I20260709 12:23:28.518047 22820 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.536512 18778 connection.cc:256] RPC call timeout handler was delayed by -2.4955717759999061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23260000000000003s for a response
I20260709 12:23:28.534785 18167 raft_consensus.cc:614] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:28.485680 22726 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:28.545271 18780 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.097s (SENT)
I20260709 12:23:28.443717 22552 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.490505 22753 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:28.548055 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080021f880 after lost signal to thread 22701
I20260709 12:23:28.546635 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2434) took 2565 ms. Trace:
W20260709 12:23:28.550858 17953 debug-util.cc:398] Leaking SignalData structure 0x7b08002551a0 after lost signal to thread 22736
I20260709 12:23:28.550742 18169 rpcz_store.cc:276] 0709 12:23:25.973009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.354679 (+2381670us) service_pool.cc:224] Handling call
0709 12:23:28.355283 (+   604us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:28.481488 (+126205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.538134 (+ 56646us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:28.538385 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":257702272}
W20260709 12:23:28.553148 18778 connection.cc:256] RPC call timeout handler was delayed by -2.3801237830002719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
W20260709 12:23:28.553692 18778 connection.cc:256] RPC call timeout handler was delayed by -1.7625171960003172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
W20260709 12:23:28.554045 18778 connection.cc:256] RPC call timeout handler was delayed by -1.7445171960002881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
W20260709 12:23:28.554533 18778 connection.cc:256] RPC call timeout handler was delayed by -1.2133145330003572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
I20260709 12:23:28.554296 22548 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:28.555152 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.556391 22548 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:28.556677 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:28.557330 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 261987us
I20260709 12:23:28.557688 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 675) took 2577 ms. Trace:
I20260709 12:23:28.558171 18170 rpcz_store.cc:276] 0709 12:23:25.918564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.919660 (+  1096us) service_pool.cc:224] Handling call
0709 12:23:25.919979 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.920216 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:25.920601 (+   385us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.396279 (+2475678us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b1800378280. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:23:28.397220 (+   941us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.455973 (+ 58753us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:28.462902 (+  6929us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:28.487083 (+ 24181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.494114 (+  7031us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.496169 (+  2055us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20003,"prepare.queue_time_us":4261,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":134,"raft.queue_time_us":4086,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":5656,"spinlock_wait_cycles":14510464,"thread_start_us":82828,"threads_started":2,"wal-append.queue_time_us":60798,"wal-append.run_cpu_time_us":1530,"wal-append.run_wall_time_us":1014170}
I20260709 12:23:28.562209 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.562765 22836 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.563386 18810 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:28.565153 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.566154 22560 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:28.566582 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:28.567662 22753 raft_consensus.cc:2764] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:23:28.568710 22753 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.570603 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:28.580107 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.537084 22765 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:28.586957 22839 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:28.588176 22865 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.079s	user 0.002s	sys 0.000s
I20260709 12:23:28.591501 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 619) took 2675 ms. Trace:
I20260709 12:23:26.746934 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:28.550978 22861 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.594743 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.595724 18094 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:28.596872 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.598147 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.599488 22486 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:28.600122 17989 leader_election.cc:336] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.971s (SENT)
I20260709 12:23:28.606353 18023 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:28.608779 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:28.609776 18169 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:28.612809 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.606393 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.587996 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:28.616315 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.616425 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:28.617424 22839 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:28.617883 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.618551 22839 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:28.618767 18815 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:28.619971 22726 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:28.620865 22713 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:28.620985 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64522.8us
I20260709 12:23:28.621519 22820 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:28.598183 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 679) took 2665 ms (client timeout 2063 ms). Trace:
I20260709 12:23:28.626708 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:28.569793 22855 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.218s	user 0.005s	sys 0.000s
I20260709 12:23:28.592063 18020 rpcz_store.cc:276] 0709 12:23:25.916195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.916378 (+   183us) service_pool.cc:224] Handling call
0709 12:23:25.916835 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.917313 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:25.982429 (+ 65116us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.484129 (+2501700us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:28.565000 (+ 80871us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.565278 (+   278us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:28.565327 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.565478 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.591076 (+ 25598us) raft_consensus.cc:1685] finished
0709 12:23:28.591103 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.591299 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":58367,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":19012,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":14689,"thread_start_us":114112,"threads_started":2,"wal-append.queue_time_us":2473558}
I20260709 12:23:28.597077 22861 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.633700 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.639294 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:28.640121 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.650250 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:28.634187 22826 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.635102 18810 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:28.661078 18170 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:28.629390 18091 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:28.642736 18166 rpcz_store.cc:269] 0709 12:23:25.932267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.949924 (+ 17657us) service_pool.cc:224] Handling call
0709 12:23:28.549225 (+2599301us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:28.597923 (+ 48698us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5341871872}
I20260709 12:23:28.662473 18095 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:23:28.652709 18171 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:28.664676 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.665241 18171 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:28.667661 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.667027 18016 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:28.667819 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2213) took 2844 ms. Trace:
I20260709 12:23:28.673611 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.674729 18169 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:28.632884 22713 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:28.677016 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:28.677882 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79857.4us
I20260709 12:23:28.652956 22751 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:28.652734 22765 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:28.682192 18168 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 6
I20260709 12:23:28.680096 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:28.680003 18807 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:23:28.644248 22548 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260709 12:23:28.683149 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:28.690382 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56811.8us
I20260709 12:23:28.693941 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:28.690660 17989 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.244s (SENT)
I20260709 12:23:28.634542 22753 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.682544 18168 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:23:28.698202 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:28.634280 22822 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.700079 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:28.699633 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 745) took 2992 ms. Trace:
I20260709 12:23:28.696321 18168 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:28.699501 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.673122 18172 rpcz_store.cc:276] 0709 12:23:25.823139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.856651 (+ 33512us) service_pool.cc:224] Handling call
0709 12:23:25.857169 (+   518us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:25.899675 (+ 42506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.899708 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:25.900387 (+   679us) log.cc:844] Serialized 52 byte log entry
0709 12:23:25.928786 (+ 28399us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:28.282546 (+2353760us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.498715 (+216169us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.628919 (+130204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.645077 (+ 16158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.645140 (+    63us) raft_consensus.cc:1685] finished
0709 12:23:28.645160 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.667742 (+ 22582us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46021,"prepare.queue_time_us":134317,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":124,"raft.queue_time_us":83555,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":20780,"thread_start_us":2493850,"threads_started":3,"wal-append.queue_time_us":415335,"wal-append.run_cpu_time_us":3226,"wal-append.run_wall_time_us":1042101}
I20260709 12:23:28.681589 22751 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.709522 22751 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.710511 22725 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.705565 22583 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:28.714145 18097 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:28.716014 22726 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:28.716643 22726 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:28.717758 18166 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:28.717628 22546 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.718550 22597 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:23:28.719945 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 180307us
I20260709 12:23:28.722198 22744 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:28.724509 22744 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.724820 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.721390 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:28.726832 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:28.724895 22744 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.730885 18172 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:23:28.731383 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.732159 18165 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:28.732405 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:28.732784 22854 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.730916 22820 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:28.632560 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:28.742626 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 110561us
I20260709 12:23:28.726997 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:28.633540 22839 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.697240 22861 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:28.708220 22552 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:28.735706 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53612.1us
I20260709 12:23:28.702559 18093 rpcz_store.cc:276] 0709 12:23:25.706888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.848460 (+141572us) service_pool.cc:224] Handling call
0709 12:23:25.849005 (+   545us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:25.849598 (+   593us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.849644 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:25.850541 (+   897us) log.cc:844] Serialized 52 byte log entry
0709 12:23:25.980588 (+130047us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.026834 (+ 46246us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.491231 (+2464397us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.679664 (+188433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.682193 (+  2529us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.690977 (+  8784us) raft_consensus.cc:1685] finished
0709 12:23:28.690997 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.699565 (+  8568us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":55208,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":107,"raft.queue_time_us":98679,"raft.run_cpu_time_us":525,"raft.run_wall_time_us":24926,"thread_start_us":2685782,"threads_started":3,"wal-append.queue_time_us":349681,"wal-append.run_cpu_time_us":3741,"wal-append.run_wall_time_us":1085117}
I20260709 12:23:28.742626 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.743536 18807 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:28.733454 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.704599 18171 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.754841 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.756445 18166 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:28.743875 22546 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:28.762194 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80382.4us
I20260709 12:23:28.753870 22765 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:28.766355 22765 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:28.767679 22753 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.768427 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.768751 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:28.769299 18093 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:28.769783 22485 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:28.770076 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:28.770323 22485 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:28.770823 22485 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:28.770941 22713 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.772753 22713 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.779778 18806 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:28.780244 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:28.786888 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:28.745884 22861 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.793383 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.810782 18094 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:28.811522 18167 raft_consensus.cc:2368] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:28.812381 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:28.813758 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.783847 22483 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:28.821249 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:28.791544 22737 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:28.804497 22861 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:28.828063 18808 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:28.828526 18171 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:28.824134 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:28.803925 22725 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.754218 22854 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 12:23:28.808830 22552 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:23:28.837221 22618 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.837965 22557 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:23:28.840436 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:28.840678 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.850462 18170 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:28.832908 18092 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:28.846587 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.851634 18093 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:28.847765 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.852470 18097 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:28.825398 22774 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.820942 22483 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:28.865375 18023 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 12:23:28.865499 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144277us
I20260709 12:23:28.823716 22842 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.858371 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:28.870601 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52719us
I20260709 12:23:28.868422 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:28.872527 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:28.853120 22830 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:28.874425 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 107631us
I20260709 12:23:28.864990 22618 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:28.878350 22774 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.879781 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:28.880688 18810 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:28.880918 22618 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:28.887321 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:28.872078 18020 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:28.767793 22765 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:28.858933 22817 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:28.944043 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.871234 22483 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:28.949084 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57085.4us
I20260709 12:23:28.891094 18173 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:28.928207 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.887727 18094 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:28.951316 18092 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:28.946863 18097 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:23:28.893273 18021 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:28.881654 22852 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.139s
W20260709 12:23:28.951077 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66042.6us
I20260709 12:23:28.956302 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:28.883687 22822 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:28.853251 22816 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:23:28.896039 22842 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:28.996276 18807 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
W20260709 12:23:28.991310 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47212us
I20260709 12:23:28.993130 22781 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:29.005795 22572 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.006656 22572 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.009248 22572 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:28.958804 22827 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:23:28.947436 22817 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:29.031764 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:29.039019 22820 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:29.039759 22820 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:29.046087 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:29.047019 18098 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:29.048827 18098 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:29.053048 22725 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.864627 22864 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.194s
I20260709 12:23:29.053869 22725 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.056447 22852 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:29.056967 22852 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:29.058228 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.063099 22854 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:29.064400 22854 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:29.064821 22854 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:29.066550 18091 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:29.070300 22827 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:29.070830 22827 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:29.071812 22827 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.073318 22827 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.056488 22725 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:29.078276 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 171684us
I20260709 12:23:29.079500 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.080653 22597 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:29.082402 22852 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.084059 22852 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:29.087507 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48521.1us
I20260709 12:23:29.087633 22572 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:29.088543 22572 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.033243 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:29.114449 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 81308.8us
I20260709 12:23:29.115585 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:29.121678 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174115us
I20260709 12:23:29.122057 22864 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:29.122852 22864 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:29.123100 22737 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:29.123414 22864 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.123638 22737 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.124161 22737 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.127077 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:29.129957 22830 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:29.130776 22830 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:29.132098 22830 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.137707 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.103808 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.139763 18020 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:29.105337 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:29.106669 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.106915 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:29.142179 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.108215 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:29.142949 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.108505 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.143688 22597 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:29.144146 18015 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:29.053355 22485 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.088557 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.109506 22854 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:29.146422 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57987.3us
I20260709 12:23:29.146600 22854 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.148006 22583 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:29.148661 22583 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.149156 22583 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.041149 22820 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.149309 22840 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:29.150287 22840 consensus_queue.cc:1243] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260709 12:23:29.098225 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65032.9us
I20260709 12:23:29.155129 22485 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:29.156893 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.161760 22751 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.165735 18098 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:29.166494 22751 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:29.172339 18095 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:29.173077 22572 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:29.174069 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47170.7us
I20260709 12:23:29.046197 18173 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:29.176952 18171 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 12:23:29.177434 18171 pending_rounds.cc:85] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 11
I20260709 12:23:29.179157 18808 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:29.169075 22744 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.000557 22842 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:29.184551 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:29.196290 18171 pending_rounds.cc:107] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:23:29.196743 18171 raft_consensus.cc:2889] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:29.193887 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.201201 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.177786 22817 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.202508 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.204950 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.206266 18808 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:29.206437 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.207475 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.208757 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.209491 18810 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.213099 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.180442 22557 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:29.221768 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:29.228718 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50391.5us
I20260709 12:23:29.229012 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.229912 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.232710 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.221423 18018 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:29.189651 22485 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.199175 22842 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.231735 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.238385 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.238677 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:29.239987 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.240000 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.240443 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.250790 18172 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:29.252306 18169 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:29.247781 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:29.231164 22560 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:23:29.251610 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.256932 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.264802 18024 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:29.262243 18022 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:29.266122 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:29.260689 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.273960 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.274487 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.275863 18017 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:23:29.275647 22548 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:29.275918 18167 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:29.276927 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.278152 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.278865 18015 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:29.280922 18099 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:29.274959 18166 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:29.283180 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.284106 18172 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:29.285451 18023 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:29.267299 18165 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:29.244243 22835 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:29.294523 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.297513 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.307240 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.314610 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:29.322432 18806 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:29.326287 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:29.332461 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:29.333843 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.312143 22726 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.595s	user 0.017s	sys 0.000s
I20260709 12:23:29.348655 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.352139 18023 raft_consensus.cc:2368] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:29.298331 22557 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:29.362810 18172 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:29.327432 18165 raft_consensus.cc:2376] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16. Re-sending same reply.
I20260709 12:23:29.303445 22572 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:29.355188 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:29.357007 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:29.303544 22835 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:29.268899 22877 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:23:29.384493 18809 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:29.333092 22598 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:29.393579 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:29.398505 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 197360us
I20260709 12:23:29.312518 22861 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.415791 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.367580 22572 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.426415 22572 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:29.422286 22861 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:29.423815 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.435060 18808 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:29.406738 22839 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.016s	sys 0.023s
W20260709 12:23:29.425784 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43851us
W20260709 12:23:29.425318 22875 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.468s	user 0.000s	sys 0.002s
I20260709 12:23:29.403074 22845 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:29.286902 22485 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.367375 22483 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.371438 22726 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.448673 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:29.456840 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99914.8us
I20260709 12:23:29.433625 22552 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:29.462975 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:29.465246 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:29.442009 22845 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:29.440907 22839 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.017s	sys 0.023s
I20260709 12:23:29.482055 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.482950 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:29.486621 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.460182 22744 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:29.485683 18020 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:29.496353 18812 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:23:29.499370 22483 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:29.503214 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.507478 22485 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:29.472309 22737 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.509977 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.466962 22552 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:29.512861 22822 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:29.513253 22552 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.514873 22861 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:29.490458 22713 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.717s	user 0.019s	sys 0.000s
I20260709 12:23:29.515791 22861 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:29.516609 22861 consensus_queue.cc:1243] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:29.515383 22822 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:29.517155 22839 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:29.517575 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.518028 22839 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:29.519107 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.519529 22839 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.520484 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.521633 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.522346 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.522486 22861 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:29.522949 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.523185 22861 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:29.523675 18807 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:29.524682 22861 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.521493 22839 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:29.526260 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.528229 18094 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:29.474133 22845 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:29.530020 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.530821 22861 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.521690 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.513305 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.545392 18808 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:29.546563 18809 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:29.548180 18814 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:29.472370 22598 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:29.462777 22548 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.523104 18813 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:29.553512 22822 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:29.555225 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.556034 18815 raft_consensus.cc:2376] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11. Re-sending same reply.
I20260709 12:23:29.547689 22751 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:29.514899 22552 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:29.558333 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.559197 18811 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:29.562390 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.563997 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:29.564836 18023 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:29.570209 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:29.581440 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68237.7us
I20260709 12:23:29.581707 18815 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:29.570365 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.583037 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.583405 18023 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:29.585031 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:29.586726 18810 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:29.567530 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.587721 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:29.589761 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:29.591634 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.592458 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.593950 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.597663 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.599355 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.600186 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.601063 18811 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:29.602974 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.605188 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:29.607729 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.591801 22824 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:29.566956 22557 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:29.551244 22598 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.568410 22726 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:29.619830 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:29.623279 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.628646 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.626474 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.602697 22822 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:29.570650 22864 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.568145 22817 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.619413 22598 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:29.565618 22572 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.651327 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.646685 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.664767 18808 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:23:29.585705 22548 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.666378 18808 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:29.666908 18808 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:29.668334 22548 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:29.619856 22713 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:29.614889 22818 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:29.620549 22557 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:29.650113 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:29.670156 22713 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.670397 22557 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.670895 22713 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.668418 18808 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.672099 18808 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:29.594458 22583 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.548290 22835 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:29.698268 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:29.699373 18811 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:29.644927 22485 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:29.709209 18018 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:29.714560 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:29.663429 22822 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:29.723181 18815 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:29.711036 22485 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:29.739377 18814 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260709 12:23:29.741482 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.748093 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.721050 22840 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.751929 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 123153us
W20260709 12:23:29.754453 22827 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.593s	user 0.013s	sys 0.003s
I20260709 12:23:29.756347 22583 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:29.760749 18806 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:29.739503 22744 raft_consensus.cc:2764] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:23:29.743616 22485 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:29.778087 22694 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.010s	sys 0.013s
W20260709 12:23:29.778672 22888 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:29.795145 22864 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:29.804064 22572 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:29.805687 22817 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:29.752245 22704 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:23:29.806758 22822 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:29.821727 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68519.4us
W20260709 12:23:29.812183 22694 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.133s	user 0.011s	sys 0.013s
W20260709 12:23:29.836090 18092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.018s	sys 0.035s
W20260709 12:23:29.847889 18092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.019s	sys 0.036s
I20260709 12:23:29.825342 18808 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
I20260709 12:23:29.831799 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:29.765331 22840 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.854525 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2340) took 1286 ms. Trace:
I20260709 12:23:29.855615 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:29.855276 18092 rpcz_store.cc:276] 0709 12:23:28.568330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.029879 (+461549us) service_pool.cc:224] Handling call
0709 12:23:29.181260 (+151381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.188141 (+  6881us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:29.248056 (+ 59915us) log.cc:844] Serialized 14 byte log entry
0709 12:23:29.850485 (+602429us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:29.850587 (+   102us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:29.854074 (+  3487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.854202 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.854439 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":64625,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":62,"thread_start_us":539354,"threads_started":1}
I20260709 12:23:29.827780 22820 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:29.870239 18096 pending_rounds.cc:85] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 2
I20260709 12:23:29.894284 18096 pending_rounds.cc:107] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:23:29.894644 18096 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:29.895002 18096 pending_rounds.cc:107] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 9.4
I20260709 12:23:29.895247 18096 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:29.885475 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 420304us
I20260709 12:23:29.886735 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:29.890632 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 218070us
I20260709 12:23:29.857793 22817 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:29.861861 22836 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:29.899252 22852 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.815s	user 0.016s	sys 0.002s
I20260709 12:23:29.854444 22845 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:29.911206 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 303547us
I20260709 12:23:29.888281 22901 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:29.920186 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64675.2us
I20260709 12:23:29.901482 22817 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.921947 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:29.914633 22842 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.005s
I20260709 12:23:29.925486 18021 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:23:29.930274 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:29.923086 22817 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:29.938566 22894 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:29.961297 18095 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:29.939694 22840 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:29.941478 22845 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:29.923796 22901 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:29.989637 22901 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.947142 22713 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:29.947985 22827 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:29.994087 22827 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:29.994625 22827 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:29.997037 22737 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:29.948844 18810 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:29.954363 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:29.955544 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:29.974321 22900 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.910025 22864 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:29.978418 18024 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:29.981741 18064 leader_election.cc:400] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:30.005861 18064 leader_election.cc:403] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:29.984591 18093 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
W20260709 12:23:30.006215 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50774.9us
W20260709 12:23:30.007016 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83730.8us
W20260709 12:23:29.984894 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 254488us
I20260709 12:23:29.911700 22572 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:23:29.982519 22842 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:30.011538 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57278.7us
I20260709 12:23:30.012351 22820 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:30.011281 22904 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:30.013758 22904 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:30.013978 22840 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:30.015036 22904 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.019109 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.028286 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:30.044638 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.057346 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:30.065027 22560 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:23:30.067556 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46679.1us
W20260709 12:23:30.068795 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.068552 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:30.027078 22911 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:30.074880 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.073431 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:30.058122 22914 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.078s	user 0.000s	sys 0.003s
I20260709 12:23:30.074061 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.058768 22572 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:30.035127 22840 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:23:30.027042 22864 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.068387 22774 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.081331 18096 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:30.083568 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:30.069471 22845 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:30.087750 22861 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.556s	user 0.017s	sys 0.000s
I20260709 12:23:30.009765 22900 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:30.093206 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.098407 18016 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:23:30.088270 18095 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:23:30.110828 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2356) took 1006 ms. Trace:
I20260709 12:23:30.078696 22560 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.089169 22820 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.105273 22845 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:30.110657 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.145092 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:30.118131 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.113512 22726 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:30.100554 22898 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:30.119104 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.119801 22842 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.120810 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:30.157982 22726 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.158692 18018 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:30.160540 22726 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:30.160784 18018 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:30.158113 22737 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 12:23:30.162268 22737 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:30.162647 22737 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:30.163916 22737 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.164439 22694 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:30.164856 22694 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:30.165338 22737 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.122987 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:30.055269 22557 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.116474 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.139249 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:30.145991 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:30.153386 18090 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:30.155998 22483 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:30.122220 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53016.1us
I20260709 12:23:30.162740 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3179) took 1512 ms. Trace:
I20260709 12:23:30.171908 18023 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.172048 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:30.133718 22839 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.607s	user 0.016s	sys 0.000s
I20260709 12:23:30.130827 18021 rpcz_store.cc:276] 0709 12:23:29.104858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.105000 (+   142us) service_pool.cc:224] Handling call
0709 12:23:30.103519 (+998519us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:30.110722 (+  7203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1802663296}
I20260709 12:23:30.086921 22485 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.171912 18094 rpcz_store.cc:276] 0709 12:23:28.658486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.955929 (+297443us) service_pool.cc:224] Handling call
0709 12:23:30.162520 (+1206591us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400296dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:30.162647 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312632192}
I20260709 12:23:30.149981 22845 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.182125 18099 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:23:30.173616 18095 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:30.151887 22618 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:30.184396 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 109638us
I20260709 12:23:30.188150 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.187117 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.190099 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.193708 18095 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.196300 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.181584 22560 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:30.198611 18094 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:30.199005 18021 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:30.201372 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.190783 22618 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:30.196127 22598 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.185076 22548 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:23:30.202075 18094 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.210644 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.208287 22618 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:30.226544 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:30.227015 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:30.227408 22902 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.289s	user 0.000s	sys 0.004s
I20260709 12:23:30.227825 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.228554 18096 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:30.228777 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:30.229166 22892 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.290s	user 0.001s	sys 0.001s
I20260709 12:23:30.216890 22774 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.230669 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3109) took 1044 ms. Trace:
I20260709 12:23:30.231109 22774 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.232384 22485 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.233342 22548 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.234282 22548 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.202379 22486 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:30.165755 22694 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:30.235831 18780 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.508s (SENT)
I20260709 12:23:30.236057 18019 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:30.236316 22694 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:30.236433 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.200176 22560 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.238240 22560 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:30.189702 22911 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.049s
I20260709 12:23:30.231165 18020 rpcz_store.cc:276] 0709 12:23:29.186684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.186814 (+   130us) service_pool.cc:224] Handling call
0709 12:23:30.230451 (+1043637us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:30.230595 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":638058752}
I20260709 12:23:30.242205 18020 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:23:30.245184 22552 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.237632 18019 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:30.252101 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.253001 18099 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.252707 22897 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:30.254540 22548 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:30.256060 22548 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:30.256547 22548 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:30.257328 22713 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:30.266963 22912 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.325s	user 0.002s	sys 0.000s
W20260709 12:23:30.268286 22890 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.296s	user 0.002s	sys 0.001s
W20260709 12:23:30.275933 22891 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.303s	user 0.001s	sys 0.002s
I20260709 12:23:30.201930 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.289146 18092 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:30.215814 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:30.295457 18021 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:30.298384 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.257704 22548 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.224021 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:30.264056 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:30.234457 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:30.302589 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78750.5us
I20260709 12:23:30.303439 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:30.298408 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89809.5us
I20260709 12:23:30.259925 22900 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.305241 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.320309 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.333190 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:30.336802 18093 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:30.337684 18017 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
W20260709 12:23:30.342094 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 113431us
W20260709 12:23:30.207736 22889 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.468s	user 0.003s	sys 0.003s
W20260709 12:23:30.349798 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46458.3us
W20260709 12:23:30.214517 22916 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:23:30.362893 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:30.341595 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.368321 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.370714 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.372871 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:30.374202 22548 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:30.374899 22864 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:30.376027 18019 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:30.378171 22839 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:30.379833 22839 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:30.380322 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.208299 22835 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:30.383303 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.384384 22830 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:30.385239 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.385262 22830 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:30.386010 18064 leader_election.cc:400] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11.
I20260709 12:23:30.351145 22820 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.387741 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:30.388206 22861 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:30.390224 22830 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:30.390793 22817 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.392385 22839 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:30.392964 22839 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:30.393224 22817 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.395599 22827 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.365634 22897 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.402117 22835 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:30.406136 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.406569 22901 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.407068 18093 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:30.408010 22931 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:23:30.408298 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.409305 18093 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:30.410727 22774 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:30.411062 22839 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.412503 22903 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:30.414036 22903 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:30.414448 22903 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:30.409162 22901 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.426776 18020 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:30.217052 22852 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.377745 18019 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:30.434732 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66556.2us
I20260709 12:23:30.423390 18095 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:30.263036 17921 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 5 to 8, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:30.211298 22598 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:30.310578 22483 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:30.430136 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:30.339802 22911 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:30.430159 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:30.444859 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 192544us
I20260709 12:23:30.444926 18092 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:30.475054 18097 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:30.439322 22830 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.415457 22903 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.440125 22842 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:30.491937 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:30.469764 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.437995 18094 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:23:30.481657 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.479276 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.502317 18167 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13. Re-sending same reply.
I20260709 12:23:30.512966 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:30.514117 18807 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:30.528831 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.529294 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.529572 18172 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:30.529775 18166 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:30.530005 18168 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:30.530809 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.532248 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.533463 18169 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:30.451767 22921 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 12:23:30.536242 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106176us
W20260709 12:23:30.453136 22824 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.013s	sys 0.029s
I20260709 12:23:30.549608 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.553570 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.561574 18165 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:30.562296 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.563087 18093 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:30.555496 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:30.565438 18020 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:30.545616 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.569480 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.570752 22774 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:30.571175 22774 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.571570 22774 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:30.571694 18099 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:30.572965 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.573071 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.564514 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.481249 22911 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.587180 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:30.479550 22726 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.542184 22901 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.084s
I20260709 12:23:30.627084 22901 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:30.627554 22901 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:30.628647 18168 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:30.472245 22906 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:30.594611 18807 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:30.595536 18094 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:30.599787 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.602254 18814 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:30.588111 22827 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.608924 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:30.616597 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.617439 22836 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.707s	user 0.011s	sys 0.005s
I20260709 12:23:30.621973 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:30.622191 22911 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:30.585739 22824 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.911s	user 0.014s	sys 0.035s
I20260709 12:23:30.430253 22548 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.123s
W20260709 12:23:30.588938 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 97154.3us
I20260709 12:23:30.631050 18099 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:30.590003 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.537873 22557 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:30.646864 18169 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:30.550494 22737 raft_consensus.cc:2764] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:23:30.663870 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:30.621974 22891 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:23:30.694319 18780 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.943s (SENT)
I20260709 12:23:30.490587 22483 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.591398 22485 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:30.704907 18169 raft_consensus.cc:614] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:30.628575 22901 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.630488 22598 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.711136 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:30.740217 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 271716us
I20260709 12:23:30.441318 22852 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.690994 22836 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.734773 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2343) took 2146 ms. Trace:
I20260709 12:23:30.775028 18091 rpcz_store.cc:276] 0709 12:23:28.569199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.067642 (+498443us) service_pool.cc:224] Handling call
0709 12:23:29.068081 (+   439us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:29.068595 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.068624 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:29.069330 (+   706us) log.cc:844] Serialized 52 byte log entry
0709 12:23:29.535063 (+465733us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:29.535133 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.535468 (+   335us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.206796 (+671328us) spinlock_profiling.cc:243] Waited 6.51 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:30.698656 (+491860us) raft_consensus.cc:1685] finished
0709 12:23:30.698693 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.715361 (+ 16668us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":844,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":136,"spinlock_wait_cycles":14970240,"thread_start_us":465189,"threads_started":1,"wal-append.queue_time_us":583115}
W20260709 12:23:30.777125 18090 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.004s
I20260709 12:23:30.777768 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:30.741380 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.742077 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:30.645076 22864 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:30.661955 22704 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:30.746359 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.748914 22901 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:30.749385 22598 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:30.750638 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56388.2us
W20260709 12:23:30.799315 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.800268 18780 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.063s (SENT)
I20260709 12:23:30.801256 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.802006 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.752795 22852 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:30.753541 18024 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:30.754562 22552 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.693341 22824 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:30.812626 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:30.798831 22618 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.572s	user 0.009s	sys 0.009s
I20260709 12:23:30.763469 22483 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.764367 22485 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:30.768582 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.770028 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.771023 22774 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:30.771934 22918 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.591s	user 0.006s	sys 0.007s
I20260709 12:23:30.706136 22816 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:30.785851 18020 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
W20260709 12:23:30.787353 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46069.5us
I20260709 12:23:30.793198 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.860657 18095 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11.
I20260709 12:23:30.861871 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.863502 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.864283 18095 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:30.860687 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:30.808525 22827 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:30.881315 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:30.803128 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.804847 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:30.814729 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72760.9us
W20260709 12:23:30.763250 22560 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.524s	user 0.010s	sys 0.003s
I20260709 12:23:30.815013 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.816116 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.887277 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.662766 22906 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.815491 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.888895 18167 raft_consensus.cc:2368] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12.
I20260709 12:23:30.823560 22824 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.805363 22765 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:30.800156 22830 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.805898 22897 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:30.798480 22899 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:30.834472 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.835119 22910 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:30.902447 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.851944 18097 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:30.852263 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:30.853394 18096 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.624s	user 0.016s	sys 0.000s
I20260709 12:23:30.906342 18099 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:30.906500 18096 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:23:30.795946 18021 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:23:30.886399 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 240322us
I20260709 12:23:30.888134 18090 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11.
I20260709 12:23:30.900635 22827 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:30.887737 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72799us
W20260709 12:23:30.889858 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86822.6us
I20260709 12:23:30.800770 22704 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.887708 18169 raft_consensus.cc:2376] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12. Re-sending same reply.
I20260709 12:23:30.909663 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.890028 22483 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:30.852880 22618 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.918143 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:30.924633 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:30.925118 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:30.922339 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:30.900866 22816 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:30.932224 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.932224 18097 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:30.904452 22899 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.890443 22560 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.940411 22485 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.941108 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3194) took 1018 ms. Trace:
I20260709 12:23:30.903673 22830 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.941502 18098 rpcz_store.cc:276] 0709 12:23:29.923731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.923886 (+   155us) service_pool.cc:224] Handling call
0709 12:23:29.927280 (+  3394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:29.927745 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.927779 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:29.928599 (+   820us) log.cc:844] Serialized 52 byte log entry
0709 12:23:30.036418 (+107819us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:30.036491 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.036608 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.771137 (+734529us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:30.940731 (+169594us) raft_consensus.cc:1685] finished
0709 12:23:30.940755 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.941033 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1929,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":5103360,"thread_start_us":107240,"threads_started":1,"wal-append.queue_time_us":168817}
I20260709 12:23:30.938058 22899 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:30.943872 22830 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:30.944298 18098 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:30.836591 18165 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 12:23:30.947350 22845 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.755s	user 0.014s	sys 0.000s
I20260709 12:23:30.947974 22845 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.896533 22765 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.951421 22704 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:30.914121 22905 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.952338 22905 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.952639 22827 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:30.927906 22852 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.955149 22852 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:30.958585 22774 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:30.960732 22943 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:30.961541 22943 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:30.963240 18173 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:30.957513 22852 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:30.895908 18094 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:30.936954 22737 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:30.942836 22560 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:30.901387 18166 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:30.918116 22833 raft_consensus.cc:2764] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:30.938189 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:30.976615 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.979658 18093 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:30.977082 18095 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:30.929925 22817 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:31.000425 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:31.003641 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 126365us
I20260709 12:23:31.004105 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.978227 22830 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.005580 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.935725 22897 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.007470 18165 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:31.011286 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:31.015815 18167 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:31.018385 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.951822 22765 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:31.020979 18168 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:31.029043 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.002698 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:31.033150 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:31.031596 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:30.983790 22817 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.982551 22704 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:31.044914 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:31.047643 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.053745 18096 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:23:31.054296 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:31.050097 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:31.060012 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 125849us
I20260709 12:23:31.054553 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:31.059589 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.551s (SENT) [suppressed 48 similar messages]
I20260709 12:23:31.169728 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.020836 22737 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:30.966289 22937 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.114s
I20260709 12:23:31.167543 18167 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:31.031213 22816 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:31.020147 22822 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:31.047590 22833 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:31.036669 22765 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:31.170903 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:31.047276 22726 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:31.179164 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.006433 22830 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.019723 22900 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:23:31.212376 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.214159 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3208) took 1069 ms. Trace:
I20260709 12:23:31.224174 18095 rpcz_store.cc:276] 0709 12:23:30.144530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.983153 (+838623us) service_pool.cc:224] Handling call
0709 12:23:30.983538 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.984115 (+   577us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:31.019899 (+ 35784us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.213537 (+193638us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.213631 (+    94us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:31.213721 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.213865 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.214062 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":149973,"prepare.queue_time_us":35762,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":11231,"raft.queue_time_us":4476,"thread_start_us":193096,"threads_started":1}
I20260709 12:23:31.048552 22817 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:31.222383 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 211192us
W20260709 12:23:31.215377 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 161181us
W20260709 12:23:31.226828 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 167271us
I20260709 12:23:31.188388 22753 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:31.169700 22552 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:31.233773 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.577s (SENT)
I20260709 12:23:31.244913 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:31.234561 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:31.246559 18172 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:31.309190 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.057416 22704 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:31.313982 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:31.323756 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91704.9us
I20260709 12:23:31.326732 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.232637 22845 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:31.229739 22765 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:31.325574 22900 consensus_queue.cc:1243] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:31.521445 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.049242 22943 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.541540 18165 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:31.550946 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:31.559899 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:31.567557 18099 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:31.569150 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2367) took 1137 ms. Trace:
I20260709 12:23:31.523169 22852 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:31.552924 22899 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.609s	user 0.013s	sys 0.003s
I20260709 12:23:31.579293 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:31.589354 18165 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:31.581804 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.570086 18099 rpcz_store.cc:276] 0709 12:23:30.431491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.566608 (+1135117us) service_pool.cc:224] Handling call
0709 12:23:31.567243 (+   635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.568287 (+  1044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.569048 (+   761us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.590802 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.583839 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:31.535331 22845 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:31.591580 18171 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:31.596249 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.597132 18099 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:31.551007 22552 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:31.569105 22833 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:31.585634 22877 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:31.574646 22618 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:31.583408 22840 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:31.581964 22852 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:31.612808 22947 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.381s	user 0.000s	sys 0.001s
W20260709 12:23:31.613389 22954 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 12:23:31.614547 22958 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:23:31.618583 22830 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:31.620426 22911 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.885s	user 0.012s	sys 0.003s
I20260709 12:23:31.621688 22704 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:31.545658 22737 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:31.623098 22833 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:31.624960 22901 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.824s	user 0.011s	sys 0.000s
I20260709 12:23:31.583904 22952 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:23:31.626183 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:31.626791 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:31.625561 22935 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.578s	user 0.001s	sys 0.000s
W20260709 12:23:31.686321 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80324.4us
W20260709 12:23:31.624995 22939 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:23:31.528206 22816 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:31.672549 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52998.6us
W20260709 12:23:31.677541 18090 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.000s	sys 0.006s
I20260709 12:23:31.632268 22840 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:31.640237 22852 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.683789 18171 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.601699 22845 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.683182 18173 raft_consensus.cc:606] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:31.675896 22941 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.436s	user 0.001s	sys 0.001s
W20260709 12:23:31.724104 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 553255us
W20260709 12:23:31.700843 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 375894us
I20260709 12:23:31.594134 22899 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:31.709193 18090 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.001s	sys 0.006s
W20260709 12:23:31.718570 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.042s	sys 0.064s
I20260709 12:23:31.704963 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:31.626614 22552 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:31.723387 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:31.726243 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:31.734701 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 929) took 1205 ms. Trace:
I20260709 12:23:31.727828 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:31.674386 22949 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:23:31.735244 18808 rpcz_store.cc:276] 0709 12:23:30.519151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.519356 (+   205us) service_pool.cc:224] Handling call
0709 12:23:30.519880 (+   524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.542312 (+ 22432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:30.542345 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.554492 (+ 12147us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.939725 (+385233us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:30.939775 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.939929 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:31.724639 (+784710us) raft_consensus.cc:1685] finished
0709 12:23:31.724666 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.724862 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21843,"raft.queue_time_us":23053,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":34548,"thread_start_us":242045,"threads_started":1,"wal-append.queue_time_us":396833}
W20260709 12:23:31.746982 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 513266us
I20260709 12:23:31.737663 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:31.729537 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.678695 22830 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:23:31.753356 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:31.753726 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:31.678969 22911 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.749446 17989 leader_election.cc:400] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12.
I20260709 12:23:31.754547 18099 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:31.757467 17989 leader_election.cc:403] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:31.771236 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.769171 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:31.785115 18165 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:31.760710 22966 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.761668 22485 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:31.778607 22704 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:31.816658 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78643.7us
I20260709 12:23:31.826968 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:31.755179 22953 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:31.817517 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:31.833086 18169 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:31.830909 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107615us
I20260709 12:23:31.822830 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.837713 18173 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:31.819932 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:31.838208 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:31.839301 18015 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:31.828457 18021 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:31.824599 18778 leader_election.cc:400] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11.
I20260709 12:23:31.841595 18778 leader_election.cc:403] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:23:31.844866 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:31.844935 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:31.848758 18024 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:31.822690 22969 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
W20260709 12:23:31.852425 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 126382us
W20260709 12:23:31.853384 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.855257 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:31.818367 22971 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.861171 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:23:31.861371 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44273.5us
I20260709 12:23:31.859915 18171 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:31.863431 18173 raft_consensus.cc:606] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:31.864213 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:31.821645 22765 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:31.822331 22952 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:31.827441 22972 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.890293 18165 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:31.896461 18165 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:31.902196 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:31.904407 18099 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:31.907348 18021 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:31.906795 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:31.911518 18099 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:31.832085 22965 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:31.860675 22901 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12.
I20260709 12:23:31.778338 22840 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.923832 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:31.912737 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:31.926705 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:31.866343 22953 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:31.933935 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93886.6us
I20260709 12:23:31.988880 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:31.934677 18097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.003s	sys 0.030s
W20260709 12:23:31.989852 18097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.003s	sys 0.030s
I20260709 12:23:31.932669 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:31.921808 18172 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.674s	user 0.011s	sys 0.003s
I20260709 12:23:31.901896 22949 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.974808 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:31.904207 22877 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:32.015008 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.016443 18021 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:32.030439 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:32.034111 18808 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:31.978222 22957 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:23:31.978767 22964 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:31.979017 22983 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:31.980707 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:31.904580 22765 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.983652 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:31.846428 22485 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.990840 18172 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:23:31.991200 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 167325us
I20260709 12:23:32.049080 18171 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:31.939406 22969 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:23:32.052712 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:31.930331 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.001391 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:31.833598 22966 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.037504 22840 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.057049 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2462) took 1080 ms. Trace:
I20260709 12:23:31.908747 22952 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:31.893988 22977 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11.
I20260709 12:23:32.055528 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:31.866675 22971 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.057610 18172 rpcz_store.cc:276] 0709 12:23:30.968136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.191859 (+223723us) service_pool.cc:224] Handling call
0709 12:23:32.048847 (+856988us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:32.068814 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:31.936115 18814 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:32.042448 22903 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:31.907909 22972 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.071458 18021 raft_consensus.cc:2376] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11. Re-sending same reply.
I20260709 12:23:32.074793 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.071694 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:32.057492 22483 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.134s	user 0.017s	sys 0.003s
W20260709 12:23:32.078857 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 252052us
W20260709 12:23:32.079353 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 310295us
I20260709 12:23:31.974542 22965 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.059046 22552 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.089154 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:32.059244 22964 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:32.089032 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.094663 18019 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:32.095404 18024 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:32.068374 22845 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.091555 18023 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:32.033111 22877 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.028666 22949 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.090634 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: true dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:32.100152 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:32.108354 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.108876 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:32.029343 22953 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.087886 22483 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:32.057060 22765 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.109277 18024 raft_consensus.cc:2368] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
W20260709 12:23:32.104439 22973 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:23:32.134063 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:32.115249 22817 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.856s	user 0.012s	sys 0.004s
I20260709 12:23:32.103803 22984 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:32.105679 22989 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:32.153986 17970 auto_rebalancer.cc:893] Removing replace marker failed: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:32.154390 22989 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:32.155269 22965 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.100366 22911 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.103341 22990 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:32.157152 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3207) took 2017 ms. Trace:
I20260709 12:23:32.160051 18098 rpcz_store.cc:276] 0709 12:23:30.139827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.965686 (+825859us) service_pool.cc:224] Handling call
0709 12:23:30.996780 (+ 31094us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.018670 (+ 21890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.018726 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.031000 (+ 12274us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.734841 (+703841us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:31.734889 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.734971 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.156846 (+421875us) raft_consensus.cc:1685] finished
0709 12:23:32.156874 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.157066 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":503,"raft.queue_time_us":7848,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":6178,"thread_start_us":519562,"threads_started":1,"wal-append.queue_time_us":728267}
I20260709 12:23:32.103440 22976 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:32.144958 22996 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:32.146351 22952 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.147351 22949 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.104797 22966 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.147861 22980 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
W20260709 12:23:32.149134 22974 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.288s	user 0.002s	sys 0.000s
I20260709 12:23:32.169312 22990 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:32.150118 22979 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:23:32.170466 22990 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.171048 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3212) took 1870 ms. Trace:
I20260709 12:23:32.171442 18095 rpcz_store.cc:276] 0709 12:23:30.301055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.244950 (+943895us) service_pool.cc:224] Handling call
0709 12:23:31.245930 (+   980us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.312148 (+ 66218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.312204 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.494091 (+181887us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.638265 (+144174us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:31.638355 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.638487 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.170776 (+532289us) raft_consensus.cc:1685] finished
0709 12:23:32.170801 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.170978 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65039,"raft.queue_time_us":65444,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":148258,"thread_start_us":77968,"threads_started":1,"wal-append.queue_time_us":217156}
I20260709 12:23:32.150934 22485 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.144963 22964 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.175416 22765 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.152060 22953 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.152797 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.177640 22972 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.177784 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:32.154655 17970 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 12:23:32.178890 17970 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:23:32.178771 22899 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.142855 22984 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.180923 22984 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.155570 22852 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.103886 22986 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:32.157946 18092 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:32.138039 18015 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:32.183709 22984 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.184692 18022 raft_consensus.cc:2368] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:32.158214 22483 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:32.185915 22852 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.162395 18814 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:32.103173 22901 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:23:32.168473 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:32.192719 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.161592 22971 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.192879 22983 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.203162 22911 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:32.208894 22986 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.209431 22598 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.210979 22976 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:32.213943 22949 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:32.231235 22983 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.157712 22817 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.155673 22989 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.218323 22910 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:32.203945 22901 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:32.218502 22982 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:23:32.219205 22997 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:32.196650 22996 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:32.221767 22897 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.223608 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:32.227394 22753 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.976s	user 0.015s	sys 0.003s
I20260709 12:23:32.227232 22877 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:32.228434 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 139348us
I20260709 12:23:32.269134 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3118) took 3000 ms. Trace:
I20260709 12:23:32.215811 22572 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:32.245428 22911 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.280939 22827 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:23:32.281713 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.303468 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:32.304137 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.304438 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:32.304960 18095 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:32.305195 18099 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.310289 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:32.311931 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:32.284901 22949 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:32.291208 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 198784us
I20260709 12:23:32.285852 22910 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.295475 18024 raft_consensus.cc:2368] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:32.287289 22901 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:32.299517 18018 rpcz_store.cc:276] 0709 12:23:29.250617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.183334 (+932717us) service_pool.cc:224] Handling call
0709 12:23:30.195349 (+ 12015us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:30.242036 (+ 46687us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:30.242077 (+    41us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:30.311409 (+ 69332us) log.cc:844] Serialized 78 byte log entry
0709 12:23:30.645727 (+334318us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:23:30.645773 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.242877 (+1597104us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b5400250b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:32.242950 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.243030 (+    80us) raft_consensus.cc:1685] finished
0709 12:23:32.243048 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.250757 (+  7709us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1043,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":132,"spinlock_wait_cycles":105937280,"thread_start_us":271394,"threads_started":1,"wal-append.queue_time_us":370883,"wal-append.run_cpu_time_us":3194,"wal-append.run_wall_time_us":1061365}
I20260709 12:23:32.327068 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:32.328078 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:32.317895 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:32.269057 22976 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.298687 22753 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:32.331634 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:32.316226 18095 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:32.342121 18099 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:32.375216 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.343027 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.343706 22835 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.344102 22911 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:32.344569 22948 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.753s	user 0.003s	sys 0.003s
W20260709 12:23:32.344859 22999 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 12:23:32.335489 23004 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:23:32.345329 22901 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.346168 22910 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.395023 22910 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.346978 22817 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.348541 22976 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.348685 22949 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.349184 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.400096 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:32.350541 22956 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.732s	user 0.002s	sys 0.002s
I20260709 12:23:32.324726 22598 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.330451 23002 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:23:32.402793 23002 consensus_queue.cc:1243] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:23:32.402686 22598 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.325518 22897 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.303747 22827 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:32.352550 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:32.405295 18098 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.401492 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.406834 18022 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:23:32.353672 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:32.356776 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:32.407713 18095 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:32.357852 22737 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.326442 22864 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:23:32.356161 22572 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:32.367326 18780 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.229s (SENT)
I20260709 12:23:32.374526 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.388657 22964 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.417785 22964 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.420706 22964 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.376667 18018 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:23:32.424656 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 693) took 1962 ms. Trace:
I20260709 12:23:32.425288 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.385535 22835 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.390165 18807 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:23:32.390689 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 275716us
I20260709 12:23:32.299829 22997 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.452688 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.453208 22906 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.453552 18017 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:32.324445 22971 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.407897 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51304.8us
I20260709 12:23:32.330518 22943 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:32.412665 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:32.413226 18165 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:32.464589 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.465898 18165 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:32.466372 18165 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.467209 18165 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.375901 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:32.415997 22827 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.360654 22986 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.425256 18164 rpcz_store.cc:276] 0709 12:23:30.462568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.462736 (+   168us) service_pool.cc:224] Handling call
0709 12:23:30.555554 (+ 92818us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.638583 (+ 83029us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:30.638643 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.704173 (+ 65530us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.935167 (+230994us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:30.935210 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.935282 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.231915 (+1296633us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:32.424284 (+192369us) raft_consensus.cc:1685] finished
0709 12:23:32.424313 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.424567 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17715072,"thread_start_us":175279,"threads_started":1,"wal-append.queue_time_us":426056}
I20260709 12:23:32.426793 22572 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.412345 22897 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.431447 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2242) took 1516 ms. Trace:
I20260709 12:23:32.426806 22978 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:32.476460 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.477296 18017 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:32.478538 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.478443 22572 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.479518 18017 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:32.480538 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.481336 18017 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:32.468724 22971 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.484941 22971 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.468466 22943 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.486124 22943 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.486464 22827 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:32.463470 22774 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:32.461159 22901 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.490739 22774 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.496579 22835 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.436163 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.437060 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:32.438143 22988 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.359s	user 0.002s	sys 0.000s
I20260709 12:23:32.498445 18166 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.373003 22996 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.444787 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:32.434157 22951 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.833s	user 0.002s	sys 0.003s
I20260709 12:23:32.448153 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:32.435315 18814 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:32.502321 18808 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.503289 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:32.413159 23002 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:32.471035 18098 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:32.472404 18169 rpcz_store.cc:276] 0709 12:23:30.915021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.915247 (+   226us) service_pool.cc:224] Handling call
0709 12:23:32.431153 (+1515906us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:32.431338 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1975100288}
I20260709 12:23:32.450582 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:32.474903 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.476634 22997 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.476239 22978 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.432806 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.490057 22897 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.496856 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2354) took 2387 ms. Trace:
W20260709 12:23:32.505625 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55214.1us
W20260709 12:23:32.506080 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61399us
I20260709 12:23:32.505795 18098 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:32.474611 18099 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:32.507637 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 140353us
I20260709 12:23:32.547168 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.493341 22774 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:32.553587 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:32.554085 18016 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 6.
I20260709 12:23:32.555285 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:32.556743 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.511569 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3210) took 2286 ms. Trace:
I20260709 12:23:32.567322 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:32.507457 22877 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.511190 22911 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:32.546679 22900 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.575599 22774 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:32.578068 23014 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:32.508949 22964 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:32.572196 18097 rpcz_store.cc:276] 0709 12:23:30.225003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.035179 (+810176us) service_pool.cc:224] Handling call
0709 12:23:31.055715 (+ 20536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.056927 (+  1212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.056972 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.211260 (+154288us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.990298 (+779038us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:31.990341 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.990410 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.437300 (+446890us) spinlock_profiling.cc:243] Waited 9.04 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:32.500794 (+ 63494us) raft_consensus.cc:1685] finished
0709 12:23:32.500821 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.511480 (+ 10659us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":714,"raft.queue_time_us":1185,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":102124,"spinlock_wait_cycles":20803328,"thread_start_us":680304,"threads_started":1,"wal-append.queue_time_us":802147}
I20260709 12:23:32.585078 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.522579 22903 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:32.514869 22971 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:32.448899 22737 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.549199 18096 rpcz_store.cc:276] 0709 12:23:30.109066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.923240 (+814174us) service_pool.cc:224] Handling call
0709 12:23:30.936285 (+ 13045us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.054973 (+118688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.496505 (+1441532us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54002963d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:32.496768 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269329664}
I20260709 12:23:32.448484 18168 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:32.586650 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:32.555426 22598 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.603469 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.614795 18017 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:32.605707 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.617970 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:32.568990 22985 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.616317 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.567764 22817 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:32.622094 18166 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
W20260709 12:23:32.623205 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2458) took 1656 ms (client timeout 2017 ms). Trace:
W20260709 12:23:32.623682 18166 rpcz_store.cc:269] 0709 12:23:30.967010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.047093 (+ 80083us) service_pool.cc:224] Handling call
0709 12:23:32.436876 (+1389783us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:32.623129 (+186253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3794304}
I20260709 12:23:32.508363 23002 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:32.631048 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:32.639129 22943 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.640558 18164 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:32.584280 22911 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:32.642274 18020 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:32.562561 22966 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:23:32.653338 22966 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:32.634905 22774 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:32.580358 22977 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 12:23:32.583411 22552 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 12:23:32.655470 22977 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:32.655939 22977 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:32.656004 22552 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:32.656478 22552 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:32.611982 22835 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:32.658483 22835 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.605191 22897 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:32.628131 22864 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:32.662513 22911 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.641086 23002 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.610078 22997 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:32.664793 22997 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.613521 22900 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.641562 22903 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:32.674463 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.674793 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.675400 18017 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:32.675781 18016 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:32.676597 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.676968 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.661543 22897 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.678455 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:32.629184 22985 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:32.679042 22959 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.129s	user 0.002s	sys 0.000s
I20260709 12:23:32.679801 22897 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:32.681193 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 694) took 2218 ms. Trace:
I20260709 12:23:32.682717 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:32.615607 22971 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.681618 18170 rpcz_store.cc:276] 0709 12:23:30.462953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.480756 (+ 17803us) service_pool.cc:224] Handling call
0709 12:23:30.551788 (+ 71032us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.897906 (+346118us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:30.918509 (+ 20603us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.211597 (+293088us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.212206 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.212381 (+   175us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:31.212422 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.212508 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.678221 (+1465713us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:32.680813 (+  2592us) raft_consensus.cc:1685] finished
0709 12:23:32.680948 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.681112 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":246720,"prepare.queue_time_us":28040,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":1656,"raft.queue_time_us":365069,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":18257,"spinlock_wait_cycles":135411968,"thread_start_us":357320,"threads_started":2,"wal-append.queue_time_us":239993}
W20260709 12:23:32.686830 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.691486 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2475) took 1073 ms. Trace:
I20260709 12:23:32.695749 18166 rpcz_store.cc:276] 0709 12:23:31.618131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.635369 (+1017238us) service_pool.cc:224] Handling call
0709 12:23:32.690839 (+ 55470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.691097 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.691127 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.691174 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:32.691217 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.691282 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.691417 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:32.703688 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.705655 18808 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:32.634320 23014 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:32.677676 22967 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.074s	user 0.004s	sys 0.006s
I20260709 12:23:32.724938 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:32.724938 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:32.657138 22977 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.738198 22977 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.740255 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.740559 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:32.741832 18017 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:32.744787 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.745868 18173 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:32.703444 22971 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:32.746881 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.688527 22966 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:32.597165 22877 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.657447 22552 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.754202 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.755088 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:32.756939 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:32.757750 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 140915us
I20260709 12:23:32.741925 22830 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:32.708014 22901 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:32.763312 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:32.762727 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.764086 18166 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:32.757601 22989 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.760483 22830 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:32.773927 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:32.776100 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 198727us
I20260709 12:23:32.776340 18170 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:32.775786 18173 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:32.781289 22966 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:32.784404 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.785580 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.770802 22552 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.813882 18172 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:32.822556 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.821245 18814 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:32.828497 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:32.828740 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:32.829406 18807 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:23:32.829494 18170 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.831573 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:32.834371 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:32.849809 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:32.850936 18173 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:32.864494 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.879177 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2257) took 1206 ms. Trace:
I20260709 12:23:32.879313 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:32.879822 18166 rpcz_store.cc:276] 0709 12:23:31.672574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.864047 (+1191473us) service_pool.cc:224] Handling call
0709 12:23:32.864609 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.864925 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.864969 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.878784 (+ 13815us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:32.878855 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.878933 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.879093 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:32.821740 22943 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:32.885579 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:32.895107 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.778399 22989 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:32.848434 22966 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:32.923501 18173 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:23:32.968109 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.926752 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:33.010605 18166 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:33.012650 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:33.015062 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:33.016122 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:33.009932 18170 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:33.022846 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.894806 22985 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:33.048548 22984 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.014s	sys 0.042s
W20260709 12:23:33.049261 22984 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.014s	sys 0.042s
I20260709 12:23:33.050055 18173 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:23:33.064345 23022 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 12:23:33.064649 22905 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:33.066741 23019 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:32.825866 23014 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.016212 22997 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.084118 18166 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:33.091440 18170 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:23:33.061025 22910 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.006s
I20260709 12:23:33.105887 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.106917 18173 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:33.134516 18170 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:32.918136 22830 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:33.096035 23024 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.098s	user 0.003s	sys 0.000s
I20260709 12:23:33.094630 23020 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:33.165735 23020 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:33.166566 23020 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:33.162721 23029 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:23:33.163316 22989 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.201478 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 761) took 2580 ms. Trace:
I20260709 12:23:33.202087 18092 rpcz_store.cc:276] 0709 12:23:30.620869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.554625 (+933756us) service_pool.cc:224] Handling call
0709 12:23:32.157646 (+603021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.182806 (+ 25160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.201190 (+1018384us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54002959d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.201395 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385475840}
I20260709 12:23:33.205752 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:33.244694 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.207458 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 280836us
W20260709 12:23:33.212316 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.298498 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.310086 22877 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:33.311568 23002 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.647s	user 0.017s	sys 0.000s
I20260709 12:23:33.312054 23002 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.314632 23002 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:33.309582 22971 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.001s
I20260709 12:23:33.316309 22971 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.317541 18017 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:33.319300 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2384) took 1274 ms. Trace:
I20260709 12:23:33.319682 18017 rpcz_store.cc:276] 0709 12:23:32.044999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.752676 (+707677us) service_pool.cc:224] Handling call
0709 12:23:32.753163 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.318011 (+564848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.319001 (+   990us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:33.319221 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297396992}
I20260709 12:23:33.320128 22971 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.321039 18018 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:33.323669 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3142) took 1500 ms (client timeout 1696 ms). Trace:
W20260709 12:23:33.324158 18023 rpcz_store.cc:269] 0709 12:23:31.822867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.823018 (+   151us) service_pool.cc:224] Handling call
0709 12:23:33.323410 (+1500392us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.323550 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":561241728}
I20260709 12:23:33.325153 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:33.327785 18021 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:33.328303 18021 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:33.330367 22991 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.331032 22991 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.333662 22991 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:33.335654 18020 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:33.336472 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2402) took 1183 ms. Trace:
I20260709 12:23:33.336800 18020 rpcz_store.cc:276] 0709 12:23:32.153488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.334859 (+1181371us) service_pool.cc:224] Handling call
0709 12:23:33.335230 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.336160 (+   930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.336412 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:33.339347 22827 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.852s	user 0.012s	sys 0.006s
I20260709 12:23:33.340421 18016 raft_consensus.cc:2410] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 12:23:33.344645 22552 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.005s
I20260709 12:23:33.354875 18017 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: [8.7-10.8]
I20260709 12:23:33.356628 18015 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:23:33.357371 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2400) took 1204 ms. Trace:
I20260709 12:23:33.329198 18021 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.357697 18015 rpcz_store.cc:276] 0709 12:23:32.152907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.333415 (+1180508us) service_pool.cc:224] Handling call
0709 12:23:33.356204 (+ 22789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.357083 (+   879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.357310 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.361454 18023 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:23:33.362311 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2399) took 1209 ms. Trace:
I20260709 12:23:33.362649 18023 rpcz_store.cc:276] 0709 12:23:32.152650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.325287 (+1172637us) service_pool.cc:224] Handling call
0709 12:23:33.325620 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.361955 (+ 36335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.362239 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.363982 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3146) took 1052 ms. Trace:
I20260709 12:23:33.364377 18023 rpcz_store.cc:276] 0709 12:23:32.311730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.363295 (+1051565us) service_pool.cc:224] Handling call
0709 12:23:33.363585 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.363742 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.363762 (+    20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.363795 (+    33us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:33.363823 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.363857 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.363937 (+    80us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.365702 18023 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:33.367137 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2396) took 1215 ms. Trace:
I20260709 12:23:33.390349 18017 rpcz_store.cc:276] 0709 12:23:32.151731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.354106 (+1202375us) service_pool.cc:224] Handling call
0709 12:23:33.354558 (+   452us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:33.355883 (+  1325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.367060 (+ 11177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.393409 18173 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:33.366492 18020 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:33.435272 18015 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:33.436129 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 638) took 1271 ms. Trace:
W20260709 12:23:33.375366 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59533.2us
I20260709 12:23:33.436800 18015 rpcz_store.cc:276] 0709 12:23:32.165296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.361242 (+1195946us) service_pool.cc:224] Handling call
0709 12:23:33.435045 (+ 73803us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.435832 (+   787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.436056 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.371567 18021 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:33.374130 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.434298 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 637) took 1280 ms. Trace:
I20260709 12:23:33.366487 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3148) took 1054 ms. Trace:
I20260709 12:23:33.439571 18023 rpcz_store.cc:276] 0709 12:23:32.312292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.365136 (+1052844us) service_pool.cc:224] Handling call
0709 12:23:33.365500 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.366224 (+   724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.366429 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.372879 22827 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:33.454493 22898 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.464524 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:33.470520 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:33.439091 18020 rpcz_store.cc:276] 0709 12:23:32.153796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.337802 (+1184006us) service_pool.cc:224] Handling call
0709 12:23:33.366306 (+ 28504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.433930 (+ 67624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.434214 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.438493 22991 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:33.477047 23038 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:33.477658 23032 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:33.498202 23038 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:33.475672 22898 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.498731 23038 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:33.500178 23032 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.500859 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.501014 23032 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.472002 22827 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:33.503170 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.516191 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.523787 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.524696 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
W20260709 12:23:33.514174 22987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.409s	user 0.008s	sys 0.000s
I20260709 12:23:33.528016 18021 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:33.530097 18173 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:23:33.514425 22877 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.532382 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2380) took 1602 ms. Trace:
I20260709 12:23:33.532938 18021 rpcz_store.cc:276] 0709 12:23:31.931755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.081968 (+150213us) service_pool.cc:224] Handling call
0709 12:23:33.327541 (+1245573us) spinlock_profiling.cc:243] Waited 83.8 ms on lock 0x7b5400541120. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662edc0 00007f6c166bae18 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:33.531817 (+204276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192834816}
I20260709 12:23:33.537130 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:33.528431 22987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.423s	user 0.009s	sys 0.000s
W20260709 12:23:33.534818 22485 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:23:33.543421 22995 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:33.550140 22864 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:33.550770 22864 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:33.506564 22991 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.555928 22864 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.557372 22864 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:33.559595 23032 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.536098 22898 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.570000 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.570930 18173 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:33.552079 23042 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:33.538720 22877 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.593801 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2474) took 1976 ms. Trace:
W20260709 12:23:33.587286 22911 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.922s	user 0.015s	sys 0.002s
I20260709 12:23:33.595327 18164 rpcz_store.cc:276] 0709 12:23:31.617846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.570971 (+953125us) service_pool.cc:224] Handling call
0709 12:23:32.598075 (+ 27104us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.643972 (+ 45897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.593445 (+949473us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:33.593712 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97122688}
I20260709 12:23:33.582821 23042 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.575587 23020 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.616562 22978 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.626299 22827 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:33.628080 22955 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 2.149s	user 0.011s	sys 0.000s
W20260709 12:23:33.639918 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2467) took 2636 ms (client timeout 1566 ms). Trace:
W20260709 12:23:33.640672 18168 rpcz_store.cc:269] 0709 12:23:31.054872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.084804 (+ 29932us) service_pool.cc:224] Handling call
0709 12:23:32.223388 (+1138584us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:33.639557 (+1416169us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.639787 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318906112}
I20260709 12:23:33.643922 22737 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.645192 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:33.650108 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:33.651793 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2393) took 1501 ms. Trace:
I20260709 12:23:33.652311 18018 rpcz_store.cc:276] 0709 12:23:32.150828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.435838 (+285010us) service_pool.cc:224] Handling call
0709 12:23:33.651538 (+1215700us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b540024de54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.651719 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1891817216}
I20260709 12:23:33.653806 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:33.674925 18018 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.654142 18021 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:33.679080 18021 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 10. Ops: []
I20260709 12:23:33.680786 18021 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:23:33.682744 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3143) took 1859 ms (client timeout 1752 ms). Trace:
W20260709 12:23:33.683284 18019 rpcz_store.cc:269] 0709 12:23:31.823176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.823552 (+   376us) service_pool.cc:224] Handling call
0709 12:23:33.682480 (+1858928us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b5400108bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.682655 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":571585920}
I20260709 12:23:33.691324 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:33.692806 18809 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:23:33.693248 22830 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:33.694370 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2247) took 2661 ms. Trace:
I20260709 12:23:33.694854 18171 rpcz_store.cc:276] 0709 12:23:31.033373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.866334 (+832961us) service_pool.cc:224] Handling call
0709 12:23:32.048842 (+182508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.049710 (+   868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.694132 (+1644422us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400043dd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.694293 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362137472}
I20260709 12:23:33.697170 18165 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:33.697635 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:33.698679 18814 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:33.700753 22900 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.660111 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.660234 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2385) took 1597 ms. Trace:
W20260709 12:23:33.709669 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2398) took 1557 ms (client timeout 1705 ms). Trace:
W20260709 12:23:33.710299 18024 rpcz_store.cc:269] 0709 12:23:32.152315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.177418 (+ 25103us) service_pool.cc:224] Handling call
0709 12:23:33.709412 (+1531994us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54003476d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.709574 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268161408}
I20260709 12:23:33.709587 18016 rpcz_store.cc:276] 0709 12:23:32.062816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.678049 (+615233us) service_pool.cc:224] Handling call
0709 12:23:33.659990 (+981941us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54001095d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:33.660163 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1520151936}
I20260709 12:23:33.720253 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.660820 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:33.722626 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:33.723799 22839 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.724397 22839 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.725651 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.726061 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.726367 18092 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:33.727263 22839 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.728217 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.729588 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.730432 18096 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:33.731446 18812 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:33.600402 23042 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:33.735561 18814 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:33.736653 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.741214 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.702790 18810 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:33.657740 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2471) took 2089 ms. Trace:
W20260709 12:23:33.706216 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:33.831231 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 125112us
W20260709 12:23:33.624949 22977 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.886s	user 0.011s	sys 0.006s
I20260709 12:23:33.793435 18092 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:23:33.846997 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.847892 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.743709 18172 rpcz_store.cc:276] 0709 12:23:31.568229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.076853 (+508624us) service_pool.cc:224] Handling call
0709 12:23:32.077282 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.814505 (+737223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.657440 (+842935us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:33.657660 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1693369216}
W20260709 12:23:33.829473 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.829909 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.850914 18099 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:33.851469 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.852005 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.852480 18098 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 9 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
I20260709 12:23:33.852887 18099 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:23:33.748739 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:33.652380 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:33.839864 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.853926 18016 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:33.854379 18097 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:33.857847 18024 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:33.858583 18814 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:33.600509 22911 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.860610 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:33.863171 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.863723 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.864784 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.865859 18092 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:33.865079 18091 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:33.839470 22842 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.877628 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:33.878427 18809 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:33.702913 22835 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:33.892915 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:33.669723 22877 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.668192 22991 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.877017 22964 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 12:23:33.923833 22964 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:33.924386 22964 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:33.925063 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2386) took 1590 ms. Trace:
I20260709 12:23:33.911299 17921 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 3 to 10. New cstate: current_term: 10 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:33.925806 22905 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.926036 22929 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:33.926466 22905 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.926740 22929 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.927400 23020 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.862550 22906 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.929474 22906 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.861995 22560 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:33.930560 22905 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.930718 22560 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.932128 22906 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.932343 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.933091 18097 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:23:33.933187 22560 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:33.929047 22840 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.934707 22840 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.938736 18091 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:23:33.939198 18092 pending_rounds.cc:85] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:23:33.903648 22985 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.939636 18092 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:23:33.939268 18091 pending_rounds.cc:85] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 6
I20260709 12:23:33.941448 18091 pending_rounds.cc:107] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:23:33.941799 18091 raft_consensus.cc:2889] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:33.849146 22726 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.943238 22726 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:33.910077 22842 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.925587 18095 rpcz_store.cc:276] 0709 12:23:32.334429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.415059 (+ 80630us) service_pool.cc:224] Handling call
0709 12:23:32.463319 (+ 48260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.463608 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:32.553449 (+ 89841us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.877936 (+1324487us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.924555 (+ 46619us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.924627 (+    72us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:33.924687 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.924791 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.924978 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1233491,"prepare.queue_time_us":181176,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":113,"raft.queue_time_us":1250578,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":6336,"spinlock_wait_cycles":36438272,"thread_start_us":195430,"threads_started":2,"wal-append.queue_time_us":325881}
I20260709 12:23:33.940037 18092 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:33.912498 18165 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:33.892599 22486 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.929047 17921 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:33.877386 22774 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.895450 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.940022 22985 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:33.947600 22842 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:33.911299 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:33.950160 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2473) took 2333 ms. Trace:
I20260709 12:23:33.970521 18165 rpcz_store.cc:276] 0709 12:23:31.617370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.465294 (+847924us) service_pool.cc:224] Handling call
0709 12:23:32.465672 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.696868 (+1231196us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b58004343d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166130aa 00007f6c16695447 00007f6c166adfe7 00007f6c166af7de 00007f6c166b9b49 00007f6c166ba416 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4
0709 12:23:33.949656 (+252788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.950084 (+   428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2744614784}
I20260709 12:23:33.950325 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3149) took 1636 ms. Trace:
I20260709 12:23:33.841961 22943 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:33.949486 18096 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:33.951895 18098 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:23:33.895179 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.875524 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:33.976610 18097 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:33.978096 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.978893 18097 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
W20260709 12:23:33.909286 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.953752 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91675.5us
W20260709 12:23:33.980269 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.905604 22965 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.966043 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.978468 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3157) took 1426 ms. Trace:
I20260709 12:23:33.900370 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:33.987775 18099 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:23:33.989140 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88958.2us
I20260709 12:23:33.990828 18018 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:33.995823 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:33.989647 18099 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:34.015213 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.016059 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.981976 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:33.989434 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114037us
W20260709 12:23:33.956882 22694 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:23:33.983158 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:34.002132 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.021749 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:34.001945 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:33.892202 23032 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:23:34.006517 18814 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:34.022804 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:34.007212 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.024116 18097 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:34.024344 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.006538 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.025534 18019 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:34.010121 18809 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260709 12:23:34.026718 18809 pending_rounds.cc:85] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 9
I20260709 12:23:34.025625 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.013988 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:34.031195 18810 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:33.992270 22996 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.014196 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3209) took 3789 ms. Trace:
I20260709 12:23:34.016243 18098 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:34.017133 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:34.019883 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:34.027032 18809 pending_rounds.cc:107] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 12:23:33.949486 22726 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.022913 18091 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:34.108747 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86031.6us
W20260709 12:23:34.108949 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 87287.9us
W20260709 12:23:34.110097 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 90296.3us
I20260709 12:23:33.962033 22486 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.112190 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:34.122304 22486 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:34.133347 22903 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:34.133729 22995 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:34.134999 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:34.136490 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:34.140901 18812 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:34.142096 22978 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:34.142721 22978 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:34.143224 22817 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:34.143842 22817 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.144348 22817 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:34.150374 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:34.004561 22560 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:34.031123 18099 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:34.112434 18809 raft_consensus.cc:2889] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.113863 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:33.925521 22964 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:34.114396 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.015133 22977 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:33.965317 22774 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.105768 22842 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:34.127795 22911 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.128422 22929 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.130568 22953 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:34.147882 22903 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.153450 18097 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:34.108659 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 112928us
I20260709 12:23:34.020556 22598 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.156273 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44181.7us
W20260709 12:23:34.160000 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.991479 22943 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:34.173630 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.033918 22737 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:23:34.162446 22864 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.598s	user 0.005s	sys 0.010s
W20260709 12:23:34.207252 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.146962 22989 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:34.132444 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:34.134418 22548 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:23:34.134320 23026 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:34.113163 22483 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:34.160518 22966 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.157s	user 0.008s	sys 0.017s
I20260709 12:23:34.243731 22953 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.028412 22765 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.158s
I20260709 12:23:34.252362 18172 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:34.250239 22964 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:34.158390 22816 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:34.179227 22842 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.255211 22618 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:34.283263 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.134413 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.283931 22929 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.287099 18167 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.030s	sys 0.024s
W20260709 12:23:34.331261 18167 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.030s	sys 0.024s
I20260709 12:23:34.332415 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2472) took 2724 ms. Trace:
I20260709 12:23:34.145962 22840 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:34.300879 18097 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:34.334261 18098 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:34.237409 22836 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:34.302433 18094 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.012s	sys 0.037s
I20260709 12:23:34.307971 18807 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:34.157469 22560 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:34.342202 18094 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.013s	sys 0.037s
I20260709 12:23:34.143868 22978 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.002700 22965 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.147476 22996 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.251670 22903 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:34.162042 22944 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.359191 22944 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.287047 22864 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.281806 23026 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.323155 18091 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:34.167047 22977 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.334558 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.370399 18097 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:34.242673 22827 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:23:34.110702 18093 rpcz_store.cc:276] 0709 12:23:30.224633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.026517 (+801884us) service_pool.cc:224] Handling call
0709 12:23:31.035532 (+  9015us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.035841 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:31.465788 (+429947us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.519980 (+ 54192us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.560477 (+2040497us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b1800478c60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:33.617626 (+ 57149us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b1800478c60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:23:33.618161 (+   535us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.013318 (+395157us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.013391 (+    73us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:34.013462 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.013973 (+   511us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.014117 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2462193,"prepare.queue_time_us":155515,"prepare.run_cpu_time_us":183,"prepare.run_wall_time_us":5721,"raft.queue_time_us":2110951,"raft.run_cpu_time_us":640,"raft.run_wall_time_us":39971,"spinlock_wait_cycles":71961088,"thread_start_us":596977,"threads_started":3,"wal-append.queue_time_us":181691,"wal-append.run_cpu_time_us":7927,"wal-append.run_wall_time_us":1401720}
I20260709 12:23:34.285395 22737 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:34.249637 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:34.332882 18167 rpcz_store.cc:276] 0709 12:23:31.607864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.300155 (+692291us) service_pool.cc:224] Handling call
0709 12:23:32.300569 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:32.313125 (+ 12556us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:32.411778 (+ 98653us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.561381 (+149603us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.561510 (+   129us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:32.562107 (+   597us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.572892 (+1010785us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b1800022120. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:33.646968 (+ 74076us) spinlock_profiling.cc:243] Waited 6.4 ms on lock 0x7b1800022120. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c1626dd70 00007f6c16261293 00007f6c16256cea 00007f6c162570d1 00007f6c16672a0f 00007f6c1662e621 00007f6c166b792b
0709 12:23:33.647214 (+   246us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:34.331749 (+684535us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.332012 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.332118 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.332174 (+    56us) raft_consensus.cc:1685] finished
0709 12:23:34.332186 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.332350 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42901,"prepare.queue_time_us":543906,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":146,"raft.queue_time_us":23240,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":6144,"spinlock_wait_cycles":29903104,"thread_start_us":840685,"threads_started":3,"wal-append.queue_time_us":112629}
I20260709 12:23:34.287457 22483 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:34.426349 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 176923us
I20260709 12:23:34.305670 22953 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:34.346151 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.021s	sys 0.014s
W20260709 12:23:34.458087 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.022s	sys 0.014s
W20260709 12:23:34.358114 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.019s	sys 0.019s
W20260709 12:23:34.362938 18020 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.025s	sys 0.062s
I20260709 12:23:34.350317 22618 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:34.386173 18172 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:23:34.402290 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 420402us
I20260709 12:23:34.332208 22975 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:34.423038 18099 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:34.350538 22560 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:34.473347 18099 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:34.458724 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.019s	sys 0.020s
I20260709 12:23:34.357205 22966 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.458881 18020 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.025s	sys 0.062s
I20260709 12:23:34.254632 22598 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.483932 22842 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:34.485146 23046 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:23:34.485355 22598 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.456697 18091 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 10. Ops: []
I20260709 12:23:34.491318 22977 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.491989 22977 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.492251 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.492465 22977 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.492938 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:34.497061 22774 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.497941 22975 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.498831 22943 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:34.499171 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 936) took 1614 ms (client timeout 1896 ms). Trace:
W20260709 12:23:34.499687 18807 rpcz_store.cc:269] 0709 12:23:32.885053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.885211 (+   158us) service_pool.cc:224] Handling call
0709 12:23:34.498937 (+1613726us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54004b46d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:34.499066 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3246630784}
W20260709 12:23:34.473170 23017 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:23:34.368746 23026 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:33.972371 18017 rpcz_store.cc:276] 0709 12:23:32.313859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.391188 (+1077329us) service_pool.cc:224] Handling call
0709 12:23:33.391544 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.391745 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:33.522546 (+130801us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.949597 (+427051us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.949924 (+   327us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.949982 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:33.950042 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.950128 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.950258 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":133607,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":49,"raft.queue_time_us":3360,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":12097,"spinlock_wait_cycles":11470592,"thread_start_us":556645,"threads_started":2}
W20260709 12:23:34.477098 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 745969us
I20260709 12:23:34.350880 22929 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.358404 22836 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.490156 18099 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.522893 18099 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:34.481689 22966 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.364622 22816 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.138s
I20260709 12:23:34.361491 22864 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.460209 22483 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:34.466121 22830 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:34.428965 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:33.986605 18015 rpcz_store.cc:276] 0709 12:23:32.551750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.462069 (+910319us) service_pool.cc:224] Handling call
0709 12:23:33.462542 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.462830 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:33.525196 (+ 62366us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.977902 (+452706us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.977999 (+    97us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:33.978083 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.978191 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.978383 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":62645,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":26338,"thread_start_us":477344,"threads_started":2}
W20260709 12:23:34.540740 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111914us
I20260709 12:23:34.503043 22986 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.480754 22944 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:34.527240 23042 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.687s	user 0.008s	sys 0.006s
W20260709 12:23:34.555931 18806 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.015s	sys 0.029s
I20260709 12:23:34.501343 22975 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.515543 22906 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.561266 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 410987us
I20260709 12:23:34.513140 22985 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:34.549021 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 531825us
I20260709 12:23:34.553190 18093 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 12:23:34.556457 18806 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.015s	sys 0.029s
I20260709 12:23:34.571437 18166 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:34.527477 22842 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.539970 22830 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.575896 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:34.564653 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:34.565802 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4297545330000503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
I20260709 12:23:34.545444 22694 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:34.580889 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:34.599048 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3829545329999746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
I20260709 12:23:34.599437 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:34.599516 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:34.588697 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:34.600914 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:34.620388 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44641.2us
W20260709 12:23:34.599573 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3730545329999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
I20260709 12:23:34.591647 18809 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260709 12:23:34.629437 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3613545329999397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1579s for a response
I20260709 12:23:34.623548 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:34.631136 18172 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
W20260709 12:23:34.544030 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 380558us
I20260709 12:23:34.632385 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:34.638569 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.634018 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:34.567824 23042 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:34.644811 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.562990 22977 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:34.647076 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:34.547355 22943 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:23:34.557991 22986 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.579023 22618 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.643394 22900 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:34.579650 23056 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:34.645354 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:34.637701 22830 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:34.645551 22835 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:34.654338 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:34.715456 18815 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:23:34.716637 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71490.2us
I20260709 12:23:34.656085 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:34.659389 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:34.736822 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80858.6us
I20260709 12:23:34.660022 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:34.649957 18099 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:34.581319 22944 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.130s
I20260709 12:23:34.743156 22986 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:34.644507 22842 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.721482 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:34.744669 18814 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:23:34.662755 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 414107us
I20260709 12:23:34.661659 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:34.711159 22900 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:34.746611 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85026.7us
I20260709 12:23:34.746776 22900 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:34.649091 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49730.4us
I20260709 12:23:34.664400 22774 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:34.659554 22996 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.749221 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:34.708370 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:34.668908 18812 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:34.750366 18098 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:34.751116 23042 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.749213 22774 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:34.751492 22842 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.751768 22774 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:34.650030 22827 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.102s
I20260709 12:23:34.752305 22842 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.753724 22842 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:34.755625 18096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.008s	sys 0.017s
W20260709 12:23:34.684386 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3532545329999266s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 12:23:34.650738 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51998.1us
I20260709 12:23:34.759163 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:34.569001 22906 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:34.765848 22817 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.621s	user 0.014s	sys 0.001s
I20260709 12:23:34.766533 22817 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.767315 22906 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.770906 22842 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:34.656085 23053 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
W20260709 12:23:34.779268 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.706825 22975 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:34.625892 22737 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.681529 22977 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:34.742777 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 762) took 4121 ms. Trace:
I20260709 12:23:34.711484 18165 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:34.759203 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3469545329999164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
W20260709 12:23:34.790693 18064 connection.cc:256] RPC call timeout handler was delayed by -2.3469545329999164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
W20260709 12:23:34.791460 18064 connection.cc:256] RPC call timeout handler was delayed by -2.32445453299988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16200000000000003s for a response
W20260709 12:23:34.792101 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2587545329997738s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
W20260709 12:23:34.792476 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2479545329997563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20260709 12:23:34.792776 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2479545329997563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
I20260709 12:23:34.790434 18094 rpcz_store.cc:276] 0709 12:23:30.621253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.561294 (+940041us) service_pool.cc:224] Handling call
0709 12:23:31.561644 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.723752 (+2162108us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.723790 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.724398 (+   608us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.342770 (+618372us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:34.342816 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.342914 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.742510 (+399596us) raft_consensus.cc:1685] finished
0709 12:23:34.742538 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.742696 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2161561,"raft.queue_time_us":2162334,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":108,"thread_start_us":617908,"threads_started":1,"wal-append.queue_time_us":873142}
W20260709 12:23:34.793515 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2083545330001471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20260709 12:23:34.793957 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2083545330001471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
I20260709 12:23:34.761096 22911 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:34.794261 18064 connection.cc:256] RPC call timeout handler was delayed by -2.20385453300014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
W20260709 12:23:34.794572 18064 connection.cc:256] RPC call timeout handler was delayed by -2.16695453300008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
W20260709 12:23:34.794854 18064 connection.cc:256] RPC call timeout handler was delayed by -2.16695453300008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
W20260709 12:23:34.795122 18064 connection.cc:256] RPC call timeout handler was delayed by -2.079654532999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20260709 12:23:34.763041 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:34.663232 22965 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:34.701303 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.739272 22966 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.670213 23032 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:34.809404 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46519.2us
I20260709 12:23:34.778052 23042 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:34.787314 18096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.008s	sys 0.017s
I20260709 12:23:34.651631 22694 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.126s
I20260709 12:23:34.662289 22726 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.121s
W20260709 12:23:34.795374 18064 connection.cc:256] RPC call timeout handler was delayed by -2.069754532999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20260709 12:23:34.814931 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0472545329998866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
W20260709 12:23:34.815533 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0445545329998822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19310000000000002s for a response
W20260709 12:23:34.816037 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0445545329998822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19310000000000002s for a response
I20260709 12:23:34.817015 22944 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:23:34.816614 18064 connection.cc:256] RPC call timeout handler was delayed by -2.01845453299984s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
W20260709 12:23:34.818382 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0130545329998313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
I20260709 12:23:34.780972 23066 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:34.813597 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:34.821110 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.785403 22975 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:34.729015 23056 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.700208 23055 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.139s
W20260709 12:23:34.819101 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9716545329997643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20120000000000002s for a response
W20260709 12:23:34.841308 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9716545329997643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20120000000000002s for a response
W20260709 12:23:34.844331 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9671545329997571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:23:34.845098 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9671545329997571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:23:34.845650 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9176545330001318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
I20260709 12:23:34.828020 22836 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.821491 22965 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:23:34.820859 22903 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.825030 22560 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.818329 23032 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.848052 23061 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.890978 23032 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.890162 23061 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.806571 22977 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.896799 22977 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.842684 23066 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:34.898321 23066 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.850399 23063 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:34.902133 23059 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.195s	user 0.001s	sys 0.001s
I20260709 12:23:34.902679 23063 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:34.791221 22827 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:34.785395 22996 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:23:34.863373 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2351) took 4755 ms. Trace:
W20260709 12:23:34.911751 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 83886.1us
I20260709 12:23:34.839250 22911 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:34.795881 22985 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:23:34.935777 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.941401 18017 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:34.941291 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:34.840003 22995 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.867707 23038 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.368s	user 0.019s	sys 0.000s
I20260709 12:23:34.920732 18090 rpcz_store.cc:276] 0709 12:23:30.108332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.920157 (+811825us) service_pool.cc:224] Handling call
0709 12:23:30.937009 (+ 16852us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.937461 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:31.006030 (+ 68569us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.923708 (+2917678us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.322309 (+398601us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:34.343878 (+ 21569us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.563822 (+219944us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:34.563872 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.563958 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.862941 (+298983us) raft_consensus.cc:1685] finished
0709 12:23:34.863083 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.863290 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2292100,"prepare.queue_time_us":81953,"prepare.run_cpu_time_us":164,"prepare.run_wall_time_us":11710,"raft.queue_time_us":2193561,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":240,"spinlock_wait_cycles":139315968,"thread_start_us":945322,"threads_started":2,"wal-append.queue_time_us":1314491,"wal-append.run_cpu_time_us":2800,"wal-append.run_wall_time_us":1276506}
W20260709 12:23:34.946128 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8780545330000677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20260709 12:23:34.946837 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8753545330000634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21190000000000003s for a response
I20260709 12:23:34.888326 22929 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.878756 22835 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:34.854719 23056 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.844094 23064 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.104s
I20260709 12:23:34.955622 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.944873 22986 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.957135 18022 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 12:23:34.967504 18022 pending_rounds.cc:85] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 8
I20260709 12:23:34.967854 18022 pending_rounds.cc:107] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 12:23:34.888636 22560 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.969884 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.883378 22965 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.078s
I20260709 12:23:34.970768 18019 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:34.861960 22864 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:34.862512 22975 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.919402 23042 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:23:34.876384 22817 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.949046 22995 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.987131 22995 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:34.886212 22903 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.929338 22943 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.988400 22943 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:34.955461 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:34.956696 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:34.968155 18022 raft_consensus.cc:2889] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.946728 23061 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.975814 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.999564 18015 raft_consensus.cc:2368] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:35.000568 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.001386 18016 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
W20260709 12:23:35.014151 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 266751us
I20260709 12:23:34.881289 22483 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.021025 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:34.992641 22975 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:34.960567 22598 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:23:34.960543 22944 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:34.953516 23038 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:34.947733 22694 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 12:23:34.960296 23063 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:34.983317 22964 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.661s	user 0.014s	sys 0.000s
W20260709 12:23:34.984094 22989 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.726s	user 0.015s	sys 0.000s
I20260709 12:23:34.995268 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 649) took 1015 ms. Trace:
I20260709 12:23:34.976949 22986 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.026398 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.023335 23061 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:23:35.030687 22995 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:35.035657 22840 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:35.044185 22944 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.024845 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:35.038520 22943 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:35.065150 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:34.967375 22827 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
W20260709 12:23:34.957978 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7898545329999251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260709 12:23:35.066139 18064 connection.cc:256] RPC call timeout handler was delayed by -1.781754532999912s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
W20260709 12:23:35.066571 18064 connection.cc:256] RPC call timeout handler was delayed by -1.781754532999912s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
W20260709 12:23:35.067273 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3938545329997396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2654s for a response
W20260709 12:23:35.067734 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3938545329997396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2654s for a response
W20260709 12:23:35.068943 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3596545330001391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26920000000000005s for a response
I20260709 12:23:35.056579 22989 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:35.059887 22995 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.070153 22995 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.071657 22986 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:35.074321 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49612.5us
W20260709 12:23:35.069793 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2588545329999761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2804s for a response
W20260709 12:23:35.043406 18097 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.016s	sys 0.016s
I20260709 12:23:35.017633 22560 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:35.051144 18023 rpcz_store.cc:276] 0709 12:23:33.979977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.980653 (+   676us) service_pool.cc:224] Handling call
0709 12:23:33.981038 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.981319 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:34.010710 (+ 29391us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.462024 (+451314us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.580946 (+118922us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b18003b82a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:23:34.994632 (+413686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.994729 (+    97us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:34.994822 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.994966 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.995158 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31449,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":124,"raft.queue_time_us":158620,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":16555,"spinlock_wait_cycles":16453504,"thread_start_us":832792,"threads_started":2,"wal-append.queue_time_us":616076}
W20260709 12:23:34.990597 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 466304us
I20260709 12:23:34.992272 22864 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
W20260709 12:23:35.041790 22953 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.539s	user 0.012s	sys 0.004s
I20260709 12:23:35.039484 22774 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.060844 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.061576 18811 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 8: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:35.020543 22996 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.995070 22485 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:35.077041 22483 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:34.875762 23026 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.079118 23038 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:35.088697 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2588545329999761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2804s for a response
I20260709 12:23:35.098851 23061 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:35.091555 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:35.066375 22944 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:35.094822 22560 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:35.114010 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.100790 23085 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.117852 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:35.121562 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.125878 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:35.133771 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2209311970000272s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
W20260709 12:23:35.165696 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0347545330000685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3053s for a response
I20260709 12:23:35.167997 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.065946 22900 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:35.089104 18097 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.017s	sys 0.017s
I20260709 12:23:35.143549 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:35.171582 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80133.8us
I20260709 12:23:35.171988 18815 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:23:35.173133 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.173414 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.174041 18165 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:35.174530 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.116724 18019 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:35.106029 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.164049 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:35.163200 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 378745us
I20260709 12:23:35.126319 22996 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.174286 18172 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:35.176470 18808 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:35.176393 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58713.1us
I20260709 12:23:35.163008 18810 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:35.174160 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.180907 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:35.162711 18807 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.021s	sys 0.007s
W20260709 12:23:35.183662 18807 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.021s	sys 0.007s
I20260709 12:23:35.135414 22877 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:35.189740 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:35.108281 22929 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:23:35.205165 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.164091 22560 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:35.200695 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:35.205875 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.108453 23056 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.070s
W20260709 12:23:35.206352 18806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.015s	sys 0.030s
I20260709 12:23:35.145807 23061 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:35.108022 22953 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.214778 18173 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:35.214828 18091 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:23:35.215977 18091 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.216516 18091 raft_consensus.cc:614] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:35.212827 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:35.179489 22836 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:35.215432 18806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.015s	sys 0.031s
I20260709 12:23:35.201747 22986 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:35.217566 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:35.220351 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:35.222141 23061 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.225929 22989 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:35.226609 22989 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.226783 23042 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:35.227069 22989 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.227250 23042 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:35.227700 22953 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:35.227984 22952 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:35.228479 22952 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:35.228881 22952 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:35.227708 23042 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.134475 22485 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.182444 23038 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:35.247717 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:35.248409 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:35.260365 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.015s	sys 0.044s
I20260709 12:23:35.201336 22877 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:35.261932 18811 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:23:35.262354 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.016s	sys 0.044s
W20260709 12:23:35.266171 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83193.7us
I20260709 12:23:35.269172 18064 leader_election.cc:400] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:35.269737 18064 leader_election.cc:403] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:35.269110 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50333us
I20260709 12:23:35.249445 18172 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:35.262959 18167 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:35.274472 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.234149 22986 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.179596 23085 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.295833 18023 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:35.296358 22911 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:35.298623 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.300417 18064 leader_election.cc:400] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:35.300812 18064 leader_election.cc:403] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:23:35.301381 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:35.263300 23082 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:23:35.302137 18810 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.271673 23062 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:35.302469 23073 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:35.302812 23062 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.303304 23062 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:35.303336 23073 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:35.305127 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.305840 22900 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:35.313077 17921 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 5 to 6, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 6 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.314397 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:35.314626 23083 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:35.315371 23083 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.315706 17921 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 4 to 6, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 6 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:23:35.249877 22618 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.003s	sys 0.050s
I20260709 12:23:35.284045 18093 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:35.283420 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:35.244683 22852 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:35.288686 22906 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:35.291700 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.294960 22996 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:35.331631 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.010s	sys 0.008s
I20260709 12:23:35.328302 18813 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:35.329849 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.263361 23038 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.285878 22964 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:35.389521 18810 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:23:35.318104 22694 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:35.339000 18093 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:35.285521 22986 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:35.355891 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.360482 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.397295 18092 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:35.397431 18099 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:35.318148 23073 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:35.391702 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.404651 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:35.406033 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.406810 18096 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.400756 22694 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:35.408151 22694 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.410566 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.411456 18090 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:35.413872 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 108803us
I20260709 12:23:35.403200 23014 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:35.415390 22830 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.681s	user 0.006s	sys 0.009s
I20260709 12:23:35.416019 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.390972 22977 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:35.417197 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.399995 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3159) took 1755 ms. Trace:
I20260709 12:23:35.332762 22598 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:23:35.398564 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.390861 17921 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 7 to 13, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 13 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.325456 22485 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:35.402277 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.436081 18093 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:35.402359 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.437515 18091 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:35.347114 22852 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:35.368451 22964 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:35.325639 22618 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.006s	sys 0.054s
I20260709 12:23:35.420742 18020 rpcz_store.cc:276] 0709 12:23:33.644468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.644775 (+   307us) service_pool.cc:224] Handling call
0709 12:23:33.645099 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.645398 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:33.645781 (+   383us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.834635 (+1188854us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.925286 (+ 90651us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.331494 (+406208us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:35.398377 (+ 66883us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:35.398751 (+   374us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:35.399168 (+   417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.399687 (+   519us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.399920 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103696,"prepare.queue_time_us":6109,"prepare.run_cpu_time_us":178,"prepare.run_wall_time_us":2886,"raft.queue_time_us":355053,"raft.run_cpu_time_us":9332,"raft.run_wall_time_us":356163,"spinlock_wait_cycles":7762688,"thread_start_us":1659463,"threads_started":3,"wal-append.queue_time_us":988743}
I20260709 12:23:35.436350 18092 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:35.477278 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:35.393090 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 370062us
I20260709 12:23:35.429162 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:35.368129 23038 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:35.328303 23032 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.449615 18167 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:23:35.335700 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.011s	sys 0.009s
W20260709 12:23:35.458595 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 184199us
W20260709 12:23:35.487466 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 694889us
I20260709 12:23:35.489128 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.493393 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:35.494335 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:35.497803 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:35.498569 18807 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:35.480137 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:35.485934 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.007s	sys 0.018s
W20260709 12:23:35.500859 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.007s	sys 0.018s
I20260709 12:23:35.502871 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:35.510627 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.511116 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.511530 18023 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:35.511801 18019 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:35.511850 18022 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:35.512002 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.512867 18024 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:35.514127 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.515709 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:35.518913 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.520655 22985 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:35.523118 22953 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:35.523922 18017 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:35.523844 22953 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.525254 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.526034 18015 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:35.531165 18022 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:35.521520 22985 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.534638 22985 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:35.538839 18020 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:35.435002 22906 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:35.526870 22953 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:35.544664 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:35.550522 18814 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:35.550318 18096 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:35.559090 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3241) took 1264 ms. Trace:
W20260709 12:23:35.557056 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56664us
W20260709 12:23:35.554148 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49856.8us
I20260709 12:23:35.451443 22598 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:35.603081 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.416052 22830 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.599931 23063 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:23:35.614514 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.614737 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:35.615473 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 138735us
I20260709 12:23:35.603652 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.600936 18808 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:35.559806 18097 rpcz_store.cc:276] 0709 12:23:34.294737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.457522 (+162785us) service_pool.cc:224] Handling call
0709 12:23:34.501961 (+ 44439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.502223 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:35.171628 (+669405us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.558637 (+387009us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:35.558708 (+    71us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:35.558798 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.558877 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.559022 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3634,"prepare.queue_time_us":40901,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":6850,"raft.queue_time_us":3800,"thread_start_us":1024010,"threads_started":2}
I20260709 12:23:35.604498 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.495764 22774 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:35.605321 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.624940 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:35.606799 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:35.626937 18020 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.551275 23091 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:35.492306 22965 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.631839 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:35.559769 22929 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.660079 18099 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:35.637727 22965 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.643278 22877 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.644769 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.649529 22953 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:35.634760 22964 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.482746 22966 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:35.679394 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:35.662949 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.711227 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.686209 18807 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:35.722276 18091 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:35.681742 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:35.721932 18096 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.563333 17921 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.725235 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.780332 18092 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:35.691943 18020 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:35.739825 18097 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:35.728080 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:35.780997 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 302109us
I20260709 12:23:35.634299 23091 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.633239 22985 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:35.550676 22900 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:35.756035 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.803833 18090 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:35.765470 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.804699 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:35.758997 23042 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.527s	user 0.015s	sys 0.003s
I20260709 12:23:35.748289 22977 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:35.542524 23032 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:35.781810 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53814.2us
I20260709 12:23:35.599267 23064 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260709 12:23:35.788081 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 955) took 1127 ms. Trace:
I20260709 12:23:35.753238 22966 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.710997 22906 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 12:23:35.780859 22877 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:35.766203 22774 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:23:35.763166 22953 consensus_queue.cc:1243] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:35.772444 22996 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.752858 22852 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.713246 23056 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:23:35.564232 22943 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.209s
I20260709 12:23:35.771018 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.896695 18099 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.816728 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.898353 18092 raft_consensus.cc:2368] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:35.899436 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.900249 18092 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:35.711504 22836 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:23:35.903194 22836 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:35.903774 22836 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:35.904546 23098 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.818876 23042 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.872607 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:35.914173 18091 raft_consensus.cc:2376] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12. Re-sending same reply.
I20260709 12:23:35.834890 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.917364 18097 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 9 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
I20260709 12:23:35.836935 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:35.793696 22965 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:35.834138 23087 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:23:35.803573 22900 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.682560 22726 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
W20260709 12:23:35.835067 23080 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
W20260709 12:23:35.834713 23079 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.221s	user 0.000s	sys 0.000s
I20260709 12:23:35.784729 23091 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.833527 23089 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.875903 18809 rpcz_store.cc:276] 0709 12:23:34.660216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.660361 (+   145us) service_pool.cc:224] Handling call
0709 12:23:34.660748 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.711691 (+ 50943us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.711735 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.777528 (+ 65793us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.501590 (+724062us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:35.501661 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.501792 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.787780 (+285988us) raft_consensus.cc:1685] finished
0709 12:23:35.787810 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.787992 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40324,"raft.queue_time_us":188783,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":164,"thread_start_us":624471,"threads_started":1,"wal-append.queue_time_us":722587}
I20260709 12:23:35.905472 23098 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.911540 18807 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:35.821645 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 190950us
I20260709 12:23:35.977896 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:35.978612 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:35.834138 22598 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.980283 23109 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:35.980878 23109 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:35.981277 23109 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.901317 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.983727 23110 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:35.918762 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:35.919653 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.988205 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.988982 18812 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:35.877691 23085 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:35.993036 18812 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: [15.11-15.11]
W20260709 12:23:35.921272 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 84414.5us
I20260709 12:23:35.926635 22953 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:35.927456 18808 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260709 12:23:35.928462 23081 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
W20260709 12:23:35.929452 22560 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.708s	user 0.010s	sys 0.000s
I20260709 12:23:35.931772 23042 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:35.933390 22975 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.891s	user 0.008s	sys 0.004s
W20260709 12:23:35.934705 23094 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:23:36.036301 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.835141 23092 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:35.834972 23090 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:35.943064 18093 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.506s	user 0.013s	sys 0.000s
I20260709 12:23:36.040283 18093 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:23:36.041950 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.042337 18093 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:35.904829 22836 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:35.943835 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:35.945922 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:35.892699 22977 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:35.922600 22995 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.852s	user 0.008s	sys 0.011s
I20260709 12:23:35.947036 18099 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:23:36.030035 22952 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.801s	user 0.014s	sys 0.005s
I20260709 12:23:35.948144 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.962011 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.959951 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.068231 18165 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:36.068683 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:36.069196 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.070052 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:36.070359 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.068708 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:36.071031 18165 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:36.071383 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.071900 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:36.072492 18165 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:35.960453 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.967319 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.967115 23089 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:35.968367 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.969095 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.971653 22726 consensus_queue.cc:1243] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:35.951126 22900 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:35.997524 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:36.084268 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 86855.9us
I20260709 12:23:36.085256 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:36.000128 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.024680 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:35.966095 23061 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.743s	user 0.015s	sys 0.007s
I20260709 12:23:35.896384 22996 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:36.030651 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 348977us
I20260709 12:23:36.055835 23090 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.032544 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:35.897521 23086 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260709 12:23:36.161335 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:36.034600 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:35.947333 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.012s	sys 0.008s
W20260709 12:23:36.163403 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.013s	sys 0.008s
I20260709 12:23:36.065431 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:36.066882 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:36.012738 22953 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.073112 18813 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:36.077066 18168 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:36.074690 18173 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
W20260709 12:23:36.092554 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.144558 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:36.145112 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59982.8us
I20260709 12:23:36.149021 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.272424 18170 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:36.272591 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:36.149642 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 149611us
I20260709 12:23:36.150828 18064 leader_election.cc:400] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:36.273574 18064 leader_election.cc:403] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:36.274160 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 123537us
I20260709 12:23:36.031375 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.275856 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.162503 18166 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:36.174767 18165 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:36.238088 23106 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.166239 22483 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:36.230131 18094 raft_consensus.cc:2955] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:36.239884 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 205402us
I20260709 12:23:36.244093 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:36.236692 22953 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.047967 22560 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.144791 23056 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:36.150429 22827 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:36.271056 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 944) took 2238 ms. Trace:
I20260709 12:23:36.247695 22726 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.273006 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.542s (SENT)
I20260709 12:23:36.311772 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:36.275714 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:36.276612 18164 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:36.296602 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.073338 22995 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.069639 23092 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:36.265666 23038 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.722s	user 0.013s	sys 0.000s
W20260709 12:23:36.266366 22694 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.858s	user 0.009s	sys 0.005s
I20260709 12:23:36.078646 22952 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:36.304188 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 272918us
I20260709 12:23:36.271787 22906 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:36.313549 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.568s (SENT)
W20260709 12:23:36.334715 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59064.5us
I20260709 12:23:36.318949 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:36.356253 18168 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:36.309978 18806 rpcz_store.cc:276] 0709 12:23:34.013300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.013455 (+   155us) service_pool.cc:224] Handling call
0709 12:23:34.013879 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.569937 (+556058us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.569991 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.597500 (+ 27509us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.217444 (+619944us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:35.217493 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.217579 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.251299 (+1033720us) raft_consensus.cc:1685] finished
0709 12:23:36.251321 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.251485 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9281,"raft.queue_time_us":109274,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"thread_start_us":1084671,"threads_started":2,"wal-append.queue_time_us":827579}
I20260709 12:23:36.163805 23061 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.163646 23090 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:36.274209 23115 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:36.347690 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.371618 18171 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:36.353801 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.350690 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:36.355254 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:36.347466 22840 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:36.370853 18167 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:36.373554 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73079.3us
I20260709 12:23:36.377401 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.309254 22965 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:36.383971 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:36.323280 23020 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:36.317562 23056 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:36.394266 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:36.398138 18168 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:36.325137 23111 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:36.360862 22830 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:36.371009 23032 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:36.422938 18136 leader_election.cc:400] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:36.405591 23020 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:36.371696 22906 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.408726 23123 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.324615 22726 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.410287 23117 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:23:36.326535 23112 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:36.394899 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:36.384665 22840 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:36.404609 18166 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:36.362618 22694 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.425535 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:36.353374 23038 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.417387 22943 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:36.427925 22989 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.200s	user 0.005s	sys 0.011s
I20260709 12:23:36.410321 23089 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:36.487501 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 98502.5us
I20260709 12:23:36.480293 18166 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:36.497411 22827 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:36.351115 22977 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.448941 23126 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:36.504899 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 191384us
W20260709 12:23:36.519984 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.651s (SENT)
I20260709 12:23:36.524055 18807 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:36.507184 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 81734us
I20260709 12:23:36.525346 18173 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:36.526924 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.509258 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:36.499042 22827 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:36.531602 23108 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.237s	user 0.001s	sys 0.001s
I20260709 12:23:36.533144 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2414) took 1171 ms. Trace:
I20260709 12:23:36.533552 18098 rpcz_store.cc:276] 0709 12:23:35.361646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.361833 (+   187us) service_pool.cc:224] Handling call
0709 12:23:35.418870 (+ 57037us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.432203 (+ 13333us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:35.432244 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:35.432941 (+   697us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.163935 (+730994us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:36.163986 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.164071 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.532885 (+368814us) raft_consensus.cc:1685] finished
0709 12:23:36.532910 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.533071 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12786,"raft.queue_time_us":14316,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6626,"thread_start_us":730541,"threads_started":1,"wal-append.queue_time_us":640471}
I20260709 12:23:36.450839 22840 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:36.387388 23115 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:36.538239 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.765s (SENT)
I20260709 12:23:36.549451 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.485999 23117 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:36.512877 23106 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:36.512431 23112 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:36.580006 18166 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:36.514629 23126 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:36.554368 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 246767us
I20260709 12:23:36.516709 22965 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:36.570185 22560 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:36.572196 22694 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:36.555091 23127 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:23:36.555470 23113 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:36.444079 23020 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:36.579077 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 69927.9us
W20260709 12:23:36.549283 22986 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.115s	user 0.011s	sys 0.004s
I20260709 12:23:36.664399 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:36.580758 23131 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:36.587378 22483 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:36.604058 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.672292 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:36.582396 18092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.011s	sys 0.022s
W20260709 12:23:36.622076 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 348258us
W20260709 12:23:36.673249 18092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.015s	sys 0.028s
W20260709 12:23:36.673650 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 279475us
I20260709 12:23:36.556854 23117 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:36.674578 23117 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.619524 23125 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:36.626367 22944 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.444s	user 0.016s	sys 0.000s
I20260709 12:23:36.650285 23083 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.652369 22726 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.656885 23042 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:36.657193 23109 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.000s
W20260709 12:23:36.619874 22929 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.936s	user 0.017s	sys 0.000s
I20260709 12:23:36.665982 18809 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:36.514910 23116 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:23:36.691172 22944 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.670620 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:36.553876 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.947s (SENT)
I20260709 12:23:36.704806 22953 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.705145 23020 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.705418 22944 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:36.678785 23061 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.676275 22986 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.505682 22989 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:36.626367 23062 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.322s	user 0.019s	sys 0.000s
I20260709 12:23:36.673722 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.676692 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:36.681716 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.633916 23126 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:36.660380 23129 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:23:36.706162 18169 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:36.720130 18168 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:36.734550 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60936.2us
I20260709 12:23:36.703109 22929 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.705495 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.671159 23112 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:36.728127 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 878) took 1401 ms. Trace:
I20260709 12:23:36.746392 18810 rpcz_store.cc:276] 0709 12:23:35.326632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.390542 (+ 63910us) service_pool.cc:224] Handling call
0709 12:23:35.392682 (+  2140us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:35.909659 (+516977us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:35.909697 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:35.922316 (+ 12619us) log.cc:844] Serialized 52 byte log entry
0709 12:23:36.254915 (+332599us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:36.457557 (+202642us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.461597 (+  4040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.727471 (+265874us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54005429d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:36.727560 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.727670 (+   110us) raft_consensus.cc:1685] finished
0709 12:23:36.727792 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.728029 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24875,"prepare.queue_time_us":271089,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":58,"raft.queue_time_us":25063,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":37380,"spinlock_wait_cycles":1132462848,"thread_start_us":382229,"threads_started":2,"wal-append.queue_time_us":449690}
W20260709 12:23:36.751595 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 200513us
I20260709 12:23:36.699528 23127 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.733556 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57003.8us
I20260709 12:23:36.722499 22737 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:36.766682 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:36.767391 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 12:23:36.743578 22900 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.581s	user 0.014s	sys 0.003s
W20260709 12:23:36.700435 23091 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.730s	user 0.007s	sys 0.008s
I20260709 12:23:36.700692 23092 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:36.707409 22906 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:36.757405 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:36.758611 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:36.763468 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:36.744933 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 191125us
W20260709 12:23:36.778888 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.067s (SENT) [suppressed 17 similar messages]
I20260709 12:23:36.768091 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:36.755215 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:36.675112 18064 leader_election.cc:400] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:36.771070 23127 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.715627 23038 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.718482 23115 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.797497 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:36.780136 18064 leader_election.cc:403] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:36.780277 18094 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:36.713406 23125 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:36.744433 22560 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.775373 22900 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.781179 18098 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:36.819226 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:36.748119 23086 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:36.773691 22737 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:36.798792 18024 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:36.830538 18022 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:36.803296 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:36.805141 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:36.782769 22877 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.898s	user 0.014s	sys 0.002s
I20260709 12:23:36.834846 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:36.777631 22852 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:36.828217 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3260) took 1032 ms. Trace:
I20260709 12:23:36.749073 23020 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:36.706115 22483 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:36.827359 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 152381us
I20260709 12:23:36.705747 23109 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.838217 18019 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:36.847014 18172 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:36.839447 18097 rpcz_store.cc:276] 0709 12:23:35.796056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.925616 (+129560us) service_pool.cc:224] Handling call
0709 12:23:35.955150 (+ 29534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.965485 (+ 10335us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:36.048988 (+ 83503us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.524898 (+475910us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.525479 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.525657 (+   178us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:36.525694 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.525784 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.827918 (+302134us) raft_consensus.cc:1685] finished
0709 12:23:36.827946 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.828129 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36603,"prepare.queue_time_us":59358,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":200310,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":21487,"thread_start_us":363450,"threads_started":1,"wal-append.queue_time_us":445504}
I20260709 12:23:36.838954 22852 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:36.822248 23106 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.700282 23134 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.113s
I20260709 12:23:36.746795 23062 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.779042 22944 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:23:36.706722 22965 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.859946 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.818192 22827 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:36.830024 23141 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.831585 23014 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.828706 22835 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:36.834295 23086 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.908572 23086 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.909092 22835 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.910527 18811 raft_consensus.cc:2368] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
W20260709 12:23:36.870677 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.834775 23115 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:36.879801 22726 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:36.912878 22835 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:36.913623 23141 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.914323 22986 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:36.914489 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:36.915270 18810 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:36.862040 22852 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:36.769073 23142 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.111s
I20260709 12:23:36.777138 23091 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.815137 22977 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:23:36.901789 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:36.906580 22900 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:36.907891 23062 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.917943 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:36.944888 18810 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:36.795078 23092 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:23:36.845659 22737 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:36.903494 22995 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:36.915480 22827 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:36.924288 23119 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:36.976223 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74596.3us
I20260709 12:23:36.983042 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.003823 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.002993 23115 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:37.007571 22929 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:37.005522 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.009791 18810 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:37.018172 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:36.923997 23014 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:36.989527 22877 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.989341 22726 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:37.004150 22737 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:36.888833 23106 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.012275 22953 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.894753 23112 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.029510 23091 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:37.031060 23127 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:37.034204 22827 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.045923 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:37.035640 23109 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:37.038458 23106 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:37.020319 22989 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:36.971791 23125 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:37.065874 23151 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:23:37.049252 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:37.068617 22694 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:37.023497 22929 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:37.051919 23105 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.699s	user 0.002s	sys 0.000s
I20260709 12:23:37.051766 22840 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.049371 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.053550 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:37.075300 18098 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:37.056545 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.076751 18099 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:37.080838 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3262) took 1284 ms. Trace:
I20260709 12:23:37.081508 18092 rpcz_store.cc:276] 0709 12:23:35.796620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.984446 (+187826us) service_pool.cc:224] Handling call
0709 12:23:35.984877 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.062139 (+ 77262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:36.062192 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.063168 (+   976us) log.cc:844] Serialized 26 byte log entry
0709 12:23:37.065488 (+1002320us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b1800478840. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1
0709 12:23:37.065698 (+   210us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:37.065744 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.069647 (+  3903us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.069704 (+    57us) raft_consensus.cc:1685] finished
0709 12:23:37.069719 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.080721 (+ 11002us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":248868,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":137,"spinlock_wait_cycles":7374720,"thread_start_us":686811,"threads_started":2,"wal-append.queue_time_us":607814}
I20260709 12:23:37.062603 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.022135 23119 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:37.069221 22694 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:37.092159 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.074940 23141 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.027709 22877 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.104353 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:37.098610 23090 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.706s	user 0.016s	sys 0.001s
W20260709 12:23:37.136324 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.090318 23119 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:37.094909 23111 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:37.090705 23042 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:37.107788 23115 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:37.177287 18098 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:37.156700 22929 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 12:23:37.183713 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:37.191078 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:37.169071 23119 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.204979 22840 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:37.207584 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:37.210999 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.195226 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:37.213246 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:37.213982 18098 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:37.215660 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 436775us
I20260709 12:23:37.226058 18812 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:37.162864 23090 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.236514 22877 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.238672 22906 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:37.242404 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:37.177201 22694 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.254245 22989 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:37.255082 22989 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:37.256044 22989 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:37.258770 23117 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.007s
I20260709 12:23:37.259511 23117 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.261116 23090 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.227909 23143 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:37.262384 23143 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:37.254448 23014 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.222400 23067 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:37.210075 23111 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
W20260709 12:23:37.267434 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.060s (SENT)
I20260709 12:23:37.257390 22852 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.262351 23117 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.268069 23067 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.310194 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.271492 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2426) took 1570 ms. Trace:
I20260709 12:23:37.319734 18095 rpcz_store.cc:276] 0709 12:23:35.701422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.701579 (+   157us) service_pool.cc:224] Handling call
0709 12:23:35.712472 (+ 10893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.712735 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:35.724024 (+ 11289us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.990927 (+1266903us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b18003d6840. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:23:37.270881 (+279954us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.270972 (+    91us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:37.271059 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.271168 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.271391 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11306,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":9575,"raft.queue_time_us":178683,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"spinlock_wait_cycles":61572992,"thread_start_us":279724,"threads_started":1}
I20260709 12:23:37.323477 18098 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:37.334448 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67045.4us
W20260709 12:23:37.335564 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.133s (SENT)
I20260709 12:23:37.336198 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.290180 23111 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:37.343914 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.265980 23042 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.322871 23067 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:37.353248 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.003s	sys 0.015s
W20260709 12:23:37.353600 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.355535 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.004s	sys 0.016s
W20260709 12:23:37.355173 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.003s	sys 0.016s
W20260709 12:23:37.357678 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.003s	sys 0.016s
I20260709 12:23:37.337358 22877 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:37.341248 23102 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:37.378047 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 722) took 1347 ms. Trace:
I20260709 12:23:37.388839 18168 rpcz_store.cc:276] 0709 12:23:36.030983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.735146 (+704163us) service_pool.cc:224] Handling call
0709 12:23:36.751878 (+ 16732us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:36.752119 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:36.834218 (+ 82099us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.377382 (+543164us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.377629 (+   247us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.377682 (+    53us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:37.377741 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.377839 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.377973 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4740,"prepare.queue_time_us":193984,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":4115,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":27575,"thread_start_us":578717,"threads_started":2,"wal-append.queue_time_us":513197}
I20260709 12:23:37.387758 18098 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:23:37.393671 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.392220 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56709.8us
I20260709 12:23:37.384673 23162 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:37.337893 23143 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.352336 23111 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:37.393265 22830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.014s	sys 0.045s
I20260709 12:23:37.413111 18168 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 12:23:37.418172 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 213045us
I20260709 12:23:37.427868 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:37.412165 22830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.014s	sys 0.045s
I20260709 12:23:37.396879 23162 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:37.419798 23115 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:37.438946 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.442180 18099 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:37.443666 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:37.435592 22965 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.449107 23166 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
W20260709 12:23:37.452251 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50759.5us
I20260709 12:23:37.441412 23026 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.450821 22877 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.490698 23168 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:23:37.491348 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 870) took 2379 ms. Trace:
I20260709 12:23:37.491808 18814 rpcz_store.cc:276] 0709 12:23:35.111423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.112092 (+   669us) service_pool.cc:224] Handling call
0709 12:23:35.144307 (+ 32215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.599058 (+454751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.491011 (+1891953us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:37.491266 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933190912}
W20260709 12:23:37.492425 23169 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:23:37.483003 23162 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.495452 23162 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.503181 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:37.510360 23167 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:23:37.511411 23026 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.514695 23113 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:37.515421 23113 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:37.483314 22852 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.534711 18096 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 12:23:37.501583 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73835.3us
I20260709 12:23:37.540313 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:37.500046 23086 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.591s	user 0.013s	sys 0.000s
W20260709 12:23:37.546443 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43792us
W20260709 12:23:37.549290 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.522454 22966 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.550853 23026 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:37.522317 23170 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.001s
W20260709 12:23:37.522158 23160 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.180s	user 0.002s	sys 0.000s
I20260709 12:23:37.560139 18810 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:37.598997 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:37.601936 18814 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:37.603466 22989 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:37.479676 22965 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.610137 22989 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.598577 23115 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.610983 23174 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:37.633488 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:37.612198 23176 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:23:37.612134 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71946.7us
W20260709 12:23:37.642985 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.615309 23179 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.137s	user 0.000s	sys 0.002s
I20260709 12:23:37.615530 22864 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.617592 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:37.552695 23086 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.668354 23183 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:37.605038 23175 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:37.639806 23173 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.643517 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.634332 22985 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.650581 18814 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:37.660852 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.641202 23111 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:37.661769 22864 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.684865 23111 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:37.685415 23111 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:37.663545 23185 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:37.685894 23111 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:37.686308 23185 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:37.686712 23185 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:37.686731 22864 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.687780 23185 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.665091 23184 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 12:23:37.666409 23161 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:23:37.667291 23174 consensus_queue.cc:1243] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 12.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:23:37.691920 23174 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:37.692315 23174 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:37.667479 23089 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:23:37.605283 23119 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.694360 23089 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:37.694798 23089 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:37.695704 23089 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.696523 23119 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.607193 22966 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.697775 22966 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.698913 23174 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.689610 23185 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:37.700541 23111 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.700858 22966 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.701992 18093 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:37.544569 22877 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.644459 22737 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:37.684772 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:37.647406 22827 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260709 12:23:37.704945 22737 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:37.706375 22836 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:37.691128 22975 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:37.706885 22836 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:37.707510 22836 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:37.697132 23089 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:37.742867 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 129731us
W20260709 12:23:37.750677 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65984.6us
I20260709 12:23:37.711804 22965 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.762749 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.762648 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:37.770622 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.772266 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.716758 22774 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:23:37.773782 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:37.778537 18814 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:37.774140 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.779807 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.775321 18098 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:37.780157 18094 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:37.781853 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:37.781014 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99903.7us
I20260709 12:23:37.781061 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.784039 18097 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:37.706899 22737 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:37.716481 22985 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.786689 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.791152 18098 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:37.775701 22965 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.795064 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.796983 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.773800 23026 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.798475 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.780594 22989 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.787412 23117 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:37.712199 23173 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:37.818958 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2484) took 1808 ms (client timeout 1806 ms). Trace:
W20260709 12:23:37.820919 18167 rpcz_store.cc:269] 0709 12:23:36.010723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.071122 (+ 60399us) service_pool.cc:224] Handling call
0709 12:23:37.818641 (+1747519us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b540007bf54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:37.818863 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":686518528}
I20260709 12:23:37.819901 18166 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:37.823944 23099 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 2.027s	user 0.002s	sys 0.004s
I20260709 12:23:37.824877 22996 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:37.794507 23111 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.830258 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.831185 18097 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:37.832715 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 875) took 2667 ms. Trace:
I20260709 12:23:37.837038 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:37.837455 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 156058us
I20260709 12:23:37.837929 18810 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:37.838932 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.840032 23086 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.840672 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:37.841537 18092 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:37.840875 18809 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:37.841574 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:37.842885 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:37.833418 18815 rpcz_store.cc:276] 0709 12:23:35.164960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.214409 (+ 49449us) service_pool.cc:224] Handling call
0709 12:23:35.230797 (+ 16388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.263720 (+ 32923us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:35.263757 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:35.285461 (+ 21704us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.459123 (+173662us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:35.459194 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.459310 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.733343 (+2274033us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:37.825891 (+ 92548us) raft_consensus.cc:1685] finished
0709 12:23:37.825920 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.832590 (+  6670us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32445,"raft.queue_time_us":80265,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":122,"spinlock_wait_cycles":15929216,"thread_start_us":167653,"threads_started":1,"wal-append.queue_time_us":341788}
I20260709 12:23:37.844599 18092 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.842991 18813 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:37.820765 23117 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:37.853166 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:37.854404 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:37.832800 23075 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:37.800366 22965 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:37.859611 23187 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:23:37.859995 23186 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:37.861089 22964 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.862349 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2295) took 1857 ms. Trace:
W20260709 12:23:37.858352 23126 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.089s	user 0.019s	sys 0.007s
I20260709 12:23:37.873855 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:37.873067 18164 rpcz_store.cc:276] 0709 12:23:36.010366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.323546 (+313180us) service_pool.cc:224] Handling call
0709 12:23:36.324016 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.327711 (+  3695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:36.327756 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.361017 (+ 33261us) log.cc:844] Serialized 26 byte log entry
0709 12:23:37.552076 (+1191059us) spinlock_profiling.cc:243] Waited 9.44 ms on lock 0x7b18003ab040. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:37.690835 (+138759us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:37.690884 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.690985 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.861920 (+170935us) raft_consensus.cc:1685] finished
0709 12:23:37.861947 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.862250 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14491,"raft.queue_time_us":63233,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":4824,"spinlock_wait_cycles":21717888,"thread_start_us":138510,"threads_started":1,"wal-append.queue_time_us":1389836}
I20260709 12:23:37.882403 23062 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:37.883217 23062 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:37.884541 23062 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.851599 23056 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.887020 23056 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:37.887899 22965 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.858539 23117 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:37.889285 23146 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.047s	user 0.001s	sys 0.004s
I20260709 12:23:37.863015 23075 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.898443 23075 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:37.903539 23098 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:37.918684 22985 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:37.872581 23186 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:37.927277 23062 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.942090 23173 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:37.929605 23186 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:37.905335 23126 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:37.942090 22952 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:38.032871 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.034582 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 196144us
W20260709 12:23:38.035200 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 180744us
I20260709 12:23:38.037021 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.038694 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.039469 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.045596 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.046890 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.050386 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2294) took 2045 ms. Trace:
I20260709 12:23:38.056104 22903 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 12:23:38.059871 22952 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:38.060904 22952 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:38.061602 22952 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:38.053174 22774 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.073011 22774 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.051246 18165 rpcz_store.cc:276] 0709 12:23:36.004328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.294950 (+290622us) service_pool.cc:224] Handling call
0709 12:23:36.295349 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.374625 (+ 79276us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:36.374673 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.375220 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.677428 (+302208us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:36.677472 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.677552 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.888385 (+1210833us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:38.049138 (+160753us) raft_consensus.cc:1685] finished
0709 12:23:38.049228 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.049937 (+   709us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9003,"raft.queue_time_us":31803,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":6801,"spinlock_wait_cycles":5813504,"thread_start_us":318573,"threads_started":2,"wal-append.queue_time_us":367855}
W20260709 12:23:38.067605 17988 connection.cc:256] RPC call timeout handler was delayed by -1.5051266629998281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20260709 12:23:38.090598 17988 connection.cc:256] RPC call timeout handler was delayed by -1.2045266630002516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
W20260709 12:23:38.091089 17988 connection.cc:256] RPC call timeout handler was delayed by -1.1883085900003607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260709 12:23:38.091576 17988 connection.cc:256] RPC call timeout handler was delayed by -1.130344943000182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1539s for a response
I20260709 12:23:38.094300 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:38.095142 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2482) took 2131 ms (client timeout 1511 ms). Trace:
I20260709 12:23:38.095850 18170 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:23:38.096915 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2288) took 2094 ms (client timeout 1790 ms). Trace:
W20260709 12:23:38.097397 18171 rpcz_store.cc:269] 0709 12:23:36.002427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.002574 (+   147us) service_pool.cc:224] Handling call
0709 12:23:38.096364 (+2093790us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54001b4254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.096661 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508148992}
W20260709 12:23:38.095685 18169 rpcz_store.cc:269] 0709 12:23:35.963060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.966979 (+  3919us) service_pool.cc:224] Handling call
0709 12:23:38.094474 (+2127495us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b5400032ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.094808 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1453994368}
I20260709 12:23:38.104031 22774 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.109627 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2440) took 1003 ms. Trace:
I20260709 12:23:38.110387 18093 rpcz_store.cc:276] 0709 12:23:37.106050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.377426 (+271376us) service_pool.cc:224] Handling call
0709 12:23:37.384037 (+  6611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.702540 (+318503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.108956 (+406416us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:38.109375 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":730450304}
I20260709 12:23:38.115697 18815 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:38.126224 23126 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.128072 23175 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:23:38.129076 23067 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.752s	user 0.016s	sys 0.006s
I20260709 12:23:38.129290 23175 consensus_queue.cc:1243] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 12.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:23:38.153561 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:38.145112 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.155895 18094 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:38.158120 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:38.158406 23113 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.642s	user 0.027s	sys 0.000s
I20260709 12:23:38.159091 23113 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.137293 22774 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:38.130342 23067 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.160528 22483 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:38.161147 22483 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.161525 22483 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:38.139648 23161 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:23:38.159610 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1100) took 2203 ms (client timeout 1570 ms). Trace:
W20260709 12:23:38.163219 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64714.5us
I20260709 12:23:38.155951 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.164151 18807 rpcz_store.cc:269] 0709 12:23:35.956363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.978219 (+ 21856us) service_pool.cc:224] Handling call
0709 12:23:38.159048 (+2180829us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54007823d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.159329 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1251672960}
I20260709 12:23:38.179064 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.191870 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:38.193327 23193 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:38.192618 23089 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:23:38.192792 23186 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.203977 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.208202 23073 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:38.208803 23073 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:38.223963 18094 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:38.231379 23073 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.233304 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.239146 18091 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:38.242784 23178 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.635s	user 0.002s	sys 0.000s
I20260709 12:23:38.253881 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:38.259946 22877 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:38.262014 22774 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:38.262812 22774 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:38.264339 22774 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:38.268036 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.268467 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.278893 22944 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:38.279731 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2307) took 2265 ms (client timeout 2276 ms). Trace:
I20260709 12:23:38.280074 22944 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:38.280398 18173 rpcz_store.cc:269] 0709 12:23:36.014196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.277669 (+263473us) service_pool.cc:224] Handling call
0709 12:23:38.279175 (+2001506us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.279468 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":483845888}
I20260709 12:23:38.282363 22911 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.283236 22840 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.283612 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2427) took 1238 ms. Trace:
I20260709 12:23:38.283448 22911 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.284178 22840 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.284132 18090 rpcz_store.cc:276] 0709 12:23:37.045437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.046578 (+  1141us) service_pool.cc:224] Handling call
0709 12:23:37.046976 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.552765 (+505789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.552811 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.552872 (+    61us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:37.552919 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.282932 (+730013us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:38.283018 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.283357 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1162575232}
I20260709 12:23:38.268625 22774 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.293146 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:38.298839 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:38.301570 22911 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.291626 23073 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:38.287621 18092 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:38.282464 22835 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.287114 23067 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.307888 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:38.312563 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.311563 18094 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
W20260709 12:23:38.320688 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 162946us
I20260709 12:23:38.322355 22877 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:38.324391 22840 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.326741 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.336879 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:38.333786 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.341797 18807 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.350709 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.349200 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.354379 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.356873 23159 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:38.362919 18173 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:38.358031 23159 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.333130 22952 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.371160 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:38.351202 23113 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:38.378152 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.378547 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.382755 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2431) took 1332 ms. Trace:
I20260709 12:23:38.387478 18808 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:38.390791 22985 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:38.400871 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48863.5us
I20260709 12:23:38.349067 22835 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.355624 22877 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:38.394635 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 160154us
I20260709 12:23:38.410667 18098 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:38.403645 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.409739 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:38.411604 18093 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
W20260709 12:23:38.393996 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2479) took 2434 ms (client timeout 1908 ms). Trace:
I20260709 12:23:38.381582 23174 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:38.403168 18091 rpcz_store.cc:276] 0709 12:23:37.050508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.051741 (+  1233us) service_pool.cc:224] Handling call
0709 12:23:37.052089 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.239842 (+1187753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.382325 (+142483us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54002c9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:38.382655 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2728245504}
I20260709 12:23:38.413193 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2309) took 2398 ms. Trace:
I20260709 12:23:38.401114 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:38.348608 23067 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:38.355574 23089 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:38.416946 18090 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:38.421914 23020 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.415479 18172 rpcz_store.cc:269] 0709 12:23:35.959887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.960076 (+   189us) service_pool.cc:224] Handling call
0709 12:23:38.393432 (+2433356us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.393730 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1778518784}
I20260709 12:23:38.423414 18173 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260709 12:23:38.423669 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.430694 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.430931 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:38.418928 18170 rpcz_store.cc:276] 0709 12:23:36.014860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.647775 (+632915us) service_pool.cc:224] Handling call
0709 12:23:38.094914 (+1447139us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.096608 (+  1694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.408186 (+311578us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400032ad0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.413056 (+  4870us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3326646016}
I20260709 12:23:38.353678 22830 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.393344 23186 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.435382 18169 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:38.425102 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2444) took 1178 ms. Trace:
I20260709 12:23:38.437865 23020 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.400434 22694 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.443053 18095 rpcz_store.cc:276] 0709 12:23:37.246864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.322298 (+ 75434us) service_pool.cc:224] Handling call
0709 12:23:38.424547 (+1102249us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.424853 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1045866240}
I20260709 12:23:38.456326 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.453701 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.458308 23201 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:23:38.459095 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.459942 18098 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:38.460925 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.463001 23117 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.574s	user 0.017s	sys 0.006s
I20260709 12:23:38.463661 23117 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.466241 18097 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:38.474403 18092 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
W20260709 12:23:38.478994 22975 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.772s	user 0.001s	sys 0.024s
I20260709 12:23:38.479743 23089 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.514608 23159 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:38.480674 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.482095 18091 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:38.515799 23089 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.509464 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:38.516477 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:38.520854 23181 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.908s	user 0.000s	sys 0.013s
I20260709 12:23:38.483701 22835 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:38.487452 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.490293 23067 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.491813 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:38.492851 22877 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:38.498304 23113 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.502521 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:38.505263 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:38.506263 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:38.507115 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:38.507975 22483 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.508599 18164 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:38.509886 18814 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:38.510291 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.511241 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:38.524173 18811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 964) took 1852 ms (client timeout 2109 ms). Trace:
I20260709 12:23:38.528609 23193 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:38.483060 22952 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.535130 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.538568 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.539578 22975 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:38.564635 23090 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:23:38.568162 23086 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:23:38.572808 17922 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 293) took 33869 ms (client timeout 15000 ms). Trace:
I20260709 12:23:38.574940 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:38.582602 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2306) took 2565 ms. Trace:
I20260709 12:23:38.569389 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:38.583473 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.624450 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 169271us
I20260709 12:23:38.543387 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.472450 22944 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.587016 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3183) took 3068 ms. Trace:
I20260709 12:23:38.611665 18807 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:38.605122 18090 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:23:38.604580 18811 rpcz_store.cc:269] 0709 12:23:36.671790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.671988 (+   198us) service_pool.cc:224] Handling call
0709 12:23:38.523611 (+1851623us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b540070f8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.523924 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545505024}
I20260709 12:23:38.620582 18093 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:23:38.537806 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 182251us
W20260709 12:23:38.630515 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.622161 18166 rpcz_store.cc:276] 0709 12:23:36.013891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.581329 (+567438us) service_pool.cc:224] Handling call
0709 12:23:37.819563 (+1238234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.823094 (+  3531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.579243 (+756149us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540007bf50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.579510 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2846517632}
I20260709 12:23:38.609524 23193 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:38.632189 18023 rpcz_store.cc:276] 0709 12:23:35.518783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.518914 (+   131us) service_pool.cc:224] Handling call
0709 12:23:35.519242 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.774594 (+1255352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.586280 (+1811686us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540010a9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:38.586704 (+   424us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2862354176}
I20260709 12:23:38.629678 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:38.644698 18165 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:38.651213 18091 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260709 12:23:38.637637 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1103) took 2558 ms (client timeout 2188 ms). Trace:
W20260709 12:23:38.654017 18808 rpcz_store.cc:269] 0709 12:23:36.079321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.237201 (+157880us) service_pool.cc:224] Handling call
0709 12:23:38.620556 (+2383355us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b540064e454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:38.637541 (+ 16985us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4801387776}
I20260709 12:23:38.661736 18170 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:38.661514 18172 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:38.662753 18173 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:38.664326 18138 connection.cc:256] RPC call timeout handler was delayed by -1.4913974850001068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
W20260709 12:23:38.664827 18138 connection.cc:256] RPC call timeout handler was delayed by -1.3745474369998192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
I20260709 12:23:38.666429 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:38.667471 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50136.7us
I20260709 12:23:38.669013 23186 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:38.665129 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2790462159996423s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20260709 12:23:38.674901 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:38.676340 18811 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:38.677770 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:38.683929 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:38.623951 22830 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:38.687717 23075 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.775s	user 0.018s	sys 0.005s
I20260709 12:23:38.688786 23075 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.690120 23173 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:38.690320 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2450) took 1177 ms. Trace:
I20260709 12:23:38.690407 22929 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:38.690981 23173 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.691356 22929 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.653271 23086 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:38.690152 22726 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.699091 22726 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.708525 18098 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:23:38.671662 23186 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:38.705471 22726 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.690943 18096 rpcz_store.cc:276] 0709 12:23:37.512702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.522151 (+  9449us) service_pool.cc:224] Handling call
0709 12:23:37.522617 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.537031 (+ 14414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.641996 (+1104965us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b540032b9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:38.690100 (+ 48104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11701120}
I20260709 12:23:38.661433 23117 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.649866 23113 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:38.718158 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62098.1us
I20260709 12:23:38.723333 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:38.715150 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.724934 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 213851us
I20260709 12:23:38.713516 23056 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.740027 18808 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:38.700394 23075 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.713848 22929 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.721458 22694 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:38.737537 23067 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.763634 23130 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:38.767990 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:38.769197 23117 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:38.769649 23117 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.770073 23117 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:38.773327 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.621155 17922 rpcz_store.cc:269] 0709 12:23:04.703524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.703771 (+   247us) service_pool.cc:224] Handling call
0709 12:23:38.572437 (+33868666us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:19.773670 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:20.030837 (+257167us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:20.030910 (+    73us) write_op.cc:435] Acquired schema lock
0709 12:23:20.030922 (+    12us) tablet.cc:662] Decoding operations
0709 12:23:20.186502 (+155580us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:20.208955 (+ 22453us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:20.242130 (+ 33175us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:23:20.267059 (+ 24929us) tablet.cc:701] Row locks acquired
0709 12:23:20.267134 (+    75us) write_op.cc:260] PREPARE: finished
0709 12:23:20.267433 (+   299us) write_op.cc:270] Start()
0709 12:23:20.267671 (+   238us) write_op.cc:276] Timestamp: P: 1783599800267318 usec, L: 0
0709 12:23:20.267719 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 12:23:20.268817 (+  1098us) log.cc:844] Serialized 6266 byte log entry
0709 12:23:21.995698 (+1726881us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b
0709 12:23:22.458237 (+462539us) op_driver.cc:464] REPLICATION: finished
0709 12:23:22.666210 (+207973us) write_op.cc:301] APPLY: starting
0709 12:23:22.715102 (+ 48892us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:23.006998 (+291896us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:23.016020 (+  9022us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:23.584474 (+568454us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:23.898032 (+313558us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 12:23:23.898117 (+    85us) write_op.cc:312] APPLY: finished
0709 12:23:23.898776 (+   659us) log.cc:844] Serialized 139 byte log entry
0709 12:23:24.633967 (+735191us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:24.665578 (+ 31611us) write_op.cc:454] Released schema lock
0709 12:23:24.675534 (+  9956us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:24.814179 (+138645us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:24.819154 (+  4975us) write_op.cc:454] Released schema lock
0709 12:23:25.842940 (+1023786us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
Metrics: {"mutex_wait_us":454643,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":182692,"apply.run_cpu_time_us":63990,"apply.run_wall_time_us":2067381,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66917,"num_ops":15,"prepare.queue_time_us":556725,"prepare.run_cpu_time_us":33710,"prepare.run_wall_time_us":2428075,"raft.queue_time_us":291185,"raft.run_cpu_time_us":6423,"raft.run_wall_time_us":88881,"replication_time_us":2190275,"spinlock_wait_cycles":144403840,"thread_start_us":3199014,"threads_started":5,"wal-append.queue_time_us":2390361,"wal-append.run_cpu_time_us":47336,"wal-append.run_wall_time_us":4327617}]]}
I20260709 12:23:38.741086 22830 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.782500 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:38.782974 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:38.784168 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:38.785097 18020 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:38.785619 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:38.699167 23086 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:38.760437 22836 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.923s	user 0.015s	sys 0.018s
I20260709 12:23:38.785455 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:38.794281 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:38.786842 23020 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:23:38.753772 23056 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.806367 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:38.806861 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:38.807392 18098 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 8.
I20260709 12:23:38.807686 18096 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:38.811276 23073 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:38.815049 23073 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:38.817457 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.811748 23186 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.818531 18019 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.821193 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:38.808970 18098 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:38.787160 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:38.822151 18813 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:38.811426 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:38.841317 18811 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:38.841485 18170 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:38.809968 23056 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:38.840312 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2455) took 1038 ms. Trace:
I20260709 12:23:38.864205 18015 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20260709 12:23:38.845450 22929 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.867508 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 183946us
I20260709 12:23:38.845757 22483 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.875844 18019 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
W20260709 12:23:38.872752 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.605s (SENT)
I20260709 12:23:38.879791 18015 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:38.880646 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:38.834843 22836 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.859720 23113 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.854842 23173 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.793424 23193 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:38.877219 23073 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:38.855597 18095 rpcz_store.cc:276] 0709 12:23:37.787221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.609155 (+821934us) service_pool.cc:224] Handling call
0709 12:23:38.680155 (+ 71000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.680791 (+   636us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.680827 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.681596 (+   769us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.752955 (+ 71359us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:38.753002 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.753086 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.824056 (+ 70970us) raft_consensus.cc:1685] finished
0709 12:23:38.824092 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.825902 (+  1810us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":672,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":157,"thread_start_us":70791,"threads_started":1,"wal-append.queue_time_us":120193}
I20260709 12:23:38.888566 22694 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.907603 23020 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:38.908663 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:38.916756 18092 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:38.920352 18098 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:38.906435 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:38.925922 22911 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:38.893353 23056 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.928551 18020 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:23:38.931734 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.940085 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:38.941064 18015 raft_consensus.cc:2368] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13.
I20260709 12:23:38.942751 18095 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:38.928311 23193 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:38.891431 22929 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.938738 23073 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:38.962157 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 105872us
W20260709 12:23:38.968128 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 229677us
I20260709 12:23:38.976266 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2458) took 1164 ms. Trace:
I20260709 12:23:38.988039 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2459) took 1187 ms. Trace:
I20260709 12:23:38.978778 18093 rpcz_store.cc:276] 0709 12:23:37.799888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.639056 (+839168us) service_pool.cc:224] Handling call
0709 12:23:38.711017 (+ 71961us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.721094 (+ 10077us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:38.911687 (+190593us) log.cc:844] Serialized 14 byte log entry
0709 12:23:38.963510 (+ 51823us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:38.963579 (+    69us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:38.963646 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.963727 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.963884 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":230919,"raft.queue_time_us":3094,"thread_start_us":192616,"threads_started":2}
W20260709 12:23:38.963577 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 63659.9us
W20260709 12:23:38.991190 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.988582 18092 rpcz_store.cc:276] 0709 12:23:37.801102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.640924 (+839822us) service_pool.cc:224] Handling call
0709 12:23:38.721324 (+ 80400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.929706 (+208382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.987713 (+ 58007us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:38.987967 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":424124800}
I20260709 12:23:38.982872 22835 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:38.951416 23056 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:38.985319 23073 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:39.012022 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:39.047235 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.012383 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.020679 22830 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:39.023648 22929 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:39.029435 23020 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:39.034392 18098 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:23:39.035782 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 174674us
I20260709 12:23:39.036260 18096 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:39.036840 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:39.055305 18020 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.055099 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:39.052873 18098 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:39.040796 23174 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.002s
I20260709 12:23:39.021556 18015 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:39.050230 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46033.4us
I20260709 12:23:39.069600 18094 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:39.073733 18099 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:39.051666 22830 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:39.077210 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72350us
I20260709 12:23:39.092703 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2462) took 1017 ms. Trace:
W20260709 12:23:39.116751 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 135558us
I20260709 12:23:39.077512 18091 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260709 12:23:39.077347 18096 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20260709 12:23:39.098210 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93076.7us
W20260709 12:23:39.121241 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.078043 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 782) took 1275 ms. Trace:
I20260709 12:23:39.099442 22911 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:39.120745 22836 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:39.115548 23020 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.066854 22483 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.053s
W20260709 12:23:39.130378 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.076501 22929 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.152693 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.150446 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53228.9us
W20260709 12:23:39.151183 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79737.4us
W20260709 12:23:39.147162 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.132934 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2456) took 1344 ms. Trace:
I20260709 12:23:39.130188 18098 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.136058 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:39.155615 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:39.125910 18099 rpcz_store.cc:276] 0709 12:23:37.802147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.035824 (+233677us) service_pool.cc:224] Handling call
0709 12:23:39.075740 (+1039916us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:39.077952 (+  2212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379119872}
I20260709 12:23:39.129369 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 793) took 1073 ms. Trace:
I20260709 12:23:39.183538 18091 rpcz_store.cc:276] 0709 12:23:38.056351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.666140 (+609789us) service_pool.cc:224] Handling call
0709 12:23:39.077138 (+410998us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.117770 (+ 40632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.125307 (+  7537us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b5400410fd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:39.129250 (+  3943us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899515648}
I20260709 12:23:39.186884 23174 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:39.187062 22830 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:39.187804 23174 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.189038 22830 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.180190 22737 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:23:39.131842 23073 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:39.192178 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.151736 23113 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.145269 23020 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:39.133358 22911 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:39.136050 23056 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:39.177623 18090 rpcz_store.cc:276] 0709 12:23:37.788833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.636449 (+847616us) service_pool.cc:224] Handling call
0709 12:23:38.719348 (+ 82899us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.759520 (+ 40172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.759563 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.785110 (+ 25547us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.907870 (+122760us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b1800200920. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:38.987931 (+ 80061us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:38.987980 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.001320 (+ 13340us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:39.001411 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.132595 (+131184us) raft_consensus.cc:1685] finished
0709 12:23:39.132624 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.132858 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3876,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":1323,"spinlock_wait_cycles":83302528,"thread_start_us":79482,"threads_started":1,"wal-append.queue_time_us":284059}
W20260709 12:23:39.203835 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50089.6us
I20260709 12:23:39.143677 23186 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:39.115984 18097 rpcz_store.cc:276] 0709 12:23:38.075689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.671948 (+596259us) service_pool.cc:224] Handling call
0709 12:23:38.707103 (+ 35155us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.757023 (+ 49920us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:38.916487 (+159464us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.077879 (+161392us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:39.077959 (+    80us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:39.092356 (+ 14397us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.092461 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.092619 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4520,"prepare.queue_time_us":245622,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":22530,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":146,"thread_start_us":229106,"threads_started":2}
I20260709 12:23:39.143436 23117 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.182722 23090 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.154901 22483 consensus_queue.cc:1243] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:39.210101 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:39.220332 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66504.4us
I20260709 12:23:39.180672 23086 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:23:39.209864 22911 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.179924 22929 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.218111 23159 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:39.217329 23075 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:39.246019 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:39.248593 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:39.249964 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4844492689999242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20260709 12:23:39.263104 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2189492689999497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
W20260709 12:23:39.263662 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1271492689998013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
I20260709 12:23:39.250334 22737 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:39.251286 23130 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:39.221637 23113 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:39.272420 22485 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:39.273228 22485 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:39.274009 23113 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:39.274492 22485 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:39.253484 23086 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:39.253849 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:39.246068 22827 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 12:23:39.255065 23174 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:39.229759 23186 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.259745 18090 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:39.259971 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:39.260243 23117 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:39.266589 23075 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.235009 23067 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:39.283736 18091 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
W20260709 12:23:39.283895 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 148035us
I20260709 12:23:39.305331 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:39.307394 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:39.292227 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:39.228642 23090 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.286550 23075 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:39.313370 22548 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:39.350126 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:39.350622 22548 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:39.282533 23086 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:39.351461 23086 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:39.352720 23086 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:39.323154 18099 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:39.304913 22737 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:39.349417 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:39.352263 22548 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.286731 23067 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:39.381699 22975 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:39.407716 22975 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:39.314317 22827 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
W20260709 12:23:39.382364 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 156345us
I20260709 12:23:39.387566 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.421088 18091 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:39.390226 23090 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:39.392202 22864 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:39.382418 23086 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:39.401973 23159 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:39.415313 23067 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:39.443133 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.447394 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:39.501464 18811 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:39.405685 18137 leader_election.cc:400] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:39.502275 18137 leader_election.cc:403] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:39.502797 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:39.509265 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.512977 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2470) took 1275 ms. Trace:
I20260709 12:23:39.539502 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:39.543059 18098 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:39.513721 18095 rpcz_store.cc:276] 0709 12:23:38.238019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.943861 (+705842us) service_pool.cc:224] Handling call
0709 12:23:38.944234 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.944544 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:39.070535 (+125991us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.512380 (+441845us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:39.512464 (+    84us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:39.512550 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.512658 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.512885 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":60811,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"thread_start_us":446508,"threads_started":2}
I20260709 12:23:39.540623 18813 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:39.442905 22864 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:39.549942 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.552184 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 194292us
I20260709 12:23:39.500718 23090 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.559712 23225 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.147s	user 0.002s	sys 0.000s
I20260709 12:23:39.510463 23159 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:39.560513 22694 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:39.555871 23020 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.614416 23020 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:39.561482 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 121084us
I20260709 12:23:39.409243 22975 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.621265 18173 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:23:39.566927 18091 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 5. (term mismatch)
I20260709 12:23:39.623183 18091 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 4
I20260709 12:23:39.623605 18091 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20260709 12:23:39.623963 18091 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:39.556491 22827 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:39.580271 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.582422 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:39.592629 22830 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.630993 22830 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:39.634223 22830 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:39.593617 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.635535 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:39.595922 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.636569 23117 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:39.566934 22864 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.637084 23117 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:39.594843 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.637594 23117 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:39.638111 18168 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:39.638312 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.638080 22864 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:39.651473 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 348482us
I20260709 12:23:39.659626 18165 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:39.665580 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.627434 22827 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:39.666482 18090 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:39.670267 18813 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:39.573040 23090 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:39.677976 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 285601us
I20260709 12:23:39.677486 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:39.676388 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 491792us
I20260709 12:23:39.682994 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.686172 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.673874 22827 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.684034 18165 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:39.694447 18019 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:39.661008 23075 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.703074 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.677942 22929 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:39.704435 18020 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:39.704938 18090 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:39.704618 18808 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:39.712630 23159 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.716768 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.708806 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:39.720323 18095 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:39.720202 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.723726 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.727380 18094 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:39.731007 23186 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:39.731432 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.732470 18099 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:39.736647 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.733678 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:39.739719 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.740442 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.741021 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.741382 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.741698 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.738008 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.747032 18811 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:23:39.764693 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 448680us
I20260709 12:23:39.770478 18015 raft_consensus.cc:2410] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 12:23:39.773677 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.810537 18020 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.810299 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:39.812227 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 103563us
I20260709 12:23:39.814134 22929 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:39.814924 22929 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.816599 23186 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:39.817044 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:39.817945 18091 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:39.818674 18095 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:39.820168 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:39.820996 18815 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:39.822239 23020 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:39.817284 23186 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.706040 22726 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.089s
I20260709 12:23:39.850389 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.851001 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:39.853984 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:39.859129 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 349124us
I20260709 12:23:39.828922 23186 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:39.871726 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.874751 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.875869 18170 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:39.878098 18099 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: [10.10-12.11]
I20260709 12:23:39.878330 18015 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:39.863068 23230 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:23:39.892527 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.884055 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.893554 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.893996 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.895701 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.896529 18165 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:39.886399 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.879281 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:39.888634 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.890126 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.891357 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.893509 18164 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:39.901685 18168 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:39.898272 18165 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:39.899789 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.893564 18021 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:39.899492 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:39.892342 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:39.897714 18171 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:39.914883 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.898936 18167 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 12:23:39.916921 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.917652 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.919862 18023 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:39.920809 18173 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:23:39.922333 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:39.923769 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.924659 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:39.926112 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.928038 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.849771 23117 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:39.899904 22929 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:39.936168 18022 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:39.930475 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.930786 18815 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.931022 18019 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:39.937304 18170 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:39.941547 18165 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:39.947746 18811 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 12:23:39.946322 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54114.3us
I20260709 12:23:39.946828 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:39.952286 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.952307 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.957501 18093 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:39.930192 23073 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:39.959360 18168 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:39.940354 23020 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:39.970008 18171 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:39.963383 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:39.972926 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:39.936692 23113 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:39.926741 22483 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:23:39.979554 18164 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:23:39.979720 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:39.978528 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:39.943898 22975 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:39.990746 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 119375us
I20260709 12:23:39.991179 18165 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:39.994845 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.999707 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:40.002873 18815 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:39.952507 22485 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:23:39.981284 23113 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:40.006258 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.004719 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.007762 18168 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.002391 23193 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:40.015822 18170 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:40.021291 23075 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:40.022198 23090 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:40.013664 22911 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.725s	user 0.016s	sys 0.000s
I20260709 12:23:40.028172 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.029075 22864 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.032984 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:40.035916 22483 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:23:40.060708 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87931.7us
I20260709 12:23:40.061959 18166 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:40.062708 18098 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:23:40.064296 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 810) took 1688 ms. Trace:
I20260709 12:23:39.965691 18090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:23:40.066455 18090 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:40.066931 18090 raft_consensus.cc:614] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:40.008680 23113 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:40.064656 18098 rpcz_store.cc:276] 0709 12:23:38.376112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.990874 (+614762us) service_pool.cc:224] Handling call
0709 12:23:39.003784 (+ 12910us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.386294 (+382510us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b58008207d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166130aa 00007f6c16695447 00007f6c166adfe7 00007f6c166af7de 00007f6c166b9b49 00007f6c166ba416 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4
0709 12:23:40.063979 (+677685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.064231 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38081920}
I20260709 12:23:40.077023 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.077430 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.078722 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.078982 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.080111 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:40.080575 23226 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.231s	user 0.003s	sys 0.000s
I20260709 12:23:40.037384 23193 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:40.084475 23020 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:40.086287 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75605.1us
I20260709 12:23:40.087391 22483 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:40.087929 22483 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:40.090224 18170 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:40.093497 23067 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.000s
I20260709 12:23:40.089217 22864 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.094272 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.080575 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.109156 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:23:40.095924 17989 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.790s (SENT)
I20260709 12:23:40.098368 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2473) took 1329 ms. Trace:
I20260709 12:23:40.071316 18093 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:40.104084 18165 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:40.088581 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:40.035920 23117 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:40.099265 23239 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:23:40.115954 18164 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:40.118644 18019 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:23:40.122126 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.134011 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.110483 18092 rpcz_store.cc:276] 0709 12:23:38.768994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.041796 (+272802us) service_pool.cc:224] Handling call
0709 12:23:39.065685 (+ 23889us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:39.093252 (+ 27567us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:39.093657 (+   405us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.446560 (+352903us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:39.508752 (+ 62192us) log.cc:844] Serialized 78 byte log entry
0709 12:23:39.511386 (+  2634us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:39.516483 (+  5097us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.540028 (+ 23545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.540126 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.079759 (+539633us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:40.098012 (+ 18253us) raft_consensus.cc:1685] finished
0709 12:23:40.098053 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.098273 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23158,"prepare.queue_time_us":4471,"prepare.run_cpu_time_us":209,"prepare.run_wall_time_us":7937,"raft.queue_time_us":24783,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":8472,"spinlock_wait_cycles":83502976,"thread_start_us":352466,"threads_started":1,"wal-append.queue_time_us":613638}
I20260709 12:23:40.124449 18091 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:40.125953 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:40.142788 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.152983 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:40.154330 18808 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:40.155652 23235 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
W20260709 12:23:40.158878 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.163619 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:40.166179 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.167819 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.154170 23073 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:40.066162 22911 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.200698 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.203326 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:40.203938 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78141.4us
I20260709 12:23:40.149587 22726 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:40.170073 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74185.1us
I20260709 12:23:40.168594 18166 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:40.142611 22548 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:40.226918 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.228595 18170 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:40.237532 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.247418 18096 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:23:40.247709 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.120287 23067 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.277107 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.250566 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.279325 18172 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:40.276515 22485 consensus_queue.cc:1243] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:23:40.287382 23244 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:23:40.287918 22485 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.288753 22485 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.289300 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.290789 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.292243 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.292410 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:40.293570 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.297829 18169 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:40.301635 18166 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:23:40.304869 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 152896us
I20260709 12:23:40.311811 18022 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:40.143474 22827 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.249321 22975 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:40.318024 23248 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:40.319276 23174 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:40.321897 23073 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.168550 22835 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.155s
I20260709 12:23:40.324424 18018 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:40.329602 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.301542 23117 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.330492 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:40.331199 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84253.1us
I20260709 12:23:40.334772 18017 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
W20260709 12:23:40.337639 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.334420 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 245903us
I20260709 12:23:40.330942 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:40.252661 23186 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.353261 23020 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:40.353942 23020 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:40.354215 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:40.355152 23020 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:40.357187 23252 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:23:40.358182 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.359282 18016 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:23:40.360188 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54789.4us
I20260709 12:23:40.358062 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.377228 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.379499 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.387008 18015 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:40.378183 18166 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
W20260709 12:23:40.393250 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.265348 22726 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.403741 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.413301 18170 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:40.390902 23086 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.890s	user 0.021s	sys 0.003s
I20260709 12:23:40.338066 22929 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
W20260709 12:23:40.423450 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.426038 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.426707 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.426863 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:40.428449 18093 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:40.378345 23259 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:40.429762 18170 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:40.434574 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.386337 23117 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.440052 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.425238 23020 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.441475 18171 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:40.440985 18169 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:40.453390 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.454511 18015 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:40.455696 22911 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:40.456735 23248 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:40.428983 23257 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:40.432291 22827 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:40.470477 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116360us
I20260709 12:23:40.471978 18016 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:40.434950 22864 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:40.471701 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 179444us
I20260709 12:23:40.473793 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.434576 23263 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:40.480652 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:40.483189 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.485100 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:40.486474 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:40.487569 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.487888 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:40.489009 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.489131 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:40.489785 18021 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:40.490154 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.490278 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.490757 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.490999 18173 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:40.491554 18021 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:40.492413 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.492555 18173 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:40.493134 18021 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:40.494164 23259 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:40.494328 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.494679 23259 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:40.495260 23259 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.495966 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.496872 18016 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:40.494745 18020 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:40.499518 18173 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:40.499423 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.498323 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.504374 18015 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:40.438427 22835 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:23:40.465255 23248 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:40.530337 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.532620 22911 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:40.533638 18164 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:40.534591 22598 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:23:40.535539 18164 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:40.536068 18164 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:40.538234 23263 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:40.538944 23263 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:40.539321 23186 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:40.539549 23263 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.542373 23130 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:40.542987 23130 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:40.544250 23130 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.546860 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.547739 18020 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:40.548020 23193 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.548889 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.550182 22485 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:40.551107 23255 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.384s	user 0.001s	sys 0.001s
I20260709 12:23:40.551882 23248 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.553387 23248 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.546847 22830 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:40.553666 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:40.554028 22830 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:40.554490 18811 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:40.555155 22830 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.555961 22726 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:40.524488 23174 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:40.557067 22830 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.511607 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:40.510432 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.517462 18019 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:40.560681 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.560802 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:40.561862 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.517452 18165 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:40.553390 18171 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:23:40.565317 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.537202 18164 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.568380 18164 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.569871 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.549749 23193 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.572018 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.440552 22483 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.512740 23258 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:40.557978 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.586292 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:40.510998 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.582185 18016 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:40.566685 18172 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:40.523634 23067 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:40.584331 18808 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:40.522472 23257 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.589963 18811 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:40.509670 18170 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:40.590441 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.590857 18173 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:40.587954 18023 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:40.592878 18020 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:40.593168 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:40.595090 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.595274 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.555238 23117 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:40.601065 18171 raft_consensus.cc:2376] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12. Re-sending same reply.
I20260709 12:23:40.604403 18172 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:23:40.604657 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128721us
I20260709 12:23:40.607306 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.610973 18024 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:40.608122 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.612118 18016 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12. Re-sending same reply.
I20260709 12:23:40.606786 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:40.593384 23130 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.622388 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:40.582940 23268 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:40.619747 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:40.621802 18023 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:40.622974 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.628304 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.629104 18015 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:40.605283 22483 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:40.608647 23159 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:40.653488 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69316us
I20260709 12:23:40.658802 22975 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:40.627745 23186 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:40.669857 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.679534 23260 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:23:40.687592 23269 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:40.616154 22911 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:40.658874 18172 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:40.707208 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:40.709451 18021 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:23:40.713415 18168 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.652s	user 0.013s	sys 0.017s
I20260709 12:23:40.738365 18168 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:40.712082 22726 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:40.711243 22975 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:40.724428 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80717.5us
I20260709 12:23:40.743762 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:40.725379 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.734571 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.741089 18166 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:40.722719 23264 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:23:40.745595 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.711858 23073 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:40.703195 23159 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:40.746191 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.746833 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:40.753505 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.754554 18172 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:40.757431 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:40.736641 23186 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:40.744858 23249 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:40.744693 22827 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:40.756515 22485 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:40.779359 22485 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:40.779632 22975 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.780392 22485 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.781417 22975 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.762297 23268 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 12:23:40.757293 23073 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:40.806689 23186 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.807781 23073 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.810663 18166 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:40.811144 18166 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:40.812057 18166 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:40.813005 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 301471us
I20260709 12:23:40.778414 22598 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:40.749552 22911 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:40.783351 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.790356 23159 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.794144 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:40.795609 23272 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.001s	sys 0.002s
W20260709 12:23:40.802170 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46728.8us
I20260709 12:23:40.803596 18169 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:23:40.803889 23113 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.676s	user 0.008s	sys 0.005s
I20260709 12:23:40.813308 18166 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.768654 22827 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:40.814818 23275 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:23:40.818241 18138 leader_election.cc:400] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13.
W20260709 12:23:40.862646 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75144.9us
I20260709 12:23:40.863015 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.871325 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2485) took 1055 ms. Trace:
I20260709 12:23:40.871879 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:40.872334 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:40.872836 23159 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.869434 22827 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:40.878074 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.871822 18098 rpcz_store.cc:276] 0709 12:23:39.815536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.070621 (+255085us) service_pool.cc:224] Handling call
0709 12:23:40.071044 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.214622 (+143578us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.214655 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.308987 (+ 94332us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.499546 (+190559us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:40.499593 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.499663 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.794769 (+295106us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:40.870968 (+ 76199us) raft_consensus.cc:1685] finished
0709 12:23:40.870994 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.871249 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":159990,"raft.queue_time_us":215278,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":17001,"spinlock_wait_cycles":7899520,"thread_start_us":175770,"threads_started":1,"wal-append.queue_time_us":270386}
W20260709 12:23:40.886848 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.889765 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 101047us
I20260709 12:23:40.891084 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:40.891685 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:40.818992 22726 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:40.905563 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87497.2us
I20260709 12:23:40.906314 18808 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:40.913343 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:40.912348 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:40.818475 23086 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.086s
W20260709 12:23:40.916627 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.917414 18099 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:23:40.923331 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.926672 23277 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:40.927626 23277 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:40.930451 22598 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:40.905766 23020 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:40.932210 23020 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.932643 23020 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.934005 23262 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20260709 12:23:40.940860 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.928872 23277 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:40.945513 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:40.954952 18022 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:40.958366 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:40.959657 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111396us
I20260709 12:23:40.882246 23113 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:40.971268 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 79667us
I20260709 12:23:40.861522 23263 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.975992 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:40.981058 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:40.989243 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:40.911055 22726 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:40.998122 18166 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:23:41.003682 18166 pending_rounds.cc:85] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 6
I20260709 12:23:41.004001 18166 pending_rounds.cc:107] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:23:41.004263 18166 raft_consensus.cc:2889] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:41.002846 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:41.007899 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.011464 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.013772 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.015789 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:41.016776 18168 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:41.017599 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:40.970901 23268 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:41.018288 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.018675 18091 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:41.025952 18811 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:40.962010 23269 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:41.019532 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46332.3us
I20260709 12:23:41.014725 18167 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:41.027498 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.019724 18169 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:23:41.021081 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:41.021225 18173 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:41.021621 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:41.023808 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.029856 18165 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12. Re-sending same reply.
I20260709 12:23:41.030134 18171 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:40.972249 23258 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:23:40.970470 22975 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:40.943775 23247 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.102s
W20260709 12:23:41.030808 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74707.6us
I20260709 12:23:41.029070 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:41.036144 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57546.3us
I20260709 12:23:41.042273 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.043708 18168 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:41.061117 23283 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:41.063297 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:41.064288 18164 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:41.068346 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.073186 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.034899 22975 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.080066 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.099462 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:41.103788 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62694.5us
I20260709 12:23:41.107331 18019 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:41.108211 18018 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:41.086166 22975 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:41.098896 23269 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.077031 22548 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.162447 22548 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.098843 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:41.126407 23269 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.132256 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.134341 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:41.135278 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:41.176209 18168 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:41.142907 23287 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:41.147241 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.148343 23193 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:41.149763 23113 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:41.210306 18167 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:41.149894 23265 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.231s
I20260709 12:23:41.150674 23258 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.151204 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61204.4us
I20260709 12:23:41.152030 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:41.152416 23259 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.657s	user 0.016s	sys 0.000s
I20260709 12:23:41.091241 23282 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:23:41.152987 22737 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:41.050421 22911 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.248225 22548 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:41.252535 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105422us
I20260709 12:23:41.159269 22726 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:41.160595 23279 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.344s	user 0.004s	sys 0.000s
I20260709 12:23:41.161761 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:23:41.163120 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.128490 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:41.165983 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.189770 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.225337 23186 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:41.218006 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:41.266368 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.110790 22598 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:23:41.212934 23252 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:23:41.247030 23283 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:41.247985 22737 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:41.253705 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.106264 22830 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:41.273500 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.277582 18017 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:41.279253 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:41.288072 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:41.288579 18016 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:41.290316 18021 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:41.277935 22929 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:41.292649 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:41.293036 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.265349 23174 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:41.302098 18022 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:41.302608 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.303117 18808 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:41.302215 18020 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:41.277834 23291 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.310416 18020 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:41.281241 22830 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.273945 23283 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:41.307709 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:41.279945 22737 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:41.276679 23268 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:41.243604 23259 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.307430 22929 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.239095 23258 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:41.314767 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61177.9us
I20260709 12:23:41.305289 23174 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:41.308425 23263 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:41.331301 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:41.333711 18022 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:23:41.357204 18022 pending_rounds.cc:85] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 9
I20260709 12:23:41.357635 18022 pending_rounds.cc:107] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:23:41.357980 18022 raft_consensus.cc:2889] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:41.336735 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:41.341456 23259 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:41.338274 22737 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.367565 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:41.328785 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 265618us
W20260709 12:23:41.348032 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 275127us
I20260709 12:23:41.307018 23247 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:41.293668 23231 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.348623 18811 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:41.349371 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.323448 22830 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:41.337024 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:41.371902 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.373490 18813 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:41.377162 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:41.399266 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.428972 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51939.5us
I20260709 12:23:41.374822 23073 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:41.433290 23248 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:41.383563 22929 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:41.436566 23193 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:41.438026 22485 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:41.437477 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:41.439829 22485 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:41.354712 22598 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
W20260709 12:23:41.412235 23159 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.517s	user 0.016s	sys 0.000s
I20260709 12:23:41.444207 18813 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:41.450304 18064 leader_election.cc:400] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:41.518697 18064 leader_election.cc:403] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:41.461097 23296 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:41.452143 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90441.4us
I20260709 12:23:41.321540 23291 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.441380 23193 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:23:41.391598 23231 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:41.510686 23294 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 12:23:41.436602 23289 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:23:41.460249 18815 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: [10.10-12.11]
I20260709 12:23:41.517731 22485 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:41.546662 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 216270us
I20260709 12:23:41.459196 22737 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:41.574523 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:41.576391 18813 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:41.529747 22827 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.654s	user 0.012s	sys 0.002s
I20260709 12:23:41.572031 22911 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:41.530126 23269 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:41.534759 23291 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:41.546077 23290 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:41.559541 23020 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.626s	user 0.015s	sys 0.001s
I20260709 12:23:41.582015 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.531437 23258 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:41.596611 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159264us
W20260709 12:23:41.606508 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 176255us
W20260709 12:23:41.586891 23261 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
W20260709 12:23:41.581557 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 209795us
W20260709 12:23:41.586701 23301 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:23:41.537948 23174 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:41.625733 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43871.1us
I20260709 12:23:41.575838 23073 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:41.438539 23268 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.577281 23231 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:41.633390 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65486.7us
I20260709 12:23:41.630183 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:41.632313 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.635942 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.606693 23159 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:41.594704 23249 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:41.610420 23290 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:41.506667 23283 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.612468 22598 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:23:41.658977 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:41.594357 23308 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:23:41.608633 22827 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.670943 23312 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:41.673210 23312 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:41.628055 23020 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.657614 23249 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:41.685320 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.687078 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.697459 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.687556 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.689807 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.690953 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.693094 23283 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:41.693681 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48318.5us
I20260709 12:23:41.694461 23263 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:41.696481 23293 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:41.698926 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:41.700536 23290 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:41.684260 22975 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.552s	user 0.014s	sys 0.002s
I20260709 12:23:41.705503 22835 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:41.648859 23269 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:41.683653 23258 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:41.728140 18812 raft_consensus.cc:2368] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11.
I20260709 12:23:41.703276 22827 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:41.688779 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:41.664578 22598 consensus_queue.cc:1243] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:41.733739 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.722506 18813 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:41.657346 23159 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.732517 22835 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:41.726282 23296 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:41.773613 23248 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:41.776054 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:41.777282 23248 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:41.777741 23248 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:41.778152 23248 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:41.779881 23159 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:41.781432 22835 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.782531 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:41.783141 22835 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:41.744031 18814 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:41.744252 23259 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:23:41.745455 18806 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:41.746568 23290 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:41.746748 23304 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:23:41.747869 23287 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:41.732710 22975 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:41.754436 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57047.5us
I20260709 12:23:41.770821 22929 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:41.771941 23186 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:41.790987 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:41.791925 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 159785us
I20260709 12:23:41.795393 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:41.793038 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 141001us
W20260709 12:23:41.801201 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.802376 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.808810 18172 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:41.792891 23073 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:41.802465 23277 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.012s	sys 0.013s
I20260709 12:23:41.816385 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:41.806298 23186 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:41.820235 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:41.823375 23186 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.823937 23186 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:41.825145 22975 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:41.789589 22830 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:41.822597 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:41.810765 23296 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:41.808533 23020 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:23:41.803355 22598 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.016s
W20260709 12:23:41.817683 23277 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.012s	sys 0.014s
I20260709 12:23:41.836408 23073 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:41.839181 23257 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:41.866762 23073 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.876178 22830 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:41.830983 22929 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:41.877374 22929 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.831990 22548 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:41.878257 22548 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:41.878844 22548 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:41.879176 22929 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:41.874662 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.823939 22726 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:23:41.880869 22975 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:41.879591 23322 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:23:41.891813 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53068us
I20260709 12:23:41.893054 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.896763 23020 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:41.897634 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.897961 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.898620 18814 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:41.888197 23322 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:41.921525 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.922430 18813 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:41.918376 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.903805 23292 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:41.926178 23292 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:41.926430 22864 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:41.927086 22864 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:41.927318 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.927591 23292 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.928087 22864 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.906014 22975 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.926340 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.930732 22975 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:41.931445 18811 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:41.928081 18812 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:41.933542 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.901291 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.948597 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.951346 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:41.929594 23292 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:41.899473 23247 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.515s	user 0.007s	sys 0.008s
I20260709 12:23:41.965411 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:41.954871 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:41.914625 23258 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:41.966410 18812 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
W20260709 12:23:41.950366 23314 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.136s	user 0.000s	sys 0.002s
I20260709 12:23:41.979912 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:41.958292 23308 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:41.929700 22864 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:41.988354 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 165832us
I20260709 12:23:41.972259 23247 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:41.993371 18090 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20260709 12:23:41.995971 18090 pending_rounds.cc:85] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 10
I20260709 12:23:41.996275 18090 pending_rounds.cc:107] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20260709 12:23:41.996505 18090 raft_consensus.cc:2889] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:41.979331 22830 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:42.011947 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2444) took 1469 ms. Trace:
I20260709 12:23:42.014010 23248 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.016853 23248 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:42.017786 18023 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:23:42.018326 18023 pending_rounds.cc:85] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 10
I20260709 12:23:42.018563 18023 pending_rounds.cc:107] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:23:42.018810 18023 raft_consensus.cc:2889] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.004150 18814 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:42.022030 22598 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:42.022624 22598 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.024106 22975 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.023973 18019 rpcz_store.cc:276] 0709 12:23:40.542911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.564988 (+ 22077us) service_pool.cc:224] Handling call
0709 12:23:42.011635 (+1446647us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:42.011827 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1243902848}
I20260709 12:23:42.023269 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:42.021615 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:42.030571 18812 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:42.030915 18167 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: [12.13-12.13]
I20260709 12:23:42.030498 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.032202 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:42.033346 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.033650 18812 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:42.034339 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:42.034946 18099 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:42.035395 18811 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:42.036705 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2452) took 1067 ms. Trace:
I20260709 12:23:42.037240 18023 rpcz_store.cc:276] 0709 12:23:40.958227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.959152 (+   925us) service_pool.cc:224] Handling call
0709 12:23:42.017359 (+1058207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.019317 (+  1958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.021571 (+  2254us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000fe950. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:42.025214 (+  3643us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2432797056}
I20260709 12:23:42.040021 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:42.040693 18092 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.043406 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.044374 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.048120 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.048452 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.052389 18811 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:42.053972 18814 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:42.061952 23318 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:23:42.063808 23313 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:42.045516 22830 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:42.065995 23268 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:42.067101 23268 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:42.067443 23268 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:42.068454 23268 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.13, Last appended by leader: 13, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.072185 23322 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.074937 18815 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:42.048033 23159 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:42.079114 23159 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:42.079543 23159 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:42.080557 23159 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.025271 23277 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.980144 23258 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:42.091511 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.009141 23186 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.042120 22975 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:42.097172 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 467143us
W20260709 12:23:42.105619 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 505917us
I20260709 12:23:42.105854 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:42.107127 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.107935 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.087553 23268 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:42.103680 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.091887 23159 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.043473 23312 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:23:42.121850 22830 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.127996 22830 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.128732 22737 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:42.129500 23186 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:42.130777 23319 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:23:42.105366 23262 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:42.118037 22911 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.086984 23313 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:23:42.118971 23296 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:42.147991 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.146507 18092 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:23:42.148520 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.149904 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:42.127118 22598 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.148414 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:42.153241 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.092864 23277 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.156229 22485 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:42.156497 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 158757us
I20260709 12:23:42.137954 22929 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.156951 22485 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:42.172273 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 117977us
I20260709 12:23:42.174561 18022 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:42.174815 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.175156 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.175765 18021 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.176860 18019 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:42.177099 18096 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.5-11.5]
I20260709 12:23:42.181890 22485 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.190192 18024 pending_rounds.cc:85] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 10
I20260709 12:23:42.190665 18024 pending_rounds.cc:107] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:23:42.190956 18024 raft_consensus.cc:2889] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.164541 22483 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.149374 22911 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.183156 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.193501 22827 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:42.198563 23303 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:42.177284 23262 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:42.208016 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:42.163565 22830 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:42.235145 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53483.4us
I20260709 12:23:42.197105 22548 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.225351 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.176766 22598 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.217810 18021 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:42.203388 22929 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:42.230671 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2134709629999634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
I20260709 12:23:42.189781 22835 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:42.256752 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.292269 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:42.288343 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 99238.9us
W20260709 12:23:42.257390 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 738) took 1269 ms (client timeout 1595 ms). Trace:
W20260709 12:23:42.293463 18172 rpcz_store.cc:269] 0709 12:23:40.988376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.988503 (+   127us) service_pool.cc:224] Handling call
0709 12:23:42.257213 (+1268710us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:42.257333 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1799711488}
I20260709 12:23:42.297441 18164 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:42.289559 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.531s (SENT) [suppressed 48 similar messages]
W20260709 12:23:42.298326 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69910.2us
I20260709 12:23:42.304119 18017 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:42.304219 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:42.218125 23329 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.102s
I20260709 12:23:42.259977 23130 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:42.320533 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:42.224138 22483 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.246923 23291 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:23:42.330628 22835 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.330931 23331 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:23:42.334213 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:42.325635 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.306516 23296 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:42.319377 23316 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.297379 23073 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.300345 23322 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:42.335801 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:42.350261 23325 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:23:42.339039 23324 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:42.246961 23303 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.354141 23296 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.362146 23316 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.369516 23263 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.378410 23263 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.379683 23316 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.380414 23316 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:42.383946 23316 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:42.288817 22598 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.337273 23258 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.373533 22929 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:23:42.395848 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90896.9us
I20260709 12:23:42.396164 22929 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:42.396729 22929 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:42.397202 23086 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:42.369377 23073 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.397696 23086 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.398116 23086 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.400189 23073 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.402281 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:42.403314 22548 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.403965 22598 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.387696 23322 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.376001 23296 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:42.383325 23283 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.007s
I20260709 12:23:42.383438 23303 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.380792 23263 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:42.406682 23322 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:42.408242 18808 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:42.385773 22911 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.330664 22975 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:23:42.413408 22483 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:42.414187 18813 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:42.319286 23317 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:42.402593 23287 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.601s	user 0.015s	sys 0.000s
I20260709 12:23:42.417688 23287 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.417572 23317 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.420320 23317 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.420768 23277 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.421552 22835 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.424239 18806 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:42.384661 23249 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.651s	user 0.010s	sys 0.004s
I20260709 12:23:42.396828 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:42.333612 23186 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:23:42.369486 23324 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:42.435923 17988 leader_election.cc:400] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11.
W20260709 12:23:42.441434 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 151913us
I20260709 12:23:42.441864 17988 leader_election.cc:403] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:23:42.442876 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:42.454902 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120778us
I20260709 12:23:42.366583 23333 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
W20260709 12:23:42.455296 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0568709630001649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260709 12:23:42.461338 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 166312us
I20260709 12:23:42.463868 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.466960 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:42.470602 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.472728 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.473611 18170 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:42.466787 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.477195 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:42.474200 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:42.491266 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:42.441298 23291 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.497406 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:42.492558 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 198768us
I20260709 12:23:42.436482 23287 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:42.507253 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.508221 18168 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:42.511801 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.514863 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.515875 18165 raft_consensus.cc:2376] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13. Re-sending same reply.
I20260709 12:23:42.518839 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:42.493428 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 98201.4us
I20260709 12:23:42.477520 18167 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:42.452066 22864 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11.
I20260709 12:23:42.523556 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.523972 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.524657 22485 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:42.525146 22485 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.525606 22485 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:42.443692 22483 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:42.495728 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.529920 18169 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:42.521641 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.522735 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.525054 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.494583 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.530752 18807 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:42.530769 18173 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:42.531419 18172 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:42.526418 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:42.524662 18171 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:42.527498 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.533716 18811 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:23:42.469328 23277 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.506533 22548 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:42.530929 18810 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:42.551842 18166 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:42.491833 23283 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:42.555801 18809 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:23:42.562721 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85609.1us
I20260709 12:23:42.565193 23193 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:42.566286 23303 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:42.566738 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47982us
I20260709 12:23:42.608585 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:42.567956 23174 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:42.578097 22548 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:42.528755 23282 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:42.540614 23268 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.585240 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:42.436578 23249 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.583300 23277 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.628743 22827 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 12:23:42.637293 23249 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.523111 23331 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:42.623368 18168 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:23:42.600154 23304 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:23:42.631554 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.636049 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.613409 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.657791 18021 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:42.640210 23282 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:42.639370 23268 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:42.660903 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 139919us
I20260709 12:23:42.495458 23316 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.662477 18173 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:42.522588 23287 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.646938 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.647385 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.650875 18019 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:42.671788 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:42.639842 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.639079 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.645072 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.655651 18167 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:42.659201 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:42.612380 23332 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.309s	user 0.000s	sys 0.005s
I20260709 12:23:42.642048 22548 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.664027 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:42.675268 18021 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:42.677058 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.607842 23303 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.683645 18022 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:42.684188 18015 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:42.662680 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:42.645619 23247 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:42.627905 22864 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:42.627460 23322 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.673776 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.677719 18021 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 6.
I20260709 12:23:42.656337 23331 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.679067 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:42.691773 18024 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:42.625795 23283 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:42.693886 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:42.696758 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.554704 22929 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.704921 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:42.706278 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.661121 23282 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:42.726370 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47433.4us
I20260709 12:23:42.725409 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.727191 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:42.714458 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51875.8us
I20260709 12:23:42.731338 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:42.731881 18090 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.731010 18097 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:42.735466 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.736687 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.695344 23303 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.731509 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.745774 18021 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:42.738937 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.746210 18094 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:42.691468 23186 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:42.696074 23292 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.725s	user 0.012s	sys 0.004s
I20260709 12:23:42.760658 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:42.748936 18168 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:42.753676 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.663367 23343 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:23:42.730307 23331 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.758261 18096 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:23:42.763605 18095 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:42.700541 22864 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:42.838342 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.761325 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:42.839234 18165 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:42.760326 22485 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.744304 23293 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:42.743363 23263 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:42.553463 23324 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.694622 23333 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
W20260709 12:23:42.855590 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 106179us
I20260709 12:23:42.823961 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.861477 18016 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:42.862802 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.831979 18021 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:42.831492 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:42.844043 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.831429 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:42.846540 18097 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:42.866840 22485 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:42.847846 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.726119 23130 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:23:42.757005 23186 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.854684 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.846477 22864 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:42.874684 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:42.859845 23304 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:23:42.831019 23357 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.740754 23322 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:42.883271 23186 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:42.911486 23159 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.784s	user 0.009s	sys 0.004s
I20260709 12:23:42.912290 23257 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:42.913257 23268 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.915791 23193 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:42.917217 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.918339 23341 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:42.919358 23355 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:42.919811 23287 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:42.881633 23359 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:42.922747 23322 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.923940 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.902735 23283 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:42.872788 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112259us
W20260709 12:23:42.888517 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 208194us
I20260709 12:23:42.903842 23357 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.926359 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:42.926545 23276 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:42.927279 23276 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:42.927615 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.799s (SENT)
I20260709 12:23:42.928501 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:42.850699 23263 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:42.929347 23333 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:42.930348 22929 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:42.930658 23333 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.932376 23352 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:42.932920 23352 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:42.935839 18092 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:23:42.894856 22485 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:42.907501 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:42.890434 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:42.850878 22827 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:42.928754 23357 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:42.930483 23346 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 12:23:42.964753 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:42.880851 18020 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:42.938421 18015 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:23:42.924926 18019 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:42.945217 18090 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:42.931727 23333 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:42.953181 18018 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:42.929397 23354 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:23:42.929414 23263 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:42.971836 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:42.972829 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65497.5us
I20260709 12:23:42.952919 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:42.986186 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:42.986460 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:42.957216 23073 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:42.996650 18093 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:42.996670 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.009904 18020 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
W20260709 12:23:43.001639 18806 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.571s	user 0.009s	sys 0.001s
I20260709 12:23:42.992022 18815 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:23:43.005180 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:43.006117 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61159.2us
I20260709 12:23:43.016269 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:42.934037 23352 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:42.987823 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.020593 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:43.017936 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:42.971357 23333 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:42.989192 22929 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:43.022445 18806 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:42.988318 23357 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.034461 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.038448 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:43.009095 23073 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.016058 23159 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:43.028430 23351 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.028236 23247 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:43.066120 23247 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.066769 23247 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:43.068687 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.928498 23276 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.020431 23193 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
W20260709 12:23:43.049365 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:43.027009 22929 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:43.071969 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69631.4us
I20260709 12:23:43.045460 22827 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:43.054811 23356 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:23:43.056106 23339 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:23:43.040246 23358 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:43.039429 23359 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.047006 23073 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.017844 23343 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:23:43.081701 22929 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.027581 23333 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.072942 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.074962 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:43.084403 22835 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.051023 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:43.039386 23357 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.089705 23359 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.093731 18018 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:43.102342 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.126015 23365 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:23:43.127751 23086 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.729s	user 0.011s	sys 0.002s
I20260709 12:23:43.128087 23357 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.128404 23086 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.130415 18022 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:43.130666 23359 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.085651 23159 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:43.132287 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87396.8us
I20260709 12:23:43.133330 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:43.135381 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60529.1us
I20260709 12:23:43.072391 23352 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.139317 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.143486 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.158668 18017 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:43.085928 23341 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.165688 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.093288 23358 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.141496 23297 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.158792 23317 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:43.139698 23258 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:43.191888 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48538.8us
I20260709 12:23:43.202497 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.161626 23312 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.156936 23351 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:43.140708 23174 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.204028 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.162250 23343 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:43.174821 23322 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:43.209280 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 170949us
I20260709 12:23:43.181766 23357 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.141294 22835 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.187139 23358 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:43.188656 23339 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.185130 23355 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.230468 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72320.1us
I20260709 12:23:43.215886 23343 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.225488 23290 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:43.203312 23317 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.216987 23276 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.214485 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:43.208563 23248 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:43.216262 23258 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.202651 23297 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.182057 23341 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.245157 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:43.236567 23290 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:43.215734 23174 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.230525 23159 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.220482 22827 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.196588 23361 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:43.249791 18019 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:43.241317 23317 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:43.250157 23292 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:43.221997 23357 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.260293 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:43.306216 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46038.7us
I20260709 12:23:43.262863 23248 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.266083 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:43.220645 23312 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.285355 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:43.236896 23343 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.289047 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:43.231166 23353 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:43.253078 23282 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.507s	user 0.013s	sys 0.000s
I20260709 12:23:43.251838 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1122) took 1577 ms. Trace:
I20260709 12:23:43.263737 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.231432 23339 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:43.329267 23303 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.558s	user 0.013s	sys 0.000s
I20260709 12:23:43.330353 23303 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.239693 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.257103 23258 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.234513 22864 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:43.293885 23380 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:23:43.257496 23297 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:43.277539 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63232.9us
I20260709 12:23:43.287279 23283 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:43.312415 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46437.8us
I20260709 12:23:43.312602 18807 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:43.315295 23268 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:43.331444 23067 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.352643 23385 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.353482 23385 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.355469 23073 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.356216 23385 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.356966 18094 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:43.358207 23073 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.358742 23385 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.358964 23073 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:43.359539 23073 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:43.359345 23385 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.281713 23020 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:23:43.322124 18809 rpcz_store.cc:276] 0709 12:23:41.674882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.688206 (+ 13324us) service_pool.cc:224] Handling call
0709 12:23:43.251571 (+1563365us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b540034fdd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:43.251746 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1506674944}
W20260709 12:23:43.332866 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47592.3us
I20260709 12:23:43.302834 23292 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:43.342212 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:43.310791 23248 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.236532 23355 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.274963 23276 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.260375 23316 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.315654 23231 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:43.315927 23086 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.362079 18016 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:43.393731 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51615.2us
W20260709 12:23:43.364269 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 124684us
I20260709 12:23:43.363643 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:43.322713 23351 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:43.372097 18015 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:43.374481 23384 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.410107 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121172us
I20260709 12:23:43.345219 22598 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:43.407909 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44365.3us
I20260709 12:23:43.362409 23268 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:43.384341 23297 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:43.361860 23385 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.335947 23258 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:23:43.370585 23365 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.426218 18099 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: [11.12-11.12]
I20260709 12:23:43.361378 23277 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.397535 23249 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.418226 18096 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: [12.13-12.13]
I20260709 12:23:43.418892 18098 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:43.401616 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.337841 23282 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.427502 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.428030 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:43.445112 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.446719 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.336406 23290 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.373090 23067 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.428347 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.430140 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:43.442919 23186 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.006s
I20260709 12:23:43.459008 23247 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.409320 23020 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.460276 23020 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.451498 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.463112 23020 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.463585 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.464208 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:43.464982 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.466598 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:43.463616 23390 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.421653 23268 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:43.468889 23390 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.470171 22835 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.471539 23390 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.471899 23296 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:43.462050 23247 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.472479 23296 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:43.474268 23086 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:43.481972 23350 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:43.482474 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2525) took 1005 ms. Trace:
I20260709 12:23:43.482762 23350 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:43.483067 18167 rpcz_store.cc:276] 0709 12:23:42.477700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.527650 (+ 49950us) service_pool.cc:224] Handling call
0709 12:23:42.655149 (+127499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.674533 (+ 19384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.482104 (+807571us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400383450. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:43.482374 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292310912}
I20260709 12:23:43.483918 23350 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:43.486572 23375 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:23:43.421623 23297 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:43.488509 23297 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.489436 23342 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:43.490376 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45655.9us
I20260709 12:23:43.492873 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:43.493598 18807 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:43.494663 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3305) took 1494 ms. Trace:
I20260709 12:23:43.495124 18091 rpcz_store.cc:276] 0709 12:23:42.000540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.000729 (+   189us) service_pool.cc:224] Handling call
0709 12:23:42.001043 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.087953 (+1086910us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:43.088006 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:43.088072 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:43.088140 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.494382 (+406242us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400052654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:43.494431 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.494592 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2381,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"spinlock_wait_cycles":2498552064}
I20260709 12:23:43.418767 22911 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:43.498412 22911 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.499151 22911 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:43.501151 23355 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.502651 23357 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:43.504025 23383 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:23:43.436328 23365 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.420178 23384 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.510748 23282 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.512231 23312 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.513962 18020 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:23:43.473762 23296 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.456476 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.515842 23341 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.517330 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:43.386952 23263 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.498291 23351 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.459182 23186 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.455385 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:43.518975 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:43.550613 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 236065us
I20260709 12:23:43.551681 18814 raft_consensus.cc:2376] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14. Re-sending same reply.
I20260709 12:23:43.552899 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:43.454268 18093 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:43.546046 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.554769 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.555238 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.557353 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:43.558929 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.560180 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:43.563385 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.565354 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:43.566412 18806 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:43.568585 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.569625 18094 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:23:43.570515 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.572249 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.360659 23073 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:43.583041 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127736us
I20260709 12:23:43.583956 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.584712 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.585284 18813 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:43.518249 22830 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.589217 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.520112 23231 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:43.594488 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:43.594786 18090 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 12:23:43.543046 22598 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:43.591382 18809 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:43.596149 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:43.594625 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.605698 18020 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:43.595424 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.605799 18814 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:43.606580 18095 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:43.602041 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.607149 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:43.615095 18807 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:43.545572 23287 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:43.557685 23293 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.617014 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:43.571322 23290 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:43.640491 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 212570us
I20260709 12:23:43.643483 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:43.645422 18806 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:43.452978 23249 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.441917 23292 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.682509 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.607157 22835 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:43.687388 18091 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:43.634701 22548 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:43.650731 23174 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.689174 23263 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.689597 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.706404 18097 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:43.691092 17920 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 6 to 8, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 8 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:43.675455 23287 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.626457 23303 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.628913 23361 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:43.723665 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:43.728196 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 169350us
I20260709 12:23:43.730942 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:43.737169 23263 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:43.703589 23249 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.737769 23159 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:43.738308 23263 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:43.738871 23365 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.739619 23389 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:43.740368 23389 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:43.745450 23345 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:43.738901 23263 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.748610 23268 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.746248 23345 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.738125 23296 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.698037 23390 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:23:43.738930 23159 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:43.755594 23390 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:43.755753 23159 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:43.756098 23390 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:43.758564 23287 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.758879 23067 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.759372 23287 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:43.760145 23345 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.757233 23159 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.763830 23067 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:43.778553 23067 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.781342 23268 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.741539 23389 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.767601 18098 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:23:43.769224 17920 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 9 to 10, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 10 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:43.757176 23390 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:43.764606 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.782757 23258 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:43.783310 23067 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:43.769309 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:43.786298 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.802522 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 209823us
I20260709 12:23:43.787885 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:43.788091 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51176.6us
I20260709 12:23:43.707753 22835 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.790179 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:43.606812 23342 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.118s
I20260709 12:23:43.804497 18090 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 10: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:43.608050 23355 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.129s
W20260709 12:23:43.805749 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 385949us
I20260709 12:23:43.805037 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.806109 18808 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:43.814986 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:43.810446 17920 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:43.822867 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:43.671281 23293 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:43.836743 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.842602 18022 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:43.843974 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.842334 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:43.833113 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.813171 23388 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:43.805931 22975 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.862087 23388 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.862463 22975 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:43.799355 23384 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.813493 23361 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:43.819893 23268 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:43.866189 23361 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.866636 23361 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.830833 23231 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:23:43.869246 23186 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.872593 23339 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.799221 22864 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:43.880755 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.799187 23258 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:43.881784 18024 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:43.882160 23258 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:43.883152 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.883824 23390 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:43.885367 23292 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.885787 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.809929 22835 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.786643 22911 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:43.888203 18811 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.020s	sys 0.038s
I20260709 12:23:43.889109 18015 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:43.817350 23324 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:43.936439 18811 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.024s	sys 0.040s
I20260709 12:23:43.929365 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:43.822103 23159 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.902647 18814 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:43.910614 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.817730 23276 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:43.910358 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:43.917429 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88279.9us
I20260709 12:23:43.920101 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:43.982664 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.984289 18024 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:43.997887 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:43.948927 18813 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:44.006414 18020 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:43.943926 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:44.007715 18810 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.865700 22975 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:44.011965 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:44.012820 18020 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
I20260709 12:23:43.975385 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:43.819211 23343 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:44.039863 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.027014 17920 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 5 to 7, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:44.022600 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:43.937374 18808 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:43.988274 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:43.916640 23353 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.577s	user 0.013s	sys 0.000s
I20260709 12:23:43.822464 23248 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.042909 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:43.988888 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:44.012552 23331 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.153s	user 0.017s	sys 0.000s
I20260709 12:23:44.130564 18021 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.984086 23339 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.898840 23086 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:43.885389 23392 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:44.033350 18090 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 11: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:44.036027 18020 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:44.143110 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.143798 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:43.915417 22548 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.198s
I20260709 12:23:44.120682 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:44.120465 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:44.122213 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.122890 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.124614 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.123698 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:44.132783 18096 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:44.135305 18015 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:44.124465 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:44.145263 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:44.031522 22726 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.000s	sys 0.021s
I20260709 12:23:44.164609 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:44.165329 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:44.149413 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.166486 18164 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:44.154184 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:43.989962 22830 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.143s
I20260709 12:23:43.983166 23384 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.147s
I20260709 12:23:44.167675 18168 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:43.944703 23308 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.199s
I20260709 12:23:44.156622 18021 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
W20260709 12:23:44.130179 22929 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.039s	user 0.016s	sys 0.000s
I20260709 12:23:44.222678 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:44.229105 18090 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 8.
I20260709 12:23:44.230170 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.223515 18093 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:44.164592 22726 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.000s	sys 0.024s
I20260709 12:23:44.030045 23387 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.135s
I20260709 12:23:44.147305 23353 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:44.243613 18168 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10. Re-sending same reply.
I20260709 12:23:44.245397 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:44.212509 23333 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.120s	user 0.014s	sys 0.010s
I20260709 12:23:44.246456 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:44.301235 18090 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:23:44.301245 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.144098 23282 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:44.243899 18091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.022s	sys 0.034s
I20260709 12:23:44.117129 23193 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:44.311187 18164 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:44.166417 23086 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:44.302049 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.309479 18172 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:44.309460 18091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.024s	sys 0.039s
I20260709 12:23:44.146216 23331 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.330988 18165 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:44.245524 23400 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:44.331653 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.151404 23339 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.343086 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.350342 18165 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:44.349479 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.354449 18172 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:44.355551 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:44.360702 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:44.352676 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 377378us
I20260709 12:23:44.362118 18164 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:44.373013 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:44.374310 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:44.375665 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.376473 18172 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:44.377748 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.380172 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:44.245707 22929 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:44.349586 23316 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.954s	user 0.018s	sys 0.000s
I20260709 12:23:44.482007 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:44.385090 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:44.389689 23297 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.901s	user 0.006s	sys 0.007s
I20260709 12:23:44.398080 23257 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.403276 23248 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:44.351033 23400 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:44.405568 23396 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:23:44.500367 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:44.407533 23293 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:44.341506 23193 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.410409 23339 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:44.352967 23322 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.135s	user 0.016s	sys 0.003s
I20260709 12:23:44.333078 23333 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.414963 23398 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.418355 22911 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:44.420532 23399 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 12:23:44.421839 23343 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:44.422284 23086 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:44.389810 23317 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.079s	user 0.005s	sys 0.010s
I20260709 12:23:44.341508 23393 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:44.434607 18173 raft_consensus.cc:2368] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
W20260709 12:23:44.468228 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58472.5us
I20260709 12:23:44.634716 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:44.498044 23384 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.634553 18090 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:44.417589 23263 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.670s	user 0.013s	sys 0.000s
W20260709 12:23:44.494624 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 252979us
I20260709 12:23:44.512687 22827 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.526216 22975 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:44.536077 23268 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:44.547048 18095 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:44.495294 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 999) took 1167 ms. Trace:
I20260709 12:23:44.498131 18165 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:44.498733 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.482007 23186 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:44.494427 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.498559 18810 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
W20260709 12:23:44.487118 23296 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.728s	user 0.009s	sys 0.007s
I20260709 12:23:44.345580 23282 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:44.577869 23408 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:23:44.546422 23257 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.583861 22929 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:44.646018 22975 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:44.665022 23352 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.502s	user 0.019s	sys 0.000s
I20260709 12:23:44.584762 23398 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.489416 23020 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:44.612036 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.641462 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 997) took 1333 ms. Trace:
W20260709 12:23:44.691965 23358 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.455s	user 0.019s	sys 0.000s
I20260709 12:23:44.647593 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:44.586920 23293 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:44.570271 23322 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:44.603829 23287 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:44.656412 18064 leader_election.cc:400] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
W20260709 12:23:44.729430 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73208.6us
I20260709 12:23:44.654191 22827 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.662066 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.637836 23316 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:44.496202 23297 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.641938 23317 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.667912 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.808151 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:44.649147 23263 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.658046 23277 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.674217 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:44.840449 18809 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:44.674494 23331 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:44.644840 23393 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.667138 23186 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.663105 23350 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:23:44.648561 23384 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.667176 23342 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.676281 23417 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:44.644872 22911 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
W20260709 12:23:44.692430 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.011s	sys 0.003s
I20260709 12:23:44.585166 23303 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:44.707192 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 206907us
I20260709 12:23:44.712702 18167 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20260709 12:23:44.644410 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:44.705385 23361 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.838s	user 0.010s	sys 0.004s
I20260709 12:23:44.711870 18815 rpcz_store.cc:276] 0709 12:23:43.307956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.309695 (+  1739us) service_pool.cc:224] Handling call
0709 12:23:43.310102 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.316330 (+  6228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.316372 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.320604 (+  4232us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.835013 (+514409us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:43.835080 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.835166 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.641136 (+805970us) raft_consensus.cc:1685] finished
0709 12:23:44.641163 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.641372 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8609,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":14196,"thread_start_us":443891,"threads_started":1,"wal-append.queue_time_us":768500}
I20260709 12:23:44.720407 18090 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:44.730998 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:44.652828 18811 rpcz_store.cc:276] 0709 12:23:43.313001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.313480 (+   479us) service_pool.cc:224] Handling call
0709 12:23:43.319561 (+  6081us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:43.320008 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.320042 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:43.333105 (+ 13063us) log.cc:844] Serialized 52 byte log entry
0709 12:23:43.937251 (+604146us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:43.990355 (+ 53104us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.017533 (+ 27178us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.352027 (+334494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.352126 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.480427 (+128301us) raft_consensus.cc:1685] finished
0709 12:23:44.480464 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.480682 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78545,"prepare.queue_time_us":283149,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":7605,"raft.queue_time_us":738369,"raft.run_cpu_time_us":605,"raft.run_wall_time_us":184363,"thread_start_us":895975,"threads_started":2,"wal-append.queue_time_us":797033}
I20260709 12:23:44.735386 18165 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:44.736958 18097 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:23:44.718981 23159 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.681s	user 0.012s	sys 0.001s
W20260709 12:23:44.733577 23355 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.009s	sys 0.008s
W20260709 12:23:44.862190 23290 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.176s	user 0.011s	sys 0.005s
I20260709 12:23:44.718783 23358 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.732307 23287 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:44.890996 23317 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:44.893450 23390 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.009s	user 0.010s	sys 0.004s
I20260709 12:23:44.896466 18164 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:44.897364 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:44.806656 23408 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:23:44.736655 23316 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.006384 23316 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:45.009240 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2537) took 1245 ms. Trace:
I20260709 12:23:44.923277 23342 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.011269 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:45.011987 18167 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:45.012517 23342 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:45.014698 23418 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 12:23:45.016170 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2524) took 1593 ms. Trace:
I20260709 12:23:45.016777 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1143) took 1128 ms. Trace:
I20260709 12:23:45.017505 18807 rpcz_store.cc:276] 0709 12:23:43.888856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.889064 (+   208us) service_pool.cc:224] Handling call
0709 12:23:43.889486 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.898676 (+  9190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.898723 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.984889 (+ 86166us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.498987 (+514098us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:44.499034 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.499111 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.016460 (+517349us) raft_consensus.cc:1685] finished
0709 12:23:45.016495 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.016688 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15960,"raft.queue_time_us":8840,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":28178,"thread_start_us":366547,"threads_started":1,"wal-append.queue_time_us":660329}
I20260709 12:23:45.020658 23290 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.014381 18097 rpcz_store.cc:276] 0709 12:23:43.763922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.764187 (+   265us) service_pool.cc:224] Handling call
0709 12:23:45.008949 (+1244762us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54004123d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:45.009145 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2167489408}
I20260709 12:23:44.946136 23410 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.023010 23413 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.359s	user 0.001s	sys 0.001s
I20260709 12:23:45.022836 23410 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.016642 18093 rpcz_store.cc:276] 0709 12:23:43.423220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.572481 (+149261us) service_pool.cc:224] Handling call
0709 12:23:43.604010 (+ 31529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.270861 (+666851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.015855 (+744994us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b5400215054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:45.016091 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1424179968}
I20260709 12:23:44.908059 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:44.910254 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.011s	sys 0.003s
I20260709 12:23:44.925134 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:45.075855 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150808us
I20260709 12:23:44.967712 23193 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:44.971897 22864 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.737094 23248 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.168s
I20260709 12:23:44.949071 18166 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 12:23:44.837878 23411 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:44.949625 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:44.954830 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 223946us
I20260709 12:23:45.082723 18064 leader_election.cc:400] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:44.666208 23400 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:44.904628 23350 consensus_queue.cc:1243] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:23:44.945482 23355 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.010s	sys 0.008s
I20260709 12:23:44.900327 22975 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.021539 23290 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:44.911643 23392 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.699s	user 0.002s	sys 0.011s
W20260709 12:23:44.998183 23258 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.115s	user 0.015s	sys 0.001s
I20260709 12:23:44.998279 18813 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:44.964928 23389 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:44.899401 23277 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:44.909402 18806 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:45.109038 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:44.970837 22835 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.017s	user 0.007s	sys 0.007s
I20260709 12:23:45.061180 18094 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [8.10-8.10]
W20260709 12:23:44.958618 23292 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.073s	user 0.011s	sys 0.004s
I20260709 12:23:44.945866 23427 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:45.055325 18098 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7.
I20260709 12:23:44.969704 23287 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.076810 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:44.859979 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:44.946687 23361 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:45.090173 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 182210us
I20260709 12:23:45.104983 23422 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:44.959824 23303 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.087006 23020 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:45.104609 22830 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:45.070889 23312 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:45.069096 23333 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.072896 23257 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.114678 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:45.119016 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 474665us
I20260709 12:23:45.087291 22911 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.098778 23193 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:45.132190 18172 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:45.100835 23339 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.106801 23392 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.152359 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:45.152776 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:45.153213 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2468) took 1164 ms. Trace:
I20260709 12:23:45.121781 22975 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.102895 23411 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.097685 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:45.227236 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 129677us
I20260709 12:23:45.129990 23400 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:45.168462 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.234891 18168 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:45.137669 23355 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.175571 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:45.098960 22864 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.179028 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:45.248529 18091 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:23:45.181937 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:45.187881 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.189698 18090 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:45.191007 18019 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:23:45.111413 23258 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.198861 18016 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:45.173110 23268 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:45.173440 23422 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.130056 22835 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:45.154636 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77928.5us
W20260709 12:23:45.110460 23331 consensus_queue.cc:1175] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "910f8c8d787e4abbb092413c1f90ebf3" responder_term: 11 status { last_received { term: 11 index: 10 } last_committed_idx: 9 last_received_current_leader { term: 11 index: 10 } }
I20260709 12:23:45.223814 23159 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.228235 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:45.188899 22830 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.238191 23308 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.177032 23312 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:45.191540 23430 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.303006 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:45.214725 23420 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:23:45.208319 23257 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:45.247198 18814 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:45.197162 23193 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.202261 23352 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.250417 18022 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:45.196332 23398 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.267428 23422 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:45.152065 23427 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.219965 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:45.259665 22827 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.242542 23248 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:45.289988 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61870.3us
I20260709 12:23:45.308254 23390 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:45.233134 23297 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.299299 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:45.360229 18813 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:45.251362 23086 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.204427 22911 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.248382 23186 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.217641 23339 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.307049 18167 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:23:45.315536 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:45.217686 23263 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.278893 23392 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.325130 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:45.333509 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61029.1us
I20260709 12:23:45.312856 23423 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:45.294518 23020 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:45.334856 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.342191 18091 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:23:45.342828 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 142596us
I20260709 12:23:45.381912 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:45.259392 23355 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.210409 18017 rpcz_store.cc:276] 0709 12:23:43.988363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.988532 (+   169us) service_pool.cc:224] Handling call
0709 12:23:44.029272 (+ 40740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.029664 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:44.482427 (+452763us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.152786 (+670359us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:45.152852 (+    66us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:45.152922 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.152997 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.153132 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":651686,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":155685,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":126,"spinlock_wait_cycles":797056,"thread_start_us":1036254,"threads_started":3,"wal-append.queue_time_us":625508}
I20260709 12:23:45.384173 18091 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
W20260709 12:23:45.331349 23324 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.376s	user 0.014s	sys 0.004s
I20260709 12:23:45.310022 23393 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.330638 23193 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:45.330921 23437 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.323009 23358 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.335631 23398 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.303666 23159 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.374719 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.376396 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:45.376839 18811 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:23:45.434188 18167 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.310380 23308 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:45.365060 23428 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.293s	user 0.000s	sys 0.002s
I20260709 12:23:45.314821 23257 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.431648 23437 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:45.453487 23426 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.299s	user 0.000s	sys 0.002s
I20260709 12:23:45.417184 23277 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.458807 23303 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.460388 23257 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.385717 18091 pending_rounds.cc:85] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:23:45.311501 23430 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:45.394834 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69798us
I20260709 12:23:45.468142 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:45.333566 23352 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.402894 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:45.469686 18021 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:45.363260 23439 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:45.384605 23425 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
W20260709 12:23:45.488029 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 85324.8us
I20260709 12:23:45.491534 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.493188 18093 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:45.412118 23339 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.415148 23446 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:45.417351 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2526) took 1981 ms. Trace:
I20260709 12:23:45.497354 18096 rpcz_store.cc:276] 0709 12:23:43.436162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.595564 (+159402us) service_pool.cc:224] Handling call
0709 12:23:43.595916 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.215880 (+619964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.417016 (+1201136us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54002ca954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:45.417265 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1234082560}
I20260709 12:23:45.373565 23086 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.430166 23333 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:45.502151 23333 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.502679 23333 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:45.378629 23290 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.447665 18167 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:45.506930 23427 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.509110 23430 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.511296 23159 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.514509 22830 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.515317 23258 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.506930 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:45.517351 23411 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.518409 22835 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.465312 18090 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:45.473801 23361 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.519037 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.519912 22864 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.519783 23333 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.520357 23355 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.391840 23020 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:45.521763 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.523131 23020 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:45.524621 23402 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.167s	user 0.010s	sys 0.005s
I20260709 12:23:45.528658 23410 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:45.530988 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2531) took 1412 ms (client timeout 1531 ms). Trace:
W20260709 12:23:45.564232 18173 rpcz_store.cc:269] 0709 12:23:44.118809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.122600 (+  3791us) service_pool.cc:224] Handling call
0709 12:23:45.530691 (+1408091us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400382f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:45.530904 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":535179520}
I20260709 12:23:45.516693 22911 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.518257 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:45.520277 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.532713 23333 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.534090 23257 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:45.540635 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:45.540963 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 830) took 1734 ms (client timeout 1609 ms). Trace:
I20260709 12:23:45.543654 18807 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:45.462285 23398 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.556437 23282 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.558210 23388 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.554399 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:45.626266 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71984.8us
W20260709 12:23:45.626547 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116422us
I20260709 12:23:45.463388 18091 pending_rounds.cc:107] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:23:45.628886 18091 raft_consensus.cc:2889] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:45.630700 23308 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.631116 23350 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:45.631748 23282 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.496517 23439 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.632277 23324 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:45.422405 22827 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.633844 22827 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.634794 23439 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.636624 23388 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:45.499331 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.489358 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:45.581610 18168 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
W20260709 12:23:45.639055 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 149834us
I20260709 12:23:45.390910 23392 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.557549 23352 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:45.370405 23312 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 15.15, Last appended by leader: 15, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:45.593905 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.597779 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:45.474462 23283 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:45.529657 18099 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:45.560506 23446 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:45.612560 18098 rpcz_store.cc:269] 0709 12:23:43.806950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.814571 (+  7621us) service_pool.cc:224] Handling call
0709 12:23:45.540687 (+1726116us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540035f2d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:45.540830 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2715728256}
I20260709 12:23:45.663786 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.664752 18098 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:45.614359 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:45.619486 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 270683us
I20260709 12:23:45.623019 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.623420 18021 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:45.668964 18093 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:45.625949 23257 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:45.639154 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.669728 18168 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:45.670118 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.602345 23355 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.675675 18098 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:45.679139 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2377) took 1378 ms. Trace:
I20260709 12:23:45.679602 18170 rpcz_store.cc:276] 0709 12:23:44.301015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.309633 (+  8618us) service_pool.cc:224] Handling call
0709 12:23:44.310110 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.311297 (+  1187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.311333 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.354917 (+ 43584us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.720133 (+365216us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:45.118232 (+398099us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.321125 (+202893us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.321229 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.678892 (+357663us) raft_consensus.cc:1685] finished
0709 12:23:45.678918 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.679069 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42058,"prepare.queue_time_us":527161,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":80,"raft.queue_time_us":45415,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":40767,"thread_start_us":890099,"threads_started":3,"wal-append.queue_time_us":660642}
I20260709 12:23:45.681176 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:45.681348 23444 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:45.682201 18173 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:45.682077 23444 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.682746 23257 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.682780 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2394) took 1276 ms. Trace:
I20260709 12:23:45.683871 23439 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:45.684509 23439 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.683908 18173 rpcz_store.cc:276] 0709 12:23:44.406248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.680863 (+1274615us) service_pool.cc:224] Handling call
0709 12:23:45.682713 (+  1850us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.677443 23283 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.686829 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.687136 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.687852 18173 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:45.688195 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.688392 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2543) took 1056 ms. Trace:
I20260709 12:23:45.688814 18173 rpcz_store.cc:276] 0709 12:23:44.632490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.686832 (+1054342us) service_pool.cc:224] Handling call
0709 12:23:45.688326 (+  1494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.690893 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.691514 18173 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:45.686187 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:45.655251 23424 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.283s	user 0.002s	sys 0.000s
I20260709 12:23:45.657505 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2375) took 1357 ms. Trace:
W20260709 12:23:45.661370 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.485325 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:45.586019 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.639827 23390 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:45.636626 22827 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.658459 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:45.697638 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:45.733199 18170 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:45.706055 23231 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:45.688834 18168 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:45.738524 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2545) took 1045 ms. Trace:
I20260709 12:23:45.710187 23297 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:45.712044 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:45.726979 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 241770us
I20260709 12:23:45.740949 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:45.727927 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 299265us
W20260709 12:23:45.743016 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84667.9us
I20260709 12:23:45.743896 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:45.745944 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:45.748301 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:45.739024 18168 rpcz_store.cc:276] 0709 12:23:44.693072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.687901 (+994829us) service_pool.cc:224] Handling call
0709 12:23:45.738426 (+ 50525us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.733975 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2378) took 1432 ms. Trace:
I20260709 12:23:45.749593 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.750228 18170 rpcz_store.cc:276] 0709 12:23:44.301424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.681767 (+1380343us) service_pool.cc:224] Handling call
0709 12:23:45.733869 (+ 52102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.754160 18813 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:45.698433 18171 rpcz_store.cc:276] 0709 12:23:44.300268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.300418 (+   150us) service_pool.cc:224] Handling call
0709 12:23:44.354782 (+ 54364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.368490 (+ 13708us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.368543 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.418132 (+ 49589us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.726665 (+308533us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:45.199128 (+472463us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.199439 (+   311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.199544 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.657183 (+457639us) raft_consensus.cc:1685] finished
0709 12:23:45.657223 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.657417 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23735,"prepare.queue_time_us":546411,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":103803,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":20678,"thread_start_us":603256,"threads_started":2,"wal-append.queue_time_us":649980}
W20260709 12:23:45.757712 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45752.9us
W20260709 12:23:45.667778 23316 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.652s	user 0.015s	sys 0.000s
I20260709 12:23:45.749605 18811 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:45.666097 23308 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.769603 18093 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:45.773567 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:45.774370 18813 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:45.672323 23409 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:45.671504 23446 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:45.793665 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:45.632333 23282 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.781935 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:45.672545 23352 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.111s
W20260709 12:23:45.782812 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 154610us
I20260709 12:23:45.686120 23441 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:45.740522 23312 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:45.737315 23430 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:45.800426 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:45.784394 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 81659us
I20260709 12:23:45.805907 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.808145 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:45.770339 23268 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:45.766754 23331 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.836824 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.772475 22864 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:45.772820 23290 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.766381 23447 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:45.777570 23361 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:45.884135 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50046.7us
W20260709 12:23:45.802067 23293 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.074s	user 0.014s	sys 0.004s
I20260709 12:23:45.826776 18808 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:45.700670 23355 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.796660 23341 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:45.792732 23437 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:45.780400 22737 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
W20260709 12:23:45.941710 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50085us
I20260709 12:23:45.943516 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:45.841367 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.841431 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:45.814078 22835 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:45.858165 23449 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:23:45.804919 23409 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.839567 23411 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:45.874583 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:45.838400 22911 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:45.878008 23436 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.106s
I20260709 12:23:45.880313 23446 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.882730 23448 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:45.778254 23316 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.885143 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:46.014565 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 173222us
W20260709 12:23:46.014596 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.811024 23231 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:45.856788 22864 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.023748 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.891408 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.025321 18093 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 7: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:45.885787 22975 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.653s	user 0.013s	sys 0.002s
I20260709 12:23:46.003648 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:45.861131 18813 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:46.007257 18098 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:46.002956 23268 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.890771 23418 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:45.885390 23430 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.018257 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:45.884305 23290 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:45.870183 23392 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:23:45.884649 23361 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:45.866856 23398 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:45.831540 23350 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:45.940981 23341 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.940490 23293 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:46.031000 18090 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:46.035202 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:45.876120 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1003) took 1937 ms. Trace:
I20260709 12:23:46.105507 18814 rpcz_store.cc:276] 0709 12:23:43.939046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.941249 (+  2203us) service_pool.cc:224] Handling call
0709 12:23:43.941837 (+   588us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.311888 (+1370051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.875784 (+563896us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54004df654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:45.876041 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3001361280}
I20260709 12:23:46.038617 18167 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 12:23:46.108350 23316 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:46.043915 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.110144 23308 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.110800 23308 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:45.871649 23390 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.003868 23441 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:46.112857 23441 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.114957 23290 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.068789 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.117105 23409 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.117723 18023 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:46.119043 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.119635 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:46.119807 18023 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:46.069669 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.120751 22726 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:46.019821 23411 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.123369 23446 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.123996 23411 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.117197 23441 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.077657 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:46.124624 23361 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.125516 23355 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.125619 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.126132 23361 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:46.127225 18019 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:46.127324 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:46.127967 23287 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.921s	user 0.014s	sys 0.001s
I20260709 12:23:46.128458 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.128235 23459 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.106s
I20260709 12:23:46.129295 23287 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.082705 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.131033 23282 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.099294 23392 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:46.131999 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.085520 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.030359 22864 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:46.027315 23086 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.078425 23430 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.070371 23317 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.160849 23317 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.161454 23317 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.106280 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.121690 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:46.018322 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 202085us
W20260709 12:23:46.169108 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3258) took 2259 ms (client timeout 1742 ms). Trace:
W20260709 12:23:46.169680 18016 rpcz_store.cc:269] 0709 12:23:43.909482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.909759 (+   277us) service_pool.cc:224] Handling call
0709 12:23:46.168777 (+2259018us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:46.168958 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2735636864}
I20260709 12:23:46.171783 23287 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.172048 23443 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:46.172716 23443 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:46.174132 23389 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.055s	user 0.015s	sys 0.017s
W20260709 12:23:46.174616 23193 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.743s	user 0.015s	sys 0.003s
I20260709 12:23:46.175597 23389 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:46.176446 23174 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:46.177273 23174 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:46.178833 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2536) took 2606 ms. Trace:
I20260709 12:23:46.179327 18095 rpcz_store.cc:276] 0709 12:23:43.571964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.727662 (+155698us) service_pool.cc:224] Handling call
0709 12:23:43.746659 (+ 18997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.657153 (+910494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.178322 (+1521169us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:46.178679 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1839999232}
I20260709 12:23:46.180691 18096 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:23:46.182953 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2399) took 1775 ms (client timeout 1777 ms). Trace:
I20260709 12:23:46.175091 23193 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.134147 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.184580 18018 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:45.871430 23454 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.192s
I20260709 12:23:46.185554 22548 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:46.186309 22548 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:46.132150 23392 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.187569 22548 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.189253 22548 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:46.189657 23392 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.129709 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:46.096151 23422 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.742s	user 0.008s	sys 0.006s
I20260709 12:23:46.191432 23422 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.193104 22598 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:46.143509 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:46.152819 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.193727 22598 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:46.158504 23436 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:46.194267 18023 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:23:46.158921 23456 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.272s	user 0.001s	sys 0.000s
I20260709 12:23:46.194269 18090 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:46.195084 22598 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.196523 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3266) took 1046 ms. Trace:
I20260709 12:23:46.160168 23430 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.197212 22598 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:46.199260 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 817) took 2785 ms (client timeout 1611 ms). Trace:
I20260709 12:23:46.200263 18098 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:45.903900 23447 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.201975 23422 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.176376 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:46.203550 23447 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.207419 23430 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.208374 23193 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.209442 18015 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:46.213322 23384 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.213318 23430 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.133226 18806 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:46.183393 18172 rpcz_store.cc:269] 0709 12:23:44.407996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.498387 (+ 90391us) service_pool.cc:224] Handling call
0709 12:23:46.182762 (+1684375us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54002026d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:46.182885 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1097620736}
I20260709 12:23:46.197058 18021 rpcz_store.cc:276] 0709 12:23:45.152345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.152506 (+   161us) service_pool.cc:224] Handling call
0709 12:23:46.196291 (+1043785us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b540010a9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:46.196442 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595968256}
I20260709 12:23:46.229970 18017 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:46.243718 18092 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:46.199887 18099 rpcz_store.cc:269] 0709 12:23:43.413571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.440539 (+ 26968us) service_pool.cc:224] Handling call
0709 12:23:46.198917 (+2758378us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54002cae54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:46.199094 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4437774080}
W20260709 12:23:46.244710 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2529) took 2178 ms (client timeout 2202 ms). Trace:
W20260709 12:23:46.245602 18166 rpcz_store.cc:269] 0709 12:23:44.066975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.115698 (+ 48723us) service_pool.cc:224] Handling call
0709 12:23:46.244362 (+2128664us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:46.244608 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1850926976}
I20260709 12:23:46.254941 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:46.261544 18172 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:46.262059 18808 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:46.262550 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2386) took 1950 ms. Trace:
I20260709 12:23:46.262982 18172 rpcz_store.cc:276] 0709 12:23:44.311872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.260827 (+1948955us) service_pool.cc:224] Handling call
0709 12:23:46.261263 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.262230 (+   967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.262480 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:46.264299 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 379242us
I20260709 12:23:46.261484 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:46.266098 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.134037 23341 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:46.278213 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.042s	sys 0.048s
I20260709 12:23:46.077950 23268 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:46.278851 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.042s	sys 0.048s
W20260709 12:23:46.282615 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 541777us
W20260709 12:23:46.283754 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107548us
I20260709 12:23:46.286947 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.293032 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:46.229504 23308 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.221673 23283 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.273294 23447 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.310559 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:46.316624 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:46.317202 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.295900 23430 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:46.317909 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:46.322060 18811 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.323750 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:46.324644 18165 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:23:46.325215 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2398) took 1917 ms (client timeout 2052 ms). Trace:
I20260709 12:23:46.317766 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:44.651857 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:46.326651 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:23:46.325605 18165 rpcz_store.cc:269] 0709 12:23:44.407654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.092695 (+685041us) service_pool.cc:224] Handling call
0709 12:23:46.323538 (+1230843us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:46.325125 (+  1587us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5070080}
I20260709 12:23:46.327925 22483 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.328418 22483 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.328900 22483 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:46.328472 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.332405 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:46.310014 23231 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.314015 23414 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:46.315546 23193 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:46.349457 18165 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:46.357226 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2389) took 2024 ms. Trace:
I20260709 12:23:46.358001 18165 rpcz_store.cc:276] 0709 12:23:44.332722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.348800 (+2016078us) service_pool.cc:224] Handling call
0709 12:23:46.349261 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.356880 (+  7619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.357134 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.361117 18165 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:46.363178 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2390) took 2030 ms. Trace:
I20260709 12:23:46.364436 18165 rpcz_store.cc:276] 0709 12:23:44.333046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.360525 (+2027479us) service_pool.cc:224] Handling call
0709 12:23:46.360912 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.362865 (+  1953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.363110 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:46.372310 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 106319us
I20260709 12:23:46.381080 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.396600 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:46.405200 18811 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:46.347642 23282 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.390316 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.294553 23443 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.353502 23290 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:46.417900 22483 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.418562 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.318539 23447 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.419236 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.316236 23174 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.401981 23353 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:46.389531 23414 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:46.320223 23350 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.437582 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:46.447433 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.449714 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.420884 23290 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.424072 23193 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:46.427361 23446 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:46.426438 23086 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:46.426364 23287 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.458287 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.463827 22483 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.429402 23358 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:46.423213 23372 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 12:23:46.482583 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:46.483168 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:46.484140 18811 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.436842 23353 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:46.442849 23443 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:46.453864 23454 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:46.459220 23449 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.458190 23290 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:46.460281 23193 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.494217 18811 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:46.460871 23446 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.498489 18815 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:46.501152 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.502187 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.510331 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:46.497936 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.484239 23086 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:46.513128 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.490012 23353 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:46.516973 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:46.494556 23449 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.528973 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.530313 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.497051 23372 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.498337 23457 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.535806 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:46.498179 23193 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:46.533808 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.530619 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:46.461576 23469 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:23:46.508692 23446 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:46.479143 23287 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:46.548481 23464 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:46.515792 23283 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:46.537982 23328 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.526044 23268 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.539283 23449 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.566082 23086 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:46.568186 23429 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.402s	user 0.007s	sys 0.000s
I20260709 12:23:46.568650 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 819) took 3143 ms. Trace:
I20260709 12:23:46.538342 23389 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:23:46.577023 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2383) took 2266 ms. Trace:
I20260709 12:23:46.553859 23351 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.589988 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:46.651058 18815 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:46.592080 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1144) took 1322 ms. Trace:
I20260709 12:23:46.595520 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.627249 18808 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:46.652907 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:46.639691 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.634531 18094 rpcz_store.cc:276] 0709 12:23:43.425134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.604317 (+179183us) service_pool.cc:224] Handling call
0709 12:23:43.604629 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.113051 (+1508422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.568278 (+1455227us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540011d854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:46.568564 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3267844864}
I20260709 12:23:46.499733 23358 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:46.644187 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2381) took 2334 ms. Trace:
I20260709 12:23:46.642107 18170 rpcz_store.cc:276] 0709 12:23:44.310954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.751451 (+1440497us) service_pool.cc:224] Handling call
0709 12:23:45.751974 (+   523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.781255 (+ 29281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.781285 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:45.781752 (+   467us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.889122 (+107370us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.002783 (+113661us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.576276 (+573493us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b18004810c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:23:46.576520 (+   244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.576613 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.576662 (+    49us) raft_consensus.cc:1685] finished
0709 12:23:46.576761 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.576939 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55892,"prepare.queue_time_us":114629,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":614208,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":16794,"spinlock_wait_cycles":93426432,"thread_start_us":78089,"threads_started":1,"wal-append.queue_time_us":230048}
W20260709 12:23:46.648347 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 111341us
W20260709 12:23:46.656654 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105288us
I20260709 12:23:46.636667 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2387) took 2305 ms. Trace:
I20260709 12:23:46.652643 18806 rpcz_store.cc:276] 0709 12:23:45.298064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.298272 (+   208us) service_pool.cc:224] Handling call
0709 12:23:45.298696 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.230376 (+931680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.591687 (+361311us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:46.591983 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1918403328}
I20260709 12:23:46.658193 18166 rpcz_store.cc:276] 0709 12:23:44.331739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.263326 (+1931587us) service_pool.cc:224] Handling call
0709 12:23:46.270629 (+  7303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.270972 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:46.296456 (+ 25484us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.636235 (+339779us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.636296 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:46.636363 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.636434 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.636595 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":150868,"prepare.queue_time_us":31366,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"thread_start_us":188446,"threads_started":1,"wal-append.queue_time_us":249495}
I20260709 12:23:46.662384 18064 leader_election.cc:400] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:46.664795 18064 leader_election.cc:403] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:46.520231 23414 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.587903 23439 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.598961 23328 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:46.675292 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:46.590760 23447 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.677767 18096 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:46.679263 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.681612 18170 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:46.690181 18097 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:46.690429 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2538) took 2312 ms. Trace:
I20260709 12:23:46.690930 18170 rpcz_store.cc:276] 0709 12:23:44.377669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.669158 (+2291489us) service_pool.cc:224] Handling call
0709 12:23:46.669574 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.690062 (+ 20488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.690355 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.656384 23283 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:46.669883 23358 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:46.550823 23372 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.669597 23248 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:46.582480 23317 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.686137 23328 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.581773 22864 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.701696 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2400) took 2284 ms. Trace:
I20260709 12:23:46.684114 23463 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.701103 18093 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:46.714115 23476 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.165s	user 0.001s	sys 0.001s
I20260709 12:23:46.716506 23372 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.718472 23351 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:46.692291 23439 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:46.722554 18170 rpcz_store.cc:276] 0709 12:23:44.408324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.691941 (+2283617us) service_pool.cc:224] Handling call
0709 12:23:46.692291 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.692531 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.692560 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.692607 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:46.692672 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.692756 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.692900 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.727934 23450 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:46.728860 23447 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.730407 23287 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.759697 18170 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:46.761276 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2384) took 2450 ms. Trace:
I20260709 12:23:46.768859 18171 rpcz_store.cc:276] 0709 12:23:44.311252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.772267 (+1461015us) service_pool.cc:224] Handling call
0709 12:23:45.772752 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.781098 (+  8346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.781135 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:45.798654 (+ 17519us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.273261 (+474607us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.302554 (+ 29293us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.593429 (+290875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.593555 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.760974 (+167419us) raft_consensus.cc:1685] finished
0709 12:23:46.761002 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.761190 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":328853,"prepare.queue_time_us":29353,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":332611,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":9110,"spinlock_wait_cycles":28889856,"thread_start_us":434580,"threads_started":1,"wal-append.queue_time_us":604006}
I20260709 12:23:46.735693 23248 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:46.771767 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2374) took 2465 ms. Trace:
W20260709 12:23:46.779589 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 126789us
I20260709 12:23:46.736382 23463 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:46.791668 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 129461us
I20260709 12:23:46.773981 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 759) took 2365 ms. Trace:
I20260709 12:23:46.792615 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.792742 18170 rpcz_store.cc:276] 0709 12:23:44.408796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.758486 (+2349690us) service_pool.cc:224] Handling call
0709 12:23:46.759326 (+   840us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.773516 (+ 14190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.773897 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.776096 18171 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:23:46.794672 18171 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 9
I20260709 12:23:46.794965 18171 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:23:46.795197 18171 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:46.796003 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2401) took 2127 ms. Trace:
I20260709 12:23:46.771644 23447 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:46.776700 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 260109us
W20260709 12:23:46.800199 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 298484us
W20260709 12:23:46.800981 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:46.801950 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:46.782601 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.782673 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:46.802407 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:46.803212 18806 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:46.803607 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:46.803829 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:46.761768 23312 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.927s	user 0.016s	sys 0.000s
I20260709 12:23:46.796325 18171 rpcz_store.cc:276] 0709 12:23:44.668896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.772238 (+2103342us) service_pool.cc:224] Handling call
0709 12:23:46.772938 (+   700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.795689 (+ 22751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.795939 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.806767 18093 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:46.810173 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:46.801608 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.762611 23450 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:46.790422 23317 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.791112 23287 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:46.797988 23463 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.845372 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:46.858528 22737 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:46.859140 22737 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:46.780385 18169 rpcz_store.cc:276] 0709 12:23:44.299866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.300027 (+   161us) service_pool.cc:224] Handling call
0709 12:23:44.353662 (+ 53635us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.367547 (+ 13885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.367603 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.428431 (+ 60828us) log.cc:844] Serialized 52 byte log entry
0709 12:23:44.842912 (+414481us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:44.842966 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.843060 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.567339 (+1724279us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:46.756860 (+189521us) raft_consensus.cc:1685] finished
0709 12:23:46.756897 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.765031 (+  8134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13288,"raft.queue_time_us":36597,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":120,"spinlock_wait_cycles":9607552,"thread_start_us":413857,"threads_started":1,"wal-append.queue_time_us":659137}
I20260709 12:23:46.848042 23287 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:46.875099 22548 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.632s	user 0.008s	sys 0.008s
I20260709 12:23:46.813810 23389 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:46.884635 23488 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.891985 23488 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:46.892521 23488 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.860146 22737 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.894464 22737 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:46.896255 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.900415 23469 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:46.901144 23469 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:46.908619 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98528.4us
I20260709 12:23:46.909952 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:46.900537 23365 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:46.826053 23283 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:46.920985 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 113198us
I20260709 12:23:46.907977 23450 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.890755 23483 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.937964 23487 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:46.947551 23487 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:46.948005 23487 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:46.951663 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2532) took 3491 ms (client timeout 1911 ms). Trace:
W20260709 12:23:46.952491 18092 rpcz_store.cc:269] 0709 12:23:43.460302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.464715 (+  4413us) service_pool.cc:224] Handling call
0709 12:23:46.951331 (+3486616us) spinlock_profiling.cc:243] Waited 2.78 s on lock 0x7b5400214154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:46.951540 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6388862848}
I20260709 12:23:46.954329 23447 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:46.956061 23287 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.943599 23483 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:46.958949 23483 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.959280 22827 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:46.959960 22827 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.963732 23193 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.943850 23469 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.965737 23469 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:46.967028 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 689) took 2839 ms. Trace:
I20260709 12:23:46.969565 18807 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:46.968588 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106731us
I20260709 12:23:46.904633 23389 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:46.978614 23290 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:46.980537 23444 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:46.982982 22864 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:46.938781 23257 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:46.985575 23257 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:46.986159 23257 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:46.987212 23257 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.905719 23248 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:46.988816 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 876547us
I20260709 12:23:46.992157 23328 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.993294 23446 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:46.993888 23490 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:46.995639 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 192745us
I20260709 12:23:46.998868 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2391) took 2665 ms. Trace:
I20260709 12:23:47.000036 17920 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 7 to 9, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 9 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.018741 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.019841 18092 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:47.020905 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 762) took 2351 ms. Trace:
I20260709 12:23:47.022008 18170 rpcz_store.cc:276] 0709 12:23:44.669288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.793654 (+2124366us) service_pool.cc:224] Handling call
0709 12:23:46.794053 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.794298 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:46.834800 (+ 40502us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.020438 (+185638us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:47.020528 (+    90us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:47.020615 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.020696 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.020834 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":118397,"prepare.queue_time_us":43006,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"thread_start_us":75925,"threads_started":1,"wal-append.queue_time_us":101519}
I20260709 12:23:47.001157 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:47.017036 18165 rpcz_store.cc:276] 0709 12:23:44.333359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.369582 (+2036223us) service_pool.cc:224] Handling call
0709 12:23:46.370641 (+  1059us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.634663 (+264022us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.634725 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.635368 (+   643us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.793613 (+158245us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:46.797350 (+  3737us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.998477 (+201127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.998583 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.998632 (+    49us) raft_consensus.cc:1685] finished
0709 12:23:46.998648 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.998798 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":460112,"prepare.queue_time_us":3249,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":76,"raft.queue_time_us":271297,"spinlock_wait_cycles":21191424,"thread_start_us":157736,"threads_started":1,"wal-append.queue_time_us":228238}
W20260709 12:23:47.030053 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 102124us
I20260709 12:23:46.967552 18024 rpcz_store.cc:276] 0709 12:23:44.127569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.223826 (+ 96257us) service_pool.cc:224] Handling call
0709 12:23:44.224325 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.242535 (+ 18210us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:46.184744 (+1942209us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b1800032ec0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:23:46.497711 (+312967us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.542715 (+ 45004us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.958209 (+415494us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:46.958792 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.959045 (+   253us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:46.959093 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.959200 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.966737 (+  7537us) raft_consensus.cc:1685] finished
0709 12:23:46.966763 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.966944 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41115,"prepare.queue_time_us":439115,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":29057,"raft.run_cpu_time_us":444,"raft.run_wall_time_us":29568,"spinlock_wait_cycles":13760512,"thread_start_us":348115,"threads_started":2,"wal-append.queue_time_us":93205}
I20260709 12:23:47.045790 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2388) took 2713 ms. Trace:
I20260709 12:23:46.998206 23257 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:47.051111 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141261us
I20260709 12:23:47.047812 18172 rpcz_store.cc:276] 0709 12:23:44.332089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.263754 (+1931665us) service_pool.cc:224] Handling call
0709 12:23:46.264096 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.592480 (+328384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.592537 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.630235 (+ 37698us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.826171 (+195936us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.972069 (+145898us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.045362 (+ 73293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.045472 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.045527 (+    55us) raft_consensus.cc:1685] finished
0709 12:23:47.045541 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.045710 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":755222,"prepare.queue_time_us":28064,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":3941,"raft.queue_time_us":335196,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":14064,"thread_start_us":280947,"threads_started":2,"wal-append.queue_time_us":242341}
I20260709 12:23:47.037258 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:47.057165 18806 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:47.000272 23389 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:47.060834 18807 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:47.066589 18024 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:47.067937 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:47.072621 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:47.074038 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:47.030467 23490 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:47.075269 18093 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:47.076537 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:47.078907 18024 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:47.034889 23388 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:47.083619 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.092301 18018 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:47.078971 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:47.084167 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3078857340001377s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
W20260709 12:23:47.092207 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91108.4us
I20260709 12:23:47.093240 18020 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:47.076699 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:47.094008 18097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:23:47.095934 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.094053 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:47.098150 18092 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:47.099395 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:47.057564 23353 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.511s	user 0.011s	sys 0.004s
W20260709 12:23:47.085726 23430 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.751s	user 0.016s	sys 0.000s
I20260709 12:23:47.110997 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:47.111490 23341 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.813s	user 0.013s	sys 0.004s
I20260709 12:23:47.111423 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:47.112504 23317 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:47.112706 23341 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.114964 18016 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:47.115273 23276 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:47.115978 18024 pending_rounds.cc:85] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 9
I20260709 12:23:47.116391 18024 pending_rounds.cc:107] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:23:47.116317 23312 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:47.117033 23341 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:47.117061 18024 raft_consensus.cc:2889] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.115939 23276 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:47.119000 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:47.119062 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:47.047940 23444 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:23:47.088445 23454 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.126188 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:47.048182 23489 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:23:47.127506 18807 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:47.122020 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:47.089313 23490 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:47.092795 23248 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.108878 22548 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:47.106981 23193 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.088363 23287 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:23:47.135481 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.148296 23328 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:47.102057 23388 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.123081 23341 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.153303 23290 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.154716 23174 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:47.154790 23446 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:47.153931 23276 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.143355 23441 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:47.157991 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2537) took 2826 ms (client timeout 2086 ms). Trace:
I20260709 12:23:47.120698 23308 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:47.174630 18164 rpcz_store.cc:269] 0709 12:23:44.331692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.368593 (+ 36901us) service_pool.cc:224] Handling call
0709 12:23:47.157663 (+2789070us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54001b6054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:47.157877 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1189212416}
I20260709 12:23:47.123214 23409 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:47.184203 18807 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:47.124218 23430 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:47.162264 23355 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:47.166481 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.199470 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.134189 23312 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:47.108656 23353 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.143268 22548 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.654637 18173 rpcz_store.cc:276] 0709 12:23:44.309953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.692247 (+1382294us) service_pool.cc:224] Handling call
0709 12:23:45.692619 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.693062 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.693100 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:45.693689 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.888437 (+194748us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.333416 (+444979us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.580905 (+247489us) spinlock_profiling.cc:243] Waited 9.63 ms on lock 0x7b18000a45c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:46.627764 (+ 46859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.627890 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.627962 (+    72us) raft_consensus.cc:1685] finished
0709 12:23:46.643884 (+ 15922us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.644101 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":358851,"prepare.queue_time_us":133870,"prepare.run_cpu_time_us":184,"prepare.run_wall_time_us":4598,"raft.queue_time_us":300863,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":15217,"spinlock_wait_cycles":22145408,"thread_start_us":397461,"threads_started":2,"wal-append.queue_time_us":375883}
W20260709 12:23:47.194053 18780 connection.cc:256] RPC call timeout handler was delayed by -1.4889781419997234s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
W20260709 12:23:47.171461 23474 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.526s	user 0.000s	sys 0.004s
I20260709 12:23:47.202678 23312 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.168712 23488 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:47.209343 18780 connection.cc:256] RPC call timeout handler was delayed by -1.4826781419997133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
I20260709 12:23:47.174168 23441 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.184881 23409 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.162158 23248 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.224385 18099 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:23:47.198001 23384 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:47.141880 23454 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.213469 22548 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:47.216101 23312 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:47.233842 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:47.182514 23496 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:23:47.247080 22483 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:47.323299 22483 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:47.323657 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 195697us
I20260709 12:23:47.324105 18096 raft_consensus.cc:2376] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 7. Re-sending same reply.
I20260709 12:23:47.324365 22483 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.325524 22483 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:47.273808 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.275523 18169 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:47.279008 18173 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:47.280061 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.281164 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:47.283210 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.290191 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.295614 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1153) took 1048 ms. Trace:
I20260709 12:23:47.334015 18814 rpcz_store.cc:276] 0709 12:23:46.246782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.246946 (+   164us) service_pool.cc:224] Handling call
0709 12:23:46.247431 (+   485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.319303 (+ 71872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.319341 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.319388 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:46.319445 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.295227 (+975782us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b540034fdd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:47.295304 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.295521 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2332,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":11548,"spinlock_wait_cycles":153959296}
I20260709 12:23:47.289659 18020 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:47.174980 23448 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:23:47.305601 18091 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:47.312767 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.210484 23355 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:47.221331 23358 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.504s	user 0.015s	sys 0.000s
I20260709 12:23:47.225855 23460 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:47.215561 23333 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.235812 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:47.224627 23441 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:47.222715 23409 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:47.326404 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92708.7us
W20260709 12:23:47.251667 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2392) took 2917 ms (client timeout 2245 ms). Trace:
W20260709 12:23:47.411242 18167 rpcz_store.cc:269] 0709 12:23:44.333876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.517848 (+1183972us) service_pool.cc:224] Handling call
0709 12:23:47.251225 (+1733377us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54003016d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:47.251518 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1048638336}
I20260709 12:23:47.329001 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:47.412680 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:47.333544 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.391220 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.414122 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.414587 18164 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:47.414731 18167 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.403405 23483 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:47.419365 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90481.7us
I20260709 12:23:47.420013 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.421103 18169 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:47.398690 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.422158 18173 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:47.423144 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.086s (SENT) [suppressed 19 similar messages]
I20260709 12:23:47.397727 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.358986 23463 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.428695 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:47.405898 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126897us
W20260709 12:23:47.407919 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 191074us
W20260709 12:23:47.388546 23494 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:23:47.431834 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:47.361956 23328 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:47.388973 23361 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:47.435554 18167 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:23:47.536669 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 113570us
I20260709 12:23:47.441253 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.453446 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.537724 18164 raft_consensus.cc:2376] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12. Re-sending same reply.
I20260709 12:23:47.454413 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.538872 18020 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:47.456810 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:47.539872 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 83171.6us
I20260709 12:23:47.428435 23313 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.423494 23448 consensus_queue.cc:1243] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:23:47.442579 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.174s (SENT)
I20260709 12:23:47.317211 23384 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.548473 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:47.553172 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.577837 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:47.558281 18173 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:47.559329 18808 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:47.561802 18811 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260709 12:23:47.579878 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.580291 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:47.580827 18164 raft_consensus.cc:2376] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15. Re-sending same reply.
I20260709 12:23:47.562546 18809 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 13 index: 13. (term mismatch)
I20260709 12:23:47.563283 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:47.582060 18809 pending_rounds.cc:85] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 12
I20260709 12:23:47.582441 18809 pending_rounds.cc:107] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 12.13
I20260709 12:23:47.565330 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:47.582973 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.583470 18812 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:47.584519 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:47.567636 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:47.582759 18809 raft_consensus.cc:2889] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.584705 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:47.585512 18815 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:47.585951 18811 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:47.582891 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:47.586831 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:47.533854 23487 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.585s	user 0.007s	sys 0.007s
I20260709 12:23:47.360150 23491 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.175s
I20260709 12:23:47.596335 18812 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:47.598311 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.606158 18167 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:47.417505 23358 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.604878 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.458447 23502 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:47.598315 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.417999 23333 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.606360 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.615088 18099 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:23:47.577709 23499 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.351s	user 0.000s	sys 0.001s
W20260709 12:23:47.606715 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.609612 18020 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:47.601436 23494 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:23:47.611187 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.613544 18815 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:47.418228 23460 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.669119 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.607414 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:47.597700 23476 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:23:47.611110 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 178219us
I20260709 12:23:47.598510 23511 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.676256 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:47.681054 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.535827 23350 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.115s
I20260709 12:23:47.600481 23313 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.687634 18099 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:47.688715 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:47.741034 18164 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 12:23:47.675861 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:47.689546 23308 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:47.546607 23328 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:47.700321 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.702378 23434 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:47.710263 23414 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:47.712770 23500 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.510s	user 0.001s	sys 0.000s
I20260709 12:23:47.714996 23333 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.716564 23355 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.716706 23510 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:47.722110 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.777108 18093 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
W20260709 12:23:47.722997 23317 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.610s	user 0.007s	sys 0.005s
I20260709 12:23:47.723476 23388 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.724277 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2385) took 3412 ms. Trace:
I20260709 12:23:47.657964 23526 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:47.576336 23361 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:47.636004 23502 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:47.746794 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:47.753280 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74370us
I20260709 12:23:47.760278 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.763058 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:47.669889 23516 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:47.731261 23505 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:23:47.794884 23317 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.795885 23463 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.834543 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.796981 23313 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.799018 23384 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:47.799796 23312 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.801671 23328 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.802029 18167 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:47.803505 23449 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.158s	user 0.014s	sys 0.003s
I20260709 12:23:47.624825 23487 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.810742 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3276) took 1738 ms. Trace:
I20260709 12:23:47.869748 18017 rpcz_store.cc:276] 0709 12:23:46.071966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.084193 (+ 12227us) service_pool.cc:224] Handling call
0709 12:23:47.809294 (+1725101us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b5400268754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:47.810516 (+  1222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148251520}
W20260709 12:23:47.813903 22737 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.919s	user 0.007s	sys 0.007s
I20260709 12:23:47.784869 23290 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:23:47.775422 23350 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.743187 23511 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.830390 23502 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.819491 23518 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:47.778455 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.892596 18164 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:47.791884 18168 rpcz_store.cc:276] 0709 12:23:44.311508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.790213 (+1478705us) service_pool.cc:224] Handling call
0709 12:23:45.825428 (+ 35215us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.938673 (+113245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.938715 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.012997 (+ 74282us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.453148 (+440151us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:46.476343 (+ 23195us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.636175 (+159832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.723686 (+1087511us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b54001b4254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:47.723777 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.723854 (+    77us) raft_consensus.cc:1685] finished
0709 12:23:47.723992 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.724190 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":188658,"prepare.queue_time_us":24754,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":531907,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":63962,"spinlock_wait_cycles":111244544,"thread_start_us":373020,"threads_started":1,"wal-append.queue_time_us":488586,"wal-append.run_cpu_time_us":2862,"wal-append.run_wall_time_us":1103989}
W20260709 12:23:47.818660 23506 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 12:23:47.826578 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:47.902731 23484 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.815279 23483 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.838968 18020 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:47.842100 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 958) took 1346 ms. Trace:
W20260709 12:23:47.849615 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 407032us
I20260709 12:23:47.767617 23414 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.830374 23248 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:47.775792 23257 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.002s
I20260709 12:23:47.837890 23488 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:47.828855 23510 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:23:47.877753 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 201591us
I20260709 12:23:47.835063 23355 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:47.818725 23409 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.933944 18169 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12.
I20260709 12:23:47.883107 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:47.839205 23516 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.895709 18170 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:47.895557 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.898522 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:47.904304 18097 raft_consensus.cc:606] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:47.939465 18097 raft_consensus.cc:614] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:47.767315 23308 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:47.832569 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 69584.5us
I20260709 12:23:47.938993 18168 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:47.943324 23523 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:47.944379 23513 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:23:47.914813 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.609s (SENT)
W20260709 12:23:47.915877 18018 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.009s	sys 0.008s
I20260709 12:23:47.912679 18813 rpcz_store.cc:276] 0709 12:23:46.496063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.498939 (+  2876us) service_pool.cc:224] Handling call
0709 12:23:46.499348 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.509015 (+  9667us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:46.799994 (+290979us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.020763 (+220769us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:47.021348 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:23:47.021514 (+   166us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:47.021547 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.021616 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.838474 (+816858us) raft_consensus.cc:1685] finished
0709 12:23:47.841726 (+  3252us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.842002 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":396741,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":390,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":24410,"spinlock_wait_cycles":177920,"thread_start_us":455895,"threads_started":2,"wal-append.queue_time_us":261589}
I20260709 12:23:47.921663 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.945907 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:47.902824 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76439.9us
W20260709 12:23:47.945936 18018 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.010s	sys 0.008s
I20260709 12:23:47.929783 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2404) took 2742 ms. Trace:
I20260709 12:23:47.947187 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.948211 18020 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:47.947383 18171 rpcz_store.cc:276] 0709 12:23:45.187446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.805769 (+1618323us) service_pool.cc:224] Handling call
0709 12:23:46.806311 (+   542us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:46.997115 (+190804us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.997153 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:46.997859 (+   706us) log.cc:844] Serialized 52 byte log entry
0709 12:23:47.214698 (+216839us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:47.231617 (+ 16919us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.419361 (+187744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.419459 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.929400 (+509941us) raft_consensus.cc:1685] finished
0709 12:23:47.929440 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.929660 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":208185,"prepare.queue_time_us":16579,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":252534,"raft.run_cpu_time_us":1263,"raft.run_wall_time_us":22449,"thread_start_us":403717,"threads_started":2,"wal-append.queue_time_us":392368}
I20260709 12:23:47.948959 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:47.838819 23526 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:47.950363 23526 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:47.950927 23522 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.951812 23450 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:47.951565 23522 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.952580 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:47.876371 23449 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:47.947216 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:47.955852 23449 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:47.956588 23449 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:47.936410 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.949730 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.957792 18091 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 14 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 13.
I20260709 12:23:47.958194 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:47.958811 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.958841 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.959820 18171 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:47.892261 23350 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:47.960394 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.944200 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.961560 18168 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:23:47.949100 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.922154 18167 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:47.945897 23409 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:47.936487 18016 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.004s	sys 0.025s
W20260709 12:23:47.975397 18016 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.005s	sys 0.025s
I20260709 12:23:47.960839 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.976263 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:47.925360 23276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.044s	sys 0.038s
I20260709 12:23:47.992525 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:47.975445 23518 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.997044 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:48.000487 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:48.001595 18168 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 12:23:47.960119 23353 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.015151 23448 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:48.015868 23448 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:48.020961 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.963060 18020 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:48.023424 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:48.024168 18020 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:48.025167 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:47.977068 18171 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:23:48.027915 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:48.028636 18171 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:23:48.032976 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:48.034773 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:48.036293 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:48.029546 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:48.037887 18171 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:23:48.039083 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:48.040342 18171 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:47.972296 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:48.006335 23525 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.006160 23313 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.007853 23317 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:48.000337 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:48.055382 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:48.056273 18017 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 9: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:47.988641 23328 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:48.058107 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57882.2us
I20260709 12:23:48.057914 23328 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.059077 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:48.009840 23409 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:23:48.013875 23488 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
W20260709 12:23:48.060946 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 88822.2us
I20260709 12:23:48.060508 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:47.981809 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:47.962671 23312 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:47.920747 23361 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.051563 18020 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:48.049019 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:47.987030 23276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.044s	sys 0.038s
W20260709 12:23:48.053741 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.097312 18023 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:48.060143 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:48.084522 18171 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:48.061414 23287 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:48.099646 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:48.086375 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:48.096906 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48099.9us
I20260709 12:23:48.084749 18091 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
W20260709 12:23:48.053889 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48055us
I20260709 12:23:48.053221 23388 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.063249 23510 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.099349 23287 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:48.106659 18171 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:23:48.060930 23525 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.059314 23384 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:48.104352 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:48.118932 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:48.112746 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:48.107559 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:48.090143 23518 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:48.090487 23333 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:48.136554 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:48.116417 23174 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.943s	user 0.017s	sys 0.000s
I20260709 12:23:48.099941 23527 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:48.142526 23527 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.143013 23527 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:48.147639 23414 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.132618 23488 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:48.148818 23488 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:48.149266 23488 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:48.150864 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 839) took 1051 ms. Trace:
I20260709 12:23:48.151407 18096 rpcz_store.cc:276] 0709 12:23:47.099427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.109560 (+ 10133us) service_pool.cc:224] Handling call
0709 12:23:48.150612 (+1041052us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540035f2d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.150781 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370129664}
I20260709 12:23:48.115537 23312 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:48.124166 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:48.125624 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:48.128229 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:48.120290 18091 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:48.128407 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:48.108139 23328 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:48.130728 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:48.115811 23313 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:48.111063 23529 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:48.167758 23448 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.131544 23276 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:23:48.131937 23333 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.123538 23384 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.038276 23511 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.157938 23287 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:48.247682 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.006s (SENT)
I20260709 12:23:48.132876 23518 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.360746 23518 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:48.167366 23531 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.108379 23388 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.251585 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 848) took 1146 ms. Trace:
I20260709 12:23:48.369462 18091 rpcz_store.cc:276] 0709 12:23:47.105217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.112512 (+1007295us) service_pool.cc:224] Handling call
0709 12:23:48.251499 (+138987us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.283877 23437 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:48.129053 23355 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.107s
I20260709 12:23:48.285709 23522 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.158557 23312 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.254776 23515 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.399636 23515 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.402742 23515 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:48.286859 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:48.265623 23341 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.094s	user 0.011s	sys 0.010s
I20260709 12:23:48.403841 23341 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:48.405134 23372 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:48.406464 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 119787us
I20260709 12:23:48.352790 23516 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:48.336372 23490 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.187s	user 0.010s	sys 0.006s
I20260709 12:23:48.278604 23282 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:48.283793 23333 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:48.286923 23384 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:48.370718 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:48.376283 23437 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:48.250824 18813 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:48.282979 23502 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.378686 18090 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:48.431428 18091 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 8. Current term is 9. Ops: [8.10-8.10]
I20260709 12:23:48.271083 23317 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.386044 23448 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:48.386580 23535 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:23:48.387012 23355 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:48.387444 23312 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:48.389165 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2423) took 1089 ms. Trace:
I20260709 12:23:48.389712 23527 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.447494 18169 rpcz_store.cc:276] 0709 12:23:47.299951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.456879 (+156928us) service_pool.cc:224] Handling call
0709 12:23:48.388874 (+931995us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54002026d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.389076 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804961664}
I20260709 12:23:48.390717 23358 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.392745 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2408) took 1119 ms. Trace:
I20260709 12:23:48.395982 22827 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:48.352300 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 224033us
W20260709 12:23:48.355334 23533 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.177s	user 0.002s	sys 0.006s
W20260709 12:23:48.353404 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 105810us
I20260709 12:23:48.434566 23511 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:48.431747 23282 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:48.413734 23308 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:48.457048 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:48.433682 23457 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.372063 23350 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:48.423221 23490 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.471891 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:48.468298 23523 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.524s	user 0.014s	sys 0.000s
I20260709 12:23:48.476791 23523 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:48.476984 23532 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.275s	user 0.002s	sys 0.002s
I20260709 12:23:48.479539 23490 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.476791 23522 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:48.487913 23522 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:48.488368 23522 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:48.488587 23504 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.163s	user 0.000s	sys 0.001s
I20260709 12:23:48.447338 23355 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:48.452877 23525 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.494001 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:48.502580 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:48.504058 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:48.506840 23388 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:48.508953 23527 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:48.514055 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1166) took 1280 ms. Trace:
I20260709 12:23:48.518287 23523 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:48.518862 23523 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:48.519273 23523 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:48.448961 23448 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.513334 23463 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:48.446682 23317 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.491217 23538 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:48.533591 23469 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.567s	user 0.015s	sys 0.006s
I20260709 12:23:48.521483 23527 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:48.526079 23389 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.437s	user 0.018s	sys 0.003s
I20260709 12:23:48.543349 23502 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.475106 23457 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.498733 23437 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.273285 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:48.407640 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:48.395068 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.904208 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:48.904747 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.418395 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 151480us
I20260709 12:23:48.404881 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:48.451792 18170 rpcz_store.cc:276] 0709 12:23:47.273170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.273368 (+   198us) service_pool.cc:224] Handling call
0709 12:23:48.392520 (+1119152us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540007b554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.392655 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1186227968}
I20260709 12:23:48.514791 18806 rpcz_store.cc:276] 0709 12:23:47.233921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.235303 (+  1382us) service_pool.cc:224] Handling call
0709 12:23:47.235842 (+   539us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.236235 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:47.236770 (+   535us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.409656 (+172886us) log.cc:844] Serialized 14 byte log entry
0709 12:23:48.512578 (+1102922us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b18001e9760. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287
0709 12:23:48.512742 (+   164us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:48.512828 (+    86us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:48.512915 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.513708 (+   793us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.513962 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23362,"prepare.queue_time_us":2072,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":118,"raft.queue_time_us":296418,"raft.run_cpu_time_us":449,"raft.run_wall_time_us":39959,"spinlock_wait_cycles":55384832,"thread_start_us":319838,"threads_started":2}
W20260709 12:23:48.530158 22548 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.203s	user 0.015s	sys 0.004s
I20260709 12:23:48.723030 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2485) took 1643 ms. Trace:
I20260709 12:23:48.917006 22548 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.919795 23282 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.636464 23496 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:48.921582 23496 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:48.748904 18094 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:48.921979 18096 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:48.922497 18172 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:48.716676 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.787868 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45562 (request call id 26) took 1658 ms. Trace:
I20260709 12:23:48.722921 18022 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:48.714035 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2488) took 1633 ms. Trace:
W20260709 12:23:48.709183 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3278) took 2612 ms (client timeout 2249 ms). Trace:
W20260709 12:23:48.741887 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2479) took 1667 ms (client timeout 1946 ms). Trace:
W20260709 12:23:48.742607 18090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2584) took 1690 ms (client timeout 2140 ms). Trace:
I20260709 12:23:48.764626 22864 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.858557 22827 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.644541 23511 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:48.562227 23441 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.124s	user 0.008s	sys 0.008s
I20260709 12:23:48.923914 18097 rpcz_store.cc:276] 0709 12:23:47.093688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.093862 (+   174us) service_pool.cc:224] Handling call
0709 12:23:48.751866 (+1658004us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b54002972d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.752209 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1861728000}
I20260709 12:23:48.930815 23441 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:48.924355 18015 rpcz_store.cc:269] 0709 12:23:46.096939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.121346 (+ 24407us) service_pool.cc:224] Handling call
0709 12:23:48.708219 (+2586873us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.708678 (+   459us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100200320}
I20260709 12:23:48.585891 23524 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:48.548270 23450 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.933494 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:48.583225 23501 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.193s	user 0.003s	sys 0.002s
I20260709 12:23:48.759951 23353 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.934161 23511 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:48.935088 23353 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.938030 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2540) took 4533 ms. Trace:
I20260709 12:23:48.938227 23491 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.787755 23365 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:48.933522 23524 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:48.939656 22864 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.940879 23469 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.727128 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:48.729712 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:48.636163 23497 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:48.878996 23529 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:48.666692 23522 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.880445 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2486) took 1800 ms. Trace:
I20260709 12:23:48.882956 23317 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:48.664299 23523 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.545588 23527 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:48.572561 23454 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:48.886615 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2428) took 1559 ms. Trace:
I20260709 12:23:48.571265 23448 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.895707 23531 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.587838 23487 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.898219 23447 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:48.899282 23409 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.901578 23503 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:48.710913 17920 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 3 to 8, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 8 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:48.540223 23539 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.576959 23460 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.938520 18173 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:48.939349 23365 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.538561 23463 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:48.924798 18090 rpcz_store.cc:269] 0709 12:23:47.050411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.050628 (+   217us) service_pool.cc:224] Handling call
0709 12:23:48.740528 (+1689900us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54002963d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.740984 (+   456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2960656768}
I20260709 12:23:48.901000 23457 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.878319 18811 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:48.920836 22548 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:48.941742 23469 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:48.924494 18023 rpcz_store.cc:269] 0709 12:23:47.066629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.072168 (+  5539us) service_pool.cc:224] Handling call
0709 12:23:48.733220 (+1661052us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.733540 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1848280320}
I20260709 12:23:48.952514 18016 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:48.957209 23287 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:48.958243 23283 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:48.959882 18021 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:48.962291 23361 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:48.998973 23409 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.001077 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2478) took 1934 ms. Trace:
I20260709 12:23:48.962558 23372 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.963863 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:48.964238 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.84062e+06us
I20260709 12:23:48.916721 18019 rpcz_store.cc:276] 0709 12:23:47.079295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.079420 (+   125us) service_pool.cc:224] Handling call
0709 12:23:47.100806 (+ 21386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:47.103621 (+  2815us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:47.103655 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:47.104207 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:23:47.798696 (+694489us) spinlock_profiling.cc:243] Waited 5.27 ms on lock 0x7b180039cca0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:47.933562 (+134866us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:47.988322 (+ 54760us) log.cc:844] Serialized 14 byte log entry
0709 12:23:48.047226 (+ 58904us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.047372 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.722082 (+674710us) raft_consensus.cc:1685] finished
0709 12:23:48.722327 (+   245us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.722787 (+   460us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18298,"prepare.queue_time_us":38111,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":6747,"raft.queue_time_us":68544,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":25511,"spinlock_wait_cycles":12114176,"thread_start_us":134571,"threads_started":1,"wal-append.queue_time_us":868853}
W20260709 12:23:48.967569 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 559864us
I20260709 12:23:48.548820 23389 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.968546 22827 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.011628 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:49.012980 23324 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:48.945147 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 695) took 1518 ms. Trace:
I20260709 12:23:48.976828 23502 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:48.977677 23487 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.015332 23463 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.978742 23531 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:48.980103 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2582) took 1954 ms. Trace:
I20260709 12:23:48.974854 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 700) took 1426 ms. Trace:
I20260709 12:23:48.977525 23527 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:48.972608 18167 rpcz_store.cc:276] 0709 12:23:47.327550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.610893 (+283343us) service_pool.cc:224] Handling call
0709 12:23:48.886092 (+1275199us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54003011d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.886348 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":324291968}
I20260709 12:23:48.662089 23518 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.001596 18022 rpcz_store.cc:276] 0709 12:23:47.066275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.066435 (+   160us) service_pool.cc:224] Handling call
0709 12:23:48.722150 (+1655715us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.929879 (+207729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.000781 (+ 70902us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400109fd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.000997 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3806508416}
I20260709 12:23:49.024008 18020 rpcz_store.cc:276] 0709 12:23:47.544671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.726092 (+1181421us) service_pool.cc:224] Handling call
0709 12:23:48.970234 (+244142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.982810 23447 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:48.939263 23491 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.009840 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:49.010926 23372 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.015802 18015 rpcz_store.cc:276] 0709 12:23:47.426339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.933059 (+1506720us) service_pool.cc:224] Handling call
0709 12:23:48.945025 (+ 11966us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:49.012976 18165 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:49.002446 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 704) took 1060 ms. Trace:
I20260709 12:23:49.019349 23487 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:48.924240 18018 rpcz_store.cc:276] 0709 12:23:47.080236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.102389 (+ 22153us) service_pool.cc:224] Handling call
0709 12:23:47.106213 (+  3824us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:47.947270 (+841057us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:47.947304 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:47.947913 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 12:23:48.078216 (+130303us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:48.078283 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.078385 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.504452 (+426067us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:48.712797 (+208345us) raft_consensus.cc:1685] finished
0709 12:23:48.712925 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.713636 (+   711us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2687,"raft.queue_time_us":13939,"raft.run_cpu_time_us":4904,"raft.run_wall_time_us":1382006,"spinlock_wait_cycles":5449600,"thread_start_us":967606,"threads_started":2,"wal-append.queue_time_us":207916}
I20260709 12:23:49.087548 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:49.032418 23523 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.032611 23518 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.930538 23496 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.035462 23522 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.036355 18099 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:49.023622 18094 rpcz_store.cc:276] 0709 12:23:47.025893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.026156 (+   263us) service_pool.cc:224] Handling call
0709 12:23:47.026649 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.922707 (+1896058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.979564 (+ 56857us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400051ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:48.979963 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3958713216}
I20260709 12:23:48.941293 23524 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.013598 23324 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:48.996316 23503 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.070089 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:48.948094 18024 rpcz_store.cc:276] 0709 12:23:47.079626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.087918 (+  8292us) service_pool.cc:224] Handling call
0709 12:23:47.115706 (+ 27788us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:47.120713 (+  5007us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:47.120753 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:47.121346 (+   593us) log.cc:844] Serialized 52 byte log entry
0709 12:23:47.323131 (+201785us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:47.454950 (+131819us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.534709 (+ 79759us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.534990 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.535101 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.901560 (+366459us) raft_consensus.cc:1685] finished
0709 12:23:47.901586 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.398434 (+496848us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b18003df300. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:48.879770 (+481336us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b54000fe950. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:48.880149 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80953,"prepare.queue_time_us":168664,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":81,"raft.queue_time_us":22910,"raft.run_cpu_time_us":616,"raft.run_wall_time_us":142345,"spinlock_wait_cycles":182108288,"thread_start_us":238073,"threads_started":2,"wal-append.queue_time_us":405605,"wal-append.run_cpu_time_us":5059,"wal-append.run_wall_time_us":1351317}
I20260709 12:23:48.938567 18166 rpcz_store.cc:276] 0709 12:23:44.404530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.669156 (+2264626us) service_pool.cc:224] Handling call
0709 12:23:46.686802 (+ 17646us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:46.994107 (+307305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.994145 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:46.994920 (+   775us) log.cc:844] Serialized 52 byte log entry
0709 12:23:47.109783 (+114863us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:47.166023 (+ 56240us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.166290 (+   267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.166378 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.582283 (+1415905us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:48.937349 (+355066us) raft_consensus.cc:1685] finished
0709 12:23:48.937500 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.937731 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":295611,"prepare.queue_time_us":40190,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"raft.queue_time_us":298086,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":7045,"spinlock_wait_cycles":24500608,"thread_start_us":114393,"threads_started":1,"wal-append.queue_time_us":220347}
W20260709 12:23:49.059752 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.060271 23463 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:49.059877 17920 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 7 to 9, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 9 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:49.064888 23497 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:49.028359 23409 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:49.024682 23317 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.067781 23283 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.019261 23502 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.106578 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.118640 18811 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:49.119755 23283 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.934230 23450 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.122339 23317 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.125149 23317 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.126415 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:49.130478 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:49.130529 17987 connection.cc:256] RPC call timeout handler was delayed by -1.0354007499995532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
I20260709 12:23:49.132148 23503 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:49.135516 17921 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 281) took 46737 ms (client timeout 15000 ms). Trace:
I20260709 12:23:49.135943 18099 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:49.069757 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:49.049782 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.047613 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:48.544517 23355 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.045650 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:49.139583 18090 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.097066 23496 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:49.025231 23441 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:49.038666 23353 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.111480 23497 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.111987 17920 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 5 to 7, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 7 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:49.116285 23447 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.119874 23502 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.066277 18016 rpcz_store.cc:276] 0709 12:23:47.935728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.729011 (+793283us) service_pool.cc:224] Handling call
0709 12:23:48.995860 (+266849us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:49.092227 23531 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:48.980151 23529 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.155555 23503 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:49.157853 23529 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.157632 17920 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 4 to 6, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 6 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:49.159273 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.160336 17920 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:49.162726 23324 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.164216 23324 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:49.164729 22864 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.165908 23389 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.166270 23372 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.166766 23524 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:49.167217 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:49.168093 18099 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:49.170855 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:49.172509 23441 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.141742 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.149971 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.175222 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.176925 23447 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:49.177932 18022 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:49.161240 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:49.176503 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:49.134210 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:49.109417 23487 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.186666 18096 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:49.151134 23317 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:49.183511 18017 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:49.193884 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2414) took 1900 ms. Trace:
I20260709 12:23:49.194438 18164 rpcz_store.cc:276] 0709 12:23:47.292933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.746309 (+453376us) service_pool.cc:224] Handling call
0709 12:23:49.191722 (+1445413us) spinlock_profiling.cc:243] Waited 64.9 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.193660 (+  1938us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149309312}
I20260709 12:23:49.195518 23491 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.196475 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:49.197204 18094 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:49.200706 23450 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.201622 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.202842 18810 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 7.
I20260709 12:23:49.206859 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:49.215866 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 206706us
I20260709 12:23:49.216904 23372 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:49.221611 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.225370 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:49.222102 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61126us
W20260709 12:23:49.136173 17921 rpcz_store.cc:269] 0709 12:23:02.397618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.398761 (+  1143us) service_pool.cc:224] Handling call
0709 12:23:49.135036 (+46736275us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:38.773012 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:38.812696 (+ 39684us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:38.812763 (+    67us) write_op.cc:435] Acquired schema lock
0709 12:23:38.812785 (+    22us) tablet.cc:662] Decoding operations
0709 12:23:38.956292 (+143507us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:38.981320 (+ 25028us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:38.984895 (+  3575us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:23:38.999581 (+ 14686us) tablet.cc:701] Row locks acquired
0709 12:23:38.999661 (+    80us) write_op.cc:260] PREPARE: finished
0709 12:23:38.999878 (+   217us) write_op.cc:270] Start()
0709 12:23:39.000057 (+   179us) write_op.cc:276] Timestamp: P: 1783599818999797 usec, L: 0
0709 12:23:39.000102 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:23:39.042235 (+ 42133us) log.cc:844] Serialized 2825 byte log entry
0709 12:23:40.008713 (+966478us) op_driver.cc:464] REPLICATION: finished
0709 12:23:40.344027 (+335314us) write_op.cc:301] APPLY: starting
0709 12:23:40.374311 (+ 30284us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:40.699294 (+324983us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:40.742385 (+ 43091us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:40.937045 (+194660us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:40.937427 (+   382us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=14,mrs_lookups=0
0709 12:23:40.937491 (+    64us) write_op.cc:312] APPLY: finished
0709 12:23:40.937936 (+   445us) log.cc:844] Serialized 73 byte log entry
0709 12:23:41.301666 (+363730us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:41.313882 (+ 12216us) write_op.cc:454] Released schema lock
0709 12:23:41.353981 (+ 40099us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:41.443323 (+ 89342us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:41.451514 (+  8191us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9010839,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":286687,"apply.run_cpu_time_us":27084,"apply.run_wall_time_us":1049042,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33429,"num_ops":7,"prepare.queue_time_us":229124,"prepare.run_cpu_time_us":140750,"prepare.run_wall_time_us":501273,"raft.queue_time_us":238130,"raft.run_cpu_time_us":36917,"raft.run_wall_time_us":172849,"replication_time_us":995578,"thread_start_us":882032,"threads_started":5,"wal-append.queue_time_us":801751,"wal-append.run_cpu_time_us":18743,"wal-append.run_wall_time_us":2532215}]]}
I20260709 12:23:49.230192 23441 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:49.230924 23441 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.231387 23441 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:49.234174 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2407) took 1961 ms (client timeout 2247 ms). Trace:
W20260709 12:23:49.234795 18172 rpcz_store.cc:269] 0709 12:23:47.272779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.280812 (+  8033us) service_pool.cc:224] Handling call
0709 12:23:49.219247 (+1938435us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54001b5b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.234096 (+ 14849us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3655192960}
I20260709 12:23:49.234810 18022 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:49.236754 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2493) took 1603 ms. Trace:
I20260709 12:23:49.237098 18022 rpcz_store.cc:276] 0709 12:23:47.633779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.189135 (+1555356us) service_pool.cc:224] Handling call
0709 12:23:49.234448 (+ 45313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.236391 (+  1943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.236658 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26624}
I20260709 12:23:49.238967 18022 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:49.239743 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2500) took 1241 ms. Trace:
I20260709 12:23:49.240082 18022 rpcz_store.cc:276] 0709 12:23:47.998486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.238128 (+1239642us) service_pool.cc:224] Handling call
0709 12:23:49.238532 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.239423 (+   891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.239657 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:49.249542 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.250275 18096 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:49.250717 18814 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:49.252556 18808 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:49.254195 18094 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:49.260892 23365 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.209707 23487 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.262624 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 702) took 1692 ms. Trace:
I20260709 12:23:49.266865 18024 rpcz_store.cc:276] 0709 12:23:47.584211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.139520 (+1555309us) service_pool.cc:224] Handling call
0709 12:23:49.262538 (+123018us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:49.215173 23529 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.269629 18022 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:49.276381 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:49.278494 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 132076us
I20260709 12:23:49.279213 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:49.279702 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:49.281042 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2431) took 1606 ms. Trace:
I20260709 12:23:49.282128 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3283) took 2773 ms. Trace:
I20260709 12:23:49.246261 17920 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 6 to 9, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 9 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:49.282549 18021 rpcz_store.cc:276] 0709 12:23:46.508432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.508693 (+   261us) service_pool.cc:224] Handling call
0709 12:23:46.509095 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.028743 (+2519648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.266688 (+237945us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b540022f954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:49.282042 (+ 15354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5537916800}
I20260709 12:23:49.283900 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 5 to 10, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 10 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:23:49.287276 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.289127 18807 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:49.290011 18093 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.908s	user 0.015s	sys 0.012s
W20260709 12:23:49.290243 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2412) took 1998 ms (client timeout 2108 ms). Trace:
I20260709 12:23:49.290627 18093 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
W20260709 12:23:49.290762 18173 rpcz_store.cc:269] 0709 12:23:47.292221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.597907 (+305686us) service_pool.cc:224] Handling call
0709 12:23:49.265866 (+1667959us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.290169 (+ 24303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3043430272}
W20260709 12:23:49.291999 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.293264 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 159275us
I20260709 12:23:49.294683 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2501) took 1294 ms. Trace:
I20260709 12:23:49.295153 18022 rpcz_store.cc:276] 0709 12:23:48.000086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.241066 (+1240980us) service_pool.cc:224] Handling call
0709 12:23:49.241736 (+   670us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.294282 (+ 52546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.294591 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:49.299368 18813 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:49.306285 18016 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:49.310098 23248 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.310916 23248 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.313530 23437 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:49.293541 23384 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.855s	user 0.003s	sys 0.013s
W20260709 12:23:49.322337 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 274979us
I20260709 12:23:49.295771 23497 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:49.270005 23409 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.337702 18024 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:49.347922 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.351352 18017 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:49.357558 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2432) took 1563 ms. Trace:
W20260709 12:23:49.354563 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 365868us
I20260709 12:23:49.364914 18092 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:49.369982 23515 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.352612 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:49.374722 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.375272 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76279.4us
I20260709 12:23:49.281877 18171 rpcz_store.cc:276] 0709 12:23:47.576658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.129285 (+552627us) service_pool.cc:224] Handling call
0709 12:23:48.129681 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:48.155482 (+ 25801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:48.155523 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:48.156153 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:23:48.449324 (+293171us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:48.989598 (+540274us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.108471 (+118873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.145279 (+ 36808us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:49.151814 (+  6535us) raft_consensus.cc:1685] finished
0709 12:23:49.151835 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.182842 (+ 31007us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":122368,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":83019,"raft.run_cpu_time_us":456,"raft.run_wall_time_us":11342,"thread_start_us":725133,"threads_started":2,"wal-append.queue_time_us":348789}
I20260709 12:23:49.324146 23469 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:49.383970 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74600.4us
I20260709 12:23:49.384552 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2494) took 1742 ms. Trace:
I20260709 12:23:49.385254 18810 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:49.385673 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.387276 18017 rpcz_store.cc:276] 0709 12:23:47.634152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.209625 (+1575473us) service_pool.cc:224] Handling call
0709 12:23:49.296939 (+ 87314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.375870 (+ 78931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.376150 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:49.352223 23450 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:49.400084 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.411909 18807 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:49.358803 23487 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.417987 18813 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:49.405058 23450 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:49.358220 18168 rpcz_store.cc:276] 0709 12:23:47.615544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.261772 (+646228us) service_pool.cc:224] Handling call
0709 12:23:48.335048 (+ 73276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.335406 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:48.750468 (+415062us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.043895 (+293427us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.092920 (+ 49025us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:49.099705 (+  6785us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:49.107988 (+  8283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.142018 (+ 34030us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.179210 (+ 37192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61017,"prepare.queue_time_us":316081,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":132,"raft.queue_time_us":84714,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":424,"spinlock_wait_cycles":43148160,"thread_start_us":513493,"threads_started":2,"wal-append.queue_time_us":178329}
I20260709 12:23:49.414809 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.329511 23384 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.429242 23487 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.426462 23450 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:49.431783 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.433856 23529 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:49.439042 18093 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 842) took 2338 ms (client timeout 2006 ms). Trace:
I20260709 12:23:49.440044 17920 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 1 to 4, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 4 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:49.443683 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:49.449535 18090 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:49.455549 18098 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:23:49.455765 23248 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.456957 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:49.461103 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:49.467834 23548 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.500s	user 0.004s	sys 0.004s
W20260709 12:23:49.468431 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.498229 18098 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 5
W20260709 12:23:49.503010 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.469902 23511 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.535s	user 0.014s	sys 0.004s
W20260709 12:23:49.471058 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 837) took 2372 ms (client timeout 1874 ms). Trace:
I20260709 12:23:49.473308 23355 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:49.463028 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:49.484918 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 203397us
I20260709 12:23:49.484025 18808 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
W20260709 12:23:49.493894 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.495446 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:49.496371 18093 rpcz_store.cc:269] 0709 12:23:47.103355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.778064 (+674709us) service_pool.cc:224] Handling call
0709 12:23:49.438587 (+1660523us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.438875 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649434496}
W20260709 12:23:49.504050 18092 rpcz_store.cc:269] 0709 12:23:47.098595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.099079 (+   484us) service_pool.cc:224] Handling call
0709 12:23:49.470680 (+2371601us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.470926 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4648308608}
I20260709 12:23:49.507094 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 703) took 1566 ms. Trace:
I20260709 12:23:49.508849 22827 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.502669 18098 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:23:49.514628 18098 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:49.505589 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 873074us
I20260709 12:23:49.515359 18169 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:49.498324 23518 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.517426 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.523294 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:49.532922 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70054.2us
I20260709 12:23:49.529222 18813 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:49.536810 23365 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:49.544503 23365 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.544981 23365 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:49.545990 23457 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:49.546552 23457 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.546949 23457 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:49.550416 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106968us
I20260709 12:23:49.570868 18812 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:49.527546 23518 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.514755 18020 rpcz_store.cc:276] 0709 12:23:47.935411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.214049 (+1278638us) service_pool.cc:224] Handling call
0709 12:23:49.234447 (+ 20398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.234893 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:49.414588 (+179695us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.495492 (+ 80904us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:49.495674 (+   182us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:49.500907 (+  5233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.501078 (+   171us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.501464 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":239060,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":12536,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":11751,"spinlock_wait_cycles":24960,"thread_start_us":222169,"threads_started":2}
I20260709 12:23:49.560740 18807 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:49.559587 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.508586 23511 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:49.587657 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46721us
W20260709 12:23:49.589707 23333 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.104s	user 0.008s	sys 0.014s
I20260709 12:23:49.590337 23333 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:49.599373 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46527.7us
W20260709 12:23:49.557607 23488 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.284s	user 0.009s	sys 0.010s
I20260709 12:23:49.526510 23529 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:49.585189 23312 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.962s	user 0.012s	sys 0.010s
I20260709 12:23:49.618010 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:49.613801 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.602476 23333 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:49.622076 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 64562.9us
I20260709 12:23:49.625403 18017 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
W20260709 12:23:49.628937 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.627061 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:49.627650 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:49.637567 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:49.619514 23518 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:49.605468 17920 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: config changed from index 3 to 5, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1), VOTER 768723648a9444fda53659e6a8cc3091 (127.16.102.2), VOTER ae9641d5c3184572a102d51a75f670ee (127.16.102.3), VOTER 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 9 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:23:49.641237 23549 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:23:49.616788 23248 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:23:49.635095 22827 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:23:49.636521 23447 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:49.621306 23231 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:23:49.656246 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:49.656940 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72032us
I20260709 12:23:49.619644 23488 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.661281 17920 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 5 to 10, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 10 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:49.665555 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59575.7us
W20260709 12:23:49.662766 23287 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.666s	user 0.008s	sys 0.008s
I20260709 12:23:49.672688 23529 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:49.673868 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:49.657934 23282 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:49.629601 23312 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.653858 23447 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:49.677532 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:49.679083 23282 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:49.673410 22864 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:49.664381 23518 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:49.679993 18809 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:49.691730 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:49.675237 23384 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:49.707979 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50130.7us
W20260709 12:23:49.712889 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.712992 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93553.3us
I20260709 12:23:49.687143 23282 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.715660 17989 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12.
I20260709 12:23:49.719370 17989 leader_election.cc:403] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:49.690686 23523 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.687143 22864 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:49.684347 22827 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:23:49.686533 23447 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.707584 23463 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:49.703902 23518 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:49.709789 23487 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.712332 23511 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.722935 23248 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:49.728166 23317 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.502s	user 0.014s	sys 0.000s
I20260709 12:23:49.737365 23333 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.730685 23463 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.742126 22548 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:49.742883 22548 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.741278 23333 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:49.753997 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70535.6us
I20260709 12:23:49.758983 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:49.742201 23463 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:49.760224 18093 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:49.761771 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:49.762362 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82967.8us
I20260709 12:23:49.763190 18092 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:23:49.767203 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:49.774657 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:49.739494 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:49.769889 23248 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.781747 23248 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:49.786473 23555 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:23:49.786662 23365 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.781965 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:49.788813 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2495) took 1823 ms. Trace:
I20260709 12:23:49.780807 23317 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.795029 23511 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:49.795666 23511 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.796101 23511 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:49.796663 23333 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.797881 22548 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:49.798666 23437 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:23:49.776782 23488 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.803959 23312 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:49.769763 23372 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.000s
I20260709 12:23:49.807679 23372 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:49.733628 23518 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:49.816309 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 76883.7us
I20260709 12:23:49.817390 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:49.809460 23372 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:49.826936 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52454.3us
I20260709 12:23:49.842787 23487 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:49.843400 18169 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:49.838268 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:49.839200 23186 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:49.790021 18015 rpcz_store.cc:276] 0709 12:23:47.965415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.223534 (+1258119us) service_pool.cc:224] Handling call
0709 12:23:49.238457 (+ 14923us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:49.239086 (+   629us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:49.389546 (+150460us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.603122 (+213576us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:49.652352 (+ 49230us) log.cc:844] Serialized 52 byte log entry
0709 12:23:49.662336 (+  9984us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:49.662387 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.662496 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:49.788488 (+125992us) raft_consensus.cc:1685] finished
0709 12:23:49.788515 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.788701 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":244851,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":12823,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":9404,"thread_start_us":326139,"threads_started":2,"wal-append.queue_time_us":272321}
I20260709 12:23:49.850524 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:49.851311 23488 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:49.851819 23488 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.852236 23488 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:49.850531 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:49.847071 23186 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.854049 23186 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:49.850798 18098 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.847503 23487 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:49.858724 23487 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.726859 23523 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.887941 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:49.892786 23527 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.858s	user 0.015s	sys 0.003s
W20260709 12:23:49.896080 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 173403us
W20260709 12:23:49.894726 23441 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.663s	user 0.012s	sys 0.007s
W20260709 12:23:49.900864 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135479us
I20260709 12:23:49.897807 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:49.898376 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 187607us
I20260709 12:23:49.789602 23365 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:49.871551 23497 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.522s	user 0.009s	sys 0.004s
I20260709 12:23:49.908943 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:49.909771 18809 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:49.909886 18169 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:49.924075 18023 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:49.904557 23372 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:49.896729 23527 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.936379 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:49.944973 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:49.945345 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.928462 23312 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:49.944844 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:49.948724 18019 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:49.904735 23441 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.950872 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:49.931100 23559 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:23:49.951635 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.951699 18015 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:49.953292 18812 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:23:49.954025 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:49.957165 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:49.960083 23497 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:49.960812 23497 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:49.961320 23497 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.966079 23312 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:49.969617 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:49.972584 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44391.7us
I20260709 12:23:49.973915 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:49.974361 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77211.3us
I20260709 12:23:49.975817 23365 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.976665 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:49.978447 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2504) took 1075 ms. Trace:
I20260709 12:23:49.978857 18023 rpcz_store.cc:276] 0709 12:23:48.902679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.149526 (+246847us) service_pool.cc:224] Handling call
0709 12:23:49.978260 (+828734us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:49.978383 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1724008704}
I20260709 12:23:49.983317 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:49.979122 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:49.989962 23559 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:23:49.990634 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.993749 18090 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:49.995169 23372 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:49.998591 18097 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:49.933683 22864 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.002393 23257 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:23:50.003585 23257 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:50.003795 23527 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.004067 23257 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:49.995170 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.015947 18173 raft_consensus.cc:2368] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13.
W20260709 12:23:50.019806 23355 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.503s	user 0.004s	sys 0.017s
I20260709 12:23:50.020020 23529 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:50.020913 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44338.9us
W20260709 12:23:50.019860 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69918.2us
I20260709 12:23:49.933219 23389 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
W20260709 12:23:50.028568 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100744us
I20260709 12:23:50.013720 23287 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.032043 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.032058 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:50.036448 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.037209 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.038801 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.039750 18096 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.041281 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.042635 23372 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:50.042891 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:50.033322 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.014099 23441 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:50.049974 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.043320 23523 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.050827 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:50.054780 18171 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:23:50.005834 23457 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.005299 23257 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.067660 18015 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:50.033802 23312 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:50.070304 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.059496 23450 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.564s	user 0.011s	sys 0.008s
I20260709 12:23:50.072146 18090 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:50.082803 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.083858 18097 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.090318 23562 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:50.078090 23564 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:50.090930 23523 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:50.091557 23564 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.092279 23567 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:50.092783 23567 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.092720 23564 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.051393 23231 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:23:50.094123 23567 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.095546 23560 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:50.090318 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46482.4us
I20260709 12:23:50.090763 23568 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:50.070641 23518 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.052805 22827 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:23:50.101506 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.071821 23558 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:50.057132 23355 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.048003 23287 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.108119 23186 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.080654 23450 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.105921 23568 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.111774 23231 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:50.112277 23560 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:50.110383 22864 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.113715 23564 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:50.114028 23524 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.947s	user 0.011s	sys 0.008s
I20260709 12:23:50.059928 23463 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.103174 23312 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:50.129015 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80618.7us
I20260709 12:23:50.151921 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:50.074970 23447 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.091179 23457 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.133492 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.136067 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.093371 23511 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.143872 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:50.149932 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.129093 23257 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.159507 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.161068 23463 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.162154 18017 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:23:50.163305 23447 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.164122 23447 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:50.164512 23447 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.164855 23447 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.166071 18173 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:50.166445 18810 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:50.166008 18171 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:50.167981 23560 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.143119 23231 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:50.159551 18098 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:50.165980 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.171830 18814 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:50.155318 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112511us
I20260709 12:23:50.156672 18093 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:50.143762 23568 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:50.142509 23558 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:50.172073 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.177490 18809 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:50.180110 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.183620 23511 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.184495 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.187269 18098 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:50.187777 23457 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:50.190500 23524 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:23:50.192310 23565 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.098s	user 0.002s	sys 0.000s
I20260709 12:23:50.186327 23496 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:50.206941 18172 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:50.149120 23355 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.209800 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.190867 23558 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.211712 23497 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.214287 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.214470 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.215209 18090 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.215503 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.216821 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.217643 18096 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:50.201272 23511 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:50.217675 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.218832 18091 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.218499 23511 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.221589 23450 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.222713 23287 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:50.216935 18095 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:50.215094 23437 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12.
I20260709 12:23:50.229405 18168 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:23:50.230705 18169 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:50.233165 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:50.234634 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69087.5us
I20260709 12:23:50.218446 23447 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.236493 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.236572 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.237553 18017 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.238225 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.239609 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.239984 23497 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.240485 18024 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.240614 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.241307 18173 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.242347 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.226534 23282 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.243469 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.244196 18018 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.245208 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.237612 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.221640 23511 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.200109 23496 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.246366 18022 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.246661 18020 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:50.248243 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.252209 18016 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:50.257656 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.258268 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.259084 18019 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:50.259272 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.254876 23441 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:50.260969 23441 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.249254 23523 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:50.262487 23523 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:50.263741 23441 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:50.263617 23523 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.266664 18165 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:50.268179 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.268374 23186 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.268889 18165 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.268985 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.269340 23372 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:50.269847 23372 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:50.270257 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.270452 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.270975 18808 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:50.271229 18166 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:50.178527 23463 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:23:50.272326 23518 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.273200 18170 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:50.274447 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.275704 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.275787 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.276363 18813 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:50.278978 23496 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:50.279186 23384 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.279752 23384 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.245651 23282 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.280375 23248 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.281785 18098 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:50.281915 23365 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:50.283576 18172 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:50.247108 23287 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:50.284785 23287 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:50.285202 23287 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:50.285264 23355 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.238243 23447 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:50.170812 23231 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.217936 23312 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.295027 18167 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:50.297102 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.294291 23312 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:50.271008 23372 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.298871 23384 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.299374 23372 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.290266 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.300411 18810 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:50.300592 18167 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
W20260709 12:23:50.301129 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.300416 23231 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.302479 18017 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:50.303895 23248 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.272326 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.305027 18166 raft_consensus.cc:2368] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:50.306813 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.249805 18780 leader_election.cc:400] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:23:50.309477 18780 leader_election.cc:403] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:23:50.301842 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.308389 18170 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [12.11-14.12]
I20260709 12:23:50.313621 18171 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:50.318101 18097 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:50.320075 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:50.320675 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71024.2us
I20260709 12:23:50.321659 18167 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:50.319916 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.325470 18173 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:50.325470 18813 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:50.327172 18022 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:50.325439 18809 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:50.328847 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.348990 18170 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
W20260709 12:23:50.332491 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 133200us
W20260709 12:23:50.332643 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71177.7us
I20260709 12:23:50.350415 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.351444 18169 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:23:50.333701 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.352710 18172 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:50.347257 18810 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:50.353060 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.342181 18814 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:50.358035 18017 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:50.361543 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.332872 23457 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:23:50.332279 23450 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.363065 18021 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:50.362388 23447 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:23:50.363597 23450 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.333232 23563 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:50.362243 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.361511 23441 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.368667 18169 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:50.368695 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.356619 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.371456 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.377261 18167 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:50.378818 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:50.379899 18165 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:50.382815 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.383553 18171 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:50.381309 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.380205 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:50.391062 18091 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.392822 18810 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.13-12.13]
I20260709 12:23:50.393703 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.393713 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.394924 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.395437 18169 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:50.395821 18021 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.398976 18167 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:50.368368 23563 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.367967 23441 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.381636 23497 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:50.401518 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.404152 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.427695 18171 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:50.424714 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.428576 18810 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:50.426816 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:50.400436 23488 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.548s	user 0.008s	sys 0.009s
I20260709 12:23:50.429104 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.401337 18168 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:50.388353 23564 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:50.402339 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:50.410048 23487 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:50.382942 23469 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:23:50.455937 23469 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:50.456445 23469 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.426249 23560 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.458179 23560 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:50.460726 23560 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:50.463888 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.436813 18807 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.13-13.13]
I20260709 12:23:50.447654 18808 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:50.421116 23450 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.449287 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.449419 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.476548 18169 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:50.477272 18021 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:50.447685 18167 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.424877 23568 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.421007 23248 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:50.489519 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:50.452518 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.452996 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:50.395238 22864 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:23:50.450132 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.476624 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:50.477105 18168 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:50.478531 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:50.480000 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.486089 18806 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:50.487565 18020 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:50.492172 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.494127 18165 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:50.494526 18170 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:50.493357 18814 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:50.444875 23497 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:50.500257 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.503002 18172 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 10: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:50.500257 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.506457 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.507529 18169 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:23:50.509008 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.510262 23186 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:50.510943 23186 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:50.511473 23186 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:50.504529 23496 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.516427 23496 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.518582 23568 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.519470 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.520285 18165 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:50.521324 23570 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:50.522008 23567 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.493162 23248 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:50.503362 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.510265 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.522085 23570 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.528342 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.490077 23282 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:50.535410 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.535248 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 133014us
I20260709 12:23:50.530422 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.444298 23488 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:50.526289 23529 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.003s
I20260709 12:23:50.578228 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.504572 23524 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:50.582908 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.582979 18022 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:23:50.587158 18806 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:50.587070 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.588456 18810 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.579507 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.577649 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:50.576622 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 123733us
I20260709 12:23:50.489652 23564 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.576908 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.540941 18017 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
W20260709 12:23:50.589231 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.591630 18809 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.595865 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.596514 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.597421 18099 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:50.598397 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.596824 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.620337 18814 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.587064 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.595504 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.622154 18016 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.623299 18811 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:50.594287 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.624245 18807 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:23:50.627105 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.628397 18015 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:50.535887 23248 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:50.634601 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.065s (SENT)
I20260709 12:23:50.632362 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.635404 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:50.539739 23437 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:50.534960 23487 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.639533 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.639854 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.647377 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.648334 18806 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:50.457743 23469 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.578423 23312 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:50.664129 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.654107 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:50.595132 23488 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:50.666555 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.636355 23529 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:50.576087 23568 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 12:23:50.670459 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.673861 18015 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 10: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.674263 18780 leader_election.cc:400] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13.
I20260709 12:23:50.677348 18814 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.678375 18021 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.651721 23518 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:50.716629 18092 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:50.718709 18780 leader_election.cc:403] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:23:50.719012 18806 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:50.677866 18016 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:50.651093 23524 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.682605 18809 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.732030 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:50.732370 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.734058 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.734089 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.737025 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.658130 23437 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:50.670413 23447 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:50.557331 23563 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.749173 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:50.750478 18165 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:50.752219 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.752851 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.753229 18016 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.753594 18091 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:50.752878 18092 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:50.738422 23469 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:50.761552 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 163706us
I20260709 12:23:50.734933 23511 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.748446 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.763731 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:50.668828 23312 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.770012 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.769239 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.744505 23529 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13.
I20260709 12:23:50.743841 23488 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:50.777745 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.778609 18016 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:50.744295 23447 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:50.778210 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.874s (SENT)
I20260709 12:23:50.782362 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:50.777019 18095 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:50.786288 18814 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:50.787796 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:50.788626 18165 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:50.787568 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 117633us
I20260709 12:23:50.789637 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.790443 18809 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:50.790941 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:50.792124 18806 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:50.793043 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.793975 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:50.781544 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:50.796672 23441 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:50.796983 23576 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.167s	user 0.001s	sys 0.001s
I20260709 12:23:50.802026 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.788946 23447 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:50.874780 18167 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:50.810297 23231 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:50.796005 23524 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.737445 23497 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.884984 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.762413 23570 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.818709 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:50.819471 23563 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:50.819561 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.835839 18780 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:23:50.889628 18780 leader_election.cc:403] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:50.841445 18809 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:23:50.841562 18170 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:50.860394 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:23:50.797374 23437 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:50.876616 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.876886 23287 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.591s	user 0.017s	sys 0.005s
W20260709 12:23:50.878432 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82072.1us
I20260709 12:23:50.899107 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.879525 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.900457 18096 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:50.881027 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:50.884864 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:50.900655 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65057us
I20260709 12:23:50.901298 18092 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:50.886096 18812 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:50.876967 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.903825 18097 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:50.877667 23231 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.879137 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:50.815248 23488 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:50.877575 23529 raft_consensus.cc:2764] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
I20260709 12:23:50.775799 23511 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.905916 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.987129 18016 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:50.901232 18815 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:50.951227 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:50.934353 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 161578us
I20260709 12:23:50.988467 18018 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:50.946326 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.964108 18098 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:50.964901 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65163.5us
W20260709 12:23:50.965663 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.909546 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.902118 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:50.773702 23564 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.780407 23487 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:50.939096 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.886271 23518 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:50.974387 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:50.987749 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 170524us
I20260709 12:23:51.002534 23518 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:51.003058 23518 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.003473 23518 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:51.005431 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:51.001426 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:51.006875 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.008692 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.013716 18017 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:51.013077 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:50.971338 23231 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:51.015061 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:51.001678 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.014608 18022 raft_consensus.cc:2376] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12. Re-sending same reply.
I20260709 12:23:51.043864 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:51.046595 18021 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 11: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:51.048864 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:50.904929 23287 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:50.975412 23587 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:23:51.013046 23558 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.058840 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.007526 23312 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:51.040863 23457 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:51.051699 23282 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:51.075577 18813 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:51.081781 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 179751us
W20260709 12:23:51.083034 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.079411 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.085016 18021 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:51.083154 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:51.088771 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.054723 22864 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:23:51.067695 23457 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.117663 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:51.082337 23186 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.570s	user 0.015s	sys 0.000s
W20260709 12:23:51.121250 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65725us
I20260709 12:23:51.077790 23282 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.079133 23312 raft_consensus.cc:2764] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:51.121006 23457 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.123776 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:51.122651 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55541.9us
I20260709 12:23:51.082510 23558 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.123061 23441 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:51.145946 23568 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:51.155952 23568 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.156466 23568 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:51.125555 23587 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:51.071853 23590 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:23:51.170487 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:51.148522 23575 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.198s	user 0.003s	sys 0.000s
I20260709 12:23:51.087569 23591 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.103s
I20260709 12:23:51.173422 23558 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:51.176847 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.176923 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.121980 23186 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.191797 18023 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:51.162533 23567 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.006s
W20260709 12:23:51.189446 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59835.2us
I20260709 12:23:51.122174 23282 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:51.194731 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:51.122709 23560 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:51.158857 23597 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:23:51.176877 23511 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:51.178161 23287 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:51.209952 18094 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:23:51.117508 23488 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 15.15, Last appended by leader: 15, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.232725 23457 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.211511 23497 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:51.226802 23248 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.575s	user 0.013s	sys 0.001s
I20260709 12:23:51.228722 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:51.217715 23186 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:51.239382 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:51.260571 18098 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:51.241508 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:51.202698 23558 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:51.199404 23560 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:51.272449 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:51.272764 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:51.259723 18171 raft_consensus.cc:606] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:51.273666 18171 raft_consensus.cc:614] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.263823 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:51.246466 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 241093us
I20260709 12:23:51.272456 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:51.234560 23511 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.279836 18092 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:23:51.283262 18092 pending_rounds.cc:85] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 9
I20260709 12:23:51.279394 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:51.283649 18092 pending_rounds.cc:107] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:23:51.283972 18092 raft_consensus.cc:2889] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.278074 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:51.285081 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:51.264111 23558 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.286880 18098 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:51.259907 23372 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.251557 23587 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:51.296455 23590 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:51.297042 23590 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:51.298132 23602 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.097s	user 0.002s	sys 0.000s
I20260709 12:23:51.289721 23601 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:51.273582 23248 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:51.304524 23248 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:51.321049 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 343208us
I20260709 12:23:51.321893 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:51.286936 23558 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.327917 18023 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:51.360664 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.264390 23497 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:51.325240 23604 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.124s	user 0.002s	sys 0.000s
W20260709 12:23:51.327975 23469 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.002s
I20260709 12:23:51.285523 23511 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.305699 23372 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:51.383524 23457 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:51.383795 18098 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:51.395642 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:51.410971 18806 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:51.393723 23447 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.505s	user 0.012s	sys 0.000s
I20260709 12:23:51.365800 23384 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:51.402769 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.330392 23600 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:51.414572 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.415432 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:51.415804 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93843.9us
I20260709 12:23:51.420466 23611 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:51.416790 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:51.418000 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.423249 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0915842029999112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
W20260709 12:23:51.424680 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0755680989996108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260709 12:23:51.425261 18064 connection.cc:256] RPC call timeout handler was delayed by -2.0447842029998355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
I20260709 12:23:51.391047 23597 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.428814 18021 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:51.435290 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.435096 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:51.430706 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 108924us
I20260709 12:23:51.435880 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.441840 18164 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:51.442746 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.443387 18020 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:51.443059 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.362547 23590 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.401732 23601 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:51.464071 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:51.464466 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:51.465389 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:51.484043 18165 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:51.484573 18167 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.468358 18806 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:51.469686 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:51.486541 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:51.487934 18168 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:23:51.386313 23591 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:23:51.475359 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.474942 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:23:51.461894 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.480932 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:51.481985 18099 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:23:51.418453 23384 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.486534 18016 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:51.486563 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.511564 18169 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:51.511643 18170 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.512296 18023 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:51.490155 18809 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.13-12.13]
I20260709 12:23:51.515102 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:51.515810 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 108937us
I20260709 12:23:51.518780 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:51.520546 23590 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.521870 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:51.538805 18018 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:51.522620 23312 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:51.381665 23560 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.524217 23591 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:51.400064 23587 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.450793 23597 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.564415 23597 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:51.529731 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:51.531643 23611 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:51.537245 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:51.565644 18812 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 9.
W20260709 12:23:51.571087 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.537979 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 110840us
W20260709 12:23:51.575743 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9475842030001331s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260709 12:23:51.576537 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8853535829998691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
W20260709 12:23:51.577062 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.520670 23612 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 12:23:51.538937 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:23:51.540127 18172 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:51.578859 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62298.1us
W20260709 12:23:51.580562 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.581375 23524 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:51.542053 18815 raft_consensus.cc:2410] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:51.582016 23524 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:51.583276 23524 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.589228 23597 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:51.590401 23597 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:51.545744 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:51.577739 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7516860239998095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260709 12:23:51.547232 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:51.548801 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.553833 18171 raft_consensus.cc:606] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:51.547904 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:51.542182 18138 leader_election.cc:336] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.601s (SENT)
I20260709 12:23:51.428637 23231 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:51.421586 23447 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:51.560928 23523 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.038s	user 0.012s	sys 0.032s
W20260709 12:23:51.613555 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65793.4us
I20260709 12:23:51.491206 18810 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.13-13.13]
I20260709 12:23:51.614710 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.613409 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.606015 18171 raft_consensus.cc:614] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.616437 18015 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:23:51.606587 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64474.5us
W20260709 12:23:51.618115 18138 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.628s (SENT)
W20260709 12:23:51.606722 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58055us
I20260709 12:23:51.593569 18023 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 8.
I20260709 12:23:51.595008 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:51.595108 18170 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:23:51.621134 18809 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 8: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:51.621204 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:51.596763 18167 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:51.622807 18165 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:23:51.597286 18808 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:51.598634 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.604823 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7498860239998066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
I20260709 12:23:51.604113 18021 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:51.625074 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3340860240000438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
I20260709 12:23:51.611614 18164 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:23:51.556838 23312 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:51.608752 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 153222us
W20260709 12:23:51.625543 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3115860240000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20260709 12:23:51.543027 23384 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:51.627821 18808 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 12:23:51.631934 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2647860239999318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
I20260709 12:23:51.633399 18024 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:23:51.634370 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0415217949998805s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260709 12:23:51.612913 23523 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.090s	user 0.012s	sys 0.032s
I20260709 12:23:51.614817 23287 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:51.600595 23600 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:51.557374 23463 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:23:51.665518 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:51.559840 22864 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:23:51.635169 23469 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.601060 23599 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.132s
W20260709 12:23:51.696841 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78594.3us
W20260709 12:23:51.722426 18138 leader_election.cc:336] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.711s (SENT)
I20260709 12:23:51.722965 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:51.723129 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 171712us
I20260709 12:23:51.630089 23568 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:51.636828 23603 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:51.670975 23287 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.637358 23615 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:51.715735 23523 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:51.700510 23497 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:51.746704 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 125548us
I20260709 12:23:51.748951 18019 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:51.663405 23457 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 12:23:51.727455 23372 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:51.697569 23564 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:23:51.731248 23623 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.733222 23511 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:51.746970 23523 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.745332 23287 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.734534 23587 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.745085 23615 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:51.771512 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.772212 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 992) took 1369 ms. Trace:
W20260709 12:23:51.757684 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143083us
I20260709 12:23:51.699947 23601 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.725513 23469 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:51.784979 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:51.773654 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:51.793618 18020 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:51.783046 18813 rpcz_store.cc:276] 0709 12:23:50.397228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.397358 (+   130us) service_pool.cc:224] Handling call
0709 12:23:50.426382 (+ 29024us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.076301 (+649919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.766068 (+689767us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54005901d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:51.766385 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1492292480}
I20260709 12:23:51.746763 23324 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:23:51.793696 18021 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:51.746248 23595 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:23:51.771970 23497 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:51.658892 23597 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.779273 23523 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:51.753319 23617 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
W20260709 12:23:51.878435 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114633us
I20260709 12:23:51.744524 23312 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.13, Last appended by leader: 13, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.896697 18018 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
W20260709 12:23:51.897497 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 175099us
W20260709 12:23:51.899277 18138 leader_election.cc:336] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.740s (SENT)
I20260709 12:23:51.898020 18017 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:51.899907 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.777969 23623 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.917035 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:51.946965 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:51.948508 18810 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:51.879235 23567 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:51.907644 23441 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.753s	user 0.016s	sys 0.000s
I20260709 12:23:51.879689 23628 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:51.963631 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.969278 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70042.7us
W20260709 12:23:51.971464 18138 leader_election.cc:336] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.249s (SENT)
I20260709 12:23:51.905213 23447 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:51.972041 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:51.916462 23457 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:51.975384 18064 leader_election.cc:400] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:51.996686 18064 leader_election.cc:403] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:23:51.969127 23441 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:51.978749 23625 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:23:52.027464 23564 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13.
I20260709 12:23:51.979244 23621 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:51.917539 23612 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 12:23:52.029008 23568 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:51.956743 23629 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:51.982826 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:52.030112 23629 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.030630 23629 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:51.983855 23623 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:51.987041 23469 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:52.031567 23558 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.000s
I20260709 12:23:51.905659 23231 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.031867 23622 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:51.997408 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 856) took 1248 ms. Trace:
I20260709 12:23:51.879864 23497 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:51.952865 23567 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:51.996383 23457 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:51.995489 23628 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:51.991976 23615 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.013476 23595 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:52.015439 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:52.017379 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.051530 18095 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:52.052371 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 495936us
I20260709 12:23:52.018343 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 860) took 1199 ms. Trace:
I20260709 12:23:52.053014 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:52.053794 18094 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:52.019228 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:52.019883 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48472.6us
W20260709 12:23:52.020994 23518 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.017s	user 0.016s	sys 0.000s
I20260709 12:23:52.053448 18099 rpcz_store.cc:276] 0709 12:23:50.818977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.819240 (+   263us) service_pool.cc:224] Handling call
0709 12:23:52.018093 (+1198853us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54002977d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:52.018260 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1106317568}
I20260709 12:23:52.037762 23617 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:51.988716 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.785112 23600 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.035876 18093 rpcz_store.cc:276] 0709 12:23:50.748662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.748879 (+   217us) service_pool.cc:224] Handling call
0709 12:23:51.997100 (+1248221us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b54002959d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:51.997314 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1292214656}
I20260709 12:23:52.056686 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:52.057138 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 332967us
W20260709 12:23:52.068841 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53558.7us
W20260709 12:23:52.024372 23282 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.811s	user 0.011s	sys 0.000s
I20260709 12:23:52.040737 23567 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.041657 23587 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.075670 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:52.076604 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:52.077080 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.077167 18168 raft_consensus.cc:606] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.077559 18810 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:23:52.077724 18168 raft_consensus.cc:614] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:52.088356 23372 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:52.092067 23615 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.092155 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:52.094278 23627 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:52.094393 23282 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.096954 23282 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.096835 23523 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.063078 23231 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.101122 23231 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.075476 23324 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:23:52.102708 23324 consensus_queue.cc:1243] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:52.103294 23487 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:52.103682 23231 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.103713 23487 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:52.092031 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:52.105122 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:52.105054 23487 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.105577 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.105926 18093 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:23:52.106262 23384 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.106885 23487 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:52.105159 23590 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.584s	user 0.012s	sys 0.002s
I20260709 12:23:52.108944 23384 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:52.032226 23558 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.111953 23558 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.111574 23457 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.112849 23457 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.066453 23595 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.113451 23441 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:52.116278 23595 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.117178 18096 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:52.072006 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:52.120186 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:52.120918 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:52.120218 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48338.9us
I20260709 12:23:52.122044 18098 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:52.122761 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.124187 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.124671 18166 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:52.125116 18167 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:52.127173 18166 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:52.128623 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:52.143426 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.148474 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.129482 18166 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:52.154644 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.155660 18166 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:52.158210 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.013s	sys 0.011s
W20260709 12:23:52.158581 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.013s	sys 0.011s
W20260709 12:23:52.159006 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 527862us
I20260709 12:23:52.157982 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:52.149415 18164 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 10.
I20260709 12:23:52.068011 23469 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.164616 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:52.159801 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.168329 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.157508 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:52.168329 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:52.172607 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:23:52.173524 18098 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:23:52.177201 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 103237us
I20260709 12:23:52.178469 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:52.180492 18780 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:23:52.183244 18167 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:23:52.149969 23231 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:52.150133 23568 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.187641 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.189112 18164 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:52.156471 23558 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.158914 23441 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.192809 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.216038 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:52.072587 23518 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.222555 18171 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:23:52.237008 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79173.2us
I20260709 12:23:52.223006 18170 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:52.238224 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.195282 18172 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.172057 22864 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.195254 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:52.243636 23523 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:52.244117 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51466.9us
I20260709 12:23:52.245052 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:52.246990 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:52.247916 18017 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:52.193506 23282 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.269737 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.266759 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3318) took 1319 ms. Trace:
I20260709 12:23:52.187151 23590 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:52.220417 23441 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.268350 18016 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:23:52.271149 18814 raft_consensus.cc:2368] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:52.270632 18024 rpcz_store.cc:276] 0709 12:23:50.946995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.947138 (+   143us) service_pool.cc:224] Handling call
0709 12:23:51.170293 (+223155us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:52.266406 (+1096113us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54001d1354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:52.266657 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055911040}
I20260709 12:23:52.221702 23257 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:52.245420 23529 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 12:23:52.236711 23600 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:52.278049 18172 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:52.273188 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:52.282289 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87141.7us
I20260709 12:23:52.195153 23595 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:52.191617 23587 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:52.285480 23441 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.292626 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.272061 23621 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:52.293268 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55145.8us
I20260709 12:23:52.293933 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.294943 18172 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:52.342851 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.233729 23558 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:52.293018 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 176562us
I20260709 12:23:52.287714 23597 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:52.295977 23231 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:23:52.321429 23587 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:52.365485 23524 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 12:23:52.367183 23560 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.366792 23587 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:52.369527 23648 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:23:52.328593 23312 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:52.370805 23638 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:23:52.346328 23511 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:52.374586 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 90127.8us
I20260709 12:23:52.374421 23511 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.338222 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 999) took 1095 ms. Trace:
I20260709 12:23:52.376187 23564 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:52.376058 18814 rpcz_store.cc:276] 0709 12:23:51.242365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.242565 (+   200us) service_pool.cc:224] Handling call
0709 12:23:52.337947 (+1095382us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:52.338107 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2293914880}
I20260709 12:23:52.376924 23564 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:52.377174 23511 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.378213 23564 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.381986 23558 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.382539 23497 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.280895 23384 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:52.283609 23622 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:52.284809 23257 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.295765 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:52.236536 23287 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:52.352169 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 79092.6us
I20260709 12:23:52.352301 18810 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 15. Preceding OpId from leader: term: 17 index: 16. (index mismatch)
I20260709 12:23:52.358101 18812 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:52.347839 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.352521 23646 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:52.352897 23600 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.355247 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:52.364647 18815 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:52.323221 18024 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:52.358821 23595 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:52.291452 23282 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.375692 18064 leader_election.cc:400] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:52.398617 18810 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:52.394546 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:52.413282 18064 leader_election.cc:403] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:23:52.413930 18021 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.394068 18809 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:52.396018 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:52.391110 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95466us
W20260709 12:23:52.398641 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.403128 18166 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:52.414354 18063 leader_election.cc:400] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:52.421236 18063 leader_election.cc:403] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260709 12:23:52.428915 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73764.3us
I20260709 12:23:52.389868 23257 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:52.382560 23518 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:52.443756 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.405761 23497 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:52.448992 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.443756 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.438537 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:52.456293 18808 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:52.457019 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49398.5us
I20260709 12:23:52.392907 23287 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.391019 23591 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:52.470774 23575 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:23:52.442634 23597 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.427577 23599 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.480059 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.500765 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.489312 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.434793 23622 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:52.477396 23628 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.451375 23312 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:52.508908 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:23:52.509734 18168 raft_consensus.cc:606] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:23:52.507141 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:52.492559 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43711.3us
I20260709 12:23:52.407945 23629 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.478065 23611 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.499881 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:52.389034 23567 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.510285 18168 raft_consensus.cc:614] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:52.513415 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.520740 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.513440 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.528308 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:52.485986 23437 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:52.466313 23645 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:23:52.491791 23600 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:52.488211 23497 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:52.529497 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.471987 23463 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:23:52.532418 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.550518 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.543054 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:52.502559 23646 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:52.490126 23591 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:52.553300 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.554795 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.555742 18173 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:52.510040 23488 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 9. Current term is 10.
I20260709 12:23:52.451381 23621 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:23:52.522126 23590 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:52.563232 18097 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:52.565261 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:52.557313 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.094s (SENT) [suppressed 30 similar messages]
I20260709 12:23:52.545893 23601 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:52.561939 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.586937 18164 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 12.
I20260709 12:23:52.538985 23312 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:52.530947 23524 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.545511 23617 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.517403 22864 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:52.579531 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51336.4us
I20260709 12:23:52.579891 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.580549 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.567500 23628 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:52.547646 23600 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:52.593385 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 217849us
I20260709 12:23:52.506942 23372 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:23:52.560695 23287 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.621907 18164 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
W20260709 12:23:52.618129 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114381us
I20260709 12:23:52.578533 23611 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:52.604737 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47482.1us
I20260709 12:23:52.612275 18168 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:52.613306 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.613672 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.543895 23615 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:52.509327 23597 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.518314 23599 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:52.628959 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.590045 23601 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:52.636363 18164 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:52.617017 23590 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.640569 18173 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.627517 23587 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:52.621362 23600 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:52.683058 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56164.4us
W20260709 12:23:52.634677 23487 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.521s	user 0.013s	sys 0.002s
I20260709 12:23:52.686362 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:52.689795 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159638us
I20260709 12:23:52.702675 18164 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:52.638780 23623 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.580734 23595 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.710168 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.635417 23628 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:52.639559 23601 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:52.679860 23635 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.716084 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:52.703274 23469 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.001s
I20260709 12:23:52.682543 23615 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:52.704460 23641 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.346s	user 0.000s	sys 0.002s
I20260709 12:23:52.714890 23623 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:52.630362 23518 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:23:52.722158 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:52.729254 18138 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.766s (SENT)
I20260709 12:23:52.750195 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.734195 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.686337 23590 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.737383 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.738179 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:52.619329 23497 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:52.725190 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 184014us
I20260709 12:23:52.747092 18173 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:52.639814 23488 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
I20260709 12:23:52.751875 23658 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.183s
I20260709 12:23:52.752785 23282 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:52.754601 23649 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
W20260709 12:23:52.756362 23655 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:23:52.756820 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:52.757205 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:52.757745 23652 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:23:52.758543 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.760066 23626 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:52.761437 23524 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:52.734603 23632 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.762246 23620 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.020s
W20260709 12:23:52.753742 23654 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:23:52.770542 23587 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:52.832024 23441 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.706287 23643 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.250s
I20260709 12:23:52.740465 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:52.823539 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1192) took 1036 ms. Trace:
I20260709 12:23:52.737262 23469 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:52.868086 23657 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 12:23:52.869427 23665 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:23:52.638985 23622 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:52.870579 23568 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.635s	user 0.009s	sys 0.005s
I20260709 12:23:52.871142 23568 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:52.874024 23660 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:52.875020 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:52.875298 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153326us
I20260709 12:23:52.879279 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.879948 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:52.857578 23632 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.882972 23632 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.883926 23287 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:52.884526 23324 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:52.884999 23324 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:52.839910 23567 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.749706 23623 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.843261 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:52.907419 18024 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.916195 18171 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:52.723399 23312 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.726493 23659 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.124s
I20260709 12:23:52.848786 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:52.849125 23656 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:23:52.840585 23463 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:52.854727 23653 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:23:52.855232 23664 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:52.776413 23591 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.938164 23664 consensus_queue.cc:1243] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:52.864202 18809 rpcz_store.cc:276] 0709 12:23:51.787217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.787353 (+   136us) service_pool.cc:224] Handling call
0709 12:23:51.870218 (+ 82865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.414815 (+544597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.823159 (+408344us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b540070f8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:52.823432 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1204166400}
I20260709 12:23:52.866024 23643 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:52.939715 23643 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:52.940843 23643 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.866808 23599 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:52.750837 23615 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.944802 23615 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:52.867100 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2466) took 1338 ms. Trace:
I20260709 12:23:52.945648 18165 rpcz_store.cc:276] 0709 12:23:51.529014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.623620 (+ 94606us) service_pool.cc:224] Handling call
0709 12:23:51.640136 (+ 16516us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:51.785936 (+145800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:51.785971 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:51.884534 (+ 98563us) log.cc:844] Serialized 52 byte log entry
0709 12:23:52.045617 (+161083us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:52.066095 (+ 20478us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.066489 (+   394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.066611 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:52.866806 (+800195us) raft_consensus.cc:1685] finished
0709 12:23:52.866835 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.867023 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":265,"prepare.queue_time_us":25300,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":55,"raft.queue_time_us":93931,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":33334,"thread_start_us":163282,"threads_started":2,"wal-append.queue_time_us":350423}
I20260709 12:23:52.948318 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.948527 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:52.869246 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.831296 18021 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:52.861846 23628 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.949043 18165 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:23:52.952919 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:52.953945 23599 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:52.954540 23599 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:52.956254 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:52.957764 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.949468 18167 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:52.834854 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97177.7us
I20260709 12:23:52.901633 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.886266 23324 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.738258 23635 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.965118 18168 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:23:52.782670 23457 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.669s	user 0.016s	sys 0.000s
I20260709 12:23:52.838917 23601 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:52.911362 23629 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:52.900023 23660 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:52.935379 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.930208 23463 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.950346 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:52.950762 18170 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.909588 23564 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:52.963574 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:53.003840 18138 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.771s (SENT)
I20260709 12:23:53.014233 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:53.019124 18164 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:53.019140 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:52.955878 23591 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.912158 23487 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:23:52.951352 23643 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:53.051016 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 149598us
W20260709 12:23:53.019070 23560 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.651s	user 0.015s	sys 0.000s
I20260709 12:23:53.022009 23287 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:52.949693 23618 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:53.034258 23567 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:53.047937 23441 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:53.049717 23257 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.579s	user 0.013s	sys 0.000s
I20260709 12:23:53.033900 23564 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.052258 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.057545 18024 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:53.034623 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:53.018654 23282 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:23:53.068535 18098 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:53.016611 23597 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.070544 18096 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:53.018318 23463 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:53.073045 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:53.112715 18092 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.075053 18170 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:53.079108 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:53.038239 23324 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:53.083230 18091 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:53.078611 23618 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:53.087970 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:53.122479 18093 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:53.088901 23529 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.124105 23529 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.018208 23620 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 12:23:53.127672 23663 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.128585 23663 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.130575 23618 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.132468 23618 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.127746 23529 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.138701 23590 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.050925 23629 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.139659 23629 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:52.955596 23599 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.142562 23660 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:53.096628 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63314.9us
I20260709 12:23:53.084142 23672 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.113759 18097 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:53.080289 23646 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:53.120874 23457 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:53.136569 23287 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.137953 23560 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.091372 18099 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20260709 12:23:53.064010 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.089866 23666 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:53.113844 23568 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:53.151779 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 99640.4us
I20260709 12:23:53.148435 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:53.150342 23624 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:53.115908 23626 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:23:53.165033 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:53.105268 23257 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.166030 23601 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.168715 23560 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.200949 23560 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.171207 23634 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:53.174916 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 346415us
I20260709 12:23:53.178053 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:53.206382 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.182452 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:53.187176 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:53.208439 23611 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.571s	user 0.013s	sys 0.000s
I20260709 12:23:53.208956 23611 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:53.188344 23642 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:53.134054 23441 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:53.173805 23600 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.214651 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:53.184295 23620 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:53.220798 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:53.223464 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:53.225653 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:53.214100 23642 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:53.238533 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56196.8us
I20260709 12:23:53.237769 18097 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:53.244187 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:53.181643 23568 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.262555 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:53.161019 23590 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:23:53.254953 18021 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:23:53.254822 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 144040us
I20260709 12:23:53.282186 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:53.191385 23666 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:53.184949 23646 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:53.257259 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:53.292040 18024 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:53.261022 18092 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:53.280086 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:53.293567 18018 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:53.279453 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:53.294651 18015 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:53.295924 18169 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:53.263027 18020 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:53.300153 18021 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:53.284971 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:53.263638 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:53.290385 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.301666 18019 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:53.301564 18172 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:53.306384 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:53.307961 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 130115us
W20260709 12:23:53.315189 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 311392us
I20260709 12:23:53.318488 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:53.319312 18806 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:53.319734 18093 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:23:53.264154 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:53.323299 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:53.323817 18022 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:53.323299 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.252004 23618 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:53.158476 23672 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.262053 23629 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.279009 23568 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.298488 23564 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.279006 23635 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.191994 23624 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.285840 23487 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:53.250140 23597 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 12:23:53.337714 18809 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:53.279498 23620 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:53.249091 23610 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 12:23:53.332329 18171 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: [12.10-13.11]
W20260709 12:23:53.360031 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50911.7us
I20260709 12:23:53.287541 23282 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:23:53.251468 23441 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.330382 23618 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.363047 18813 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:53.373206 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.317544 23646 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.398522 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.400826 23678 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:53.402536 23620 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.411140 23624 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.414341 23365 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:53.415093 23365 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:53.416359 23365 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.418017 23666 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.423516 23672 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.436029 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:53.436862 18098 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:53.371999 23618 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.434403 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:53.444475 18096 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 9: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:53.377228 23685 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.458056 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:53.443126 23623 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.525s	user 0.017s	sys 0.000s
W20260709 12:23:53.466094 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66868.5us
I20260709 12:23:53.467191 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:53.457018 23661 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.770s	user 0.000s	sys 0.006s
I20260709 12:23:53.438038 23678 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:53.471876 18169 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:23:53.450238 23590 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.484817 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.448588 23600 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.468343 23687 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.476611 23610 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:53.482699 23487 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:53.487931 23365 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:53.540916 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56222.3us
I20260709 12:23:53.487483 23257 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.397127 23642 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.473613 23623 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.500404 23564 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.549387 18096 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:53.545863 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:53.545522 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:53.553200 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:53.517773 23659 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:53.516057 23635 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:53.488624 23463 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.507020 23610 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:53.469962 23685 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.486769 23590 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.561750 23257 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.508104 23687 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.579959 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:53.570070 23659 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:53.568876 23666 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.571499 23642 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:53.591133 18171 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:23:53.559633 23564 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.569113 23678 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:53.608245 23679 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:23:53.566762 23635 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.594245 23487 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:53.595571 23441 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:53.608839 23632 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.725s	user 0.022s	sys 0.005s
I20260709 12:23:53.601624 23469 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:53.597632 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:53.623988 23282 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.624475 23384 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:53.625008 23384 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:53.624732 23282 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.626194 23384 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.627463 23384 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.627700 23282 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.584830 23257 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.635473 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:53.600123 23666 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.641609 23487 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:53.663918 23635 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:53.667160 23700 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:23:53.667343 23689 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 12:23:53.668139 23695 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:23:53.670579 23564 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.671871 23463 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.615613 23601 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:53.675326 23324 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.538s	user 0.013s	sys 0.006s
I20260709 12:23:53.622483 23632 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:53.715663 23694 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:23:53.717522 23696 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:23:53.695636 23487 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:53.762579 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.700646 23623 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.665836 23659 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.773460 18167 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:53.774292 23600 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.778501 23600 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.781198 23600 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.774045 23312 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.782475 23312 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.784456 23687 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.785095 23312 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.768355 23591 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:53.787331 23591 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.788044 23632 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.795522 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.796522 18172 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:53.769928 23469 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.798676 23469 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:53.803498 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3776350840003033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260709 12:23:53.804057 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2462350840000909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1653s for a response
W20260709 12:23:53.804430 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2399350840000807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20260709 12:23:53.804703 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2003350840000166s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20260709 12:23:53.804956 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1022350840003128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
I20260709 12:23:53.805580 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.797673 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.725927 23601 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.806779 23497 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.807216 18172 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:53.814286 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.766213 23590 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.823889 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:53.824501 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:53.825188 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:53.825613 18022 raft_consensus.cc:2368] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:53.826079 18024 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:53.826587 18172 raft_consensus.cc:2376] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13. Re-sending same reply.
I20260709 12:23:53.775091 23324 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.807049 23564 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.818746 23660 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:53.824359 23469 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.832635 23324 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.823300 23497 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.838196 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.838027 23487 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.839682 23601 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.840085 23591 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:53.840642 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:53.842157 23568 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:53.844969 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:53.866310 18164 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:53.855387 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:53.843374 23699 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:23:53.829764 23685 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.860422 23590 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:53.876680 18164 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:23:53.831727 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:53.847467 23469 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:53.870254 23568 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.873473 23666 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:53.874049 23591 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:53.874711 18171 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:53.878840 18172 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:53.880052 18164 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:23:53.848249 23564 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:53.889314 18164 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:23:53.895051 18164 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12. Ops: [10.6-11.7]
W20260709 12:23:53.969591 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75317.9us
I20260709 12:23:53.971001 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.973201 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.976217 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:53.884625 23324 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:53.997128 18171 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 15. Preceding OpId from leader: term: 17 index: 16. (index mismatch)
I20260709 12:23:53.994019 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2582) took 1304 ms. Trace:
I20260709 12:23:54.004313 18173 rpcz_store.cc:276] 0709 12:23:52.689327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.874603 (+185276us) service_pool.cc:224] Handling call
0709 12:23:53.987351 (+1112748us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54000424d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:53.993894 (+  6543us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779049728}
I20260709 12:23:53.998036 23601 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 12:23:54.047070 18164 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:53.986129 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:53.990460 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.979533 23590 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:53.987360 23591 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:53.981088 23621 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.065570 23601 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.071705 23497 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:54.085357 23703 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:54.161893 23518 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.162416 23564 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:54.057567 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78705.9us
I20260709 12:23:54.171540 18138 leader_election.cc:400] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:54.176705 18138 leader_election.cc:403] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:54.162876 23518 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.121675 23703 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:54.172220 23618 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.004s
W20260709 12:23:54.183002 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 795) took 1651 ms (client timeout 1755 ms). Trace:
W20260709 12:23:54.183544 18169 rpcz_store.cc:269] 0709 12:23:52.531971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.532119 (+   148us) service_pool.cc:224] Handling call
0709 12:23:54.182767 (+1650648us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:54.182923 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1712133376}
I20260709 12:23:54.182567 23618 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.187496 18016 raft_consensus.cc:2368] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13.
I20260709 12:23:54.188421 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:54.190439 18092 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:54.197110 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:54.153522 18172 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:23:54.054179 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 239977us
I20260709 12:23:53.583928 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:54.200479 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:54.203042 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.181519 23518 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:54.204675 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.241972 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.245239 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.194146 23621 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.181005 23703 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.248833 23312 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.248989 23441 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.253762 23600 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.274073 23600 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.275112 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.236032 23558 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.040s
W20260709 12:23:54.260294 23642 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.659s	user 0.015s	sys 0.000s
I20260709 12:23:54.276623 23600 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.276958 23642 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.277343 23600 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:54.277501 23642 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.254508 23623 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.254526 23457 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.249171 23705 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.190295 23618 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:54.194231 23663 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:54.264242 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74856.2us
I20260709 12:23:54.254020 23384 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.278021 23659 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:54.277791 23600 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.278505 23441 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.278612 23312 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.203653 23676 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:54.218358 23658 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.281610 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1200) took 1573 ms. Trace:
I20260709 12:23:54.305387 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.306857 23312 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:54.305612 18808 rpcz_store.cc:276] 0709 12:23:52.708028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.708316 (+   288us) service_pool.cc:224] Handling call
0709 12:23:52.708829 (+   513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:52.718064 (+  9235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.718125 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:52.728200 (+ 10075us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.884459 (+1156259us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b180026e7e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:54.198761 (+314302us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:54.198812 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:54.198879 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:54.258238 (+ 59359us) raft_consensus.cc:1685] finished
0709 12:23:54.258277 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:54.281486 (+ 23209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3268,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":9319,"spinlock_wait_cycles":12082432,"thread_start_us":314018,"threads_started":1,"wal-append.queue_time_us":1521906}
I20260709 12:23:54.307276 18169 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:54.200250 23590 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.308241 23590 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.309927 23600 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.310972 23590 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:54.248346 23615 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:54.287698 23711 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.288354 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:54.313863 23384 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.316289 23441 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.317216 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.318104 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:54.319312 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.320431 18809 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:23:54.323005 18164 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:54.296226 23623 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.202363 23463 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:54.282612 23591 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:54.301056 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.290354 23642 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:54.346335 23615 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:54.347726 23615 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.336267 23463 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.350414 23463 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:54.352409 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53263us
I20260709 12:23:54.293102 23610 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.355834 18165 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:23:54.289314 23692 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.708s	user 0.006s	sys 0.009s
I20260709 12:23:54.332463 18809 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:54.360009 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:54.289090 23487 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:54.336802 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:54.338537 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.363687 18164 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 14
W20260709 12:23:54.364018 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 163633us
I20260709 12:23:54.365072 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:54.358992 18172 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:54.360852 18808 raft_consensus.cc:2368] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:54.367543 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.361255 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.368430 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.362026 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.369143 18814 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:54.316318 23384 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.373449 18091 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:54.362154 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.374603 18809 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:54.258144 23564 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:23:54.377422 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.380292 18093 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:54.376205 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.391280 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.368443 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:23:54.392639 18099 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:54.397126 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.022s (SENT)
I20260709 12:23:54.397948 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.400174 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.401010 18809 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:23:54.402345 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:54.363157 23658 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.403224 18022 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.365531 23457 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:54.405718 18021 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:54.313637 23711 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.362625 23714 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.422677 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.412300 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.368059 23469 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.419893 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.420389 18092 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:23:54.374043 23518 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:54.421216 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:54.426505 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.421806 18019 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:54.422441 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.427390 18020 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:54.427564 18169 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:54.417666 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:54.429296 18024 raft_consensus.cc:2368] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:54.408082 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.430593 18097 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 13 because replica has last-logged OpId of term: 12 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 12.
I20260709 12:23:54.394467 23610 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.431500 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:54.431639 23623 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.367810 23705 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:23:54.432853 23705 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:54.433264 23705 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:54.434245 23705 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.435653 23705 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.367509 23659 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:23:54.438402 23487 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:54.419106 23658 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.438568 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.438961 23487 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:54.440056 23487 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.440780 23658 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:54.399482 23384 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.442263 23384 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.443423 23646 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:54.444319 23711 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:54.444468 23384 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.417511 23564 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:54.445043 23564 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:23:54.446372 23688 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.834s	user 0.002s	sys 0.014s
I20260709 12:23:54.446081 23564 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.447474 23564 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:54.449024 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2492) took 1158 ms. Trace:
I20260709 12:23:54.427641 23518 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.450335 23518 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:54.453773 18019 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:54.455304 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.455322 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:54.455538 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.403971 23685 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:54.458220 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:54.458865 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.458704 23676 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.459421 23676 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.460727 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.461458 18097 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.463132 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.464095 18093 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:54.464133 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1048) took 1181 ms. Trace:
I20260709 12:23:54.464704 18813 rpcz_store.cc:276] 0709 12:23:53.283229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.283385 (+   156us) service_pool.cc:224] Handling call
0709 12:23:53.317441 (+ 34056us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:53.368515 (+ 51074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:54.463797 (+1095282us) spinlock_profiling.cc:243] Waited 45.2 ms on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:54.464049 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103947136}
W20260709 12:23:54.466928 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69883.5us
I20260709 12:23:54.469101 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.469853 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:54.472476 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:54.473513 23711 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.474710 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:54.475169 23488 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.475792 23488 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.467963 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.478896 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.479877 18093 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:54.479733 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.480635 18090 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:23:54.449465 18166 rpcz_store.cc:276] 0709 12:23:53.290541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.290694 (+   153us) service_pool.cc:224] Handling call
0709 12:23:53.291076 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:53.304540 (+ 13464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:53.304593 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:53.327751 (+ 23158us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.490546 (+162795us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:53.490597 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.490688 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:54.445598 (+954910us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:54.448672 (+  3074us) raft_consensus.cc:1685] finished
0709 12:23:54.448698 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:54.448948 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24454,"raft.queue_time_us":27320,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":5141,"spinlock_wait_cycles":4994560,"thread_start_us":152845,"threads_started":1,"wal-append.queue_time_us":245044}
I20260709 12:23:54.441790 18169 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:54.485384 18021 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:23:54.441524 23487 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.486495 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.506755 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.490682 18022 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:54.507863 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.507047 18021 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:54.507561 18096 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:54.515764 18164 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:54.516379 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.517367 18165 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:54.518527 18018 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:54.413160 23558 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.458192 23685 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.520596 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.520596 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:54.521749 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.522434 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:54.524395 18171 raft_consensus.cc:2376] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10. Re-sending same reply.
I20260709 12:23:54.526541 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.527617 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:54.529210 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 162197us
I20260709 12:23:54.530232 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.531816 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.533162 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.529619 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:54.536891 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.538913 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.541952 18022 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14.
I20260709 12:23:54.544943 18137 leader_election.cc:400] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
I20260709 12:23:54.545341 18137 leader_election.cc:403] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:23:54.547703 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:54.538403 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.728s (SENT)
I20260709 12:23:54.554625 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:54.557512 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.740s (SENT)
I20260709 12:23:54.484879 23711 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:54.561470 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.000s (SENT)
I20260709 12:23:54.507946 23324 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:54.586357 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45945.7us
I20260709 12:23:54.573566 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.575265 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:54.566507 18097 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:54.590890 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:54.591342 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.529065 23590 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:54.600754 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84498.5us
I20260709 12:23:54.486980 23469 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.538738 23713 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:54.608592 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:54.611217 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.526962 23591 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:23:54.518005 23597 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 12:23:54.484655 23714 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:54.667526 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.547585 23384 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:23:54.604375 23590 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.539855 23558 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.668473 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.688179 18024 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:54.679623 18094 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18. Re-sending same reply.
I20260709 12:23:54.604103 23488 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.619163 23623 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 9. Current term is 10.
W20260709 12:23:54.690635 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 121599us
I20260709 12:23:54.624768 23713 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.599514 23324 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.610364 23312 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 12:23:54.697980 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.729545 18023 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:54.729585 18097 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:54.706506 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.696944 18093 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:54.708256 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 117400us
I20260709 12:23:54.731246 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:54.617465 18015 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:54.598307 23711 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.728660 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.732661 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:54.733704 18024 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:23:54.677974 23497 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:54.689527 23590 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:54.746201 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.682152 23716 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.753451 18090 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:54.753676 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.751725 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.752753 18093 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
I20260709 12:23:54.757654 18016 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:54.712154 23713 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:54.711383 23718 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.727264 23621 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:23:54.765305 23621 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:54.741405 23663 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.768778 23663 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.761795 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:54.770370 18024 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:54.764907 23718 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.761372 23716 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.775844 23324 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.779175 23469 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.746057 23597 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.781034 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.801332 18091 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:23:54.801558 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 240110us
I20260709 12:23:54.783496 23621 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:54.809667 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.011s (SENT)
I20260709 12:23:54.816414 23718 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:23:54.817440 23558 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:54.793421 23728 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:23:54.818655 23685 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.833631 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.834789 18164 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:23:54.791513 18167 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:54.712245 23441 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:23:54.793675 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.855805 18097 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:54.794449 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.749651 23623 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:54.871372 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57831.1us
I20260709 12:23:54.795662 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.874308 18093 raft_consensus.cc:2376] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9. Re-sending same reply.
I20260709 12:23:54.814328 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.876858 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.877795 18171 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:54.862493 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.751884 23497 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:54.814023 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 212028us
I20260709 12:23:54.883527 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.884815 18093 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:54.830020 23621 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:54.890110 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.893985 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.859180 23591 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.863041 18164 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:54.888388 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:54.902966 18164 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:54.905488 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:54.868563 23623 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.878962 18166 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:54.911427 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:54.912287 18166 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.814633 23723 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.891140 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:54.897961 18169 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:54.878118 23731 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:54.899338 23558 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:54.899598 23324 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.900295 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.875980 18091 raft_consensus.cc:2376] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10. Re-sending same reply.
I20260709 12:23:54.889420 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.868026 23587 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:54.899650 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:54.930929 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:54.900570 23437 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.908684 23312 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.944247 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:54.974980 18809 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 14 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:54.895193 23615 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:54.894672 23718 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.991501 23718 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.992259 23497 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.911633 23591 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.993067 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.961117 18097 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:54.994705 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.995497 23663 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.995983 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.996613 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:54.961971 18093 raft_consensus.cc:2376] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12. Re-sending same reply.
I20260709 12:23:54.998788 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.999406 18093 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:55.000303 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.000917 18093 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:54.962409 18098 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.005245 23437 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.007925 23437 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:54.960673 18166 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:23:54.970206 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79931.9us
I20260709 12:23:55.010166 18813 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:54.929301 23623 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:55.009433 23587 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.021196 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.022854 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.023530 18091 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.909998 23610 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:55.026234 23610 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.026739 23610 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:55.027662 18022 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:55.028116 18022 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:55.029022 18022 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.959566 23518 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.982885 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:54.983880 23600 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.000s
I20260709 12:23:54.984835 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:55.066713 18169 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:54.946060 23719 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:54.970316 23731 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.974112 23324 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:54.987399 23558 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:54.987607 23463 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.637s	user 0.014s	sys 0.001s
I20260709 12:23:54.989286 23312 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:54.990113 23615 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:54.990471 23597 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:23:54.990762 23705 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.004s
I20260709 12:23:54.994839 23591 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:54.968101 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62703.4us
I20260709 12:23:55.011188 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:54.963007 18171 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:23:55.001770 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.012389 18095 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:23:54.997552 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.018705 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.018916 18165 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:55.018692 23714 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.030977 23716 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.031978 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.138695 18094 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:23:55.139936 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.141477 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:54.903226 23685 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:23:55.036270 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:55.142612 18166 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:55.143005 23685 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:55.143452 23685 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:55.015084 18778 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.702s (SENT)
I20260709 12:23:55.010447 23718 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:55.037808 23698 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.353s	user 0.000s	sys 0.018s
W20260709 12:23:55.047107 23676 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.003s
I20260709 12:23:55.047981 23257 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:55.042356 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2488) took 1753 ms. Trace:
I20260709 12:23:55.152642 18168 rpcz_store.cc:276] 0709 12:23:53.288829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.305960 (+ 17131us) service_pool.cc:224] Handling call
0709 12:23:55.042126 (+1736166us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54001b5654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:55.042272 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":548127104}
I20260709 12:23:55.043648 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.033771 18022 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.155260 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.039886 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:55.157016 18090 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:55.103469 23558 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.062701 18814 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:55.068279 18098 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:55.158900 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.159652 18098 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:55.106448 18091 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:55.108348 18811 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:55.133275 23610 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.123291 18812 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:55.126116 23623 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.127900 23324 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.086829 23600 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.086432 23730 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.085973 23713 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.107733 23658 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:55.110163 23463 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:55.132349 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121237us
I20260709 12:23:55.174424 18166 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:55.194654 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:55.133689 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:55.134366 23487 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:55.124604 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.150357 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.196771 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:55.045787 23590 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.108227 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:55.106801 18813 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:23:55.163066 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 147993us
I20260709 12:23:55.163733 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:55.199877 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.771s (SENT)
I20260709 12:23:55.175485 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.201033 18091 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:23:55.192497 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.200372 18090 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:23:55.161237 23558 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:55.202216 18098 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:55.203184 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.203747 18098 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:55.203966 18097 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:55.205482 18097 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:55.206964 18097 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:23:55.207177 23740 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:23:55.208346 18097 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:55.197484 18022 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:23:55.209233 18022 pending_rounds.cc:85] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 10
I20260709 12:23:55.209522 18022 pending_rounds.cc:107] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:23:55.199586 18169 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:55.209756 18022 raft_consensus.cc:2889] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:55.211246 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 86803.9us
I20260709 12:23:55.211555 18173 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:55.202164 18095 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:55.145926 23597 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.215147 18168 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:55.216090 18166 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:55.212086 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.223205 18098 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:23:55.097362 23719 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.144469 23685 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.231596 23749 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:55.258575 23749 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:55.259665 23749 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.218274 23703 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:55.262238 23703 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:55.263217 23703 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.264987 23590 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:55.240473 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.267721 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.268100 23463 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.269297 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:55.243696 23620 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 12:23:55.243885 23597 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:55.244764 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.275714 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.276714 18098 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:55.222762 23658 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.277771 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.277935 23746 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:55.278985 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:55.279846 23748 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.280669 23748 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.281795 23685 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:55.250591 23734 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.305s	user 0.001s	sys 0.000s
I20260709 12:23:55.251992 23711 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:55.256891 18091 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:23:55.270213 23714 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:55.270747 23591 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:55.285279 23733 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.298s	user 0.001s	sys 0.001s
I20260709 12:23:55.285279 23714 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:55.286819 23714 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:55.158172 23676 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.290311 23663 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.291257 23663 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.293920 23560 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:55.294049 23620 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:55.294727 23663 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:55.296308 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101751us
I20260709 12:23:55.297484 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:55.229046 23564 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.781s	user 0.013s	sys 0.004s
I20260709 12:23:55.303514 18093 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:23:55.307287 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.026s (SENT)
I20260709 12:23:55.285552 23591 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.308146 23591 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.222632 23324 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:55.278996 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.232141 18095 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:23:55.318557 18095 pending_rounds.cc:85] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 6
I20260709 12:23:55.318826 18095 pending_rounds.cc:107] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:23:55.272167 23518 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.279784 23658 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:55.283304 23748 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.247818 23719 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.319093 18095 raft_consensus.cc:2889] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:55.330727 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115674us
W20260709 12:23:55.319271 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.644s (SENT)
I20260709 12:23:55.319815 18091 pending_rounds.cc:85] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:23:55.335083 18091 pending_rounds.cc:107] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:23:55.335521 18091 raft_consensus.cc:2889] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:55.273777 23731 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.279412 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.281385 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:55.321031 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2494) took 2029 ms. Trace:
I20260709 12:23:55.199239 23621 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:55.343040 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63713us
W20260709 12:23:55.345001 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63713.7us
I20260709 12:23:55.323333 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.346767 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:55.324621 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.331763 23720 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:23:55.342820 18170 rpcz_store.cc:276] 0709 12:23:53.291200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.425778 (+134578us) service_pool.cc:224] Handling call
0709 12:23:53.426399 (+   621us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:53.426970 (+   571us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:53.427018 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:53.445372 (+ 18354us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.559615 (+114243us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:53.559665 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.559771 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.037023 (+1477252us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:55.316296 (+279273us) raft_consensus.cc:1685] finished
0709 12:23:55.316323 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.320946 (+  4623us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11592,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10485,"spinlock_wait_cycles":129163008,"thread_start_us":82224,"threads_started":1,"wal-append.queue_time_us":146489}
W20260709 12:23:55.325553 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.325937 23463 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.363795 18019 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:55.365182 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.326012 23324 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:55.195649 23730 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.338213 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.322633 23487 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:55.321898 23678 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:55.319072 23717 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:55.333729 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.355891 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.380941 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.360803 18171 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: [12.10-13.11]
I20260709 12:23:55.361939 23601 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:55.372328 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:55.384305 23437 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:55.385149 23457 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:55.351207 23725 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:23:55.362962 18017 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:55.394428 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.362973 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.326534 23676 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.368839 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:55.414022 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45315.5us
I20260709 12:23:55.370404 23324 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:55.374091 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54869.3us
I20260709 12:23:55.363517 23664 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:55.350811 23642 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:55.397465 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63822.7us
I20260709 12:23:55.401609 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.434746 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.434873 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.436194 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.414963 18164 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:55.415112 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.437258 18809 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:55.385459 23437 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:55.422420 18813 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:55.354068 23518 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:23:55.376288 23731 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:55.362092 23658 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:23:55.364931 23590 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:55.373863 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.438982 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.412106 23469 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:55.445966 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.476838 18023 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:55.448558 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.423067 23718 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:55.423230 23727 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:55.496644 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.482903 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.432678 23642 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.422328 23643 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:55.490410 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:55.412035 23705 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:23:55.490599 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.429400 23324 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:55.370178 23463 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.491758 18810 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:23:55.498149 18812 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:23:55.499084 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 102376us
W20260709 12:23:55.498987 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 83978us
I20260709 12:23:55.394476 23678 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.411995 23717 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.422869 23659 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 12:23:55.483907 23469 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.515868 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.533052 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:55.473708 23620 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:55.474670 23590 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.535802 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:23:55.540418 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.745s (SENT)
I20260709 12:23:55.540807 18811 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:55.550179 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.550508 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.446622 23560 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:55.411315 23487 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.496941 23727 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.511137 23642 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:55.529165 23718 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:55.562295 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:55.552259 23469 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:55.559793 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.530431 23748 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:55.556257 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.568961 18813 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:55.571509 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.572510 18813 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:55.542290 23711 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.505066 23600 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.517028 23610 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.570133 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.578922 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.581719 18064 leader_election.cc:400] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 11. Current term is 12.
I20260709 12:23:55.583220 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.584583 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:55.566028 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 90430.3us
I20260709 12:23:55.554184 23590 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.554489 23620 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.489499 23497 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 12:23:55.495395 23686 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:23:55.560003 23691 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:55.587173 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.604132 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.608345 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.517230 23720 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 12:23:55.563004 23727 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:55.600226 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.622942 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:55.573752 23742 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.541s	user 0.002s	sys 0.002s
I20260709 12:23:55.602073 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.541957 23716 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:23:55.565970 23643 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:55.608467 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.610961 18138 leader_election.cc:400] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14.
I20260709 12:23:55.631054 18138 leader_election.cc:403] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:55.564841 23560 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:55.568193 23748 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.576867 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:55.636726 18809 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:23:55.561563 23597 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:23:55.628304 18091 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:55.595112 23620 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:55.587129 23457 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:55.631366 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.631801 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.634001 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.574127 23663 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:23:55.659264 18097 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:55.653900 18812 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:23:55.652190 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2637) took 1138 ms. Trace:
I20260709 12:23:55.656778 23597 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:55.625903 23610 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.617964 23600 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:55.654878 23560 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.660126 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.625324 23746 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:55.666707 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:55.596951 23711 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.667534 18092 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:55.667774 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.659932 18096 rpcz_store.cc:276] 0709 12:23:54.514076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.516458 (+  2382us) service_pool.cc:224] Handling call
0709 12:23:54.516824 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.517075 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.517102 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.275886 (+758784us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:23:55.276413 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.491162 (+214749us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:55.491216 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.491302 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.651788 (+160486us) raft_consensus.cc:1685] finished
0709 12:23:55.651813 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.652102 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10360064,"thread_start_us":214299,"threads_started":1,"wal-append.queue_time_us":282570}
I20260709 12:23:55.670354 18091 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:55.643632 23748 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:55.646435 23659 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:55.670783 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:55.674803 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:55.681777 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 130248us
I20260709 12:23:55.684245 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.700727 18092 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:55.701658 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:55.703037 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.704320 18018 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:55.688560 18091 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:55.705845 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.708391 18022 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:23:55.711870 18091 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:55.647486 23621 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.647982 23643 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:23:55.734006 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58910.4us
I20260709 12:23:55.737886 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:55.713171 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 164204us
W20260709 12:23:55.742717 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 131996us
I20260709 12:23:55.743824 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:55.745409 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43902.2us
I20260709 12:23:55.595800 23437 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.664350 23597 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:55.755316 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.756544 18022 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:23:55.757078 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.094s (SENT)
W20260709 12:23:55.758495 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 175404us
I20260709 12:23:55.762538 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.763657 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:55.757870 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.677050 23600 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14.
I20260709 12:23:55.722172 23659 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.766548 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.790979 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:55.791471 23758 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:23:55.718648 23746 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:55.752442 23730 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.801260 18020 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:55.738422 23763 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:55.670037 23610 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:55.780843 23759 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:23:55.765329 23765 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:55.663864 23718 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.799858 23659 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:55.820691 23714 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.003s
I20260709 12:23:55.797248 23716 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:55.810036 23686 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:55.874302 23743 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.888s	user 0.000s	sys 0.004s
I20260709 12:23:55.874454 23766 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:55.794943 23720 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:23:55.828325 23763 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:55.815138 23591 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.506s	user 0.013s	sys 0.000s
I20260709 12:23:55.859879 23765 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.912863 23780 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:55.913932 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 873) took 1438 ms. Trace:
I20260709 12:23:55.911741 23591 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.914426 18099 rpcz_store.cc:276] 0709 12:23:54.475995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.478905 (+  2910us) service_pool.cc:224] Handling call
0709 12:23:54.526399 (+ 47494us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.713380 (+186981us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.713418 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:54.732674 (+ 19256us) log.cc:844] Serialized 26 byte log entry
0709 12:23:54.864989 (+132315us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:54.865035 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:54.865113 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.860211 (+995098us) spinlock_profiling.cc:243] Waited 46.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:55.913515 (+ 53304us) raft_consensus.cc:1685] finished
0709 12:23:55.913541 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.913856 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4591,"raft.queue_time_us":74245,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":15584,"spinlock_wait_cycles":106376960,"thread_start_us":256923,"threads_started":2,"wal-append.queue_time_us":233214}
I20260709 12:23:55.917055 23678 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:55.897655 23658 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 12:23:55.822423 23737 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
I20260709 12:23:55.825260 23746 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.917595 23591 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:55.856940 23597 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.924796 23597 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:55.817809 23457 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.873447 23725 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:55.768354 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:23:55.940111 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 171783us
I20260709 12:23:55.941200 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:55.940299 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 183259us
I20260709 12:23:55.882498 23686 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.945210 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.767468 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:55.955726 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 189237us
W20260709 12:23:55.956342 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 188949us
W20260709 12:23:55.917055 23697 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 2.217s	user 0.000s	sys 0.021s
I20260709 12:23:55.941308 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:55.956766 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:55.880179 23714 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:55.771270 23745 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:55.770140 23676 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:55.928892 23765 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:55.988291 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.957904 23778 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:55.973342 23749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.006s	sys 0.013s
I20260709 12:23:55.993707 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:55.972815 23591 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:55.909631 23763 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:55.861254 23643 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:55.957052 23728 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:55.957790 23686 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.978296 23761 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:56.022667 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:56.032306 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:55.979125 23463 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:55.974565 23717 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:55.769482 23719 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.225s
W20260709 12:23:55.985021 23685 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.702s	user 0.019s	sys 0.004s
I20260709 12:23:56.026557 23384 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.983415 23730 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:56.030454 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:23:56.075423 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52854.3us
I20260709 12:23:55.991976 23658 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:56.075378 18171 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:23:56.100128 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69842.5us
I20260709 12:23:56.101641 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:56.102269 18171 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [14.14-14.14]
I20260709 12:23:56.081023 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:56.020205 23749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.006s	sys 0.013s
W20260709 12:23:56.071416 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80591us
I20260709 12:23:56.022816 23778 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.026862 23621 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:55.999868 23257 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:55.993592 23600 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:56.110687 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.111369 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:56.112198 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:56.093601 18173 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:23:56.026983 23728 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.023707 23591 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.081861 23745 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:56.135064 18171 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:56.181062 18018 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:56.182049 18096 raft_consensus.cc:2376] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10. Re-sending same reply.
I20260709 12:23:56.199998 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.200882 18096 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
W20260709 12:23:56.181965 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45133.7us
I20260709 12:23:56.142748 23716 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
W20260709 12:23:56.193212 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55704.7us
I20260709 12:23:56.188964 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.206255 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.139614 23257 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.209918 18171 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:55.992450 23774 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.157s
I20260709 12:23:56.134214 23717 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:56.217964 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.133957 23780 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.224661 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.204981 23728 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:56.212102 23257 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:56.204276 23591 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:56.215029 23720 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.252799 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2504) took 1889 ms. Trace:
I20260709 12:23:56.226279 23737 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:56.256831 23737 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.257337 23737 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:56.258517 23746 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:56.259239 23746 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.140851 23678 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.094s
I20260709 12:23:56.137615 23685 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.132495 23658 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:56.132814 23384 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:56.261682 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2653) took 1220 ms. Trace:
I20260709 12:23:56.253376 18167 rpcz_store.cc:276] 0709 12:23:54.363366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.364726 (+  1360us) service_pool.cc:224] Handling call
0709 12:23:56.252536 (+1887810us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400030a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:56.252707 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920148736}
I20260709 12:23:56.259459 23623 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:56.231344 23717 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:56.277974 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.199206 23725 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
W20260709 12:23:56.283416 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 79578us
I20260709 12:23:56.240679 23711 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:56.297123 23685 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:56.309685 18097 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:56.222246 23645 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:23:56.250157 23610 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:56.259734 23746 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.311954 18090 rpcz_store.cc:276] 0709 12:23:55.041278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.216078 (+174800us) service_pool.cc:224] Handling call
0709 12:23:55.216408 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.327993 (+111585us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.328039 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.328710 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.401740 (+ 73030us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:55.664829 (+263089us) log.cc:844] Serialized 14 byte log entry
0709 12:23:55.668260 (+  3431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.261159 (+592899us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54004123d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:56.261229 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.261298 (+    69us) raft_consensus.cc:1685] finished
0709 12:23:56.261417 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.261599 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106143,"prepare.queue_time_us":81732,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":7205,"raft.queue_time_us":310657,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":28180,"spinlock_wait_cycles":67343232,"thread_start_us":287553,"threads_started":2,"wal-append.queue_time_us":97440}
I20260709 12:23:56.321134 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:56.282260 23784 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:56.398780 23716 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:56.401223 23716 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:56.402516 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.398841 23784 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.336917 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:56.234849 23780 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:56.325500 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 104684us
I20260709 12:23:56.342382 23792 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.344118 23778 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:56.344743 23788 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.260s
I20260709 12:23:56.310673 23785 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:56.374301 23795 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.148s
I20260709 12:23:56.379633 23711 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.390050 23786 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.146s
W20260709 12:23:56.399192 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78184.9us
I20260709 12:23:56.325613 23730 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:23:56.449645 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:56.322690 23717 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:56.362903 23789 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:23:56.393397 23676 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:56.467108 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:56.353387 23748 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.614s	user 0.006s	sys 0.015s
I20260709 12:23:56.267379 23720 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.324908 23782 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:56.425719 23777 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.406s	user 0.000s	sys 0.002s
I20260709 12:23:56.420042 23716 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:56.534129 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:56.354131 23714 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:56.540190 18812 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.005s	sys 0.013s
W20260709 12:23:56.565093 18812 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.005s	sys 0.013s
I20260709 12:23:56.349376 23775 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.264s
I20260709 12:23:56.425143 23773 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:56.555583 23676 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:56.544193 23784 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:56.560957 23782 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:56.559253 23487 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.003s	sys 0.014s
W20260709 12:23:56.563380 23469 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.982s	user 0.012s	sys 0.004s
W20260709 12:23:56.597029 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63011.6us
I20260709 12:23:56.569033 23716 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:56.613492 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:56.548578 23711 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:56.602298 23558 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.374s	user 0.012s	sys 0.016s
I20260709 12:23:56.598105 23778 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.619588 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:23:56.469802 23792 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:56.598851 23645 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:56.546630 23615 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.610410 23705 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:56.563745 23748 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:56.610526 23487 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.004s	sys 0.015s
I20260709 12:23:56.608618 23782 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:56.659266 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.019s	sys 0.029s
I20260709 12:23:56.664134 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3423) took 1265 ms. Trace:
W20260709 12:23:56.684561 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.023s	sys 0.029s
I20260709 12:23:56.683812 18098 rpcz_store.cc:276] 0709 12:23:55.399165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.399419 (+   254us) service_pool.cc:224] Handling call
0709 12:23:55.399931 (+   512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.569280 (+169349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.569334 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.581550 (+ 12216us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.724939 (+143389us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:55.724997 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.725084 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.663762 (+938678us) raft_consensus.cc:1685] finished
0709 12:23:56.663796 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.664032 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15917,"raft.queue_time_us":138626,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":131,"thread_start_us":231064,"threads_started":2,"wal-append.queue_time_us":166716}
I20260709 12:23:56.635429 23718 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:56.672102 18813 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.001s	sys 0.048s
W20260709 12:23:56.656548 23590 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.048s	user 0.014s	sys 0.022s
W20260709 12:23:56.695539 18813 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s	user 0.001s	sys 0.053s
W20260709 12:23:56.666380 23437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.000s	sys 0.022s
I20260709 12:23:56.697161 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1020) took 1181 ms. Trace:
I20260709 12:23:56.701735 18017 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:56.635551 23558 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.622301 23469 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:56.748379 18093 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2641) took 2188 ms (client timeout 2621 ms). Trace:
I20260709 12:23:56.701081 18813 rpcz_store.cc:276] 0709 12:23:55.515990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.614061 (+ 98071us) service_pool.cc:224] Handling call
0709 12:23:55.614486 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.614820 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:55.755431 (+140611us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.696266 (+940835us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:56.696351 (+    85us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:56.696418 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.696521 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.697069 (+   548us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59597,"prepare.queue_time_us":336436,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":161,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":6843,"thread_start_us":1053328,"threads_started":2}
I20260709 12:23:56.610729 23773 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.657296 23778 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:56.686777 23615 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:56.804263 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:56.785138 18778 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.505s (SENT)
I20260709 12:23:56.800604 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:56.816052 18806 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:56.751212 18093 rpcz_store.cc:269] 0709 12:23:54.560422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.001485 (+441063us) service_pool.cc:224] Handling call
0709 12:23:56.748060 (+1746575us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:56.748279 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414230400}
I20260709 12:23:56.817116 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:56.729138 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1207) took 1783 ms. Trace:
I20260709 12:23:56.818495 18808 rpcz_store.cc:276] 0709 12:23:54.946262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.946479 (+   217us) service_pool.cc:224] Handling call
0709 12:23:54.973684 (+ 27205us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.116600 (+142916us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.116642 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:55.116704 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:55.116741 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.728802 (+1612061us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54004b46d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:56.728873 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.729062 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328007424}
W20260709 12:23:56.782558 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2495) took 2461 ms (client timeout 2194 ms). Trace:
W20260709 12:23:56.820686 18172 rpcz_store.cc:269] 0709 12:23:54.287825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.288013 (+   188us) service_pool.cc:224] Handling call
0709 12:23:56.748906 (+2460893us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b5400031bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:56.749038 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64637696}
I20260709 12:23:56.670655 23705 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.636451 23730 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.644083 23800 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:56.832620 18099 raft_consensus.cc:2368] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17.
W20260709 12:23:56.701045 23437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.000s	sys 0.023s
I20260709 12:23:56.834733 18169 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:56.866714 18021 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:23:56.868235 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2517) took 1920 ms. Trace:
I20260709 12:23:56.868676 18169 rpcz_store.cc:276] 0709 12:23:54.948120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.210240 (+262120us) service_pool.cc:224] Handling call
0709 12:23:56.833758 (+1623518us) spinlock_profiling.cc:243] Waited 6.36 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:23:56.834143 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.866019 (+ 31876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.868155 (+  2136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14628224}
I20260709 12:23:56.693425 23658 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:56.794756 23714 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:56.721756 23797 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
W20260709 12:23:56.728523 23664 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.284s	user 0.025s	sys 0.002s
I20260709 12:23:56.749455 23746 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:56.864619 18093 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:23:56.705317 23727 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.069s	user 0.018s	sys 0.008s
I20260709 12:23:56.747951 23610 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:56.704923 23780 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:56.728223 23685 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:56.865547 23621 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:56.840539 23560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.005s	sys 0.006s
W20260709 12:23:56.861539 23620 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.194s	user 0.019s	sys 0.007s
I20260709 12:23:56.881347 18806 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:56.888042 18019 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:56.702898 23590 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.906395 18093 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:56.917868 18062 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.031s	sys 0.028s
W20260709 12:23:56.918419 18062 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.032s	sys 0.028s
W20260709 12:23:56.905176 23597 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.970s	user 0.018s	sys 0.000s
I20260709 12:23:56.930385 23792 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:56.930716 23816 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:56.932061 23801 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:56.933039 23817 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:23:56.932816 23801 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:56.910308 23560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.006s	sys 0.006s
I20260709 12:23:56.889200 23720 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:56.942703 23469 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:56.943011 23760 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.316s	user 0.008s	sys 0.000s
I20260709 12:23:56.888643 23610 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:56.958976 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2666) took 1279 ms. Trace:
I20260709 12:23:56.903275 23384 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:56.904914 23664 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:56.952090 23798 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.922870 23620 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.952172 23772 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:56.952284 23796 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.232s	user 0.001s	sys 0.000s
I20260709 12:23:56.952368 23769 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.904495 23746 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:56.886744 23714 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:56.918850 23621 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:56.903836 23727 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:57.187816 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 330576us
W20260709 12:23:57.388648 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 571586us
W20260709 12:23:57.390547 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.390739 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.391245 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.391817 18169 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:23:57.393273 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.395203 18169 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:23:57.394291 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:57.394179 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2511) took 2486 ms (client timeout 2172 ms). Trace:
W20260709 12:23:57.396503 18164 rpcz_store.cc:269] 0709 12:23:54.907870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.039190 (+131320us) service_pool.cc:224] Handling call
0709 12:23:57.393963 (+2354773us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.394085 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":594172672}
I20260709 12:23:57.398404 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.399088 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:57.399305 18164 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:57.399842 18806 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:57.233898 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:57.400673 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.019563 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3364) took 1409 ms. Trace:
I20260709 12:23:57.402597 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:57.402524 18021 rpcz_store.cc:276] 0709 12:23:55.609977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.611474 (+  1497us) service_pool.cc:224] Handling call
0709 12:23:55.612084 (+   610us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.867252 (+1255168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.019146 (+151894us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001d0e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:57.019467 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2884528384}
I20260709 12:23:57.403653 18169 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:57.026350 18094 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2646) took 1987 ms (client timeout 2172 ms). Trace:
W20260709 12:23:57.404502 18094 rpcz_store.cc:269] 0709 12:23:55.038994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.141129 (+102135us) service_pool.cc:224] Handling call
0709 12:23:57.026023 (+1884894us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.026259 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3593298816}
I20260709 12:23:57.406212 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1022) took 1868 ms. Trace:
I20260709 12:23:57.409734 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:57.412508 18022 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:57.413630 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:57.410786 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.415565 18809 rpcz_store.cc:276] 0709 12:23:55.542409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.659131 (+116722us) service_pool.cc:224] Handling call
0709 12:23:57.020489 (+1361358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.020750 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.020780 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.020830 (+    50us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:57.020865 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.020927 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.021780 (+   853us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400544050. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.406121 (+384341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3129834880}
I20260709 12:23:57.416920 18806 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:57.232914 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2667) took 1123 ms. Trace:
I20260709 12:23:57.020313 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1212) took 1598 ms. Trace:
I20260709 12:23:57.418617 18090 rpcz_store.cc:276] 0709 12:23:56.109606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.408614 (+299008us) service_pool.cc:224] Handling call
0709 12:23:56.409057 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.602072 (+193015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.602132 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.606695 (+  4563us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.843973 (+237278us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:56.844024 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.013366 (+169342us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b540035e8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:57.013469 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.013555 (+    86us) raft_consensus.cc:1685] finished
0709 12:23:57.013578 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.232776 (+219198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":587781,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"spinlock_wait_cycles":25350912,"thread_start_us":399097,"threads_started":2,"wal-append.queue_time_us":290569}
I20260709 12:23:56.967718 18091 rpcz_store.cc:276] 0709 12:23:55.679742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.739626 (+ 59884us) service_pool.cc:224] Handling call
0709 12:23:55.739996 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.024627 (+284631us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:56.082363 (+ 57736us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.606382 (+524019us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.609996 (+  3614us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.610219 (+   223us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:56.610259 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.611715 (+  1456us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.958680 (+346965us) raft_consensus.cc:1685] finished
0709 12:23:56.958708 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.958895 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19201,"prepare.queue_time_us":60102,"prepare.run_cpu_time_us":193,"prepare.run_wall_time_us":15372,"raft.queue_time_us":160579,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":11009,"thread_start_us":782408,"threads_started":3,"wal-append.queue_time_us":460851}
I20260709 12:23:57.028086 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.419827 18811 rpcz_store.cc:276] 0709 12:23:55.421656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.422568 (+   912us) service_pool.cc:224] Handling call
0709 12:23:57.020064 (+1597496us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.020227 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148510976}
W20260709 12:23:57.421402 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2617) took 1375 ms (client timeout 1534 ms). Trace:
W20260709 12:23:57.437459 18165 rpcz_store.cc:269] 0709 12:23:56.045799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.046055 (+   256us) service_pool.cc:224] Handling call
0709 12:23:57.027820 (+981765us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:23:57.421306 (+393486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5965184}
W20260709 12:23:56.987161 23324 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.440s	user 0.009s	sys 0.016s
I20260709 12:23:57.441541 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:56.996351 23659 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.102s	user 0.014s	sys 0.003s
I20260709 12:23:57.458904 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.461586 18020 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:56.999627 23761 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.886s	user 0.005s	sys 0.009s
W20260709 12:23:56.949796 23737 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.692s	user 0.010s	sys 0.000s
I20260709 12:23:57.443724 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:57.444427 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:57.009876 23773 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.008157 23784 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:23:57.293191 23591 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.015s	user 0.007s	sys 0.007s
W20260709 12:23:57.084803 23257 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.818s	user 0.014s	sys 0.001s
W20260709 12:23:57.043704 23728 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.780s	user 0.010s	sys 0.002s
I20260709 12:23:56.951838 23713 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:57.294555 23691 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.672s	user 0.018s	sys 0.011s
W20260709 12:23:57.284907 23686 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.202s	user 0.012s	sys 0.003s
W20260709 12:23:57.280354 23765 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.236s	user 0.016s	sys 0.000s
W20260709 12:23:57.002779 23642 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.428s	user 0.017s	sys 0.007s
I20260709 12:23:57.396144 23623 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:57.046814 23802 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.383s	user 0.002s	sys 0.000s
I20260709 12:23:57.015756 23457 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.011461 23779 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:57.007777 23725 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:57.457980 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.546208 18015 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:57.458546 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:57.463063 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:57.463960 18171 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:57.029703 23720 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:57.464493 18093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.035s	sys 0.058s
W20260709 12:23:57.557725 18093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.035s	sys 0.058s
I20260709 12:23:57.213641 23807 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:57.470230 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:57.472725 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:57.343588 23824 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.393s	user 0.003s	sys 0.000s
I20260709 12:23:57.516427 23691 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:56.999388 23558 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.478844 23737 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.542972 23623 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.442782 23780 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:57.480749 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:57.494715 23784 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.482035 23664 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:57.396057 23801 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:57.570930 23676 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.955s	user 0.016s	sys 0.000s
I20260709 12:23:56.957634 23590 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.423491 23705 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.422631 23745 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:57.452999 23811 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:57.027578 23717 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.470338 23805 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:57.482540 23761 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:57.494036 23796 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:23:56.989167 23711 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.407815 23772 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.459894 18023 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:23:57.549708 18173 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:57.550647 18169 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:23:57.554752 23312 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:57.555341 23716 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.918s	user 0.000s	sys 0.014s
I20260709 12:23:57.556819 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.636565 18171 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:57.637699 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:57.638351 18171 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:57.407326 23798 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.407188 23769 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.250140 23610 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.563117 18165 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:23:57.563136 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90510.9us
I20260709 12:23:56.954408 23778 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.466226 23659 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.466579 23324 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.493176 23257 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.493346 23728 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:57.569659 18780 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.792s (SENT) [suppressed 13 similar messages]
I20260709 12:23:57.582119 18164 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:57.547371 23487 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.585088 18098 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:57.481701 23819 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.542517 23779 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:57.545878 23725 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:57.406934 23587 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.689893 18016 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:57.497153 23591 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.519941 23642 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.520107 23765 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.601372 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2620) took 1522 ms. Trace:
I20260709 12:23:57.701879 18166 rpcz_store.cc:276] 0709 12:23:56.078590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.078757 (+   167us) service_pool.cc:224] Handling call
0709 12:23:56.104532 (+ 25775us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.556776 (+452244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.556810 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.597580 (+ 40770us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.826536 (+228956us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:56.826579 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.826673 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.400071 (+573398us) raft_consensus.cc:1685] finished
0709 12:23:57.400095 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.598218 (+198123us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b54000424d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.601270 (+  3052us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33297,"raft.queue_time_us":133196,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":4546,"spinlock_wait_cycles":58140032,"thread_start_us":619772,"threads_started":2,"wal-append.queue_time_us":312217}
I20260709 12:23:57.601930 23737 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.602815 23714 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.603826 23784 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.604380 23623 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.565177 23807 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:57.605173 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3365) took 1932 ms (client timeout 1610 ms). Trace:
I20260709 12:23:57.678309 23610 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:57.645596 23716 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:57.721913 18019 rpcz_store.cc:269] 0709 12:23:55.673226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.673376 (+   150us) service_pool.cc:224] Handling call
0709 12:23:57.604864 (+1931488us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.605052 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2640659328}
I20260709 12:23:57.606803 23727 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.740928 23642 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:57.608772 23763 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.534s	user 0.016s	sys 0.001s
I20260709 12:23:57.519687 23686 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.557741 23720 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.631135 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:57.632772 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.747270 18814 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:57.747448 18169 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:23:57.590358 23780 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.626483 23748 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.625605 23664 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:57.654508 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84890.5us
I20260709 12:23:57.654955 18167 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:57.600402 23676 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:57.660540 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.665900 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:57.665609 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:57.641637 23800 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:23:57.655052 23782 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.958s	user 0.007s	sys 0.007s
I20260709 12:23:57.683499 18099 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:57.565729 23231 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:57.695389 18168 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:57.616024 23745 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.652686 23620 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.629835 23805 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.769769 18021 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:57.698433 18171 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
W20260709 12:23:57.690132 23615 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.849s	user 0.015s	sys 0.000s
I20260709 12:23:57.697142 23621 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.717478 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2663) took 2109 ms. Trace:
I20260709 12:23:57.668188 23312 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:57.626848 23811 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.779865 23811 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.724496 18166 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:57.730407 23769 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.731905 23659 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.733537 23324 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.734135 23779 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.754994 23558 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.682586 23819 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.747843 23763 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.735910 23591 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.801882 23763 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.736776 23807 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:57.737779 23833 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:23:57.737910 23765 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:23:57.738752 23830 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:23:57.806108 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 900) took 1694 ms. Trace:
I20260709 12:23:57.738587 23725 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.806645 18093 rpcz_store.cc:276] 0709 12:23:56.111351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.926644 (+815293us) service_pool.cc:224] Handling call
0709 12:23:56.954365 (+ 27721us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.954692 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.954727 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.955387 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.558308 (+602921us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.639231 (+ 80923us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.639570 (+   339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.639700 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.805643 (+165943us) raft_consensus.cc:1685] finished
0709 12:23:57.805671 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.806018 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":135788,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":4092,"raft.queue_time_us":105,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":1994,"thread_start_us":669620,"threads_started":2,"wal-append.queue_time_us":587924}
I20260709 12:23:57.807834 23725 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:57.808606 23686 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.809110 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:57.740219 23772 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.679787 23487 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.756980 23664 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.812614 23664 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.812922 23487 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.761430 23800 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.815670 23800 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:57.815970 23806 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.870s	user 0.002s	sys 0.000s
I20260709 12:23:57.818143 23717 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.820116 23780 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:23:57.822292 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 737) took 2462 ms (client timeout 1640 ms). Trace:
W20260709 12:23:57.822804 18024 rpcz_store.cc:269] 0709 12:23:55.360306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.360690 (+   384us) service_pool.cc:224] Handling call
0709 12:23:57.822067 (+2461377us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b540022f954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:57.822196 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":382633856}
I20260709 12:23:57.824077 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.825323 23558 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.795236 23798 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.799507 23745 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.828176 23558 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.828737 23819 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.825476 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.832839 18093 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:57.749881 18098 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:57.735972 23728 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:57.759343 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 93847.7us
I20260709 12:23:57.764446 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.837034 18015 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:57.768222 18020 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:57.763201 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:57.771752 18019 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:57.773571 18169 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:57.781953 18814 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:57.783913 18171 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:57.771198 18092 rpcz_store.cc:276] 0709 12:23:55.610261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.739903 (+129642us) service_pool.cc:224] Handling call
0709 12:23:55.740391 (+   488us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:56.191854 (+451463us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:56.192636 (+   782us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.493868 (+301232us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:56.551433 (+ 57565us) log.cc:844] Serialized 52 byte log entry
0709 12:23:56.551733 (+   300us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:56.556110 (+  4377us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.006310 (+450200us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b180038bea0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:23:57.592624 (+586314us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.592979 (+   355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.610761 (+ 17782us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.717194 (+106433us) raft_consensus.cc:1685] finished
0709 12:23:57.717221 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.717394 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24652,"prepare.queue_time_us":200642,"prepare.run_cpu_time_us":208,"prepare.run_wall_time_us":17278,"raft.queue_time_us":1387917,"raft.run_cpu_time_us":690,"raft.run_wall_time_us":31435,"spinlock_wait_cycles":61702016,"thread_start_us":1321340,"threads_started":5,"wal-append.queue_time_us":362352}
I20260709 12:23:57.841086 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.841598 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:57.842393 18092 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:23:57.842850 23790 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.635s	user 0.000s	sys 0.008s
I20260709 12:23:57.843132 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:57.843547 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:57.843781 18093 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:57.844694 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.785761 18165 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:57.788079 18166 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:57.735078 23257 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.788511 23312 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:57.848881 18165 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:57.795203 23720 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.796749 23716 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:57.797221 23805 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:57.797798 23779 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:57.799469 23676 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:57.760562 23782 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.546046 23457 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:57.832314 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:57.842722 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.786445 18021 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:57.839071 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:57.844769 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:57.878670 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.856024 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:58.520684 18020 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:58.094791 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:57.854322 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 91182.7us
W20260709 12:23:58.357316 18093 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.000s
W20260709 12:23:57.874193 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 125201us
W20260709 12:23:58.100054 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.066722 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:57.862825 17989 connection.cc:256] RPC call timeout handler was delayed by -2.1727195819998997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2657s for a response
I20260709 12:23:58.076306 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:58.016119 18096 raft_consensus.cc:2376] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13. Re-sending same reply.
I20260709 12:23:58.543116 18019 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:23:58.091348 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3422) took 2734 ms. Trace:
I20260709 12:23:58.073937 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:58.548635 18814 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:58.201022 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.525698 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.550264 18092 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:23:58.089691 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:57.895736 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.552690 18098 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:58.553241 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3435) took 1525 ms. Trace:
I20260709 12:23:58.553587 18098 rpcz_store.cc:276] 0709 12:23:57.027744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.840346 (+812602us) service_pool.cc:224] Handling call
0709 12:23:58.553169 (+712823us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.157019 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.555253 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3429) took 1534 ms. Trace:
I20260709 12:23:58.555915 23762 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.556330 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:57.786180 23713 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:58.546311 18811 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:58.558276 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:23:58.561373 18172 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.012s	sys 0.051s
W20260709 12:23:58.563337 18172 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.012s	sys 0.052s
I20260709 12:23:58.559983 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.543340 18093 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:58.563023 18809 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:58.558804 17989 connection.cc:256] RPC call timeout handler was delayed by -2.0144801369997367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20260709 12:23:58.607038 17989 connection.cc:256] RPC call timeout handler was delayed by -2.0126801369997338s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20260709 12:23:58.607539 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.558349 18091 rpcz_store.cc:276] 0709 12:23:57.020664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.156572 (+1135908us) service_pool.cc:224] Handling call
0709 12:23:58.555164 (+398592us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.155814 23497 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:58.547354 18095 rpcz_store.cc:276] 0709 12:23:55.356637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.356882 (+   245us) service_pool.cc:224] Handling call
0709 12:23:55.357233 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.357480 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:55.376153 (+ 18673us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.942384 (+1566231us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:58.082117 (+1139733us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b18003722e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:23:58.090748 (+  8631us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.090848 (+   100us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:58.090931 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.091028 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.091224 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":47263,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":60,"raft.queue_time_us":19372,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":11375,"spinlock_wait_cycles":73926912,"thread_start_us":88144,"threads_started":1,"wal-append.queue_time_us":198365,"wal-append.run_cpu_time_us":10005,"wal-append.run_wall_time_us":2375536}
I20260709 12:23:57.962234 23257 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:23:58.597239 23610 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.867s	user 0.015s	sys 0.000s
I20260709 12:23:58.594765 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:58.586488 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:58.588455 18098 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.590902 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.587168 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3433) took 1565 ms. Trace:
W20260709 12:23:57.866091 23822 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.915s	user 0.001s	sys 0.001s
I20260709 12:23:58.596990 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.642326 18166 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:23:58.643649 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:58.644447 18166 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.602766 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:58.535845 23725 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.862372 23717 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:58.535660 23312 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:58.604714 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:58.694660 23312 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:58.695226 18808 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:58.605633 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.696312 18092 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:58.697098 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3430) took 1676 ms. Trace:
I20260709 12:23:58.697453 18092 rpcz_store.cc:276] 0709 12:23:57.020978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.563612 (+1542634us) service_pool.cc:224] Handling call
0709 12:23:58.697023 (+133411us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.621382 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.699726 18094 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:58.700766 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.701439 18094 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:23:58.604578 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:23:58.703354 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98902.8us
I20260709 12:23:58.698777 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.631904 18020 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:23:58.637558 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.639000 18093 rpcz_store.cc:276] 0709 12:23:57.026968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.842847 (+815879us) service_pool.cc:224] Handling call
0709 12:23:58.571443 (+728596us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.706604 18091 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:23:58.646003 18811 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:23:58.646834 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.708424 18095 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:58.648463 18814 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:58.657611 18098 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:23:58.713183 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3438) took 1306 ms. Trace:
I20260709 12:23:58.713573 18098 rpcz_store.cc:276] 0709 12:23:57.407288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.554514 (+1147226us) service_pool.cc:224] Handling call
0709 12:23:58.713106 (+158592us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.618469 23497 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:58.665843 23713 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.687916 23782 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:58.588359 23762 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:58.711951 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3434) took 1684 ms. Trace:
I20260709 12:23:58.641139 23676 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:23:58.720734 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:58.662456 18166 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:23:58.723578 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:23:58.723762 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4411351319999994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
I20260709 12:23:58.724368 18166 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:23:58.721127 18092 rpcz_store.cc:276] 0709 12:23:57.027396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.698356 (+1670960us) service_pool.cc:224] Handling call
0709 12:23:58.711859 (+ 13503us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.725631 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:58.726679 18166 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:58.727698 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.709437 23716 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:58.732405 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:58.731088 18094 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:23:58.733804 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3427) took 1713 ms. Trace:
I20260709 12:23:58.733940 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.734225 18094 rpcz_store.cc:276] 0709 12:23:57.020059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.718371 (+1698312us) service_pool.cc:224] Handling call
0709 12:23:58.730843 (+ 12472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.733503 (+  2660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.733742 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.650858 23257 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:58.705933 23725 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:58.743597 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2622) took 2664 ms. Trace:
I20260709 12:23:58.744333 18170 rpcz_store.cc:276] 0709 12:23:56.079232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.104313 (+ 25081us) service_pool.cc:224] Handling call
0709 12:23:56.181123 (+ 76810us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.204402 (+ 23279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.204456 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.214857 (+ 10401us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.519767 (+1304910us) spinlock_profiling.cc:243] Waited 8.64 ms on lock 0x7b18004aa640. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:23:57.611493 (+ 91726us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:57.611544 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.009647 (+398103us) spinlock_profiling.cc:243] Waited 5.68 ms on lock 0x7b48000cd158. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:23:58.743036 (+733389us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b54003020d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:58.743126 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:58.743206 (+    80us) raft_consensus.cc:1685] finished
0709 12:23:58.743229 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.743487 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36290,"raft.run_cpu_time_us":8325,"raft.run_wall_time_us":1769218,"spinlock_wait_cycles":85009664,"thread_start_us":91467,"threads_started":1,"wal-append.queue_time_us":1446683,"wal-append.run_cpu_time_us":2572,"wal-append.run_wall_time_us":1025355}
W20260709 12:23:58.749337 17987 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.051s	sys 0.077s
W20260709 12:23:58.750015 17987 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.051s	sys 0.077s
W20260709 12:23:58.750483 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 914379us
I20260709 12:23:58.751701 18092 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:58.752530 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3436) took 1724 ms. Trace:
W20260709 12:23:58.752593 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.753011 18092 rpcz_store.cc:276] 0709 12:23:57.028057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.725215 (+1697158us) service_pool.cc:224] Handling call
0709 12:23:58.744083 (+ 18868us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.752205 (+  8122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.752459 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.719552 23717 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
W20260709 12:23:58.760176 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0595990359997813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2646s for a response
W20260709 12:23:58.756842 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3431) took 1735 ms (client timeout 1610 ms). Trace:
W20260709 12:23:58.761135 18099 rpcz_store.cc:269] 0709 12:23:57.021313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.418577 (+397264us) service_pool.cc:224] Handling call
0709 12:23:58.756553 (+1337976us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400296dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:58.756742 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311098624}
I20260709 12:23:58.758519 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2545) took 3019 ms. Trace:
I20260709 12:23:58.762284 18018 rpcz_store.cc:276] 0709 12:23:55.738908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.739062 (+   154us) service_pool.cc:224] Handling call
0709 12:23:55.739436 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.198959 (+459523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.758240 (+2559281us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54000ff854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:58.758447 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":921496448}
I20260709 12:23:58.768502 18024 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 12:23:58.722311 23713 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:58.772480 18097 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:23:58.728734 23676 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:58.775031 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:58.776974 18170 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:58.777535 18099 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:58.769538 23558 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.773728 23497 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:58.783164 23804 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:58.783479 23762 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:58.779614 23716 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:58.784180 23716 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:23:58.785132 23716 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:58.785866 23257 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:58.786494 23257 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.786999 23257 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:58.789288 23257 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:58.789399 23713 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:58.789959 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.790122 18166 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:23:58.790930 18169 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:58.791934 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2553) took 1001 ms. Trace:
I20260709 12:23:58.792155 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:23:58.792748 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2522) took 1402 ms (client timeout 1865 ms). Trace:
I20260709 12:23:58.792917 18166 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:58.793247 18167 rpcz_store.cc:269] 0709 12:23:57.390834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.390973 (+   139us) service_pool.cc:224] Handling call
0709 12:23:58.792572 (+1401599us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400382f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:58.792684 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":604722432}
I20260709 12:23:58.792385 18022 rpcz_store.cc:276] 0709 12:23:57.790274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.790458 (+   184us) service_pool.cc:224] Handling call
0709 12:23:57.790894 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.791345 (+   451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.791379 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.792026 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.858565 (+ 66539us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:57.858623 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.858867 (+   244us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:58.791709 (+932842us) raft_consensus.cc:1685] finished
0709 12:23:58.791736 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.791870 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1138,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":141,"thread_start_us":30143,"threads_started":1,"wal-append.queue_time_us":945663}
I20260709 12:23:58.795109 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.783502 23497 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:58.792059 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.801011 18169 raft_consensus.cc:2410] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 11 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 12:23:58.796370 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.803784 18167 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:58.807457 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.808826 18167 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 11: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:58.787496 23676 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:58.814731 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.783692 23804 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:58.805656 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:58.826114 18170 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:58.827473 18170 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [14.14-14.14]
I20260709 12:23:58.824210 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:58.810091 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:23:58.779378 23558 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:58.846463 18167 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:58.836863 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:58.830464 23713 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:58.821511 23457 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:58.839100 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:23:58.855742 18095 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:23:58.862766 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2540) took 3161 ms (client timeout 1747 ms). Trace:
W20260709 12:23:58.863608 18017 rpcz_store.cc:269] 0709 12:23:55.701502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.701706 (+   204us) service_pool.cc:224] Handling call
0709 12:23:58.862486 (+3160780us) spinlock_profiling.cc:243] Waited 934 ms on lock 0x7b54003bb0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:58.862670 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2147831552}
I20260709 12:23:58.865285 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3445) took 1101 ms. Trace:
I20260709 12:23:58.866081 18095 rpcz_store.cc:276] 0709 12:23:57.764363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.646503 (+882140us) service_pool.cc:224] Handling call
0709 12:23:58.865128 (+218625us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.868014 18167 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:58.875820 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:58.868005 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.881268 18095 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:58.883378 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.829871 23676 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:58.884341 18167 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:58.886171 18170 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260709 12:23:58.884275 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.839519 23312 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:58.889802 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:58.882129 23713 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:58.909586 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:58.911531 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.923626 18095 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:58.929981 18138 connection.cc:256] RPC call timeout handler was delayed by -2.1514492930000415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20260709 12:23:58.934449 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.935680 18090 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:23:58.942891 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.943595 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:58.944278 18092 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:23:58.944456 18090 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:23:58.940620 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:58.908368 23312 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.955826 18095 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:23:59.003896 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:58.937716 23713 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:59.014806 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2519) took 1625 ms. Trace:
I20260709 12:23:59.015547 18173 rpcz_store.cc:276] 0709 12:23:57.389748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.392583 (+  2835us) service_pool.cc:224] Handling call
0709 12:23:57.452134 (+ 59551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.633481 (+181347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.014275 (+1380794us) spinlock_profiling.cc:243] Waited 97.1 ms on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:59.014692 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223386880}
I20260709 12:23:58.910260 23676 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:59.028256 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 816) took 1633 ms (client timeout 1640 ms). Trace:
I20260709 12:23:59.028832 23676 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.030350 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2618) took 2957 ms. Trace:
W20260709 12:23:59.028842 18164 rpcz_store.cc:269] 0709 12:23:57.394831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.400336 (+  5505us) service_pool.cc:224] Handling call
0709 12:23:59.027996 (+1627660us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:59.028171 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414231808}
I20260709 12:23:59.023880 23725 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.006580 23312 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:59.030861 18168 rpcz_store.cc:276] 0709 12:23:56.073363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.073528 (+   165us) service_pool.cc:224] Handling call
0709 12:23:56.073986 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.757936 (+1683950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.029999 (+1272063us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:59.030268 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3728122752}
I20260709 12:23:59.039606 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:59.051578 18810 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:23:59.083768 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 379258us
I20260709 12:23:59.084831 23558 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:59.029758 23676 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:23:59.099624 23803 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:59.115056 23849 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
W20260709 12:23:59.128376 23848 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.185s	user 0.000s	sys 0.010s
W20260709 12:23:59.186240 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.002s	sys 0.001s
W20260709 12:23:59.187227 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.002s	sys 0.002s
W20260709 12:23:59.188874 17989 connection.cc:256] RPC call timeout handler was delayed by -1.516102877999856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2824s for a response
I20260709 12:23:59.189051 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 753) took 1365 ms. Trace:
W20260709 12:23:59.186584 18016 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.293s	user 0.000s	sys 0.003s
W20260709 12:23:59.191324 18015 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.003s	sys 0.000s
W20260709 12:23:59.192085 18015 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.003s	sys 0.000s
W20260709 12:23:59.189603 17989 connection.cc:256] RPC call timeout handler was delayed by -1.4222801369996887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21720000000000003s for a response
W20260709 12:23:59.193492 18016 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.300s	user 0.000s	sys 0.004s
W20260709 12:23:59.194377 17989 connection.cc:256] RPC call timeout handler was delayed by -1.127771864999886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
W20260709 12:23:59.194936 17989 connection.cc:256] RPC call timeout handler was delayed by -1.0829784819998167s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
I20260709 12:23:59.195755 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2554) took 1374 ms. Trace:
I20260709 12:23:59.189781 18021 rpcz_store.cc:276] 0709 12:23:57.823864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.786661 (+962797us) service_pool.cc:224] Handling call
0709 12:23:58.787071 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.787413 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.787444 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:58.787996 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.187890 (+399894us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:59.188041 (+   151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.188251 (+   210us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.188365 (+   114us) raft_consensus.cc:1685] finished
0709 12:23:59.188399 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.188777 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":54576,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":37863,"thread_start_us":399011,"threads_started":1,"wal-append.queue_time_us":335384}
I20260709 12:23:59.199584 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:23:59.202459 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.003s	sys 0.015s
W20260709 12:23:59.203274 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.004s	sys 0.015s
W20260709 12:23:59.204054 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.031s	sys 0.012s
W20260709 12:23:59.204644 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.031s	sys 0.012s
W20260709 12:23:59.207777 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.36867e+06us
I20260709 12:23:59.209594 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:59.211474 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 896) took 3101 ms (client timeout 2646 ms). Trace:
I20260709 12:23:59.211886 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:23:59.212311 18096 rpcz_store.cc:269] 0709 12:23:56.110149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.402080 (+291931us) service_pool.cc:224] Handling call
0709 12:23:59.210708 (+2808628us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b5400214b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:59.211042 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3708491648}
I20260709 12:23:59.214841 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:23:59.215651 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.216714 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3361) took 3620 ms. Trace:
W20260709 12:23:59.218257 18165 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.007s	sys 0.014s
W20260709 12:23:59.218475 18171 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.036s	sys 0.024s
W20260709 12:23:59.218803 18165 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.007s	sys 0.014s
W20260709 12:23:59.218952 18171 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.036s	sys 0.024s
I20260709 12:23:59.220851 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 825) took 1593 ms. Trace:
W20260709 12:23:59.221979 18172 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.005s	sys 0.016s
W20260709 12:23:59.222533 18172 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.005s	sys 0.016s
I20260709 12:23:59.223520 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2557) took 1198 ms. Trace:
I20260709 12:23:59.224167 18024 rpcz_store.cc:276] 0709 12:23:58.024458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.555944 (+531486us) service_pool.cc:224] Handling call
0709 12:23:59.222619 (+666675us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:59.223180 (+   561us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471005056}
I20260709 12:23:59.217262 18023 rpcz_store.cc:276] 0709 12:23:55.596480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.596653 (+   173us) service_pool.cc:224] Handling call
0709 12:23:55.613732 (+ 17079us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.633907 (+2020175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.215725 (+1581818us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400250b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:59.216373 (+   648us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4245158144}
I20260709 12:23:59.221400 18171 rpcz_store.cc:276] 0709 12:23:57.627567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.840070 (+212503us) service_pool.cc:224] Handling call
0709 12:23:57.840492 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:57.852233 (+ 11741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.852290 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:58.517570 (+665280us) log.cc:844] Serialized 52 byte log entry
0709 12:23:59.219969 (+702399us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:59.220108 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.220267 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.220356 (+    89us) raft_consensus.cc:1685] finished
0709 12:23:59.220375 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.220673 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11080,"raft.queue_time_us":21118,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":4252,"thread_start_us":618120,"threads_started":1,"wal-append.queue_time_us":562866}
I20260709 12:23:59.226709 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 755) took 1402 ms. Trace:
I20260709 12:23:59.227335 18022 rpcz_store.cc:276] 0709 12:23:57.824450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.795805 (+971355us) service_pool.cc:224] Handling call
0709 12:23:59.223223 (+427418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.223879 (+   656us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.223918 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.223991 (+    73us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:59.224033 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.224119 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.226100 (+  1981us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b54000ffd50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:23:59.226485 (+   385us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961097472}
I20260709 12:23:59.229535 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 814) took 1835 ms. Trace:
I20260709 12:23:59.230218 18172 rpcz_store.cc:276] 0709 12:23:57.394288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.772155 (+377867us) service_pool.cc:224] Handling call
0709 12:23:57.772521 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.783834 (+ 11313us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:57.835019 (+ 51185us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.223398 (+1388379us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.223565 (+   167us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:59.223747 (+   182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.229055 (+  5308us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b54003016d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:23:59.229127 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.229451 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101198,"prepare.queue_time_us":51448,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":61,"raft.queue_time_us":131534,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"spinlock_wait_cycles":25371904,"thread_start_us":1286251,"threads_started":2,"wal-append.queue_time_us":1104803}
I20260709 12:23:59.269411 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:59.276085 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:59.277125 18019 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:59.282405 23457 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:59.287199 23457 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.287933 23457 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:59.294708 23469 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:59.295913 23469 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:23:59.298569 23469 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:59.235710 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:59.196367 18016 rpcz_store.cc:276] 0709 12:23:57.821305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.829481 (+  8176us) service_pool.cc:224] Handling call
0709 12:23:57.829977 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.830440 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.830478 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.831136 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.887112 (+ 55976us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:59.127835 (+1240723us) spinlock_profiling.cc:243] Waited 7.99 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:23:59.194331 (+ 66496us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.194927 (+   596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.195071 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.195155 (+    84us) raft_consensus.cc:1685] finished
0709 12:23:59.195306 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.195620 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4437,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":3908,"spinlock_wait_cycles":18382720,"thread_start_us":1356536,"threads_started":2,"wal-append.queue_time_us":1062324}
I20260709 12:23:59.237046 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:59.363248 23720 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.364569 23720 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:59.117537 23803 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:59.363523 23782 raft_consensus.cc:2764] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
W20260709 12:23:59.363301 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.375391 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:59.383589 18139 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.008s	sys 0.012s
W20260709 12:23:59.406908 18139 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.009s	sys 0.013s
W20260709 12:23:59.468873 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 183118us
W20260709 12:23:59.469235 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 179) took 48152 ms (client timeout 15000 ms). Trace:
W20260709 12:23:59.469478 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.008s	sys 0.014s
W20260709 12:23:59.469995 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.003s	sys 0.018s
I20260709 12:23:59.470146 23762 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:59.470803 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.003s	sys 0.018s
W20260709 12:23:59.471372 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 707274us
W20260709 12:23:59.472813 18020 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.008s	sys 0.015s
I20260709 12:23:59.472548 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:23:59.473335 18020 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.009s	sys 0.015s
W20260709 12:23:59.474053 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.006s	sys 0.009s
W20260709 12:23:59.474678 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.006s	sys 0.009s
I20260709 12:23:59.475688 23685 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.476908 23685 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.481073 23685 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.482259 23856 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:59.483458 23856 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:59.484023 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 248763us
W20260709 12:23:59.482344 18064 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.006s	sys 0.009s
W20260709 12:23:59.486380 18064 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.006s	sys 0.009s
W20260709 12:23:59.486886 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 761389us
W20260709 12:23:59.487684 18064 connection.cc:256] RPC call timeout handler was delayed by -1.379035131999899s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
I20260709 12:23:59.488852 18814 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:23:59.470038 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.009s	sys 0.014s
W20260709 12:23:59.469856 17920 rpcz_store.cc:269] 0709 12:23:11.316522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.322036 (+  5514us) service_pool.cc:224] Handling call
0709 12:23:59.468835 (+48146799us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:49.989709 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:49.990573 (+   864us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:49.990638 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:23:49.990658 (+    20us) tablet.cc:662] Decoding operations
0709 12:23:49.992197 (+  1539us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:49.992354 (+   157us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:49.992387 (+    33us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:23:50.006513 (+ 14126us) tablet.cc:701] Row locks acquired
0709 12:23:50.027236 (+ 20723us) write_op.cc:260] PREPARE: finished
0709 12:23:50.051612 (+ 24376us) write_op.cc:270] Start()
0709 12:23:50.074648 (+ 23036us) write_op.cc:276] Timestamp: P: 1783599830034147 usec, L: 0
0709 12:23:50.078277 (+  3629us) op_driver.cc:348] REPLICATION: starting
0709 12:23:50.090190 (+ 11913us) log.cc:844] Serialized 5974 byte log entry
0709 12:23:50.193975 (+103785us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b
0709 12:23:50.949980 (+756005us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b180034ed00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:23:51.393151 (+443171us) op_driver.cc:464] REPLICATION: finished
0709 12:23:51.530124 (+136973us) write_op.cc:301] APPLY: starting
0709 12:23:51.530286 (+   162us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:51.659020 (+128734us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:51.659057 (+    37us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:52.068472 (+409415us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:52.291684 (+223212us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 12:23:52.296751 (+  5067us) write_op.cc:312] APPLY: finished
0709 12:23:52.354594 (+ 57843us) log.cc:844] Serialized 139 byte log entry
0709 12:23:52.354836 (+   242us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:52.355067 (+   231us) write_op.cc:454] Released schema lock
0709 12:23:52.355165 (+    98us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:52.355836 (+   671us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:52.355870 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5697445,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":121883,"apply.run_cpu_time_us":31306,"apply.run_wall_time_us":825671,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66917,"num_ops":15,"prepare.queue_time_us":200621,"prepare.run_cpu_time_us":46723,"prepare.run_wall_time_us":433968,"raft.queue_time_us":194520,"raft.run_cpu_time_us":17626,"raft.run_wall_time_us":48296,"replication_time_us":1314610,"spinlock_wait_cycles":21666944,"thread_start_us":563746,"threads_started":4,"wal-append.queue_time_us":345148,"wal-append.run_cpu_time_us":26919,"wal-append.run_wall_time_us":2575924}]]}
W20260709 12:23:59.494998 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 285099us
W20260709 12:23:59.475154 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 700707us
I20260709 12:23:59.508397 23663 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:59.508946 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 747) took 1686 ms. Trace:
I20260709 12:23:59.509810 23853 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:59.510643 23871 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:59.509552 18015 rpcz_store.cc:276] 0709 12:23:57.822110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.933007 (+110897us) service_pool.cc:224] Handling call
0709 12:23:58.527854 (+594847us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.528264 (+   410us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:59.192750 (+664486us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.482362 (+289612us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.506886 (+ 24524us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.506994 (+   108us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:59.508445 (+  1451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.508597 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.508856 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38278,"raft.queue_time_us":14014,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":11478,"thread_start_us":845768,"threads_started":2}
I20260709 12:23:59.511499 23871 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:59.471369 23762 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:59.511826 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.002s	sys 0.007s
I20260709 12:23:59.470177 23720 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:59.499286 23852 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:59.534837 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.003s	sys 0.008s
W20260709 12:23:59.522763 18018 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.011s	sys 0.025s
W20260709 12:23:59.541584 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 601077us
W20260709 12:23:59.547744 18018 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.012s	sys 0.026s
I20260709 12:23:59.523905 23871 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:59.559037 23851 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:59.559828 23851 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:23:59.560375 23851 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:59.562701 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51401.1us
I20260709 12:23:59.563244 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:23:59.564946 18093 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:23:59.537421 23663 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:23:59.510635 23853 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:59.570403 23663 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.571430 23856 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.574158 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2626) took 1011 ms. Trace:
I20260709 12:23:59.574688 18165 rpcz_store.cc:276] 0709 12:23:58.562972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.563774 (+   802us) service_pool.cc:224] Handling call
0709 12:23:58.595945 (+ 32171us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.219674 (+623729us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.219754 (+    80us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.220977 (+  1223us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.458569 (+237592us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:59.458648 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.463460 (+  4812us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.573789 (+110329us) raft_consensus.cc:1685] finished
0709 12:23:59.573837 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.574059 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69274,"raft.queue_time_us":99454,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"thread_start_us":785559,"threads_started":2,"wal-append.queue_time_us":350041}
W20260709 12:23:59.576237 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86926.3us
I20260709 12:23:59.570171 23853 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:59.577533 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:23:59.587293 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:59.587908 23803 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.587468 23469 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.537724 23852 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.596963 23469 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.600854 23469 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.605901 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:59.605903 18094 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.004s	sys 0.007s
W20260709 12:23:59.608484 18094 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.004s	sys 0.007s
I20260709 12:23:59.508818 23685 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.083s
I20260709 12:23:59.591948 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:23:59.622398 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137304us
I20260709 12:23:59.594242 23852 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:59.629426 23762 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.630379 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.631381 18015 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
W20260709 12:23:59.632642 23497 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.826s	user 0.015s	sys 0.007s
I20260709 12:23:59.634166 23881 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:59.635430 23881 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.638031 23872 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.638785 23872 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.639132 23876 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:59.640753 23725 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:59.634325 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:59.641772 23725 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.647379 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:23:59.649252 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.649705 18022 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:59.649493 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.653956 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.655035 18015 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:23:59.656529 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:23:59.656694 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 125748us
I20260709 12:23:59.658008 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:59.658329 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:59.658517 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 748) took 1836 ms. Trace:
I20260709 12:23:59.658773 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:59.659257 23856 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:59.659554 23803 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:59.659978 18021 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:59.659893 23856 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:23:59.660514 23856 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:59.661242 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.662511 18021 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:23:59.663632 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.681066 23854 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:59.681713 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 104401us
I20260709 12:23:59.665436 23803 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:59.689531 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.696419 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:23:59.682013 23855 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:59.702617 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.658984 18020 rpcz_store.cc:276] 0709 12:23:57.822404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.729072 (+906668us) service_pool.cc:224] Handling call
0709 12:23:58.729602 (+   530us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.730664 (+  1062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.730694 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:58.731414 (+   720us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.539536 (+808122us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:59.578396 (+ 38860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.589351 (+ 10955us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.658162 (+ 68811us) raft_consensus.cc:1685] finished
0709 12:23:59.658183 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.658410 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1212,"raft.queue_time_us":1048,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5061,"spinlock_wait_cycles":1355648,"thread_start_us":741443,"threads_started":1,"wal-append.queue_time_us":912255}
I20260709 12:23:59.703629 18015 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:59.696730 23803 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:59.718215 18021 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:59.720225 18021 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:59.722157 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:23:59.723443 18023 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:23:59.730423 23872 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:59.732424 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.769424 18015 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:59.771261 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:23:59.773721 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 195420us
I20260709 12:23:59.774888 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:59.777135 23881 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:59.777618 23804 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.779268 23804 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:59.780872 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.781482 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.782421 18019 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.006s
I20260709 12:23:59.782805 18019 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:23:59.785658 18016 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
W20260709 12:23:59.793274 23457 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.504s	user 0.009s	sys 0.006s
I20260709 12:23:59.794003 23725 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:23:59.809193 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:23:59.806565 23457 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:59.817108 23457 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:23:59.820130 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:23:59.821368 18811 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:59.822016 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:23:59.822211 23872 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:59.823045 23872 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:23:59.777657 23312 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.720s	user 0.015s	sys 0.010s
I20260709 12:23:59.823062 18095 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14. Re-sending same reply.
I20260709 12:23:59.824591 23872 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:59.834565 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3440) took 2222 ms. Trace:
I20260709 12:23:59.835326 18094 rpcz_store.cc:276] 0709 12:23:57.612459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.735660 (+1123201us) service_pool.cc:224] Handling call
0709 12:23:58.751213 (+ 15553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.751485 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:59.609075 (+857590us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.806958 (+197883us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.811554 (+  4596us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:59.833959 (+ 22405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.834152 (+   193us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.834430 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":863902,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":12081,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":946374,"threads_started":2,"wal-append.queue_time_us":217295}
W20260709 12:23:59.839221 23713 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.806s	user 0.016s	sys 0.007s
I20260709 12:23:59.839850 23713 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.796283 23854 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.866979 23854 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:23:59.878047 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 231792us
I20260709 12:23:59.882696 18093 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:23:59.886799 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 161066us
I20260709 12:23:59.813438 23881 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.900844 18022 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:23:59.907789 23853 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.911273 23871 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.913932 23851 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:23:59.918381 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 141847us
I20260709 12:23:59.894837 23720 raft_consensus.cc:2764] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:59.870461 23312 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.917555 23713 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:23:59.934736 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 230494us
W20260709 12:23:59.936878 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 197812us
W20260709 12:23:59.937695 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.944341 23883 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.052s	user 0.002s	sys 0.000s
I20260709 12:23:59.945886 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:23:59.947768 23852 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.948518 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:23:59.950193 23887 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:59.950693 23887 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:23:59.951092 23887 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:59.956010 23852 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:59.956957 23852 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.962911 23872 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.963863 23871 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:23:59.963724 23872 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:23:59.973029 18811 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:23:59.983562 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:23:59.957557 23852 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:23:59.921706 23881 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.005417 18169 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:00.005234 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:23:59.995956 23865 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:00.014331 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 750) took 2186 ms. Trace:
I20260709 12:24:00.017563 18018 rpcz_store.cc:276] 0709 12:23:57.823011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.771231 (+948220us) service_pool.cc:224] Handling call
0709 12:23:58.771680 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.651730 (+880050us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.664717 (+ 12987us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.682552 (+ 17835us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.771027 (+ 88475us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:59.771088 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.771510 (+   422us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.001509 (+229999us) raft_consensus.cc:1685] finished
0709 12:24:00.001541 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.009618 (+  8077us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48544,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":5018,"thread_start_us":860318,"threads_started":2,"wal-append.queue_time_us":183988}
I20260709 12:24:00.005288 23716 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.023545 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:00.026446 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107966us
I20260709 12:24:00.013401 23872 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.030186 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.031134 18171 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:00.032467 23879 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:00.033269 23879 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:00.038548 18172 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:00.038743 23854 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.022984 23725 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 12:24:00.040490 23856 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.043331 23881 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.005419 17916 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 7, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 7 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:00.052603 23855 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.053238 23855 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.053617 23855 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.053692 23720 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:00.059314 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.059790 23856 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:00.061161 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:00.061785 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2628) took 1464 ms. Trace:
I20260709 12:24:00.062418 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.062704 23881 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:00.063809 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.064646 18815 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:00.066433 18813 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:00.068075 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2556) took 2044 ms. Trace:
I20260709 12:24:00.067986 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:00.069187 18807 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:00.070809 18809 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:24:00.071137 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 442892us
I20260709 12:24:00.063407 23881 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:00.079324 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 456057us
I20260709 12:24:00.082279 23851 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.083371 23851 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.084172 23851 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.028064 23865 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:00.087883 23865 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.059430 23762 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:24:00.091734 23713 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.092427 23713 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.094717 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:00.095819 23469 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:24:00.096450 23469 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:00.070264 23558 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:24:00.097807 23558 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:00.098407 23558 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:00.098714 18165 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:00.099817 23558 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.032346 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.104027 18168 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:00.107920 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.113593 23457 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:00.109050 23872 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:00.111373 18808 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:00.101441 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.118765 23854 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:00.119791 23854 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:00.095964 23879 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.124679 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.097581 23469 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:00.132292 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72692.4us
I20260709 12:24:00.102113 18169 rpcz_store.cc:276] 0709 12:23:58.596949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.816207 (+219258us) service_pool.cc:224] Handling call
0709 12:24:00.061482 (+1245275us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:00.061689 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606694272}
I20260709 12:24:00.141623 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.149590 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.150924 18168 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:00.147336 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65573us
I20260709 12:24:00.155324 18811 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:00.048846 17916 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 11, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 11 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:00.159360 17916 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 12, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 12 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:00.161262 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.155853 23558 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:00.163961 23257 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.164471 23257 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.164752 23720 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:00.165200 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.165428 23720 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:00.165658 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93740.6us
I20260709 12:24:00.166719 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.166805 23720 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.167557 18815 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:00.168298 23720 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.169198 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.169845 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:00.170230 18168 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:00.170974 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.171200 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.182569 23876 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:00.172510 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.172334 23664 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:00.173457 23762 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:00.122797 23854 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.068660 18017 rpcz_store.cc:276] 0709 12:23:58.024029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.864724 (+840695us) service_pool.cc:224] Handling call
0709 12:23:58.897028 (+ 32304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.897648 (+   620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.897697 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:58.898433 (+   736us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.508811 (+610378us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:59.557335 (+ 48524us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.067289 (+509954us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b180048a360. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:00.067553 (+   264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.067669 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.067717 (+    48us) raft_consensus.cc:1685] finished
0709 12:24:00.067827 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.068006 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36544,"prepare.queue_time_us":68741,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":152,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":7177,"spinlock_wait_cycles":34632576,"thread_start_us":586241,"threads_started":2,"wal-append.queue_time_us":647598}
I20260709 12:24:00.179026 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.164872 23257 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.181032 18808 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:00.178233 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.173194 23853 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.168735 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:00.162961 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.189862 18809 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:00.188982 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90268us
I20260709 12:24:00.195649 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:00.198199 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.184216 23312 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.183064 23876 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.200511 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:00.201737 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.205302 18811 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:00.207193 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.209004 17916 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:00.210117 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:00.211176 17916 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:24:00.214191 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.646s (SENT)
I20260709 12:24:00.214651 23853 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:00.215969 23663 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:00.216498 23663 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:00.218447 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.219285 18169 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:00.220202 23717 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.221019 23717 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.225023 18093 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:24:00.226302 23852 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.226899 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.227777 23713 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.228262 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.229017 18095 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:00.230114 23856 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.230770 23856 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.231936 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.232782 18090 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.234287 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.235142 18096 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.234124 23664 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:00.235786 23856 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.217656 23663 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.238479 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.238698 23663 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:00.213116 17916 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 4 to 6, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 6 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:00.205761 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.238329 18806 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
W20260709 12:24:00.202078 18780 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.640s (SENT)
I20260709 12:24:00.227825 18094 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:24:00.250348 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.255225 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59138.2us
I20260709 12:24:00.206660 23762 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:00.205967 23876 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.263209 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.266599 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.269285 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.273838 18812 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:00.277990 18095 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:00.280623 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.279959 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.307260 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.264881 23762 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.255990 23782 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:00.282464 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:00.306371 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 109492us
I20260709 12:24:00.310634 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.306967 18093 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 10.
W20260709 12:24:00.313380 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.317919 23852 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:00.318501 23852 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:00.318966 23852 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:00.313670 23762 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:00.327972 23855 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.330902 23855 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:00.331987 18095 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:24:00.338006 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.339062 23855 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.339728 23855 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.342156 23782 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:00.343096 23852 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.343417 23782 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.307384 18098 raft_consensus.cc:2376] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11. Re-sending same reply.
I20260709 12:24:00.240254 17916 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.353911 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.354866 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:00.355214 17916 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.357219 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.359153 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.360018 18098 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:00.306061 23457 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:00.366515 23457 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.369171 23457 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:00.311394 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.389745 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.392094 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:00.395241 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 122386us
I20260709 12:24:00.312835 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:00.403951 18096 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:24:00.407320 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 141937us
W20260709 12:24:00.407183 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 185374us
I20260709 12:24:00.404600 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.411409 18090 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:00.423899 18094 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:00.427140 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.312266 23717 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:00.432796 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:00.433614 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.387251 23852 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.435730 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:00.435811 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 169959us
I20260709 12:24:00.437774 18095 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:24:00.439663 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.315021 23312 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260709 12:24:00.357566 17916 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 15, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:24:00.445596 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 128550us
I20260709 12:24:00.444878 18169 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.417346 23713 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:00.433895 23804 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.654s	user 0.015s	sys 0.006s
I20260709 12:24:00.424221 23664 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:00.444048 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.425836 23872 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:24:00.436578 23803 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.691s	user 0.016s	sys 0.000s
I20260709 12:24:00.462064 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.462366 23866 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:00.462914 18097 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:00.463735 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:00.464504 18806 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:00.465585 23886 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:00.465929 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.466744 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:00.466967 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.467186 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.467859 18167 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:00.467926 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.468982 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.469208 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:00.447605 23855 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.469374 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.469864 18166 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:00.470060 23881 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.470398 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.480471 18172 raft_consensus.cc:2376] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17. Re-sending same reply.
I20260709 12:24:00.481211 18171 raft_consensus.cc:2368] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14.
I20260709 12:24:00.472015 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:00.481906 17916 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 3 to 10, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 10 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.483398 23804 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:00.483872 23804 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.484324 23804 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.486161 18099 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:24:00.473655 23610 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:24:00.474938 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51127.4us
I20260709 12:24:00.490861 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:00.492666 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:00.423697 23854 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:24:00.447260 23717 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:24:00.495836 23717 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:00.496724 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.497882 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.498047 23716 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:00.498327 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.498683 18093 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:00.498987 18091 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:24:00.499753 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.499881 23716 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:00.500408 18091 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:24:00.500842 18091 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:24:00.500902 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.462955 23866 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.445315 23853 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:00.500443 18093 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:24:00.503368 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59488.4us
I20260709 12:24:00.501209 18091 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:24:00.510943 18091 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:00.460076 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:00.512462 23897 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:00.512753 23787 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:00.513316 23886 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:00.513557 23257 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.514650 23886 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.516366 23257 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:00.516535 23868 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:00.518872 23903 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:24:00.459313 23664 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.520102 23664 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.504844 23782 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:00.447474 23713 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.504098 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.534137 23866 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.461512 23872 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.553467 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.553153 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.557554 18095 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:24:00.562165 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.563335 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.564196 18090 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:00.564983 18099 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:00.569061 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 65051.4us
I20260709 12:24:00.563473 23881 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.572695 23881 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.573240 23897 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:00.574520 23897 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.575487 23881 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.576689 18097 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:00.577231 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74587.6us
I20260709 12:24:00.554402 23713 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.565632 23872 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:00.582598 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.585394 23853 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:00.503146 23803 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.559733 23868 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:00.592274 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:00.594467 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53763.7us
I20260709 12:24:00.585871 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:00.533574 23469 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:24:00.602597 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.607507 18098 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:00.612813 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120282us
I20260709 12:24:00.619971 23716 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:00.620473 18810 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:00.615152 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:00.621944 18092 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:00.631781 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:00.595480 23717 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:00.634966 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:00.609009 23457 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:00.609220 23782 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:00.484114 17916 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 9, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 9 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.649101 18099 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
W20260709 12:24:00.641887 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59453.2us
I20260709 12:24:00.615003 23896 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.666417 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.640672 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.668692 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:00.669577 18022 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:24:00.601128 23879 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:24:00.603544 23852 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:24:00.664280 18099 pending_rounds.cc:85] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 6
I20260709 12:24:00.684985 18099 pending_rounds.cc:107] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:24:00.685350 18099 raft_consensus.cc:2889] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:00.665279 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.691973 18169 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:00.677793 23787 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:00.703413 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 117636us
I20260709 12:24:00.648104 23717 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:00.679497 23887 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.728s	user 0.016s	sys 0.003s
I20260709 12:24:00.705149 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:00.666203 23896 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.670015 23782 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.710440 17916 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 9, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 9 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.711660 23762 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.721690 23854 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:00.721922 23717 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:00.671959 23855 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:24:00.728760 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.731907 18092 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:00.749305 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114425us
I20260709 12:24:00.756304 18020 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:00.758250 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.759526 23762 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.762316 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.763115 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.782609 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:00.731585 23896 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.750006 23854 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:00.787405 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76821.1us
I20260709 12:24:00.731366 23782 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:00.745635 23716 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.781863 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.722685 23887 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.790988 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.792795 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:00.763311 23663 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.008s
I20260709 12:24:00.722687 23853 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.792502 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.805648 18019 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:00.796744 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.808327 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.809698 18020 raft_consensus.cc:2410] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:24:00.812448 18024 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:00.816220 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.822916 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:00.796876 23716 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:00.839584 23716 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:00.807197 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 166601us
I20260709 12:24:00.841490 18811 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:00.844190 18015 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:00.842419 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137417us
W20260709 12:24:00.846541 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45444.9us
I20260709 12:24:00.807897 23720 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:00.855819 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 209721us
I20260709 12:24:00.857023 18021 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:00.792469 23558 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:24:00.859809 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.861976 23854 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.862874 18780 leader_election.cc:400] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14.
I20260709 12:24:00.869608 18780 leader_election.cc:403] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:24:00.869673 18020 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:00.871281 23898 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:00.873099 23915 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:24:00.843541 17916 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 11, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 11 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.880079 17916 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 11, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 11 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.878367 23717 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.857913 23610 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:24:00.883726 23717 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:00.861229 23803 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:00.886598 23917 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:24:00.888019 23558 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:00.888518 23558 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:00.856534 23787 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:00.891047 23887 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.891850 23897 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.891652 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:00.892591 23897 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.746376 23868 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.895260 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:00.896070 18015 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.852487 23663 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:00.905661 18016 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:24:00.859333 23720 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:00.908699 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.914146 18094 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:00.905953 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.917667 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:00.918746 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56099.9us
I20260709 12:24:00.918733 18016 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:00.921511 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:00.922665 18024 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:00.895205 23897 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:00.951308 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.950486 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:00.927608 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.950577 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:00.955574 18019 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:00.957322 18095 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:00.903496 23663 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.960147 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:00.961115 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.889467 23558 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.978770 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.972095 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1046) took 1010 ms. Trace:
W20260709 12:24:00.972049 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.974426 18017 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:00.973516 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:00.981851 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:00.952775 23879 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:00.984236 18094 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:00.979977 18810 rpcz_store.cc:276] 0709 12:23:59.949685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.967401 (+ 17716us) service_pool.cc:224] Handling call
0709 12:23:59.978544 (+ 11143us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.621002 (+642458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.959850 (+338848us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:00.960097 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1463298816}
I20260709 12:24:00.916848 23852 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.982569 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:00.962677 23876 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.686s	user 0.016s	sys 0.006s
I20260709 12:24:00.950881 23787 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.992347 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:00.965893 23663 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.994829 23720 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:00.998034 23927 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:00.986405 17916 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:00.999619 23312 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:00.999971 18098 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:24:01.000172 23312 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:01.000336 17916 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:01.003381 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:01.005162 23804 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.520s	user 0.014s	sys 0.000s
I20260709 12:24:01.005646 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:01.005734 23804 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.007359 18092 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.14-14.14]
I20260709 12:24:01.007507 18016 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:00.917096 23664 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.965299 23897 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:00.909519 23713 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.018252 23852 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.029109 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:01.007313 23879 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:01.030872 23919 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:01.031703 23713 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:01.032212 23879 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:00.989738 23558 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:01.033975 23803 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:01.033742 23610 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:01.034601 23803 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:01.034826 23610 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:01.035230 23610 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:01.035586 23921 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:01.035778 23803 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.036149 23610 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.037092 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.038219 23897 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:01.039251 23897 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.033399 23851 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:00.921237 23887 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14.
I20260709 12:24:01.008833 23927 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:01.041731 23927 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:01.041886 23922 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.202s	user 0.003s	sys 0.000s
I20260709 12:24:01.009196 23787 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:01.042682 23787 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.042265 23927 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.012128 23804 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:01.015249 23782 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.049847 23782 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:01.024915 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.001327 23312 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.012421 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:01.051437 23312 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.052795 23782 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:01.050074 18018 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:01.055032 23782 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:01.055605 23782 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:01.026181 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46502.3us
W20260709 12:24:01.058718 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46417.3us
I20260709 12:24:01.028796 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:01.068840 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.083046 18015 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:24:01.087687 18015 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:24:01.088235 18015 pending_rounds.cc:85] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 5
I20260709 12:24:01.088481 18015 pending_rounds.cc:107] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:24:01.088725 18015 raft_consensus.cc:2889] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:01.083920 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:01.094831 23852 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:01.095505 18812 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:01.095599 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:01.097007 18022 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:01.094873 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65898.4us
I20260709 12:24:01.100720 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:00.872138 23898 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.102591 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:01.103365 18811 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:24:01.104714 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:01.106041 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 283271us
I20260709 12:24:01.107331 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:01.065011 23610 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:01.108970 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.109635 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.108393 23868 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:01.007680 23876 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.130427 18806 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:01.130789 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.135000 18024 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:01.134888 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:01.145800 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:01.053491 17916 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: config changed from index 3 to 9, term changed from 11 to 13, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to <none>, 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) changed from NON_VOTER to VOTER. New cstate: current_term: 13 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:24:01.154383 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64537.5us
W20260709 12:24:01.166070 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.140906 23872 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.532s	user 0.006s	sys 0.007s
W20260709 12:24:01.208770 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 101471us
W20260709 12:24:01.269619 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.348968 17916 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 10, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 10 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:01.351594 23931 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:01.342839 18810 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:01.343060 23664 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:01.343879 23866 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.771s	user 0.014s	sys 0.020s
I20260709 12:24:01.153842 23905 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:01.154870 23924 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:01.131170 23469 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
W20260709 12:24:01.346319 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 336029us
I20260709 12:24:01.347575 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.351500 17916 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 7, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:01.352169 23787 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:01.370527 23716 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.512s	user 0.013s	sys 0.006s
I20260709 12:24:01.355638 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:01.398802 18807 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:01.340651 23876 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:01.359037 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:01.359953 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 217060us
I20260709 12:24:01.360213 18024 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:01.369619 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:01.370820 18812 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:01.374015 18811 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:01.418072 17916 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 10, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 10 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:01.420264 17916 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 10, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 10 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:01.390496 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:01.369979 23896 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.558s	user 0.013s	sys 0.013s
I20260709 12:24:01.354274 23872 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:01.396965 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.403218 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:01.403775 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.382313 23924 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:01.438272 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92304.8us
W20260709 12:24:01.448701 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.368991 23725 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
W20260709 12:24:01.456211 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 111001us
I20260709 12:24:01.457381 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:01.381547 23866 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.466974 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.428411 23905 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:01.453631 23935 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.079s	user 0.002s	sys 0.000s
I20260709 12:24:01.445699 23931 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:01.471414 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:01.471913 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:01.446898 23924 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:01.518338 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69730.7us
W20260709 12:24:01.519230 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.519631 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:01.522171 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:01.447988 23876 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.537581 18023 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:01.551484 18098 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:24:01.555014 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:01.556159 18169 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:01.556830 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.559295 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:01.494314 18813 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:01.494268 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.514496 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:01.515406 23664 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:01.516189 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:01.516822 23905 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.570196 18808 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:01.523036 23558 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:01.536731 23866 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:01.530959 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:01.582603 18092 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:01.424692 23716 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.537786 23876 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:01.484588 23930 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260709 12:24:01.602092 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62379.6us
I20260709 12:24:01.540567 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.542621 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.612497 18016 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:01.546429 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.613904 18019 raft_consensus.cc:2376] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16. Re-sending same reply.
I20260709 12:24:01.548274 23898 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:01.550665 23927 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.506s	user 0.011s	sys 0.004s
I20260709 12:24:01.559978 23919 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:01.492161 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:01.483191 23872 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:01.494942 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:01.563628 18813 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:01.632423 23927 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.633104 18813 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.634372 18813 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:01.634135 23921 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:01.635296 23921 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:01.635706 23921 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:01.636952 23921 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.638599 23921 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:01.496801 18812 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:01.572901 18095 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:24:01.573411 18024 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:24:01.578461 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 112292us
I20260709 12:24:01.489010 23663 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.443297 23896 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.583422 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3367) took 1001 ms. Trace:
I20260709 12:24:01.648154 18018 rpcz_store.cc:276] 0709 12:24:00.583473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.591874 (+  8401us) service_pool.cc:224] Handling call
0709 12:24:01.578937 (+987063us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:01.583313 (+  4376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1035488000}
I20260709 12:24:01.583509 18172 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:24:01.648792 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.584510 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.586277 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:01.650336 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.651098 18019 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:01.597559 18808 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:01.570946 23782 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.515s	user 0.016s	sys 0.001s
I20260709 12:24:01.565747 23931 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.539507 23855 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:01.604732 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:01.606618 18806 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:24:01.609330 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:01.617933 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:01.601094 23897 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:01.474377 23725 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:01.627641 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.627477 23716 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:01.659281 18091 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:01.629377 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:01.668408 18099 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:01.642151 23804 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:01.649804 18024 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:24:01.623271 18092 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:01.657575 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.680454 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.662918 18169 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:01.681241 18016 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:01.666110 18095 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:01.587153 23457 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:01.667652 18021 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:01.642781 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2694) took 1117 ms. Trace:
W20260709 12:24:01.674757 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 83210.1us
I20260709 12:24:01.676136 18098 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:01.659126 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.681574 18022 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:01.683888 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:01.688333 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.686124 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.689499 18024 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:01.695433 18018 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:01.659162 23931 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:01.658869 23855 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:01.711076 18096 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:24:01.711604 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111081us
W20260709 12:24:01.710984 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 297230us
I20260709 12:24:01.722402 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:01.723472 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.727370 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:01.676759 23804 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:01.688176 18093 rpcz_store.cc:276] 0709 12:24:00.525546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.564650 (+ 39104us) service_pool.cc:224] Handling call
0709 12:24:00.579884 (+ 15234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.591043 (+ 11159us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:00.619754 (+ 28711us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.409606 (+789852us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b180039ae40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:01.642332 (+232726us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:01.642404 (+    72us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:01.642467 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.642551 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.642700 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50940,"prepare.queue_time_us":58272,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":32839,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":4138,"spinlock_wait_cycles":6956416,"thread_start_us":283591,"threads_started":2,"wal-append.queue_time_us":120926}
I20260709 12:24:01.736089 23853 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:01.738334 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.740085 23876 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:01.740764 23925 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:01.787281 23925 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:01.788822 23925 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.750027 23762 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:01.754041 23924 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.756254 18024 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:01.814039 18807 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:01.815716 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.687144 23898 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:01.760943 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.667680 23872 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:24:01.658972 23782 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.722344 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:01.790055 23902 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.790503 23933 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:24:01.657178 23886 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260709 12:24:01.797879 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.702672 23854 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:01.802553 18022 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:01.471688 17916 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 12, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 12 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:01.814463 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.822846 23762 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:01.694525 23897 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.820739 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:01.817478 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:01.814553 23457 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:01.874352 18024 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:01.882987 18016 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:01.888175 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:01.889313 18022 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12. Re-sending same reply.
W20260709 12:24:01.897511 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 204140us
W20260709 12:24:01.907485 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 185317us
I20260709 12:24:01.860996 23925 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.856115 23898 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:01.910208 18813 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
W20260709 12:24:01.908020 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 87452.9us
I20260709 12:24:01.919034 18099 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11.
I20260709 12:24:01.872454 23854 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:01.876362 23457 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:01.876845 23940 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:01.937114 18813 pending_rounds.cc:85] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 10
I20260709 12:24:01.854836 23902 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:01.825769 23804 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.911280 23762 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:01.906580 23855 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.005393 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3472) took 1366 ms. Trace:
I20260709 12:24:02.175871 18097 rpcz_store.cc:276] 0709 12:24:00.639201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.639381 (+   180us) service_pool.cc:224] Handling call
0709 12:24:00.666765 (+ 27384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.726626 (+ 59861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.726672 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.761211 (+ 34539us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.525761 (+764550us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b1800427720. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:01.697785 (+172024us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:01.697865 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.697938 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.992874 (+294936us) raft_consensus.cc:1685] finished
0709 12:24:01.992913 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.005233 (+ 12320us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16722,"raft.queue_time_us":58174,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":31733,"spinlock_wait_cycles":7938304,"thread_start_us":142484,"threads_started":1,"wal-append.queue_time_us":1166371}
I20260709 12:24:01.992084 23257 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.983619 23854 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:02.188745 18091 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.002s
I20260709 12:24:02.189286 18091 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:24:02.127269 23947 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:24:02.191620 23804 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:02.095126 23927 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.181536 18813 pending_rounds.cc:107] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:24:02.194675 23855 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.049551 23896 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:02.037216 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.951498 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 217143us
I20260709 12:24:02.207545 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:01.996457 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:01.875999 17916 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:02.195421 18813 raft_consensus.cc:2889] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:01.960940 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:02.048951 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.978513 23879 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.185946 23257 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:02.079442 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100561us
I20260709 12:24:01.925993 23558 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:02.248805 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.931234 23905 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:02.252090 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.524s (SENT)
W20260709 12:24:02.253572 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46136.3us
I20260709 12:24:02.167078 23782 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:02.174538 23787 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.228266 23956 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:02.265060 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.255893 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.256973 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:02.274206 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.258328 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 276915us
W20260709 12:24:02.258791 23954 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:24:02.257853 17916 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.229247 23879 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:02.259917 23889 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:24:02.218045 23927 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:02.288570 23927 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.033543 23898 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.262207 23897 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:02.279524 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.287810 23879 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:02.299438 18024 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:02.300259 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48238.7us
I20260709 12:24:02.296279 23854 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:02.292433 23924 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.303570 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.305298 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.306180 18016 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:24:02.275246 23956 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:02.307734 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.279172 23952 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.316809 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3383) took 1449 ms. Trace:
W20260709 12:24:02.317513 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.317777 18021 rpcz_store.cc:276] 0709 12:24:00.867298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.867490 (+   192us) service_pool.cc:224] Handling call
0709 12:24:02.316520 (+1449030us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b540022f954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:02.316706 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1736989056}
I20260709 12:24:02.311405 23558 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:02.322891 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.323684 23905 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:02.313473 23956 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.330224 18171 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:02.309077 23924 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:02.309355 23856 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:02.297971 23457 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.323455 23558 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.321213 23927 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:02.319816 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.324944 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 910f8c8d787e4abbb092413c1f90ebf3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:02.349555 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.350390 18171 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:02.350140 23720 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:02.350986 23257 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:02.351465 23469 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:02.325927 18165 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 12:24:02.326267 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.325029 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:02.366225 23923 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:02.367242 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:02.367983 18016 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:24:02.378000 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55209.6us
I20260709 12:24:02.383359 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:02.286818 17916 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 10, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 10 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:02.398572 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.399489 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:02.400326 18020 raft_consensus.cc:2376] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 11. Re-sending same reply.
I20260709 12:24:02.400503 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:02.399760 17916 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 11, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 11 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.401294 18019 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:02.401769 18023 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:02.381605 23927 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:02.402238 18023 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:02.403224 18023 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.404234 18023 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.406349 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:02.407152 23957 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.209s	user 0.002s	sys 0.000s
I20260709 12:24:02.406272 23856 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.409170 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:02.410830 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:02.411463 23953 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 12:24:02.385946 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:02.413640 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2695) took 1887 ms. Trace:
I20260709 12:24:02.414234 18022 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.414316 18090 rpcz_store.cc:276] 0709 12:24:00.525916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.572297 (+ 46381us) service_pool.cc:224] Handling call
0709 12:24:00.572745 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.605386 (+ 32641us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:01.482108 (+876722us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b1800311080. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca557 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:01.784072 (+301964us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b1800311080. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b
0709 12:24:01.784263 (+   191us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.979079 (+194816us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.983976 (+  4897us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.984276 (+   300us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:01.984313 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.984443 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.410529 (+426086us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:02.413377 (+  2848us) raft_consensus.cc:1685] finished
0709 12:24:02.413405 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.413558 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73562,"prepare.queue_time_us":102375,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":41979,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":22545,"spinlock_wait_cycles":29851008,"thread_start_us":276464,"threads_started":2,"wal-append.queue_time_us":426096}
I20260709 12:24:02.418087 18018 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:02.411665 18169 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.391928 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.392855 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:02.422417 18165 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:02.394114 23924 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:24:02.423940 23958 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.212s	user 0.000s	sys 0.003s
I20260709 12:24:02.424238 23924 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:02.424930 23924 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:02.426909 23940 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:02.409198 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:02.429492 18094 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:02.389662 23955 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.207s	user 0.004s	sys 0.000s
W20260709 12:24:02.430797 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.394850 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.631s (SENT)
I20260709 12:24:02.430092 23924 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.397990 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:02.433574 18021 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:02.369585 23923 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:02.399586 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:02.432597 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.693s (SENT)
I20260709 12:24:02.434749 23923 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.435341 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.435998 23923 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.436492 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.437354 18165 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:02.398183 18024 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:24:02.437743 18806 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:02.438714 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:02.438828 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:02.439548 18165 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:02.439512 18808 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.441264 23902 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.441417 18020 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:02.442790 18165 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:02.443075 18168 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:02.386827 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:02.444238 18813 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:02.446221 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.446869 18166 raft_consensus.cc:2376] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16. Re-sending same reply.
W20260709 12:24:02.447641 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.447355 23956 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.449235 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:02.450718 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:02.452374 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:02.453069 23921 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:02.453240 18812 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:24:02.453708 18812 pending_rounds.cc:85] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 10
I20260709 12:24:02.454061 18812 pending_rounds.cc:107] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:24:02.454296 23921 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:02.454483 18812 raft_consensus.cc:2889] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:02.455402 23921 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.456077 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:02.452260 23896 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:02.423969 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.453770 23852 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:24:02.436833 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.501s (SENT)
I20260709 12:24:02.464107 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:02.472544 18171 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:02.473315 18092 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 16 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260709 12:24:02.475203 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.478014 23956 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.487308 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:02.452299 23898 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:02.479008 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.424254 18022 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:02.482597 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:02.483095 18808 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:02.483289 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:02.483088 18094 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:02.483876 18097 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:02.485062 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:02.532089 18098 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:02.534049 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.534983 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:02.550468 18172 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:02.553979 18093 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:02.470288 23905 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:02.566423 18167 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:02.568254 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 840) took 1006 ms. Trace:
I20260709 12:24:02.571605 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.572642 18168 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:02.579285 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:02.580844 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157025us
I20260709 12:24:02.487056 23852 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:02.582818 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 103947us
I20260709 12:24:02.532755 23902 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.550554 23853 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:02.591535 18169 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:02.558240 23855 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:02.577765 23803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.158s	user 0.004s	sys 0.028s
I20260709 12:24:02.568591 23962 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.570562 23257 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.568755 18170 rpcz_store.cc:276] 0709 12:24:01.561442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.561620 (+   178us) service_pool.cc:224] Handling call
0709 12:24:01.562140 (+   520us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:01.562802 (+   662us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:01.842137 (+279335us) log.cc:844] Serialized 14 byte log entry
0709 12:24:02.174732 (+332595us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:02.229924 (+ 55192us) log.cc:844] Serialized 52 byte log entry
0709 12:24:02.251099 (+ 21175us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.255786 (+  4687us) log.cc:844] Serialized 14 byte log entry
0709 12:24:02.282829 (+ 27043us) log.cc:844] Serialized 14 byte log entry
0709 12:24:02.283196 (+   367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.283327 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.395987 (+112660us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b18003c2200. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:02.423039 (+ 27052us) spinlock_profiling.cc:243] Waited 6.22 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:02.427823 (+  4784us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b48000cd158. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:24:02.567947 (+140124us) raft_consensus.cc:1685] finished
0709 12:24:02.567983 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.568175 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58774,"prepare.queue_time_us":640595,"prepare.run_cpu_time_us":985,"prepare.run_wall_time_us":4734,"raft.queue_time_us":209148,"raft.run_cpu_time_us":3307,"raft.run_wall_time_us":169382,"spinlock_wait_cycles":34235648,"thread_start_us":412453,"threads_started":2,"wal-append.queue_time_us":333647}
I20260709 12:24:02.567425 23886 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:02.599707 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.669s (SENT) [suppressed 55 similar messages]
I20260709 12:24:02.575551 23919 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:02.573774 23663 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:02.601305 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:02.600160 18098 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:24:02.603127 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 202216us
I20260709 12:24:02.491164 23866 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 12:24:02.605746 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:02.611181 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.625957 18098 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:02.626166 18171 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.628400 18093 raft_consensus.cc:2368] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14.
I20260709 12:24:02.625177 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:02.628027 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.629691 18168 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:02.602739 23803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.183s	user 0.005s	sys 0.028s
I20260709 12:24:02.648826 18171 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:02.653569 18172 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:02.604481 23782 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.660166 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:02.667068 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60287.5us
I20260709 12:24:02.432727 17916 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 10, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 10 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:02.674186 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.650453 23898 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:24:02.713027 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.774364 18172 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:02.622110 23257 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.650405 23558 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:02.758605 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 157502us
I20260709 12:24:02.623040 23905 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:24:02.624228 23886 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:02.761459 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.630362 23855 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:02.769734 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.777647 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:02.796219 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.789163 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:02.785866 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.622270 23962 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.787699 23961 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.310s
I20260709 12:24:02.777971 23803 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:02.773518 23762 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:02.789412 23925 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.874s	user 0.010s	sys 0.004s
I20260709 12:24:02.668550 23782 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.785346 23610 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.810141 23610 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.811472 23782 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.664902 23889 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.131s
I20260709 12:24:02.807701 18172 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:24:02.795539 23944 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260709 12:24:02.814914 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.788160 23898 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:02.789686 23919 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:02.822038 18090 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:24:02.784751 23933 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
W20260709 12:24:02.831872 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 157839us
W20260709 12:24:02.834631 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45582.8us
I20260709 12:24:02.835649 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:02.839015 18808 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:02.842257 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:02.845307 18023 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
W20260709 12:24:02.846272 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.851845 18023 pending_rounds.cc:85] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 6
I20260709 12:24:02.853299 18023 pending_rounds.cc:107] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:24:02.855020 18023 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:02.760514 23852 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.819451 23881 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:02.801800 23902 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:24:02.862876 23919 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.846043 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.823474 23803 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:02.873097 18172 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.798105 23879 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
W20260709 12:24:02.880019 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 220004us
W20260709 12:24:02.883245 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.884740 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45556.3us
I20260709 12:24:02.812778 23924 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:24:02.893075 18812 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:02.873994 23782 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:02.896301 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:02.900103 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.812754 23762 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.901240 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.901237 18170 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.904248 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:02.905028 18811 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:02.906838 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:02.915084 18808 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:02.916056 18808 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:02.914743 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
W20260709 12:24:02.922691 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49082.9us
I20260709 12:24:02.894955 23312 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:02.925020 18094 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 12:24:02.906359 23974 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:24:02.889969 23716 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:24:02.929744 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:02.900053 23782 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:02.955102 23782 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:02.921983 18808 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.929657 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:02.908900 23610 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:02.950738 18097 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:02.931010 23881 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:02.932353 23933 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.921198 23925 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:24:02.986109 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:02.986995 18168 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:02.987150 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57585.3us
I20260709 12:24:02.976311 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:02.974588 18098 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:02.991650 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:02.983245 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:02.970187 18808 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:02.988160 23973 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:24:02.928584 23889 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:24:03.006807 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.007133 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.007879 18093 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:02.999382 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:02.969058 23970 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260709 12:24:02.968400 23978 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:24:03.003870 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.020722 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:03.022665 23984 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:24:02.974911 23257 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:02.975751 23983 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:03.023993 23983 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.024423 23983 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:03.026341 23881 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:02.988888 23933 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:03.031036 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.032359 23933 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:02.998871 23558 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:03.009310 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114462us
I20260709 12:24:02.898391 17916 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:03.014853 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:02.959641 23803 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:02.976537 23964 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:03.016460 23965 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:24:03.007745 23855 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:03.018383 23981 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:24:03.022800 18171 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:03.043148 18092 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:03.034166 18808 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:24:03.008937 23902 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.066517 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:03.080993 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.928833 23898 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.082255 18167 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:03.033766 23854 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.731s	user 0.010s	sys 0.001s
I20260709 12:24:03.104460 18022 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:03.033890 23558 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:03.110950 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63162.2us
I20260709 12:24:03.034646 23886 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:03.046276 23964 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.131035 23964 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:03.131867 23762 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:03.134521 23986 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:03.142722 23968 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.486s	user 0.005s	sys 0.000s
I20260709 12:24:03.143847 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.145475 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.146967 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.050841 23610 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.097s
I20260709 12:24:03.039494 23725 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:03.150751 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.127040 23962 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:03.126809 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 118023us
W20260709 12:24:03.163388 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.992s (SENT)
I20260709 12:24:03.164424 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:03.171139 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:03.112839 23931 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.330s	user 0.014s	sys 0.000s
W20260709 12:24:03.178037 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59606.7us
I20260709 12:24:03.172253 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:03.183308 17987 leader_election.cc:400] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14.
I20260709 12:24:03.184027 17987 leader_election.cc:403] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:24:03.199829 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:03.202755 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.113809 23854 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:03.150502 23558 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:03.111575 23257 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:03.229671 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.230876 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.201357 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:03.235596 18807 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:03.073567 23879 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.109s
I20260709 12:24:03.240028 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.257706 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.242389 18170 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:03.191154 18172 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:24:03.281032 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:03.191620 18092 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:24:03.248835 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85514.8us
I20260709 12:24:03.281934 18808 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:03.159792 23725 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:03.179422 23610 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:24:03.236919 23720 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.288995 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:03.289775 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.236557 23889 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:03.290640 18095 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:03.271878 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:03.275395 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.248076 23961 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:03.278381 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77152.1us
I20260709 12:24:03.255594 23866 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14.
W20260709 12:24:03.269461 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.294268 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111206us
I20260709 12:24:03.296204 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:03.160151 23782 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.277482 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.201236 23962 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:24:03.298193 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:03.087945 23902 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.316642 18808 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:03.310072 18093 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:03.311769 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:03.313090 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.286921 23876 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:03.316001 18811 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:03.202025 23931 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:03.323762 18171 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:24:03.316066 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.042776 17916 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 8, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:03.344888 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:03.336651 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.364427 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:03.302625 23803 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:03.297711 23896 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.380080 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.306053 23852 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:03.389101 18015 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:03.381065 18170 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:03.264842 23257 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.385293 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:03.380414 18172 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:03.371080 23923 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.935s	user 0.017s	sys 0.002s
I20260709 12:24:03.325416 23993 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:03.370136 23803 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:03.387727 23989 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:24:03.308782 23720 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:03.467069 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 81884.7us
W20260709 12:24:03.467453 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 162671us
I20260709 12:24:03.309870 23961 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:03.518669 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 172148us
I20260709 12:24:03.533058 18090 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
W20260709 12:24:03.539489 23996 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:24:03.539757 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:03.542382 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:03.543357 18017 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:03.543782 23919 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:03.530193 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:03.544551 23919 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.545907 23855 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:03.546389 23855 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.546854 23855 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:03.547158 23902 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:03.547326 23782 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:03.548091 23782 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:03.548393 23982 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.378s
I20260709 12:24:03.548694 23782 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.550189 23982 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.550230 23923 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:03.551331 23923 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:03.551838 23923 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:03.550714 18019 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:03.551646 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:03.551331 18024 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260709 12:24:03.553869 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.555543 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.560915 18098 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:03.615895 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:03.488984 23803 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:03.626575 18021 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:03.627341 18021 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:03.629067 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:03.630380 18022 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:03.551668 23990 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.266s	user 0.001s	sys 0.000s
I20260709 12:24:03.637056 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:03.640358 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.641000 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:03.398252 23896 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:03.645720 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 244817us
I20260709 12:24:03.628443 18021 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:03.562553 23889 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:03.403257 23852 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.537986 23993 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:03.668200 18021 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:03.380201 23983 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:03.665766 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.373849 23725 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 17.16, Last appended by leader: 16, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:03.662286 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.660799 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 364472us
I20260709 12:24:03.686125 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:03.659772 23964 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.528s	user 0.014s	sys 0.000s
I20260709 12:24:03.565953 23969 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.480s
I20260709 12:24:03.695360 18171 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:03.709388 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:03.714529 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.550732 23982 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:03.739645 18019 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:03.699774 23886 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.705894 23986 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:03.768082 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.686874 23854 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
W20260709 12:24:03.821319 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.692070 23940 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
I20260709 12:24:03.853142 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2554) took 1259 ms. Trace:
I20260709 12:24:03.853734 18169 rpcz_store.cc:276] 0709 12:24:02.594199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.594487 (+   288us) service_pool.cc:224] Handling call
0709 12:24:02.594872 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.602681 (+  7809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.602732 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.622968 (+ 20236us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.055225 (+432257us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:03.055300 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.055425 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.852833 (+797408us) raft_consensus.cc:1685] finished
0709 12:24:03.852858 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.853055 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18782,"raft.queue_time_us":5435,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":10632,"thread_start_us":294010,"threads_started":1,"wal-append.queue_time_us":511718}
W20260709 12:24:03.832273 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.695667 23964 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:03.916435 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.917693 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.870430 23889 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:03.927330 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.940548 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:03.943783 18168 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:24:03.946434 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.923724 18167 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:03.949586 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:03.950866 18019 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:03.952054 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:04.000710 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.543s (SENT)
I20260709 12:24:03.953372 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.962688 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.967633 18169 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:24:03.969226 18017 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:03.819934 24001 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.182s
I20260709 12:24:03.821354 23886 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:03.987674 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:03.974072 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.921700 24016 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.926456 23931 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:04.134636 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 394680us
W20260709 12:24:04.126309 18807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.839s	user 0.002s	sys 0.005s
I20260709 12:24:04.015385 24004 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.118314 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:03.960845 23919 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:03.989636 23257 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:03.988680 23992 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.096272 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:04.102885 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:04.044736 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44071.5us
I20260709 12:24:03.965200 23782 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:03.991696 23610 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 12:24:03.975353 23961 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.104581 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:04.131883 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.973239 23969 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:04.065496 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:04.009690 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83435.7us
I20260709 12:24:04.126809 23664 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.199781 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:04.139183 18807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.003s	sys 0.005s
I20260709 12:24:04.137084 18168 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 15 because replica has last-logged OpId of term: 14 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 13.
I20260709 12:24:04.176858 18167 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:04.110603 23720 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.116304 24031 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:04.137084 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.830549 23986 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.197368 18019 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:24:04.168931 24000 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.595s	user 0.004s	sys 0.010s
I20260709 12:24:04.206907 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:04.219383 18165 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:04.228994 18173 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:04.234696 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.240469 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:04.242403 24010 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:04.204496 23902 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:04.210779 23896 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:04.248270 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48638.8us
W20260709 12:24:04.249517 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.855213 23993 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 17.16, Last appended by leader: 16, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.218809 23720 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.256966 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:04.258446 18169 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:04.259668 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:04.264453 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.264410 18021 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.596s	user 0.014s	sys 0.001s
I20260709 12:24:04.177701 24016 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.269331 18171 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:24:04.254143 23995 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.030s	sys 0.058s
I20260709 12:24:04.270607 24011 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:04.271202 23995 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.030s	sys 0.058s
I20260709 12:24:04.271452 24011 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.217407 23852 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:04.210111 23664 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.219195 23257 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:24:04.243834 23610 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 12:24:04.326994 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67465.6us
I20260709 12:24:04.257515 23902 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:04.339537 24015 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 12:24:04.341518 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 777) took 1241 ms. Trace:
I20260709 12:24:04.342128 18023 rpcz_store.cc:276] 0709 12:24:03.100699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.101439 (+   740us) service_pool.cc:224] Handling call
0709 12:24:03.102025 (+   586us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.102374 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.102417 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.161158 (+ 58741us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.682744 (+521586us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:03.682796 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.682882 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.341147 (+658265us) raft_consensus.cc:1685] finished
0709 12:24:04.341184 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.341414 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":435563,"threads_started":1,"wal-append.queue_time_us":729348}
I20260709 12:24:04.344805 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:04.348170 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70194.5us
I20260709 12:24:04.126400 23889 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.207655 23969 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.361100 18165 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:24:04.276551 24001 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:04.351944 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:04.374163 24013 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.377s	user 0.001s	sys 0.000s
I20260709 12:24:04.374686 24038 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:04.243106 24010 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.376739 23866 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:24:04.377159 24006 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.380s	user 0.000s	sys 0.002s
W20260709 12:24:04.378249 24025 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
W20260709 12:24:04.437837 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84227.6us
I20260709 12:24:04.422060 18019 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:24:04.511932 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.346750 24004 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:04.266081 24027 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.414s
I20260709 12:24:04.333297 23852 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:04.256505 24033 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.098s
I20260709 12:24:04.278280 23720 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.219585 24031 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.243637 24018 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.446s
I20260709 12:24:04.334944 23982 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:04.352948 23997 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.576s	user 0.002s	sys 0.000s
W20260709 12:24:04.452087 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.490646 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138792us
W20260709 12:24:04.359122 23558 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.066s	user 0.016s	sys 0.000s
I20260709 12:24:04.388411 23964 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.512005 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:04.364197 24019 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.129s
I20260709 12:24:04.519039 18023 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:04.346966 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:04.437950 24014 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.596611 24014 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.598337 24019 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:04.598994 24019 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:04.522275 24011 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.438385 23983 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.533011 24036 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.524705 23961 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 6, Time since last communication: 0.013s
W20260709 12:24:04.607527 23971 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.788s	user 0.006s	sys 0.005s
I20260709 12:24:04.610242 24010 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:04.610677 24011 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:04.611531 24011 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.505738 23610 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:24:04.616444 18809 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20260709 12:24:04.616870 23855 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.069s	user 0.012s	sys 0.002s
I20260709 12:24:04.617116 24024 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.236s
I20260709 12:24:04.618162 23257 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:04.539285 24004 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:04.619840 23725 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:04.621152 24041 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:24:04.622728 23923 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.009s	user 0.014s	sys 0.003s
W20260709 12:24:04.623052 24017 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.192s	user 0.000s	sys 0.000s
I20260709 12:24:04.623688 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:04.521277 24007 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:04.566357 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.625463 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 907) took 1655 ms. Trace:
I20260709 12:24:04.625164 24007 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.625939 18096 rpcz_store.cc:276] 0709 12:24:02.970148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.977276 (+  7128us) service_pool.cc:224] Handling call
0709 12:24:03.008813 (+ 31537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.115029 (+106216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.115070 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.115819 (+   749us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.644028 (+528209us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:03.644095 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.644204 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.625179 (+980975us) raft_consensus.cc:1685] finished
0709 12:24:04.625209 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.625387 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14729,"raft.queue_time_us":41843,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":22894,"thread_start_us":493951,"threads_started":1,"wal-append.queue_time_us":1135269}
I20260709 12:24:04.627525 23905 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:04.627672 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:04.628085 23905 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:04.489919 23866 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:04.590621 24020 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.591181 23856 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:04.629892 23856 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:04.630198 24020 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.619906 24004 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.630918 23856 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.632215 23856 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:04.595774 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:04.629103 23905 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.634927 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:04.635195 23905 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.635857 18023 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:04.636977 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:04.639503 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:04.641120 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:04.617339 23855 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:04.644757 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.800s	user 0.019s	sys 0.053s
W20260709 12:24:04.645339 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.801s	user 0.019s	sys 0.053s
I20260709 12:24:04.646456 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1252) took 1351 ms. Trace:
I20260709 12:24:04.633713 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 853) took 1018 ms. Trace:
I20260709 12:24:04.647178 18170 rpcz_store.cc:276] 0709 12:24:03.615328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.617505 (+  2177us) service_pool.cc:224] Handling call
0709 12:24:03.644998 (+ 27493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.696577 (+ 51579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.696611 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:03.696658 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:03.696708 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.633426 (+936718us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b54001b6054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:04.633484 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.633647 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117929856}
I20260709 12:24:04.649564 18170 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:04.603349 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 91502.1us
I20260709 12:24:04.600272 24019 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.652119 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:04.653030 18812 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:04.653118 24019 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:04.599335 24014 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.655081 24014 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.655717 24014 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.624655 18022 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:04.658741 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 927) took 1513 ms (client timeout 1750 ms). Trace:
I20260709 12:24:04.659174 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:04.659405 18098 rpcz_store.cc:269] 0709 12:24:03.144838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.146685 (+  1847us) service_pool.cc:224] Handling call
0709 12:24:04.658451 (+1511766us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:04.658612 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511503616}
I20260709 12:24:04.661940 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:04.662671 23966 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.999s	user 0.002s	sys 0.015s
I20260709 12:24:04.662828 18021 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:24:04.664427 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:04.665231 18021 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:04.655938 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:04.644791 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.674228 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.8296e+06us
I20260709 12:24:04.675431 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:04.673484 18017 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:04.588826 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:04.678282 18098 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:04.679502 18170 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:24:04.681281 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:04.687829 23987 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:04.688184 23664 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.690239 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:04.647053 18808 rpcz_store.cc:276] 0709 12:24:03.295184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.295879 (+   695us) service_pool.cc:224] Handling call
0709 12:24:04.646152 (+1350273us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b5400590e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:04.646330 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21552000}
I20260709 12:24:04.692642 24010 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:04.694187 24010 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:04.695252 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:04.692240 18813 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:04.698079 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:04.674015 18164 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:04.699564 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:04.699712 18165 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:24:04.661339 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:04.701471 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.704312 18022 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:04.704536 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:04.706285 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:04.694932 24010 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:04.717648 23879 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:04.714306 23664 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:24:04.719583 24020 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.538324 24038 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.713482 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:04.721112 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.724769 18164 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:04.725157 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:04.418056 24001 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.681888 23902 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:04.733783 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:04.701316 23987 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.537248 23852 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.739522 23664 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:04.732120 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143393us
I20260709 12:24:04.742216 18168 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:24:04.743870 23921 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:04.744894 24028 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:04.744410 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44833.8us
I20260709 12:24:04.716187 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:04.722790 24009 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.818s	user 0.000s	sys 0.004s
I20260709 12:24:04.586285 23558 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:04.775416 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:04.782325 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.721041 23787 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:04.767109 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:04.793671 18021 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:24:04.772626 18169 raft_consensus.cc:606] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:24:04.796262 18138 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:04.796752 18138 leader_election.cc:403] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 12:24:04.797420 18018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.013s	sys 0.020s
I20260709 12:24:04.723209 23610 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 12:24:04.772475 18098 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:04.769423 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:04.723058 23886 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:04.789287 18020 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:04.789527 23558 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.701588 23923 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.733184 24038 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:04.848611 24009 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:24:04.974300 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.917183 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:04.764650 24020 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:04.944209 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2657) took 1010 ms. Trace:
I20260709 12:24:04.959323 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:05.091892 18097 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:04.798000 18018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.013s	sys 0.020s
W20260709 12:24:05.094918 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135229us
I20260709 12:24:04.772751 23902 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:05.096144 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 779) took 1412 ms. Trace:
I20260709 12:24:05.096487 23902 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:05.097085 23902 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:05.041034 18164 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:05.096882 18018 rpcz_store.cc:276] 0709 12:24:03.683535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.684364 (+   829us) service_pool.cc:224] Handling call
0709 12:24:03.703993 (+ 19629us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.864424 (+160431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.864479 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.903869 (+ 39390us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.095279 (+1191410us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:05.095401 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.095569 (+   168us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.095685 (+   116us) raft_consensus.cc:1685] finished
0709 12:24:05.095726 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.096011 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13087,"raft.queue_time_us":398133,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":7526,"thread_start_us":1254489,"threads_started":2,"wal-append.queue_time_us":911062}
I20260709 12:24:05.029892 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:04.972582 24043 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:24:04.966053 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.107278 18813 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:04.915529 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:05.071127 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 346037us
I20260709 12:24:04.874293 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:04.954491 23803 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.283s	user 0.016s	sys 0.002s
W20260709 12:24:04.844278 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48194us
I20260709 12:24:04.949719 23972 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:05.029892 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:04.855427 24016 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.087836 18165 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260709 12:24:04.779393 23997 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:24:04.871085 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:05.080430 18167 rpcz_store.cc:276] 0709 12:24:03.934071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.951637 (+ 17566us) service_pool.cc:224] Handling call
0709 12:24:04.943780 (+992143us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002026d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:04.944068 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405336960}
I20260709 12:24:05.095736 23965 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:04.919399 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.136222 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.136824 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:05.138103 18022 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:05.139734 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:04.946467 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.141072 18171 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:04.779393 23457 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:04.966279 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:04.868716 24034 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.219s	user 0.003s	sys 0.000s
I20260709 12:24:04.974718 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.843819 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:05.109799 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2577) took 1174 ms. Trace:
W20260709 12:24:04.807876 23999 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.067s	user 0.075s	sys 0.156s
I20260709 12:24:04.918097 23980 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:04.869441 23931 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:04.843146 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:05.119185 23964 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.116415 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.115635 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:04.962311 23969 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:04.939167 24001 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.963958 23852 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:04.798343 23879 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:04.969540 23866 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:05.122212 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.123415 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3500) took 2506 ms. Trace:
I20260709 12:24:04.955631 24007 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:05.124904 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 355540us
I20260709 12:24:04.921886 23855 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:04.956161 23720 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:05.126793 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:04.840773 23987 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:04.822912 23986 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:04.955595 24031 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.143288 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.782840 24028 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.103809 24030 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:05.265079 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:05.144652 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:04.919435 24027 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.211s
I20260709 12:24:05.244292 18091 rpcz_store.cc:276] 0709 12:24:02.609354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.609562 (+   208us) service_pool.cc:224] Handling call
0709 12:24:02.631988 (+ 22426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.661615 (+ 29627us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:03.131712 (+470097us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b18003cd960. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:24:03.284645 (+152933us) log.cc:844] Serialized 14 byte log entry
0709 12:24:03.624757 (+340112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.648685 (+ 23928us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.648895 (+   210us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:03.648934 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.649028 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.668798 (+1019770us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:05.115525 (+446727us) raft_consensus.cc:1685] finished
0709 12:24:05.115551 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.115723 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4785,"prepare.queue_time_us":224819,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":5832,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":51510,"spinlock_wait_cycles":30228096,"thread_start_us":431832,"threads_started":2,"wal-append.queue_time_us":400679}
I20260709 12:24:05.150291 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:04.774834 23664 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:05.154022 18024 rpcz_store.cc:276] 0709 12:24:03.935829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.873482 (+937653us) service_pool.cc:224] Handling call
0709 12:24:05.109710 (+236228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.104913 24042 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:05.025348 23889 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.099s
I20260709 12:24:05.161031 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:05.167147 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.285235 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:05.287168 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.123231 23558 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.121500 23972 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:05.325137 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 855) took 1135 ms. Trace:
I20260709 12:24:05.325789 18168 rpcz_store.cc:276] 0709 12:24:04.189784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.212191 (+ 22407us) service_pool.cc:224] Handling call
0709 12:24:04.215670 (+  3479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:04.772428 (+556758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.324682 (+552254us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:05.325019 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1209885952}
W20260709 12:24:05.331049 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3403) took 3885 ms (client timeout 2807 ms). Trace:
W20260709 12:24:05.331774 18015 rpcz_store.cc:269] 0709 12:24:01.445545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.455252 (+  9707us) service_pool.cc:224] Handling call
0709 12:24:05.330657 (+3875405us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:05.330891 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4417649408}
I20260709 12:24:05.172019 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:05.335041 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.337015 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.270300 24030 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.178074 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62534.7us
I20260709 12:24:05.338529 24030 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:05.165182 23999 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.531s	user 0.079s	sys 0.164s
I20260709 12:24:05.183074 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2578) took 1247 ms. Trace:
I20260709 12:24:05.185837 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:04.812158 24010 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.167873 23931 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:05.353782 23931 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:05.355427 23931 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:05.355772 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.250705 18813 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:05.252103 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.188076 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2551) took 2688 ms. Trace:
I20260709 12:24:05.359998 24007 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.359179 18166 rpcz_store.cc:276] 0709 12:24:02.491127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.491286 (+   159us) service_pool.cc:224] Handling call
0709 12:24:02.576002 (+ 84716us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.633088 (+ 57086us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.633145 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.668345 (+ 35200us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.204500 (+536155us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:03.280850 (+ 76350us) log.cc:844] Serialized 14 byte log entry
0709 12:24:03.286202 (+  5352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.286302 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.179126 (+1892824us) raft_consensus.cc:1685] finished
0709 12:24:05.179165 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.179394 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30106,"prepare.queue_time_us":74701,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":41802,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":36502,"thread_start_us":396221,"threads_started":1,"wal-append.queue_time_us":591227}
I20260709 12:24:05.360617 24007 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:05.191926 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.361900 23987 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:05.259475 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.364976 18092 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:24:05.265437 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 149084us
W20260709 12:24:05.366135 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.366672 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.145999 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.271443 18164 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:05.349206 18021 rpcz_store.cc:276] 0709 12:24:03.936119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.810578 (+874459us) service_pool.cc:224] Handling call
0709 12:24:05.182967 (+372389us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.270005 23720 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.274870 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:05.275285 24045 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:24:05.374771 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:05.277130 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 781) took 1593 ms. Trace:
I20260709 12:24:05.125511 23803 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.164227 23980 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:05.222852 23725 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.570s	user 0.016s	sys 0.000s
I20260709 12:24:05.184957 23886 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:05.289127 18097 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:05.392009 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:05.392982 18097 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:05.289991 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.395287 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.292244 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 782) took 1608 ms. Trace:
I20260709 12:24:05.396281 18024 rpcz_store.cc:276] 0709 12:24:03.684248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.291280 (+1607032us) service_pool.cc:224] Handling call
0709 12:24:05.291619 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.291863 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.291890 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.291941 (+    51us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:05.291980 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.292038 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.292179 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.296469 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 852) took 1681 ms. Trace:
I20260709 12:24:05.398759 18172 rpcz_store.cc:276] 0709 12:24:03.614876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.617375 (+  2499us) service_pool.cc:224] Handling call
0709 12:24:03.654245 (+ 36870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.850483 (+196238us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:03.850560 (+    77us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.867330 (+ 16770us) log.cc:844] Serialized 26 byte log entry
0709 12:24:04.142253 (+274923us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:04.143355 (+  1102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.143490 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.868244 (+724754us) spinlock_profiling.cc:243] Waited 50.1 ms on lock 0x7b24001cfec8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:24:05.281767 (+413523us) raft_consensus.cc:1685] finished
0709 12:24:05.281808 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.296344 (+ 14536us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":226708,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":19986,"spinlock_wait_cycles":115289472,"thread_start_us":346200,"threads_started":2,"wal-append.queue_time_us":622402}
I20260709 12:24:05.287984 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:05.298517 23866 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:05.405524 18021 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
W20260709 12:24:05.308408 24002 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.742s	user 0.001s	sys 0.004s
I20260709 12:24:05.311378 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3431) took 1784 ms. Trace:
I20260709 12:24:05.314357 18019 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:24:05.406399 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2583) took 1445 ms. Trace:
I20260709 12:24:05.407274 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2581) took 1457 ms. Trace:
I20260709 12:24:05.316459 24014 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.318624 23924 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.319962 23856 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.687s	user 0.017s	sys 0.000s
I20260709 12:24:05.321954 24028 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.324481 23972 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:04.807474 24011 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:05.190294 23952 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:05.368491 18166 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:05.369220 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 783) took 1684 ms. Trace:
I20260709 12:24:05.378878 18016 rpcz_store.cc:276] 0709 12:24:03.684009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.276061 (+1592052us) service_pool.cc:224] Handling call
0709 12:24:05.276484 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.276725 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.276759 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.276812 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:05.276863 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.276929 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.277054 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:05.434612 24051 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20260709 12:24:05.435475 18016 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:24:05.436358 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2587) took 1435 ms. Trace:
I20260709 12:24:05.268316 23852 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.436934 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1249) took 2153 ms. Trace:
I20260709 12:24:05.438418 18811 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:05.437351 18807 rpcz_store.cc:276] 0709 12:24:03.283519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.285810 (+  2291us) service_pool.cc:224] Handling call
0709 12:24:03.286250 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:04.203065 (+916815us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.203100 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:04.203701 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:24:04.735078 (+531377us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:04.735141 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.735252 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.436665 (+701413us) raft_consensus.cc:1685] finished
0709 12:24:05.436687 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.436840 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36794,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":16729,"thread_start_us":1414302,"threads_started":2,"wal-append.queue_time_us":846033}
I20260709 12:24:05.437314 23852 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:05.434345 18018 rpcz_store.cc:276] 0709 12:24:03.684460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.291302 (+1606842us) service_pool.cc:224] Handling call
0709 12:24:05.298783 (+  7481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.368715 (+ 69932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.368762 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.368830 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:05.368884 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.368978 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.369142 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.441004 18018 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:24:05.442160 18170 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:05.436697 18016 rpcz_store.cc:276] 0709 12:24:04.000707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.434661 (+1433954us) service_pool.cc:224] Handling call
0709 12:24:05.435155 (+   494us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.436037 (+   882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.436291 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.444272 18016 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:05.441731 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2588) took 1428 ms. Trace:
I20260709 12:24:05.444736 18018 rpcz_store.cc:276] 0709 12:24:04.013100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.440249 (+1427149us) service_pool.cc:224] Handling call
0709 12:24:05.440712 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.441412 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.441662 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.441648 23852 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:05.425467 23921 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:05.444996 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2589) took 1403 ms. Trace:
I20260709 12:24:05.409787 18164 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:05.410645 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.447886 18091 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:05.448220 23803 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:05.406911 23987 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:05.449460 18091 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:05.407779 18019 rpcz_store.cc:276] 0709 12:24:03.949587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.313394 (+1363807us) service_pool.cc:224] Handling call
0709 12:24:05.313980 (+   586us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.406868 (+ 92888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.407188 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:05.418098 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 130189us
I20260709 12:24:05.406914 18021 rpcz_store.cc:276] 0709 12:24:03.960651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.368215 (+1407564us) service_pool.cc:224] Handling call
0709 12:24:05.405232 (+ 37017us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.406041 (+   809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.406316 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.182431 23964 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.381668 23876 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.405292 23886 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.446390 23921 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.406477 18020 rpcz_store.cc:276] 0709 12:24:03.526835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.554033 (+ 27198us) service_pool.cc:224] Handling call
0709 12:24:05.310954 (+1756921us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:05.311260 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2639667584}
W20260709 12:24:05.453892 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87311.8us
W20260709 12:24:05.464135 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.427489 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 302108us
W20260709 12:24:05.404320 24019 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.733s	user 0.010s	sys 0.007s
I20260709 12:24:05.455282 18016 rpcz_store.cc:276] 0709 12:24:04.041258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.443685 (+1402427us) service_pool.cc:224] Handling call
0709 12:24:05.444078 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.444697 (+   619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.444932 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.374341 23558 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.480569 18812 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:05.482472 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.483341 18164 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:05.484628 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:05.483501 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.487205 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.493042 24010 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:05.496065 18022 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:05.487866 18017 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:24:05.497782 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 784) took 1058 ms. Trace:
I20260709 12:24:05.498834 18017 rpcz_store.cc:276] 0709 12:24:04.439054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.487085 (+1048031us) service_pool.cc:224] Handling call
0709 12:24:05.487476 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.497449 (+  9973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.497703 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.383320 23720 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:05.503345 18021 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:05.508273 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2599) took 1129 ms. Trace:
I20260709 12:24:05.508692 18021 rpcz_store.cc:276] 0709 12:24:04.379148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.458649 (+1079501us) service_pool.cc:224] Handling call
0709 12:24:05.502847 (+ 44198us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.507927 (+  5080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.508201 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.501394 18019 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:05.505126 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.510318 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2591) took 1346 ms. Trace:
I20260709 12:24:05.510807 18019 rpcz_store.cc:276] 0709 12:24:04.163745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.457019 (+1293274us) service_pool.cc:224] Handling call
0709 12:24:05.501131 (+ 44112us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.509977 (+  8846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.510251 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:05.506151 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.518170 18166 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:05.468636 23921 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:05.459625 23886 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:05.483940 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:05.496083 23972 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:05.467941 23457 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:05.420763 23924 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:05.542667 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58850.6us
I20260709 12:24:05.500501 23952 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:05.502161 23923 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.553030 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:05.555073 18164 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.555303 18019 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:05.518880 23998 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:05.468299 23876 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.504518 24028 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:24:05.548247 23457 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:05.583771 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 122178us
I20260709 12:24:05.646193 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.587515 18022 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:05.573940 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.590739 18095 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:24:05.552654 24014 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:05.660823 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.547070 24011 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:05.481530 23980 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.565099 23998 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:05.691684 18166 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:05.660291 23923 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:05.569126 23855 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.725276 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.721966 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 148139us
I20260709 12:24:05.686173 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.727469 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:05.727793 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:05.703488 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.565678 23664 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:05.645328 24054 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.569154 24068 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:05.683349 23905 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.047s	user 0.014s	sys 0.006s
W20260709 12:24:05.731976 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.688416 24007 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:05.729540 23952 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:05.728855 24014 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:05.731586 24011 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:05.789470 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.786221 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.763845 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:05.803324 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:05.792224 18164 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
W20260709 12:24:05.792755 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 132058us
I20260709 12:24:05.644730 24005 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.147s
I20260709 12:24:05.808037 18098 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:05.813704 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49953.2us
I20260709 12:24:05.729708 23787 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:24:05.827903 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:05.828655 18172 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:05.795977 23725 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.893738 23725 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:05.895334 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3506) took 1121 ms. Trace:
I20260709 12:24:05.896399 23725 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.898407 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:05.899499 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:05.900331 23610 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:05.900835 23610 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.901268 23610 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:05.764824 24066 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:05.732146 23855 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:05.867600 24060 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.306s	user 0.001s	sys 0.000s
I20260709 12:24:05.915792 18023 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
W20260709 12:24:05.868796 24057 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:24:05.869136 24076 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:24:05.870229 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57157.7us
I20260709 12:24:05.876329 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2582) took 1103 ms. Trace:
I20260709 12:24:05.877997 23964 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:05.923523 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.923240 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:05.878985 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75795us
I20260709 12:24:05.924281 18096 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:05.879841 24016 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:05.880594 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2577) took 1154 ms. Trace:
I20260709 12:24:05.880689 24011 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.881119 24018 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.881417 24004 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:05.883884 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.885289 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2584) took 1094 ms. Trace:
I20260709 12:24:05.800168 23664 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:05.886325 23940 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:05.886655 23965 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:05.826999 23905 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:05.889106 23876 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.890674 23924 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:05.822576 23998 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:05.893028 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.898221 23866 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:05.823472 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2654) took 1905 ms. Trace:
I20260709 12:24:05.810304 24014 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:05.908984 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.866899 18164 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:05.895831 18090 rpcz_store.cc:276] 0709 12:24:04.774513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.774704 (+   191us) service_pool.cc:224] Handling call
0709 12:24:04.807271 (+ 32567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.147779 (+340508us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:05.166558 (+ 18779us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.462855 (+296297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.462958 (+   103us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:05.463055 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.895031 (+431976us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b540032c154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:05.895085 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.895256 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67191,"prepare.queue_time_us":23153,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":5594,"raft.queue_time_us":50848,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":25918,"spinlock_wait_cycles":816987392,"thread_start_us":249868,"threads_started":1,"wal-append.queue_time_us":348453}
I20260709 12:24:05.798596 23927 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.831333 18097 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:05.991086 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.759037 24054 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:05.867882 24055 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.361s	user 0.001s	sys 0.000s
W20260709 12:24:06.008391 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 187603us
I20260709 12:24:06.008600 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:05.789235 24068 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:05.926821 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:05.923346 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:05.923573 18169 rpcz_store.cc:276] 0709 12:24:04.772495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.362595 (+590100us) service_pool.cc:224] Handling call
0709 12:24:05.362993 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.363486 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.363518 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:05.364119 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.533659 (+169540us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:05.535533 (+  1874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.536213 (+   680us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.876024 (+339811us) raft_consensus.cc:1685] finished
0709 12:24:05.876055 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.876244 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":188,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":169054,"threads_started":1,"wal-append.queue_time_us":420056}
I20260709 12:24:05.901372 23931 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:05.943547 18170 rpcz_store.cc:276] 0709 12:24:04.790673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.368409 (+577736us) service_pool.cc:224] Handling call
0709 12:24:05.405263 (+ 36854us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.442605 (+ 37342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.884932 (+442327us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b540007b554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:05.885200 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84275584}
I20260709 12:24:05.926136 18168 rpcz_store.cc:276] 0709 12:24:04.726528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.329131 (+602603us) service_pool.cc:224] Handling call
0709 12:24:05.329531 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.408984 (+ 79453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.409040 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:05.409788 (+   748us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.562637 (+152849us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:05.562688 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.562772 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.880308 (+317536us) raft_consensus.cc:1685] finished
0709 12:24:05.880337 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.880515 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30266,"raft.queue_time_us":31464,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":127307,"threads_started":1,"wal-append.queue_time_us":379218}
I20260709 12:24:05.820473 23969 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:05.822824 24005 consensus_queue.cc:1243] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:05.914866 24007 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:06.060660 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.961410 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:05.942051 24061 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.403s	user 0.000s	sys 0.001s
W20260709 12:24:05.962654 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.970780 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:05.945392 23720 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:05.958734 23965 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:05.977504 18173 rpcz_store.cc:276] 0709 12:24:03.918043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.918186 (+   143us) service_pool.cc:224] Handling call
0709 12:24:03.960159 (+ 41973us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.231484 (+271325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.819906 (+1588422us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:05.823396 (+  3490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":546910080}
I20260709 12:24:05.957204 24063 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.073231 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102621us
W20260709 12:24:05.997164 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.009879 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.077366 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:06.077718 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.010025 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.078555 18099 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:06.079402 18092 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:06.079478 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:06.014250 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.081018 18095 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:06.017799 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.021010 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.084633 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.085616 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.085788 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:06.086443 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:06.023136 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.087615 18806 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:05.958899 23664 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.083770 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:06.024237 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.026428 18811 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:06.028496 18096 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:24:06.028785 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:06.031843 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:06.093248 18164 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:24:05.946857 24018 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.037286 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 785) took 1326 ms. Trace:
I20260709 12:24:06.144025 18022 rpcz_store.cc:276] 0709 12:24:04.711032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.139304 (+428272us) service_pool.cc:224] Handling call
0709 12:24:06.037028 (+897724us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:06.037200 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":817443200}
I20260709 12:24:06.038053 18018 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:06.039335 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.039636 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:06.039842 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2594) took 1791 ms (client timeout 1622 ms). Trace:
I20260709 12:24:06.145732 18809 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:05.953913 24004 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.054286 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.979935 24047 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:24:06.107161 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 183957us
I20260709 12:24:06.061753 18815 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:05.980661 23558 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.063287 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:05.993355 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.080441 18166 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:06.089795 18806 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:24:06.094863 24028 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:06.095386 23964 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:06.096268 23720 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.103175 24078 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:24:06.103554 24063 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:06.090050 24010 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.579s	user 0.012s	sys 0.002s
I20260709 12:24:06.106208 23866 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:06.011973 23927 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.057735 23998 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.022089 23931 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:06.111212 23921 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.564s	user 0.002s	sys 0.010s
I20260709 12:24:06.111325 23965 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:06.168475 23886 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.005s
I20260709 12:24:06.200367 23965 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.112354 18811 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:24:06.114013 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 786) took 1402 ms. Trace:
I20260709 12:24:06.200873 23886 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:06.118688 24081 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:24:06.123562 24068 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:06.128762 24030 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.790s	user 0.009s	sys 0.004s
I20260709 12:24:06.139521 24054 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.062042 23876 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:05.958509 23905 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:06.145730 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 106190us
W20260709 12:24:06.145968 18023 rpcz_store.cc:269] 0709 12:24:04.248606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.636648 (+388042us) service_pool.cc:224] Handling call
0709 12:24:06.039599 (+1402951us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:06.039744 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2620237568}
I20260709 12:24:06.148833 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.109393 23725 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:06.167562 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.169761 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.171765 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.030807 24053 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.121s
I20260709 12:24:06.287631 18097 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:06.187263 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.150975 24047 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:06.194563 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.294512 18093 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:06.172206 23987 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.723s	user 0.011s	sys 0.004s
I20260709 12:24:06.201514 18016 rpcz_store.cc:276] 0709 12:24:04.711397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.522229 (+810832us) service_pool.cc:224] Handling call
0709 12:24:05.547495 (+ 25266us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.547964 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:05.548005 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:05.561778 (+ 13773us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.908858 (+347080us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:05.908984 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.909165 (+   181us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:06.113562 (+204397us) raft_consensus.cc:1685] finished
0709 12:24:06.113596 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.113912 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4997,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":8445,"thread_start_us":219043,"threads_started":1,"wal-append.queue_time_us":376379}
I20260709 12:24:06.222328 24010 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:06.214627 23898 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:06.161876 23923 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:06.223172 24067 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:06.206348 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.216s (SENT)
I20260709 12:24:06.218564 24030 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.063524 24066 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.125s
I20260709 12:24:06.226940 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.318378 18099 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:06.286340 23905 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:06.286372 18015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.084s	sys 0.149s
I20260709 12:24:06.177837 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.289633 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:06.323313 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:06.290607 18096 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 11.
I20260709 12:24:06.286350 23725 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:06.324024 18812 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:24:06.297629 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:06.142205 23986 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.202517 24020 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:06.308408 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:06.329716 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:06.315860 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 109569us
I20260709 12:24:06.181596 23720 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.278180 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:06.331310 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53251.5us
I20260709 12:24:06.332103 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:06.215157 23921 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.328411 18097 raft_consensus.cc:2376] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14. Re-sending same reply.
I20260709 12:24:06.343703 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.344736 18097 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:06.329896 18093 raft_consensus.cc:2376] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15. Re-sending same reply.
W20260709 12:24:06.322433 18015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.085s	sys 0.149s
I20260709 12:24:06.335891 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:06.347151 18095 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:06.348824 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:06.349689 18024 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 14. Ops: []
I20260709 12:24:06.311861 23923 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:06.319988 23902 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.222s	user 0.017s	sys 0.000s
I20260709 12:24:06.313575 24010 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:06.340808 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5767088190000322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20260709 12:24:06.321228 23987 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:06.357473 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:06.366241 18167 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:24:06.367744 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5281088189999537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20260709 12:24:06.375674 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5182088189999376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
W20260709 12:24:06.376199 18064 connection.cc:256] RPC call timeout handler was delayed by -1.45160881899983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
W20260709 12:24:06.376582 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4156088190002265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20260709 12:24:06.376981 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4039088190002076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
W20260709 12:24:06.377352 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3985088190001989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260709 12:24:06.377744 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2671088189999864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260709 12:24:06.378414 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1789088189998438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
W20260709 12:24:06.378913 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1573088190002636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
W20260709 12:24:06.379375 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1294088190002185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260709 12:24:06.380836 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.369688 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.387030 18090 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:06.321786 23855 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.390012 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.327328 23725 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.394085 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.373296 23987 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:06.331100 24028 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:06.345878 24020 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.403990 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:06.405670 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.354723 23964 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:06.354408 23866 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:06.428186 23866 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.428700 23866 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.407752 18173 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.368750 23923 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.355846 24053 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:06.412165 23964 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:06.412583 24018 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.411677 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.413102 23664 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.403735 24085 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:06.403721 23988 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.414501 23921 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:06.342227 23986 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.325461 24067 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.322479 23898 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.419454 24063 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.381423 24047 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.318065 23931 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:06.423179 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.382349 23902 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.400611 24011 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:06.383025 24010 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:06.481204 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 333484us
I20260709 12:24:06.406620 24028 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:06.369128 23965 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:06.434942 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:06.460194 23972 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.909s	user 0.015s	sys 0.000s
I20260709 12:24:06.402766 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.437398 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:06.396974 24068 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:24:06.454723 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55257.4us
I20260709 12:24:06.449469 23855 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:06.438859 24053 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.483135 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:06.486553 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.579952 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 493546us
W20260709 12:24:06.580858 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97875.7us
I20260709 12:24:06.458496 23988 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.551260 24004 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.553151 23931 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.454440 23664 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:06.525364 18094 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 17. Ops: []
I20260709 12:24:06.497051 24028 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.506516 18090 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:06.594578 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 942) took 1022 ms. Trace:
W20260709 12:24:06.594761 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 237354us
I20260709 12:24:06.595053 18090 rpcz_store.cc:276] 0709 12:24:05.572377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.396203 (+823826us) service_pool.cc:224] Handling call
0709 12:24:06.594484 (+198281us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:06.524693 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.562070 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.016s	sys 0.041s
W20260709 12:24:06.597239 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.016s	sys 0.041s
W20260709 12:24:06.545149 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 459454us
I20260709 12:24:06.564705 18173 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:06.560992 18099 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:06.537554 24047 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.538386 23886 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.540037 23927 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:06.549413 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.470912 23964 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.612455 23964 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:06.615676 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49481.4us
I20260709 12:24:06.622344 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:06.588680 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.611382 24038 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.611387 24030 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.587553 24053 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:06.625483 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.933s (SENT)
I20260709 12:24:06.631839 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:06.608162 23457 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.944s	user 0.012s	sys 0.003s
I20260709 12:24:06.627281 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.637143 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:06.634586 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:06.639607 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:06.635432 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.635290 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:06.641476 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.633482 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:06.640924 18090 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:24:06.643640 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:06.648808 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.666841 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:06.668201 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2721) took 1138 ms. Trace:
I20260709 12:24:06.668823 18090 rpcz_store.cc:276] 0709 12:24:05.529636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.606384 (+1076748us) service_pool.cc:224] Handling call
0709 12:24:06.668082 (+ 61698us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:06.614645 23972 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:06.671336 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.671872 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.616864 23998 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:06.677531 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 95528.9us
I20260709 12:24:06.681751 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.673367 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:06.684661 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.685536 18099 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:06.645750 24089 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.006s
I20260709 12:24:06.692900 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.700234 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.701234 18092 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:06.638708 23876 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:06.698357 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.713840 18098 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:06.715271 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.715773 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2584) took 2739 ms. Trace:
I20260709 12:24:06.716465 18024 rpcz_store.cc:276] 0709 12:24:03.976436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.398003 (+1421567us) service_pool.cc:224] Handling call
0709 12:24:05.423130 (+ 25127us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.355599 (+932469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.715377 (+359778us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400109ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:06.715670 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2130315520}
I20260709 12:24:06.705915 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:06.650657 23927 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:06.729938 23964 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.639308 23855 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.731550 23855 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.726336 24018 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:06.690439 23998 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.736044 18098 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:06.737165 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2713) took 1370 ms. Trace:
I20260709 12:24:06.630695 23664 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:06.707296 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.727406 23898 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:06.728308 24088 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:06.651500 23924 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:06.670243 24016 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.690577 24004 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:06.690336 24068 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:06.744307 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.726336 23558 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:06.750566 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2590) took 2654 ms. Trace:
I20260709 12:24:06.810431 18018 rpcz_store.cc:276] 0709 12:24:04.096410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.445865 (+1349455us) service_pool.cc:224] Handling call
0709 12:24:06.037800 (+591935us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.152805 (+115005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.750225 (+597420us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54001090d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:06.750467 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1360205696}
W20260709 12:24:06.751511 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59561.8us
W20260709 12:24:06.753731 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60920us
I20260709 12:24:06.812887 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:06.814323 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.753150 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:06.815707 18090 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.809734 18093 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:24:06.776150 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61307.3us
I20260709 12:24:06.818975 18093 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:06.819981 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2719) took 1352 ms. Trace:
I20260709 12:24:06.711895 24040 raft_consensus.cc:2764] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20260709 12:24:06.758103 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.004s	sys 0.033s
I20260709 12:24:06.685856 23720 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.742827 23876 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.787453 18172 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.834592 18172 raft_consensus.cc:614] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:06.647320 23457 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:06.827620 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.004s	sys 0.033s
W20260709 12:24:06.839171 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 749678us
I20260709 12:24:06.830653 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.820439 18093 rpcz_store.cc:276] 0709 12:24:05.467868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.818114 (+1350246us) service_pool.cc:224] Handling call
0709 12:24:06.818679 (+   565us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.819608 (+   929us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.819902 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:06.832568 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.842839 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:06.764483 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.843807 18814 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:06.832682 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 460701us
I20260709 12:24:06.786026 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2586) took 1920 ms. Trace:
I20260709 12:24:06.760150 24092 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.845052 18165 rpcz_store.cc:276] 0709 12:24:04.865599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.388072 (+522473us) service_pool.cc:224] Handling call
0709 12:24:05.388549 (+   477us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:05.389135 (+   586us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.389179 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:05.390088 (+   909us) log.cc:844] Serialized 52 byte log entry
0709 12:24:05.715644 (+325556us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:05.791299 (+ 75655us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.720536 (+929237us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b18004161a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:24:06.720934 (+   398us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.760348 (+ 39414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.760432 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:06.785508 (+ 25076us) raft_consensus.cc:1685] finished
0709 12:24:06.785562 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.785909 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":940231,"prepare.queue_time_us":1050028,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":122,"raft.queue_time_us":129969,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":18807,"spinlock_wait_cycles":41253248,"thread_start_us":113582,"threads_started":1,"wal-append.queue_time_us":410974}
I20260709 12:24:06.784845 23924 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.746093 18098 rpcz_store.cc:276] 0709 12:24:05.367612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.735221 (+1367609us) service_pool.cc:224] Handling call
0709 12:24:06.735741 (+   520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:06.736771 (+  1030us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.737077 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:06.776151 23969 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.697808 24063 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.108s
W20260709 12:24:06.851089 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.760526 24088 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.809960 24068 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:06.761356 24099 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.861186 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 235545us
I20260709 12:24:06.849766 23927 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:06.867303 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:06.867560 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:06.786473 23898 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:06.818066 23558 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:06.699244 23972 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.868490 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2727) took 1106 ms. Trace:
I20260709 12:24:06.806763 23854 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:06.807591 24004 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.859761 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:06.876106 18095 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 11. Current term is 12. Ops: []
W20260709 12:24:06.876952 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 171217us
I20260709 12:24:06.839920 23886 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:06.886804 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:06.887921 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.888099 18090 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:06.924834 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:06.712684 24011 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.167s
I20260709 12:24:06.895057 18023 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:24:06.895124 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:06.896066 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:06.927729 18017 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:24:06.928016 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68413.7us
W20260709 12:24:06.837069 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 211618us
I20260709 12:24:06.855954 23902 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:06.774552 23866 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:06.861202 23986 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:06.838168 24016 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:06.871609 23924 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:06.909358 18099 rpcz_store.cc:276] 0709 12:24:05.760300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.707205 (+946905us) service_pool.cc:224] Handling call
0709 12:24:06.866445 (+159240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:06.921356 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:06.922727 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:06.949221 18018 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:06.930330 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:06.931159 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91239.2us
I20260709 12:24:06.924337 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2722) took 1384 ms. Trace:
I20260709 12:24:06.887291 24068 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:06.879004 23998 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.871572 24082 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:06.971805 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:06.911937 23980 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:06.940062 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:06.943189 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110724us
I20260709 12:24:06.879060 24088 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:06.934680 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:06.951555 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:06.951900 18095 rpcz_store.cc:276] 0709 12:24:05.540186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.850944 (+1310758us) service_pool.cc:224] Handling call
0709 12:24:06.871041 (+ 20097us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.923962 (+ 52921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.924265 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:06.956617 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.957185 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.934250 23940 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:06.961889 24102 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:24:06.965384 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:07.011365 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46137.5us
I20260709 12:24:06.900055 23927 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:06.950836 24084 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.377s	user 0.000s	sys 0.001s
I20260709 12:24:06.989423 18020 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:24:07.001057 18099 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 10. Current term is 11. Ops: [8.10-10.11]
I20260709 12:24:07.013481 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:07.026100 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 944) took 1453 ms. Trace:
W20260709 12:24:06.987681 24083 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:24:06.925884 24004 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:06.868213 24092 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:07.033229 24101 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:07.035061 24092 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:06.989174 23898 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:07.036286 23898 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.036861 24082 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:07.038198 24054 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:07.038739 24054 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:07.039330 23898 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:07.041930 23854 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:07.044301 24020 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.606s	user 0.010s	sys 0.003s
I20260709 12:24:07.045378 24016 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.951632 24040 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 12:24:06.966962 24089 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260709 12:24:07.046530 24040 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:07.046986 24040 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:07.047907 24040 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:07.052814 23923 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.608s	user 0.004s	sys 0.008s
I20260709 12:24:06.969305 23988 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:07.008837 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:07.009881 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:07.013691 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:07.016232 18095 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:06.908990 23887 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.080s
I20260709 12:24:07.049372 24040 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.022933 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 72) took 1182 ms. Trace:
I20260709 12:24:07.080931 18172 rpcz_store.cc:276] 0709 12:24:05.823246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.866058 (+ 42812us) service_pool.cc:224] Handling call
0709 12:24:07.005390 (+1139332us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:07.005617 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1775925504}
W20260709 12:24:07.024127 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 174186us
I20260709 12:24:07.026561 18099 rpcz_store.cc:276] 0709 12:24:05.572900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.948402 (+1375502us) service_pool.cc:224] Handling call
0709 12:24:07.000633 (+ 52231us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:07.025692 (+ 25059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.026028 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.001858 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 905) took 4072 ms. Trace:
I20260709 12:24:07.024052 24096 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:07.006604 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55185.7us
I20260709 12:24:07.030966 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1095) took 2038 ms. Trace:
I20260709 12:24:06.990918 24104 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:07.130478 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:06.968678 24095 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:07.064359 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.135324 24068 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.138711 24068 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:06.899616 24099 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.083781 18094 rpcz_store.cc:276] 0709 12:24:02.929615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.930326 (+   711us) service_pool.cc:224] Handling call
0709 12:24:05.115832 (+2185506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.593662 (+1477830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.854929 (+261267us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:07.001448 (+146519us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b5400297cd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:07.001725 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8228185856}
W20260709 12:24:07.086570 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.020972 23998 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:07.028048 23725 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.608s	user 0.015s	sys 0.000s
I20260709 12:24:07.082779 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:07.125195 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:07.191195 18099 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:07.126479 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116710us
I20260709 12:24:07.201478 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2724) took 1518 ms. Trace:
I20260709 12:24:07.128459 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 945) took 1555 ms. Trace:
I20260709 12:24:07.202090 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:07.202198 18095 rpcz_store.cc:276] 0709 12:24:05.573155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.990064 (+1416909us) service_pool.cc:224] Handling call
0709 12:24:07.015822 (+ 25758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.061120 (+ 45298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.128346 (+ 67226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.057986 24108 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:24:07.067025 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.974191 23902 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:24:07.025084 23720 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:07.223843 18095 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:24:07.224773 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2733) took 1440 ms. Trace:
I20260709 12:24:07.225142 18095 rpcz_store.cc:276] 0709 12:24:05.784343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.203491 (+1419148us) service_pool.cc:224] Handling call
0709 12:24:07.223544 (+ 20053us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.224440 (+   896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.224705 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.226064 23998 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:07.227396 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.228067 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.228219 24107 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:24:07.227486 23998 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.025496 23927 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.232585 24099 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:07.234182 23720 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:07.235304 24103 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.260s	user 0.002s	sys 0.000s
I20260709 12:24:07.235505 23720 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.044857 24020 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:07.164913 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.240058 24020 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:07.241639 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 158973us
I20260709 12:24:07.242620 23457 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:07.124864 18810 rpcz_store.cc:276] 0709 12:24:04.984722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.984897 (+   175us) service_pool.cc:224] Handling call
0709 12:24:05.027417 (+ 42520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.551291 (+1523874us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:06.551336 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:06.551893 (+   557us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:06.551970 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.851679 (+299709us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b1800415d20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:24:07.022432 (+170753us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54005e01d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:07.022498 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.022679 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7180,"raft.queue_time_us":232,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4315,"spinlock_wait_cycles":3496533632}
W20260709 12:24:07.186272 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.247175 24114 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:07.247789 24114 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:07.248183 24114 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:07.036900 23886 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:07.249233 23886 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.249687 23886 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:07.252689 24099 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:07.201413 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 206671us
I20260709 12:24:07.126753 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:07.254949 18024 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:06.926654 23664 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:07.255467 24014 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.223s	user 0.013s	sys 0.002s
I20260709 12:24:07.256037 24014 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.029040 24085 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.180s
I20260709 12:24:07.256664 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:07.186682 24016 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:06.990145 24094 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.235s
I20260709 12:24:07.258154 24016 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:07.258196 24115 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:07.258714 24016 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.259490 24094 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:07.259876 24115 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:07.260344 24115 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.160465 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:07.207563 18099 rpcz_store.cc:276] 0709 12:24:05.682972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.083551 (+1400579us) service_pool.cc:224] Handling call
0709 12:24:07.190913 (+107362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.201146 (+ 10233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.201409 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.261399 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2579) took 2534 ms. Trace:
W20260709 12:24:07.210583 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85534.1us
I20260709 12:24:07.210016 23725 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.252888 18813 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:07.256389 23855 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:07.129132 23854 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:07.283443 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:07.159545 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:07.212086 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:07.284678 18808 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:07.260710 24115 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:07.270424 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.270851 18093 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:07.271948 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 258556us
I20260709 12:24:07.272238 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.273761 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:07.278048 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.279508 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.281704 18173 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:07.282716 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2582) took 3332 ms. Trace:
I20260709 12:24:07.206977 18811 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:07.262420 18171 rpcz_store.cc:276] 0709 12:24:04.727176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.334691 (+607515us) service_pool.cc:224] Handling call
0709 12:24:05.335217 (+   526us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:05.335862 (+   645us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:05.521214 (+185352us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.760387 (+1239173us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.760617 (+   230us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:06.786777 (+ 26160us) log.cc:844] Serialized 52 byte log entry
0709 12:24:06.787079 (+   302us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:06.787143 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.787303 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.261156 (+473853us) raft_consensus.cc:1685] finished
0709 12:24:07.261180 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.261328 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":857961,"prepare.queue_time_us":1233556,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":113,"raft.queue_time_us":133775,"raft.run_cpu_time_us":600,"raft.run_wall_time_us":46640,"thread_start_us":466474,"threads_started":2,"wal-append.queue_time_us":682207}
I20260709 12:24:07.300273 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:07.300467 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3515) took 1230 ms. Trace:
I20260709 12:24:07.302130 18168 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:07.301680 18093 rpcz_store.cc:276] 0709 12:24:06.070498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.270310 (+1199812us) service_pool.cc:224] Handling call
0709 12:24:07.270652 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.300134 (+ 29482us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.300396 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.304518 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:07.305406 18099 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:07.307428 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95479.5us
I20260709 12:24:07.308565 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:07.074548 23923 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.295627 18015 rpcz_store.cc:276] 0709 12:24:03.949928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.333045 (+1383117us) service_pool.cc:224] Handling call
0709 12:24:05.333596 (+   551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.548675 (+215079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.548728 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:05.582541 (+ 33813us) log.cc:844] Serialized 26 byte log entry
0709 12:24:06.346253 (+763712us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:06.647737 (+301484us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.674167 (+ 26430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.002597 (+328430us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001d2254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:07.002666 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.267610 (+264944us) raft_consensus.cc:1685] finished
0709 12:24:07.282455 (+ 14845us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.282642 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41638,"prepare.queue_time_us":521049,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":9474,"raft.queue_time_us":89884,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":19833,"spinlock_wait_cycles":451314048,"thread_start_us":872088,"threads_started":2,"wal-append.queue_time_us":775281}
I20260709 12:24:07.161609 24007 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:07.311340 18170 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:07.311986 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.313288 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.314574 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.253388 24099 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.318270 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.312762 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:07.275476 24020 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.322983 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:07.301638 23905 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.958s	user 0.013s	sys 0.003s
I20260709 12:24:07.312600 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:07.046468 23558 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.337736 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:07.283973 24094 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:07.352869 18173 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:07.360855 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:07.362227 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.364040 18814 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.515s	user 0.012s	sys 0.000s
I20260709 12:24:07.364692 18814 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:07.330552 23965 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.000s	sys 0.015s
I20260709 12:24:07.288475 24109 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:07.354480 23610 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.452s	user 0.011s	sys 0.007s
I20260709 12:24:07.288360 23987 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:07.378631 23558 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.378880 23987 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.351502 23866 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:07.381105 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58252.1us
I20260709 12:24:07.299103 24033 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.382668 24033 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.383888 23987 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:07.375959 24094 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.387012 23610 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.269374 24104 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.121s
I20260709 12:24:07.351310 23905 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.391589 18019 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:07.392050 24054 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.393193 18091 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:07.393509 24054 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.333220 24108 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:07.395615 24099 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:07.396667 24014 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:07.318888 24007 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.368588 18167 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:24:07.369807 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.397807 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:07.299755 23787 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.406419 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.318648 24096 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:07.375303 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.375316 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:07.375218 23854 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:07.351343 24020 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.392522 23905 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:07.398526 18165 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.399171 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:07.374014 24047 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.769s	user 0.009s	sys 0.005s
I20260709 12:24:07.340559 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:07.441289 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100815us
I20260709 12:24:07.443032 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:07.305961 23924 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:07.444872 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2718) took 1987 ms (client timeout 1860 ms). Trace:
I20260709 12:24:07.451223 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:07.460726 18091 rpcz_store.cc:269] 0709 12:24:05.457172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.943836 (+486664us) service_pool.cc:224] Handling call
0709 12:24:07.444577 (+1500741us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540032b9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:07.444768 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3061469440}
W20260709 12:24:07.466557 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61425us
W20260709 12:24:07.467097 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68021.3us
I20260709 12:24:07.469520 18093 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:24:07.470077 18093 pending_rounds.cc:85] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:24:07.470384 18093 pending_rounds.cc:107] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:24:07.470659 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:07.471050 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.470661 18093 raft_consensus.cc:2889] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:07.474973 18815 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:07.415591 24084 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:24:07.367841 23457 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:07.369735 23965 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.000s	sys 0.016s
I20260709 12:24:07.436106 18172 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:07.482472 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:07.311108 23876 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20260709 12:24:07.472131 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:07.502506 18090 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:07.478861 18099 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:07.410573 18808 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
W20260709 12:24:07.482227 18166 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.013s	sys 0.009s
W20260709 12:24:07.511971 18166 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.013s	sys 0.009s
I20260709 12:24:07.480989 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.512570 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:07.513433 18164 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
W20260709 12:24:07.410933 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 524198us
I20260709 12:24:07.482774 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2591) took 1884 ms. Trace:
I20260709 12:24:07.459159 24110 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:07.470886 24028 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.845s	user 0.014s	sys 0.000s
I20260709 12:24:07.514256 18169 rpcz_store.cc:276] 0709 12:24:05.598820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.087221 (+488401us) service_pool.cc:224] Handling call
0709 12:24:06.140674 (+ 53453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.140951 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:07.132540 (+991589us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b180030e6e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf
0709 12:24:07.132730 (+   190us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.482242 (+349512us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:07.482327 (+    85us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:07.482404 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.482494 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.482689 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":620182,"prepare.queue_time_us":761054,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":16702,"raft.queue_time_us":364,"spinlock_wait_cycles":5332352,"thread_start_us":475091,"threads_started":2}
I20260709 12:24:07.516185 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.528457 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.432658 24096 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:07.524296 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:07.553058 18814 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.450270 23610 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:07.485517 18173 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:07.526777 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:07.465555 24085 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:07.531514 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:07.572311 18015 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:07.459880 23972 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:07.549963 18022 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:24:07.525231 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:07.413337 23866 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 12:24:07.474839 24066 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.556322 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:07.520102 23457 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:07.560284 18810 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:07.564472 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:07.569697 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:07.570262 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:07.478648 24099 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:07.576824 23931 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.976s	user 0.016s	sys 0.000s
I20260709 12:24:07.431612 23787 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.522156 24047 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:07.557621 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:07.558900 23980 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.004s
I20260709 12:24:07.522449 23940 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:07.528086 24110 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:07.615772 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.108s (SENT) [suppressed 10 similar messages]
I20260709 12:24:07.618633 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:07.620105 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:07.671186 18021 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.615916 18814 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:07.475476 23879 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:24:07.554200 24109 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.677011 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2740) took 1600 ms. Trace:
I20260709 12:24:07.624545 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:07.634457 18017 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:07.557729 24096 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:07.634945 18018 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:07.625424 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:07.635481 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 314917us
W20260709 12:24:07.651060 24095 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.002s
W20260709 12:24:07.639192 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68032.7us
W20260709 12:24:07.641595 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 333575us
I20260709 12:24:07.648977 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:07.574484 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47610.7us
I20260709 12:24:07.534369 24014 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.103s
W20260709 12:24:07.669373 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.674865 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:07.641986 24088 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.621s	user 0.005s	sys 0.008s
I20260709 12:24:07.679417 18170 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:24:07.624393 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:07.688201 18172 raft_consensus.cc:606] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:07.688688 18172 raft_consensus.cc:614] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:07.642658 23664 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:07.683357 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:07.677467 18093 rpcz_store.cc:276] 0709 12:24:06.076753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.302903 (+1226150us) service_pool.cc:224] Handling call
0709 12:24:07.303257 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.472205 (+168948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.676647 (+204442us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:07.676927 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381307648}
I20260709 12:24:07.624604 23855 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:07.682574 18167 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:07.681432 18095 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:07.696816 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:07.667515 23457 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.701419 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:07.678957 24053 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.031s	user 0.011s	sys 0.003s
I20260709 12:24:07.703208 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:07.675953 24110 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.617875 24066 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.556123 24028 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.640118 24068 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:24:07.694890 23725 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:07.685900 24097 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.720178 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:07.685283 24047 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:07.693965 24095 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:07.668726 24085 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:24:07.695407 23664 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:07.742197 18093 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:07.770583 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:07.771064 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51088.8us
W20260709 12:24:07.781949 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80679.4us
W20260709 12:24:07.776453 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 78787.9us
I20260709 12:24:07.772004 18021 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:07.711592 24067 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:07.706014 23887 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.092s
I20260709 12:24:07.760468 18167 pending_rounds.cc:85] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 9
I20260709 12:24:07.870169 18167 pending_rounds.cc:107] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:24:07.870591 18167 raft_consensus.cc:2889] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:07.767870 23905 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:07.744668 24119 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
W20260709 12:24:07.875864 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.879751 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:07.700451 23886 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:07.769605 24068 consensus_queue.cc:1243] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:07.760389 23923 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:07.781471 24118 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:07.857205 24091 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:24:07.882012 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:07.692777 24109 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:07.712901 24018 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:24:07.792925 24010 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.255s	user 0.011s	sys 0.006s
I20260709 12:24:07.886857 18093 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:07.767035 23952 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:07.863909 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:07.706266 23855 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:07.943425 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79863.3us
I20260709 12:24:07.944600 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:07.873471 23931 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:07.898177 18095 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:07.898082 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 282339us
I20260709 12:24:07.871294 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3514) took 1801 ms. Trace:
I20260709 12:24:07.959415 18098 rpcz_store.cc:276] 0709 12:24:06.070181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.243892 (+1173711us) service_pool.cc:224] Handling call
0709 12:24:07.244265 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.244530 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:07.535021 (+290491us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.870713 (+335692us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:07.870805 (+    92us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:07.870898 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.871002 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.871203 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16045,"prepare.queue_time_us":498106,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":99421,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7968,"thread_start_us":530571,"threads_started":2}
I20260709 12:24:07.750954 24020 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:07.695571 24088 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:07.901517 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.003s	sys 0.011s
W20260709 12:24:07.985021 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.004s	sys 0.011s
W20260709 12:24:07.758363 24121 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:24:07.881310 24123 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:07.906358 24126 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.285s	user 0.002s	sys 0.000s
I20260709 12:24:07.712800 24053 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:07.991278 18165 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.532s	user 0.013s	sys 0.000s
I20260709 12:24:07.901261 24094 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:07.900938 24118 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:08.008100 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 941) took 2436 ms. Trace:
I20260709 12:24:08.012290 18090 rpcz_store.cc:276] 0709 12:24:05.572130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.924992 (+1352862us) service_pool.cc:224] Handling call
0709 12:24:06.925353 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.502991 (+577638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.007706 (+504715us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:08.008007 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326778112}
I20260709 12:24:07.942301 23952 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:08.016296 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 303657us
I20260709 12:24:07.994601 24067 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:07.791301 24097 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.004957 18165 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:08.035207 18173 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:08.011399 24105 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:07.954418 24010 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.011662 24118 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:07.984403 24108 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:24:08.010529 24018 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:08.038532 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:08.061805 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 261692us
I20260709 12:24:08.064419 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1095) took 1178 ms. Trace:
I20260709 12:24:08.064415 18096 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:24:08.062947 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:08.069805 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:08.071398 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 862) took 1707 ms. Trace:
I20260709 12:24:08.076496 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2715) took 2698 ms. Trace:
I20260709 12:24:08.057886 23787 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.076960 18097 rpcz_store.cc:276] 0709 12:24:05.378091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.744547 (+1366456us) service_pool.cc:224] Handling call
0709 12:24:06.745021 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:06.775139 (+ 30118us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:06.775179 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:06.775874 (+   695us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.298289 (+522415us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:07.298337 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.298418 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.076174 (+777756us) raft_consensus.cc:1685] finished
0709 12:24:08.076209 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.076407 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39021,"raft.queue_time_us":31143,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":34453,"thread_start_us":445541,"threads_started":1,"wal-append.queue_time_us":647229}
I20260709 12:24:08.050202 23940 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:08.049515 24105 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:08.004292 24142 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
I20260709 12:24:08.016098 24047 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.064893 18806 rpcz_store.cc:276] 0709 12:24:06.886221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.886447 (+   226us) service_pool.cc:224] Handling call
0709 12:24:06.886882 (+   435us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:06.926950 (+ 40068us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:06.926987 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:06.955444 (+ 28457us) log.cc:844] Serialized 52 byte log entry
0709 12:24:07.432232 (+476788us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:07.432292 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.432381 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.064146 (+631765us) raft_consensus.cc:1685] finished
0709 12:24:08.064171 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.064342 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23805,"raft.queue_time_us":64909,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":47627,"thread_start_us":413745,"threads_started":1,"wal-append.queue_time_us":667148}
I20260709 12:24:08.107546 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:07.996170 23664 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.062284 24067 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:08.100183 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.164505 18018 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:08.099196 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 116688us
I20260709 12:24:08.078748 24116 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.085548 24141 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:08.092324 24028 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.133924 23965 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:08.085728 24145 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:08.130488 24105 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:08.169766 24004 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.132s	user 0.015s	sys 0.000s
W20260709 12:24:08.169872 24152 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:24:08.359311 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:08.396543 18165 raft_consensus.cc:606] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:24:08.397209 18165 raft_consensus.cc:614] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:08.364688 23980 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:24:08.230376 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 162438us
I20260709 12:24:08.328663 23940 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.324954 24094 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:08.354578 24135 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.352s	user 0.000s	sys 0.001s
W20260709 12:24:08.360188 24133 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.462s	user 0.001s	sys 0.001s
W20260709 12:24:08.343291 24132 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.321s	user 0.001s	sys 0.001s
W20260709 12:24:08.363574 24138 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.360s	user 0.000s	sys 0.002s
W20260709 12:24:08.315488 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.004s	sys 0.000s
W20260709 12:24:08.422565 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.007s	sys 0.002s
I20260709 12:24:08.385807 24053 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:08.423482 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 774593us
I20260709 12:24:08.425195 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:08.342149 24134 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.454s	user 0.000s	sys 0.002s
I20260709 12:24:08.131880 24030 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
I20260709 12:24:08.168061 24067 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:08.360285 23927 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.129s	user 0.016s	sys 0.000s
I20260709 12:24:08.269714 24142 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:08.436697 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.408564 23980 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:08.359207 24151 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:24:08.316876 24068 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:08.358255 24150 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:24:08.197517 24129 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.157s
I20260709 12:24:08.361996 24066 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:08.179421 24115 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.863s	user 0.000s	sys 0.018s
W20260709 12:24:08.176380 23969 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.288s	user 0.011s	sys 0.007s
I20260709 12:24:08.074465 23952 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:08.383316 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.354971 23855 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:08.354974 24137 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.383s	user 0.000s	sys 0.001s
W20260709 12:24:08.399049 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.372685 24154 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:08.358350 24033 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:08.417392 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.420255 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.397526 24109 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.389725 24088 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:08.396698 24147 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:24:08.427618 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.479516 18022 raft_consensus.cc:2368] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:24:08.480304 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.240381 24018 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.422931 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:08.431581 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:08.363029 24143 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
I20260709 12:24:08.293619 24028 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:08.490906 24099 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.825s	user 0.013s	sys 0.002s
I20260709 12:24:08.491554 24099 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:08.492530 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3516) took 2408 ms. Trace:
I20260709 12:24:08.492995 18093 rpcz_store.cc:276] 0709 12:24:06.084539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.954887 (+1870348us) service_pool.cc:224] Handling call
0709 12:24:07.955421 (+   534us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:07.987663 (+ 32242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.987720 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:08.000441 (+ 12721us) log.cc:844] Serialized 52 byte log entry
0709 12:24:08.178911 (+178470us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:08.178960 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.179038 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.492238 (+313200us) raft_consensus.cc:1685] finished
0709 12:24:08.492264 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.492452 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3108,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":14776,"thread_start_us":117763,"threads_started":1,"wal-append.queue_time_us":365812}
I20260709 12:24:08.495023 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.443572 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.418388 24145 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.496028 24145 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.496724 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.497651 18093 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:08.407094 23886 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:08.502596 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71512.2us
I20260709 12:24:08.439216 23927 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.507463 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.472319 23969 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:08.457671 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 350234us
W20260709 12:24:08.431804 24086 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.005s	sys 0.031s
I20260709 12:24:08.401500 24004 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.419337 24094 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.466372 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.528481 18020 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:08.467926 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:08.468930 18015 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:08.436743 24142 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:08.531750 24142 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:08.533013 24124 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.906s	user 0.005s	sys 0.000s
I20260709 12:24:08.533139 24004 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.534893 24067 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.477445 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.535882 18024 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:08.434490 23720 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:24:08.537263 23720 consensus_queue.cc:1243] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 14.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:08.481917 24154 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.539734 24154 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.482963 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1273) took 1231 ms. Trace:
I20260709 12:24:08.540274 23924 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.540441 18812 rpcz_store.cc:276] 0709 12:24:07.251718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.265887 (+ 14169us) service_pool.cc:224] Handling call
0709 12:24:07.266394 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.290182 (+ 23788us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.290217 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.290790 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.648625 (+357835us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:07.648672 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.648760 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.843565 (+194805us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b48000cc258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:24:08.476158 (+632593us) raft_consensus.cc:1685] finished
0709 12:24:08.476186 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.482867 (+  6681us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6598,"raft.queue_time_us":24157,"raft.run_cpu_time_us":3543,"raft.run_wall_time_us":543644,"spinlock_wait_cycles":12749696,"thread_start_us":293791,"threads_started":1,"wal-append.queue_time_us":525282}
W20260709 12:24:08.486590 24159 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:24:08.542035 23886 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:08.542706 23886 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.543682 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:08.544103 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2597) took 1251 ms. Trace:
I20260709 12:24:08.545089 23886 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.545308 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:08.507001 23980 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.498849 24145 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:08.499620 24114 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.251s	user 0.013s	sys 0.004s
I20260709 12:24:08.499958 18018 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:08.451835 18095 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
W20260709 12:24:08.511581 24016 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.252s	user 0.012s	sys 0.005s
I20260709 12:24:08.574318 24016 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.575738 23927 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.576856 24016 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.520156 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1097) took 1238 ms. Trace:
I20260709 12:24:08.578235 24094 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.578704 18019 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:08.459919 24068 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.545687 23886 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:08.579639 24110 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.791s	user 0.014s	sys 0.000s
I20260709 12:24:08.579813 23886 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.580152 24110 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.581091 24068 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:08.581261 23457 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.812s	user 0.011s	sys 0.004s
I20260709 12:24:08.579664 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.575944 24099 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:08.514123 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:08.522984 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3523) took 1204 ms. Trace:
W20260709 12:24:08.558312 24040 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.410s	user 0.008s	sys 0.008s
I20260709 12:24:08.521083 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:08.507404 23855 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.598536 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.470088 24115 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.508031 23952 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:08.525682 18099 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:08.581782 23457 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.626241 18021 raft_consensus.cc:2368] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14.
I20260709 12:24:08.627398 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:08.628350 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.506968 24066 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:08.547803 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2716) took 3141 ms. Trace:
I20260709 12:24:08.629855 18092 rpcz_store.cc:276] 0709 12:24:05.415492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.748646 (+1333154us) service_pool.cc:224] Handling call
0709 12:24:06.749216 (+   570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:06.776578 (+ 27362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:06.776613 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:06.805376 (+ 28763us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.284041 (+478665us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:07.284093 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.284182 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.532591 (+1248409us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b2400278258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:24:08.547421 (+ 14830us) raft_consensus.cc:1685] finished
0709 12:24:08.547447 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.547709 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29098,"raft.queue_time_us":74110,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":10618,"spinlock_wait_cycles":39132032,"thread_start_us":408982,"threads_started":1,"wal-append.queue_time_us":612902}
I20260709 12:24:08.632366 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:08.632823 24010 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.552469 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.634895 18015 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.635962 23457 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.636531 24097 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.619925 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2735) took 2544 ms. Trace:
I20260709 12:24:08.637113 18096 rpcz_store.cc:276] 0709 12:24:06.075205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.867261 (+792056us) service_pool.cc:224] Handling call
0709 12:24:08.619582 (+1752321us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400411ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:08.619828 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2690518912}
I20260709 12:24:08.536975 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.560431 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:08.570950 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 948) took 1109 ms. Trace:
I20260709 12:24:08.544543 18167 rpcz_store.cc:276] 0709 12:24:07.293023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.695800 (+402777us) service_pool.cc:224] Handling call
0709 12:24:07.760133 (+ 64333us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:07.884062 (+123929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.884123 (+    61us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:07.887780 (+  3657us) log.cc:844] Serialized 52 byte log entry
0709 12:24:08.089751 (+201971us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:08.167079 (+ 77328us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.167435 (+   356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.167532 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.543817 (+376285us) raft_consensus.cc:1685] finished
0709 12:24:08.543842 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.544027 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12311,"prepare.queue_time_us":76435,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":15369,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":11785,"spinlock_wait_cycles":9806592,"thread_start_us":92392,"threads_started":1,"wal-append.queue_time_us":300917}
I20260709 12:24:08.638624 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:08.522838 24086 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.005s	sys 0.032s
I20260709 12:24:08.520030 24028 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:08.640941 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.623910 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 792) took 1726 ms. Trace:
I20260709 12:24:08.641636 24028 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.642313 18017 rpcz_store.cc:276] 0709 12:24:06.897238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.042665 (+145427us) service_pool.cc:224] Handling call
0709 12:24:07.043074 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.679244 (+636170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.623573 (+944329us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:08.623816 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359421184}
I20260709 12:24:08.644057 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.524073 18094 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:08.644778 18017 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:08.591372 24099 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.645855 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.646250 24099 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.646591 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2731) took 2885 ms. Trace:
I20260709 12:24:08.641000 23457 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:08.647094 18094 rpcz_store.cc:276] 0709 12:24:05.761563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.185959 (+1424396us) service_pool.cc:224] Handling call
0709 12:24:07.212413 (+ 26454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.644964 (+1432551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.646311 (+  1347us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400214654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:08.646518 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3015888128}
I20260709 12:24:08.594924 23980 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.638499 18095 rpcz_store.cc:276] 0709 12:24:07.461913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.859187 (+397274us) service_pool.cc:224] Handling call
0709 12:24:08.570868 (+711681us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.648873 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:08.562453 23558 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.183s	user 0.016s	sys 0.001s
W20260709 12:24:08.651028 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.651963 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.602321 23855 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.654130 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:08.595718 24016 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:08.646598 18017 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:08.577586 18809 rpcz_store.cc:276] 0709 12:24:07.281256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.281391 (+   135us) service_pool.cc:224] Handling call
0709 12:24:07.281855 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.291133 (+  9278us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:07.291168 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.306524 (+ 15356us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.985737 (+679213us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:07.985809 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.986401 (+   592us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.519924 (+533523us) raft_consensus.cc:1685] finished
0709 12:24:08.519948 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.520090 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8782,"raft.queue_time_us":9092,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":13424,"thread_start_us":626260,"threads_started":1,"wal-append.queue_time_us":702839}
I20260709 12:24:08.649572 18095 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:08.641685 18092 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.654362 23855 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:08.639042 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80964.4us
I20260709 12:24:08.590062 18097 rpcz_store.cc:276] 0709 12:24:07.318314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.082886 (+764572us) service_pool.cc:224] Handling call
0709 12:24:08.083456 (+   570us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:08.083958 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.083992 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:08.084727 (+   735us) log.cc:844] Serialized 52 byte log entry
0709 12:24:08.405031 (+320304us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:08.521308 (+116277us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.521656 (+   348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.522580 (+   924us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.522662 (+    82us) raft_consensus.cc:1685] finished
0709 12:24:08.522681 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.522908 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9156,"prepare.queue_time_us":117108,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":7191,"raft.run_cpu_time_us":462,"raft.run_wall_time_us":71193,"spinlock_wait_cycles":1906944,"thread_start_us":210892,"threads_started":2,"wal-append.queue_time_us":283209}
I20260709 12:24:08.654922 18812 raft_consensus.cc:2376] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14. Re-sending same reply.
I20260709 12:24:08.677708 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:08.679116 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:08.679670 18023 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:08.680073 18097 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:08.546036 23876 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:08.655614 18094 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:08.525928 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.655877 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:08.640381 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.656509 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:08.610883 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.665126 18015 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:08.687208 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:08.701448 18020 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.701520 18093 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.689064 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:08.687441 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.681661 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.703704 18017 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:08.704527 18097 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.704998 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.682036 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:08.707310 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.708149 18095 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.705806 18015 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:08.704036 18096 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:08.665401 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 144412us
I20260709 12:24:08.711534 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:08.712952 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.705353 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.722780 18017 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:08.724170 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.730180 18017 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 12: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:08.726862 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.584453 24114 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.624477 24115 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:08.670574 24087 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.680204 24016 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:08.681226 24097 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:08.741644 18095 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:08.706907 24092 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.735651 23457 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.738375 23876 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:08.774308 24092 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.775079 23876 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.777258 24092 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:08.734874 23940 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:08.781311 24175 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:24:08.751464 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:08.782889 18090 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:24:08.783571 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:08.783986 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:08.784462 18095 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:08.680510 24118 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.618s	user 0.014s	sys 0.000s
I20260709 12:24:08.785902 18168 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:24:08.604288 23886 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:08.787551 18097 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
W20260709 12:24:08.739151 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.791335 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.790696 18020 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:08.687196 24141 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.790129 24118 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.793191 23610 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:08.794624 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.794994 24118 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.796074 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.800402 23986 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:08.801013 24114 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:08.739820 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57898.4us
I20260709 12:24:08.797538 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:08.805325 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2614) took 1284 ms (client timeout 1597 ms). Trace:
I20260709 12:24:08.805989 18018 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
W20260709 12:24:08.806160 18019 rpcz_store.cc:269] 0709 12:24:07.521295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.521476 (+   181us) service_pool.cc:224] Handling call
0709 12:24:08.796722 (+1275246us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:08.805225 (+  8503us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2168500992}
W20260709 12:24:08.735598 24096 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.027s	user 0.017s	sys 0.000s
I20260709 12:24:08.770546 18093 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:08.768594 24016 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:08.813169 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:08.814221 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:08.766891 18092 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:08.742136 24028 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.743144 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:08.815254 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 250832us
I20260709 12:24:08.815187 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:08.805512 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.816150 18806 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:08.808535 18097 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 18 index: 17. (index mismatch)
I20260709 12:24:08.774366 23457 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.752391 24110 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:08.811338 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:08.814144 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:08.819489 18017 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14.
I20260709 12:24:08.790271 18094 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:08.801153 23986 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.807808 18090 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 12:24:08.817091 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3524) took 1484 ms. Trace:
I20260709 12:24:08.824996 18099 rpcz_store.cc:276] 0709 12:24:07.332374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.341274 (+1008900us) service_pool.cc:224] Handling call
0709 12:24:08.389291 (+ 48017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.669404 (+280113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.816841 (+147437us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54002972d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:08.817021 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":313082240}
I20260709 12:24:08.824900 18019 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:08.816355 18020 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.821408 18017 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:24:08.828646 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:08.816143 18092 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:08.830011 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 949) took 1367 ms. Trace:
I20260709 12:24:08.830785 18092 rpcz_store.cc:276] 0709 12:24:07.462260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.815563 (+1353303us) service_pool.cc:224] Handling call
0709 12:24:08.815863 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.829708 (+ 13845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.829950 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.832413 18809 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:08.832685 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 177355us
I20260709 12:24:08.848488 23980 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:08.766125 18096 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:08.841905 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.826047 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:08.821872 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92722.5us
I20260709 12:24:08.834546 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:08.837167 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:08.848400 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:08.854660 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.855077 18810 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:08.817101 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 946) took 1355 ms. Trace:
I20260709 12:24:08.853664 18023 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.858228 18097 rpcz_store.cc:276] 0709 12:24:07.461278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.807790 (+1346512us) service_pool.cc:224] Handling call
0709 12:24:08.808175 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.816816 (+  8641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.817021 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.836334 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:08.785277 24097 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:08.839843 24007 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.394s	user 0.016s	sys 0.002s
I20260709 12:24:08.883005 18099 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:08.835152 23610 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.873018 18020 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:08.866746 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.888557 18019 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:08.887806 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.865507 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:08.889511 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.855688 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.874333 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.864661 18097 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:08.821326 24028 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.767905 24087 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.821314 23855 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:08.887308 24118 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:08.848383 23887 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:24:08.877481 23940 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:08.878719 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.813297 24096 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.876827 18096 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:08.919130 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:08.813002 24141 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:08.902213 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71601.8us
I20260709 12:24:08.923903 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:08.917527 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.862361 23902 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.898753 23898 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:08.903977 24172 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.917382 18095 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:08.920538 18096 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:08.898342 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:08.920578 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.919584 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.880770 23558 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:08.926108 18019 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:08.929032 18094 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:24:08.899405 24085 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:08.943141 18020 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:08.938184 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:08.943760 18022 raft_consensus.cc:2368] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17.
I20260709 12:24:08.890489 24007 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:08.943001 18094 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 12:24:08.946902 18094 pending_rounds.cc:85] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 11
I20260709 12:24:08.947197 18094 pending_rounds.cc:107] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 12:24:08.947549 18094 raft_consensus.cc:2889] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:08.938463 18015 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:08.914893 18023 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:08.937925 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.944831 18097 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:08.892007 23610 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.957335 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:08.951331 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:08.966204 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:08.945834 24118 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:08.968730 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:08.980214 18165 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:08.980688 18018 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:08.969934 18169 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:08.970335 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:08.983287 18167 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:08.926106 23886 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:08.942660 23965 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:08.970904 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:08.927107 24110 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:08.931236 23898 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.973253 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:09.029376 18019 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 8.
I20260709 12:24:08.977209 18021 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:08.951288 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.984738 18015 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:08.988574 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:09.032938 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:08.988662 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:08.929257 24086 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:24:08.926208 24158 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:24:09.031701 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:08.934736 23902 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.045725 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:09.046352 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:09.047026 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:09.048013 18019 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:09.040794 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:09.048349 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:08.986810 24145 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:09.058432 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:09.059767 18167 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14.
W20260709 12:24:09.044394 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93212.5us
I20260709 12:24:09.060889 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:09.061349 24191 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:09.038560 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:09.049576 24154 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:08.943526 24172 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.042253 18164 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:08.990161 23940 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:08.986173 24066 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:09.044291 24178 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:24:09.036108 18165 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.064677 18064 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14.
I20260709 12:24:09.094666 18064 leader_election.cc:403] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:24:09.064041 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:09.058585 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:08.977728 23887 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 12:24:09.027843 24033 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:09.041338 23898 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:08.993994 24085 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:08.989707 23886 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.086763 23986 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:09.087901 24174 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:24:09.088114 24172 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:09.040939 23965 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:09.090359 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:09.091442 24099 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.097311 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:09.099953 24033 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.102171 18810 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:09.105677 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:09.082439 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:09.136399 18171 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:08.987668 24167 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.106s
I20260709 12:24:09.053452 24110 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:09.138041 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77128.7us
I20260709 12:24:09.054005 23558 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:09.143563 23986 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:09.143798 24145 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:09.144731 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80771.9us
I20260709 12:24:09.144932 24004 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:09.067412 24047 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:09.211637 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 947) took 1750 ms. Trace:
I20260709 12:24:09.062330 24154 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:09.150700 18023 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:09.212194 18093 rpcz_store.cc:276] 0709 12:24:07.461597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.811666 (+1350069us) service_pool.cc:224] Handling call
0709 12:24:08.834291 (+ 22625us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.834554 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.834581 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.850312 (+ 15731us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.987405 (+137093us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:08.987447 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.987519 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.211369 (+223850us) raft_consensus.cc:1685] finished
0709 12:24:09.211393 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.211557 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":86218,"threads_started":1,"wal-append.queue_time_us":297317}
I20260709 12:24:09.213735 18023 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:09.063450 24181 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:09.215402 24028 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:09.154768 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 303735us
I20260709 12:24:09.104718 24066 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:09.172264 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 184381us
I20260709 12:24:09.103932 24040 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14.
I20260709 12:24:09.095824 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:09.175994 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78761.4us
W20260709 12:24:09.176352 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.183390 18173 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:09.193617 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:09.199849 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:09.177021 24172 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:09.205960 24004 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.206311 24123 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:09.208055 23940 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.210055 23558 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:09.217705 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:09.164057 18165 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:09.171751 23876 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:09.227442 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:09.190568 24033 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.215444 23965 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.184911 24191 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:09.215533 24097 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:09.237313 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:09.243265 18169 pending_rounds.cc:85] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 10
I20260709 12:24:09.276878 18169 pending_rounds.cc:107] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 12:24:09.277391 18169 raft_consensus.cc:2889] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:09.218572 24154 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:09.283659 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48670.9us
I20260709 12:24:09.234208 24123 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:09.233111 23986 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:09.292081 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:09.237692 23980 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:24:09.272765 24105 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.864s	user 0.012s	sys 0.004s
I20260709 12:24:09.233423 24004 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:09.287532 24047 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.284986 24007 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:09.284722 24014 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:09.295379 23887 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:09.237668 24172 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:09.336813 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44946.8us
I20260709 12:24:09.242118 24018 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:24:09.210541 24145 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.299387 24096 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.344103 18138 leader_election.cc:400] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 11 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12.
I20260709 12:24:09.305011 24158 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:09.290596 24188 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:24:09.274241 23876 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.335686 23969 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:09.337916 23887 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.200235 24118 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.356182 24114 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:09.314344 23986 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.325119 24092 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:24:09.351828 24194 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:09.369619 24007 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:09.401840 18164 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:24:09.402454 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.396739 23887 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:09.367040 24158 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:09.370007 24014 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:09.215992 24181 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.284181 24085 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:09.408950 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 191316us
I20260709 12:24:09.335760 24047 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.318320 24105 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.323522 24185 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.080s
I20260709 12:24:09.418798 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:09.392277 23902 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:09.420717 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76859.8us
W20260709 12:24:09.420686 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 324899us
I20260709 12:24:09.402293 23969 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:09.426877 24158 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.418442 24007 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:09.349156 23558 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.440456 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:09.441371 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:09.442581 24099 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:09.447968 24186 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 12:24:09.448444 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:09.449743 24189 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:24:09.451026 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:09.333225 24110 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:08.075305 18166 rpcz_store.cc:276] 0709 12:24:06.364493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.365927 (+  1434us) service_pool.cc:224] Handling call
0709 12:24:06.409444 (+ 43517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.590220 (+180776us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:07.512598 (+922378us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.062828 (+550230us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:08.063052 (+   224us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:08.071003 (+  7951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.071115 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.071309 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":331671,"prepare.queue_time_us":378271,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":230362,"raft.run_cpu_time_us":447,"raft.run_wall_time_us":50415,"thread_start_us":1248292,"threads_started":3}
I20260709 12:24:09.453464 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:09.387634 24191 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.455731 24105 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.457955 18814 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:09.507931 18809 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:09.430915 24010 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.485503 18165 raft_consensus.cc:1217] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Deduplicated request from leader. Original: 11.12->[12.13-15.14]   Dedup: 12.13->[15.14-15.14]
I20260709 12:24:09.512333 18165 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [12.13-15.14]
W20260709 12:24:09.477272 24183 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.251s	user 0.002s	sys 0.000s
I20260709 12:24:09.475469 24066 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.501073 18813 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:24:09.431185 23886 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:09.499248 18807 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:09.519448 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:09.473258 24201 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:24:09.493934 23902 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:09.526793 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:09.541169 18169 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:09.528494 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:09.459862 23969 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.542582 18165 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:09.508767 24198 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:09.564008 24047 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.564720 24172 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:09.536952 24201 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:09.549363 23886 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:09.525555 24010 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.610661 23986 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.525992 24014 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.593189 23902 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:09.640575 24016 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.801s	user 0.014s	sys 0.001s
I20260709 12:24:09.641419 24010 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.642661 23902 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.643389 24123 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.644124 24114 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:09.644821 24114 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:09.614605 24172 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:09.641165 24016 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.565342 24194 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:24:09.646325 24172 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.671803 24142 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:09.620138 24198 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:09.674862 24198 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.677309 24087 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:09.683542 24144 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.287s	user 0.005s	sys 0.000s
W20260709 12:24:09.685511 24182 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.548s	user 0.000s	sys 0.001s
I20260709 12:24:09.539803 24007 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.690692 24007 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:09.690860 23610 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.692749 23921 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:09.693326 23921 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:09.695176 24187 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.367s	user 0.001s	sys 0.000s
I20260709 12:24:09.636423 23876 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:09.701311 24129 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:09.701678 24172 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:09.701924 24129 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:09.702425 24114 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.702910 24129 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:09.703898 24114 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.704159 24129 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:09.622112 24183 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:24:09.611696 24116 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:09.615919 24201 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:24:09.719866 24149 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.343s	user 0.003s	sys 0.001s
I20260709 12:24:09.747684 24086 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:09.748385 24086 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:09.752496 24141 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:09.723604 24116 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:09.764353 24181 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:09.764470 24092 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:09.768239 24123 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:09.770480 24092 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:09.818537 24154 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.006s
I20260709 12:24:09.819375 24154 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:09.749611 24086 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:09.839299 24155 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.438s	user 0.010s	sys 0.005s
I20260709 12:24:09.843444 24092 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:09.841269 24167 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.006s	sys 0.003s
W20260709 12:24:09.867233 24167 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.007s	sys 0.003s
W20260709 12:24:09.868645 24155 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.479s	user 0.011s	sys 0.005s
I20260709 12:24:09.869424 24154 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:09.923245 24176 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.000s	sys 0.014s
I20260709 12:24:09.948113 24158 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:09.942281 24176 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.000s	sys 0.015s
I20260709 12:24:09.964329 24201 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.008844 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:10.054724 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.000s	sys 0.008s
W20260709 12:24:10.055694 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.001s	sys 0.009s
I20260709 12:24:10.059300 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2619) took 2508 ms. Trace:
W20260709 12:24:10.060232 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.061471 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.062882 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3458) took 1583 ms. Trace:
I20260709 12:24:09.504602 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.065618 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2758) took 1464 ms. Trace:
I20260709 12:24:09.506428 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:10.066592 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 581124us
W20260709 12:24:10.067158 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.506426 18097 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:10.060269 18016 rpcz_store.cc:276] 0709 12:24:07.550915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.551083 (+   168us) service_pool.cc:224] Handling call
0709 12:24:07.572873 (+ 21790us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.624941 (+ 52068us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.624988 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.639183 (+ 14195us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.017384 (+378201us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:08.051310 (+ 33926us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.056050 (+  4740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.056155 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.719087 (+1662932us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:10.058150 (+339063us) raft_consensus.cc:1685] finished
0709 12:24:10.058436 (+   286us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.058975 (+   539us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62327,"prepare.queue_time_us":32905,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":95956,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":19322,"spinlock_wait_cycles":15703296,"thread_start_us":326431,"threads_started":1,"wal-append.queue_time_us":528878}
I20260709 12:24:10.072393 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:10.074460 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.000s	sys 0.011s
I20260709 12:24:10.074800 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:10.075242 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 951) took 2601 ms (client timeout 1907 ms). Trace:
I20260709 12:24:10.076735 18096 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:10.074460 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.003s	sys 0.002s
W20260709 12:24:10.078251 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.004s	sys 0.002s
W20260709 12:24:10.078817 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 646981us
I20260709 12:24:10.079533 18098 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:10.079743 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.066322 18099 rpcz_store.cc:276] 0709 12:24:08.601563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.884234 (+282671us) service_pool.cc:224] Handling call
0709 12:24:08.884672 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.885164 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.885191 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.889308 (+  4117us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.101282 (+211974us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:09.101332 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.101407 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.694345 (+592938us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:10.064598 (+370253us) raft_consensus.cc:1685] finished
0709 12:24:10.064649 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.065297 (+   648us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"spinlock_wait_cycles":62664576,"thread_start_us":176269,"threads_started":1,"wal-append.queue_time_us":337426}
W20260709 12:24:10.077505 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 629099us
I20260709 12:24:10.063531 18017 rpcz_store.cc:276] 0709 12:24:08.479010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.828840 (+349830us) service_pool.cc:224] Handling call
0709 12:24:08.829176 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.839695 (+ 10519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.839733 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.840314 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.006772 (+166458us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:09.052252 (+ 45480us) log.cc:844] Serialized 14 byte log entry
0709 12:24:09.079598 (+ 27346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.079722 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.062105 (+982383us) raft_consensus.cc:1685] finished
0709 12:24:10.062157 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.062606 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78139,"prepare.queue_time_us":45342,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":13258,"raft.queue_time_us":38449,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":26439,"thread_start_us":90483,"threads_started":1,"wal-append.queue_time_us":242162}
I20260709 12:24:10.069574 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.086439 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3527) took 2684 ms. Trace:
I20260709 12:24:10.086943 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:10.112862 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:10.113685 18094 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:10.081120 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:10.121955 18095 rpcz_store.cc:276] 0709 12:24:07.401749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.785011 (+1383262us) service_pool.cc:224] Handling call
0709 12:24:08.822278 (+ 37267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.931493 (+109215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.084566 (+1153073us) spinlock_profiling.cc:243] Waited 94.7 ms on lock 0x7b540025e254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:10.085847 (+  1281us) spinlock_profiling.cc:243] Waited 7.17 ms on lock 0x7b540025e250. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.086196 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234348544}
I20260709 12:24:10.089039 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:10.123327 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3526) took 2713 ms. Trace:
I20260709 12:24:10.124090 18090 rpcz_store.cc:276] 0709 12:24:07.360400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.784988 (+1424588us) service_pool.cc:224] Handling call
0709 12:24:08.807498 (+ 22510us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:08.824141 (+ 16643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.073201 (+1249060us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b5400052150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.073598 (+   397us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50664320}
I20260709 12:24:10.124930 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:10.075968 18091 rpcz_store.cc:269] 0709 12:24:07.473906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.643988 (+170082us) service_pool.cc:224] Handling call
0709 12:24:10.074545 (+2430557us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.074934 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2453888256}
I20260709 12:24:10.125881 18138 leader_election.cc:400] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14.
I20260709 12:24:10.089367 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3451) took 1625 ms. Trace:
W20260709 12:24:10.091147 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3445) took 1646 ms (client timeout 1691 ms). Trace:
I20260709 12:24:10.095712 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2700) took 1136 ms. Trace:
I20260709 12:24:10.096057 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2705) took 1123 ms. Trace:
W20260709 12:24:10.097952 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2684) took 2803 ms (client timeout 1636 ms). Trace:
I20260709 12:24:10.098376 18165 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:10.088418 18097 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:10.100212 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2598) took 2806 ms. Trace:
W20260709 12:24:10.104624 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2695) took 1162 ms (client timeout 1533 ms). Trace:
I20260709 12:24:10.100977 18172 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:24:10.106588 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.106660 18806 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.014s	sys 0.021s
I20260709 12:24:10.108050 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:09.504935 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:10.127352 18022 rpcz_store.cc:269] 0709 12:24:08.444447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.480087 (+ 35640us) service_pool.cc:224] Handling call
0709 12:24:10.090517 (+1610430us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b540004c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.090872 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1064177920}
I20260709 12:24:10.154390 18092 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:10.155319 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:10.127821 18165 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:10.062667 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 869108us
I20260709 12:24:10.112924 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:10.087530 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 798) took 1205 ms (client timeout 1542 ms). Trace:
I20260709 12:24:10.122888 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3519) took 2859 ms. Trace:
W20260709 12:24:10.149892 18806 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.015s	sys 0.021s
I20260709 12:24:10.091732 18023 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: [16.10-16.10]
W20260709 12:24:10.106985 18811 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.002s	sys 0.013s
I20260709 12:24:10.127115 18021 rpcz_store.cc:276] 0709 12:24:08.463431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.627951 (+164520us) service_pool.cc:224] Handling call
0709 12:24:10.088774 (+1460823us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54001cf554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.089112 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":799909120}
W20260709 12:24:10.128518 18173 rpcz_store.cc:269] 0709 12:24:08.942242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.942570 (+   328us) service_pool.cc:224] Handling call
0709 12:24:10.104143 (+1161573us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.104387 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":260110080}
I20260709 12:24:10.117638 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:10.129238 18168 rpcz_store.cc:269] 0709 12:24:07.294101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.313016 (+ 18915us) service_pool.cc:224] Handling call
0709 12:24:10.097210 (+2784194us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54001b6054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.097612 (+   402us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3384301440}
I20260709 12:24:10.127497 18164 rpcz_store.cc:276] 0709 12:24:08.971974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.082022 (+110048us) service_pool.cc:224] Handling call
0709 12:24:10.095527 (+1013505us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54000433d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.095773 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":495589120}
W20260709 12:24:10.157570 18020 rpcz_store.cc:269] 0709 12:24:08.881496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.887490 (+  5994us) service_pool.cc:224] Handling call
0709 12:24:10.086610 (+1199120us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.086944 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96932608}
W20260709 12:24:10.187314 18811 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.002s	sys 0.013s
I20260709 12:24:10.158200 18098 rpcz_store.cc:276] 0709 12:24:07.263093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.962202 (+699109us) service_pool.cc:224] Handling call
0709 12:24:10.078878 (+2116676us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.080361 (+  1483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.122495 (+ 42134us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b5400297cd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.122776 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4866308864}
I20260709 12:24:09.839004 24086 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:10.202061 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.009s	sys 0.009s
W20260709 12:24:10.202729 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.010s	sys 0.009s
I20260709 12:24:10.207842 18096 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.210551 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.213215 24085 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:10.181710 23965 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.000s	sys 0.010s
I20260709 12:24:10.214061 18022 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:10.198648 23854 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:10.230185 24201 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:10.230875 18017 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:24:10.231967 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 794) took 1408 ms. Trace:
I20260709 12:24:10.232059 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.232530 18017 rpcz_store.cc:276] 0709 12:24:08.823946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.206826 (+1382880us) service_pool.cc:224] Handling call
0709 12:24:10.230086 (+ 23260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.231535 (+  1449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.231868 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.233659 18171 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:10.235486 23886 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:10.239065 23921 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.240119 18165 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.242548 18812 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:10.243220 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2686) took 2948 ms. Trace:
I20260709 12:24:10.243822 18172 rpcz_store.cc:276] 0709 12:24:07.294803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.779073 (+484270us) service_pool.cc:224] Handling call
0709 12:24:10.100011 (+2320938us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.157986 (+ 57975us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.242836 (+ 84850us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b5400202bd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.243098 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5176483072}
I20260709 12:24:10.246496 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3461) took 1766 ms. Trace:
I20260709 12:24:10.250473 24154 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:24:10.252363 23898 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.104s	user 0.006s	sys 0.010s
W20260709 12:24:10.253654 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 136387us
I20260709 12:24:10.254475 23610 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.254905 23866 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:10.255801 23610 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.256345 23866 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.257771 23886 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:10.258400 24016 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.127403 18167 rpcz_store.cc:276] 0709 12:24:08.958603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.187304 (+228701us) service_pool.cc:224] Handling call
0709 12:24:09.209330 (+ 22026us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:09.226453 (+ 17123us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.226488 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:09.235365 (+  8877us) log.cc:844] Serialized 52 byte log entry
0709 12:24:09.487149 (+251784us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:09.516516 (+ 29367us) log.cc:844] Serialized 14 byte log entry
0709 12:24:09.516900 (+   384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.517024 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.094539 (+577515us) raft_consensus.cc:1685] finished
0709 12:24:10.094612 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.095217 (+   605us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53872,"prepare.queue_time_us":250638,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":77773,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":58397,"thread_start_us":181901,"threads_started":1,"wal-append.queue_time_us":248281}
I20260709 12:24:10.258863 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:10.260200 18020 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:24:10.259590 23886 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.261117 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2628) took 1497 ms. Trace:
I20260709 12:24:10.261576 18020 rpcz_store.cc:276] 0709 12:24:08.764124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.192902 (+1428778us) service_pool.cc:224] Handling call
0709 12:24:10.215070 (+ 22168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.260708 (+ 45638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.261013 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.256161 24210 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:10.265285 18165 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.229557 18022 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:10.253139 23898 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.292157 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 799) took 1404 ms. Trace:
I20260709 12:24:10.294386 18022 rpcz_store.cc:276] 0709 12:24:08.887772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.228541 (+1340769us) service_pool.cc:224] Handling call
0709 12:24:10.229170 (+   629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.291786 (+ 62616us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.292084 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.247191 18019 rpcz_store.cc:276] 0709 12:24:08.479861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.048848 (+568987us) service_pool.cc:224] Handling call
0709 12:24:09.049151 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.153734 (+104583us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:09.201415 (+ 47681us) log.cc:844] Serialized 14 byte log entry
0709 12:24:09.492012 (+290597us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.746721 (+254709us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b18003e4340. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:10.094079 (+347358us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:10.095281 (+  1202us) log.cc:844] Serialized 26 byte log entry
0709 12:24:10.095656 (+   375us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:10.096290 (+   634us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.159270 (+ 62980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.183652 (+ 24382us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.245836 (+ 62184us) raft_consensus.cc:1685] finished
0709 12:24:10.245871 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.246197 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":104005,"prepare.queue_time_us":101213,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":3609,"raft.queue_time_us":124520,"raft.run_cpu_time_us":507,"raft.run_wall_time_us":29733,"spinlock_wait_cycles":27673216,"thread_start_us":247596,"threads_started":1,"wal-append.queue_time_us":466701}
I20260709 12:24:10.259496 24066 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:24:10.225363 23965 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.001s	sys 0.010s
I20260709 12:24:10.128417 18170 rpcz_store.cc:276] 0709 12:24:07.293329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.696164 (+402835us) service_pool.cc:224] Handling call
0709 12:24:07.745161 (+ 48997us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.646386 (+901225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.646423 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.647089 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.789879 (+142790us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:08.789974 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.099175 (+1309201us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:10.099265 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.099364 (+    99us) raft_consensus.cc:1685] finished
0709 12:24:10.099384 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.099907 (+   523us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4662,"raft.queue_time_us":8568,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":14970,"spinlock_wait_cycles":2071480576,"thread_start_us":142214,"threads_started":1,"wal-append.queue_time_us":254224}
W20260709 12:24:10.266523 23969 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.003s	sys 0.011s
W20260709 12:24:10.318485 24082 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.003s	sys 0.009s
W20260709 12:24:10.312045 23969 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.003s	sys 0.012s
W20260709 12:24:10.320111 24082 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.003s	sys 0.010s
I20260709 12:24:10.355031 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:10.355355 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 229932us
I20260709 12:24:10.361704 18093 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:10.363937 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.354138 23610 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:10.373099 24233 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:24:10.374260 23866 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:10.373951 24090 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 12:24:10.379458 18020 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:10.381074 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 801) took 1484 ms. Trace:
I20260709 12:24:10.382603 18022 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:10.383596 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 805) took 1466 ms. Trace:
I20260709 12:24:10.384004 18022 rpcz_store.cc:276] 0709 12:24:08.917669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.308909 (+1391240us) service_pool.cc:224] Handling call
0709 12:24:10.382194 (+ 73285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.383261 (+  1067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.383523 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.385294 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:10.385376 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.411489 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.388348 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:10.414785 18168 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:10.389004 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80271.8us
I20260709 12:24:10.418695 18022 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:10.419099 24210 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:10.419924 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 807) took 1476 ms. Trace:
I20260709 12:24:10.420095 24210 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.420513 18022 rpcz_store.cc:276] 0709 12:24:08.943389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.417753 (+1474364us) service_pool.cc:224] Handling call
0709 12:24:10.418320 (+   567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.419498 (+  1178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.419816 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.422240 24010 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:10.422335 23610 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:10.422857 18022 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:10.423676 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 808) took 1446 ms. Trace:
I20260709 12:24:10.423190 24010 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.425163 23980 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:10.425679 23980 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:10.427780 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 217767us
I20260709 12:24:10.427446 23980 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.430265 24155 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:10.430900 24155 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:10.431900 23980 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:10.424144 18022 rpcz_store.cc:276] 0709 12:24:08.977726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.421875 (+1444149us) service_pool.cc:224] Handling call
0709 12:24:10.422393 (+   518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.423304 (+   911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.423581 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.422024 18814 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:10.396132 24158 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:10.396409 24033 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.118s	user 0.016s	sys 0.000s
W20260709 12:24:10.402227 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.362334 23940 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.007s	sys 0.006s
I20260709 12:24:10.403095 18099 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:10.357447 24085 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.404002 24066 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:10.381558 18020 rpcz_store.cc:276] 0709 12:24:08.898138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.262710 (+1364572us) service_pool.cc:224] Handling call
0709 12:24:10.263425 (+   715us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.380077 (+116652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.380763 (+   686us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001d1d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:10.380993 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266020096}
I20260709 12:24:10.408891 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:10.354192 23558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.010s	sys 0.030s
I20260709 12:24:10.390331 23921 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.432636 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.432262 24155 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.430282 18019 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:24:10.433983 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:10.437489 18017 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:24:10.438984 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2623) took 1722 ms. Trace:
W20260709 12:24:10.404043 24004 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.080s	user 0.013s	sys 0.004s
I20260709 12:24:10.394579 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.370337 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2622) took 1650 ms. Trace:
I20260709 12:24:10.467712 18016 rpcz_store.cc:276] 0709 12:24:08.715377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.197215 (+1481838us) service_pool.cc:224] Handling call
0709 12:24:10.296928 (+ 99713us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.359664 (+ 62736us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.359701 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.365070 (+  5369us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:10.365156 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.365231 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.365405 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.445122 18167 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:10.443341 18814 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:10.463795 18021 rpcz_store.cc:276] 0709 12:24:08.716551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.301566 (+1585015us) service_pool.cc:224] Handling call
0709 12:24:10.396813 (+ 95247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.397286 (+   473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.397335 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.397422 (+    87us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:10.397498 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.397577 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.438895 (+ 41318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.445943 18172 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:10.446197 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:10.449460 18809 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:10.457348 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 810) took 1466 ms. Trace:
I20260709 12:24:10.474447 18022 rpcz_store.cc:276] 0709 12:24:08.988776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.447337 (+1458561us) service_pool.cc:224] Handling call
0709 12:24:10.448344 (+  1007us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.454659 (+  6315us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.454705 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.455085 (+   380us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:10.455134 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.455222 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.455407 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.474292 18809 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:10.457273 23940 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.008s	sys 0.006s
I20260709 12:24:10.480532 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 804) took 1572 ms. Trace:
I20260709 12:24:10.476574 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 800) took 1585 ms. Trace:
I20260709 12:24:10.487304 18019 rpcz_store.cc:276] 0709 12:24:08.908204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.296477 (+1388273us) service_pool.cc:224] Handling call
0709 12:24:10.355987 (+ 59510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.475699 (+119712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.480441 (+  4742us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.487421 18017 rpcz_store.cc:276] 0709 12:24:08.891085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.296512 (+1405427us) service_pool.cc:224] Handling call
0709 12:24:10.360036 (+ 63524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.466753 (+106717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.476465 (+  9712us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:10.466406 23558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.011s	sys 0.030s
I20260709 12:24:10.500140 24210 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:10.502274 23887 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.073s	user 0.010s	sys 0.011s
I20260709 12:24:10.502403 18812 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:10.503079 23887 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.504884 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 812) took 1479 ms. Trace:
I20260709 12:24:10.505409 18020 rpcz_store.cc:276] 0709 12:24:09.025569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.462006 (+1436437us) service_pool.cc:224] Handling call
0709 12:24:10.504044 (+ 42038us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.504393 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:10.504429 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.504504 (+    75us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:10.504551 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.504636 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.504801 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.506879 24141 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.507660 24141 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.508708 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:10.509521 18167 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:10.509790 18814 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.500214 23898 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.530603 18814 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:10.511897 18018 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:10.512529 18170 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:10.496855 24004 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.536628 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.537607 18099 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:10.513466 24141 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.516842 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:10.520721 24085 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:10.524507 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78519us
I20260709 12:24:10.517289 18809 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.459245 24033 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.542330 18016 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:10.560060 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.581921 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.559947 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.575307 23969 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:10.591589 23969 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:10.591709 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:10.593745 23664 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:10.573585 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:10.634398 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.543277 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2625) took 1826 ms (client timeout 1949 ms). Trace:
I20260709 12:24:10.620275 18168 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
W20260709 12:24:10.555220 18138 connection.cc:256] RPC call timeout handler was delayed by -1.44842977899998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
W20260709 12:24:10.637553 18138 connection.cc:256] RPC call timeout handler was delayed by -1.3548297790002835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
W20260709 12:24:10.638170 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.564116 18166 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:10.639539 18165 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:10.599026 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2635) took 1116 ms. Trace:
W20260709 12:24:10.636579 18015 rpcz_store.cc:269] 0709 12:24:08.717154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.726546 (+  9392us) service_pool.cc:224] Handling call
0709 12:24:10.542727 (+1816181us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.543014 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462838016}
W20260709 12:24:10.642256 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.643013 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.643997 18091 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:10.558353 23931 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:10.644678 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:10.645437 23931 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:10.646816 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55343.3us
I20260709 12:24:10.593143 23969 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.647792 23969 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.647891 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:10.648979 23940 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:10.650065 23940 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:10.652385 23931 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:10.634270 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:10.659219 18815 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:10.660539 18815 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:10.661245 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:10.661681 24176 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:10.662931 18809 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.664273 18090 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:24:10.661257 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144546us
I20260709 12:24:10.666332 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:10.667074 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.667629 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.669203 24218 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:10.670346 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:10.672374 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.654742 24142 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.982s	user 0.016s	sys 0.009s
I20260709 12:24:10.655077 18810 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:10.545342 24155 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.645694 18807 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:10.662973 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.677902 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:10.680068 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2713) took 1158 ms. Trace:
I20260709 12:24:10.680802 18096 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:10.640965 18016 rpcz_store.cc:276] 0709 12:24:09.482468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.497328 (+1014860us) service_pool.cc:224] Handling call
0709 12:24:10.525712 (+ 28384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.598637 (+ 72925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.598942 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.685330 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3467) took 2130 ms. Trace:
I20260709 12:24:10.537714 24004 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:10.686674 18165 rpcz_store.cc:276] 0709 12:24:09.481905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.096785 (+614880us) service_pool.cc:224] Handling call
0709 12:24:10.097861 (+  1076us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.640036 (+542175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.640246 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.691465 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.692605 18165 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:10.694831 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.688371 18023 rpcz_store.cc:276] 0709 12:24:08.555291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.214652 (+659361us) service_pool.cc:224] Handling call
0709 12:24:10.090879 (+876227us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.188183 (+ 97304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.676716 (+488533us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b540004c4d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:10.685233 (+  8517us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2012443008}
I20260709 12:24:10.694235 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:10.541621 24014 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.565383 24010 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:10.558684 23457 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:10.673362 24220 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.432s	user 0.000s	sys 0.001s
I20260709 12:24:10.674203 24215 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:24:10.714387 24087 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.715306 24215 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:10.715196 24087 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.715919 24215 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:10.716414 24235 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:24:10.718211 24085 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.719069 24085 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.710922 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.722424 23952 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:10.722652 18091 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:24:10.722960 23952 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:10.722656 18090 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:10.724597 23952 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.710932 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:10.745702 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:10.707011 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.669709 24218 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:10.725994 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.722805 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.764474 18095 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:10.730165 24116 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.958s	user 0.015s	sys 0.007s
I20260709 12:24:10.730365 24085 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.717397 24215 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.732870 24092 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:10.734164 24114 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.030s	user 0.025s	sys 0.000s
I20260709 12:24:10.736153 23457 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:10.737622 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87965.4us
W20260709 12:24:10.738166 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.738356 24014 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:10.742098 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2009297790000346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
I20260709 12:24:10.742524 24216 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:24:10.746305 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.746428 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:10.750815 24010 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:10.753124 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.755302 18814 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:24:10.756632 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.741994 23855 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.697686 23887 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.766295 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3463) took 2239 ms. Trace:
I20260709 12:24:10.775079 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.765285 24218 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:10.784296 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0731297790002827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21080000000000002s for a response
W20260709 12:24:10.787160 24007 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.096s	user 0.016s	sys 0.007s
I20260709 12:24:10.780403 23952 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:10.812553 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0254297790002056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
W20260709 12:24:10.814836 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0074297790001765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
I20260709 12:24:10.808135 18018 rpcz_store.cc:276] 0709 12:24:08.527450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.063119 (+535669us) service_pool.cc:224] Handling call
0709 12:24:09.135318 (+ 72199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.534378 (+1399060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.745706 (+211328us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540024de54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:10.766213 (+ 20507us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3164279680}
W20260709 12:24:10.823179 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 222752us
W20260709 12:24:10.830754 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 164979us
I20260709 12:24:10.835224 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1105) took 1402 ms. Trace:
W20260709 12:24:10.837097 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 531813us
I20260709 12:24:10.838724 24118 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:10.835832 18806 rpcz_store.cc:276] 0709 12:24:09.432378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.432564 (+   186us) service_pool.cc:224] Handling call
0709 12:24:09.433037 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.433366 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:10.224156 (+790790us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.746857 (+522701us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b48000cc258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:24:10.833373 (+ 86516us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.834697 (+  1324us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.834782 (+    85us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:10.834843 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.834950 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.835105 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45826,"prepare.queue_time_us":402569,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":987,"raft.queue_time_us":42785,"raft.run_cpu_time_us":2361,"raft.run_wall_time_us":384551,"spinlock_wait_cycles":56363904,"thread_start_us":1073987,"threads_started":3,"wal-append.queue_time_us":224850}
I20260709 12:24:10.845386 24158 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:10.847187 24118 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:10.847695 24118 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:10.848946 24118 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.816057 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:10.858858 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.861670 18093 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:10.863448 18166 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:10.864626 18165 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:10.866693 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:10.867578 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:10.780679 24116 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.865091 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.814016 24047 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.875988 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:10.878942 18064 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 13. Current term is 14.
I20260709 12:24:10.853075 24198 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 12:24:10.879560 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:10.865752 24090 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:10.819856 24142 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:10.870575 23931 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:10.812685 23855 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.885300 24215 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:10.893013 24129 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.177s	user 0.011s	sys 0.015s
I20260709 12:24:10.893129 24142 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.884687 24118 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.896001 24142 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:10.896605 24086 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.562s	user 0.015s	sys 0.000s
I20260709 12:24:10.897295 24194 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 12:24:10.899624 24116 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:10.889074 23887 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:24:10.902920 24014 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.903059 24030 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:10.903929 24030 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.906435 24216 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:10.907125 24216 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.907582 24216 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.889182 23931 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.910416 23931 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.881958 24016 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:24:10.918403 24030 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.919416 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:10.921396 23886 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:10.921980 24226 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:10.922792 24226 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.926364 24191 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:10.927122 24191 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:10.928216 24191 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.930140 23854 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:10.931169 23854 consensus_queue.cc:1243] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:10.936234 24216 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.938251 24176 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:10.903955 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:10.940629 24176 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:10.940809 23921 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:10.941304 23921 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:10.942451 23921 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.942587 24176 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.947111 23972 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:10.947996 24176 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.947962 23972 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:10.950098 24194 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.954375 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1111) took 1441 ms. Trace:
I20260709 12:24:10.954882 24216 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:10.897651 18815 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:10.954967 18811 rpcz_store.cc:276] 0709 12:24:09.513244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.513967 (+   723us) service_pool.cc:224] Handling call
0709 12:24:09.536127 (+ 22160us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.536463 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:10.193106 (+656643us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.467334 (+274228us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.549246 (+ 81912us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b48000cc258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:24:10.744927 (+195681us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b180045d2a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:10.944797 (+199870us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b180045d2a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:24:10.945115 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.945376 (+   261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.945463 (+    87us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:10.945563 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.945659 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.946703 (+  1044us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b180045d2a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:10.954220 (+  7517us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":315364,"prepare.queue_time_us":785127,"prepare.run_cpu_time_us":169,"prepare.run_wall_time_us":177,"raft.queue_time_us":69124,"raft.run_cpu_time_us":4584,"raft.run_wall_time_us":192808,"spinlock_wait_cycles":235139456,"thread_start_us":738448,"threads_started":2,"wal-append.queue_time_us":153143}
I20260709 12:24:10.949364 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:10.961645 18809 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 12:24:10.962720 18809 pending_rounds.cc:85] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 11
I20260709 12:24:10.957513 24087 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:10.980140 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:10.978691 18095 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:24:10.970378 18809 pending_rounds.cc:107] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:24:10.991328 18809 raft_consensus.cc:2889] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:10.988760 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:10.995904 18016 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:10.999763 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:11.000528 18096 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.001531 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:10.993651 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:10.972579 23921 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:11.008976 18015 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:11.010353 18097 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:11.014781 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:10.892863 24047 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:10.998925 24141 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.025306 18098 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
W20260709 12:24:11.027038 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57684.5us
W20260709 12:24:11.030035 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80816.3us
I20260709 12:24:11.034303 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:10.993665 24216 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:11.036695 24216 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:11.037654 24141 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.040969 18020 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:11.035753 24047 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.047560 18096 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:11.006887 24116 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:24:11.012668 24087 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:24:11.054232 24155 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.021167 24218 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.089443 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:11.100066 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.100530 24020 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
W20260709 12:24:11.101277 24253 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:24:11.101585 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.102746 18165 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:11.117301 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.100970 18169 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260709 12:24:11.145959 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 83182.1us
I20260709 12:24:11.148252 18170 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:11.162423 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 128846us
I20260709 12:24:11.098884 23972 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:11.169315 23664 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.572s	user 0.015s	sys 0.010s
I20260709 12:24:11.099233 24155 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.200273 24155 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.208611 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:11.170923 24047 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:11.256579 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.306926 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 96558.6us
I20260709 12:24:11.210500 24087 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.314563 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.222828 24176 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.326635 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.328006 18168 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:24:11.219810 24116 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.104s
I20260709 12:24:11.334311 18024 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:11.336015 24033 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.338353 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:11.338780 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.339207 18171 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.324491 24087 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.314688 24118 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:11.354477 18173 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:11.342456 24116 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:11.310370 24047 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.363098 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.365723 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.368322 18168 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:11.338346 24114 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:11.320250 24155 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:24:11.350000 24018 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.371789 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.361703 24033 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.330767 24176 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.363914 24194 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:11.385092 24087 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.386256 24218 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.386718 24033 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.411502 24033 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:11.411553 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.414603 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.417659 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:11.418439 18813 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
W20260709 12:24:11.420367 24215 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.506s	user 0.011s	sys 0.002s
I20260709 12:24:11.382122 24191 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.421854 24191 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.424559 24191 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.381685 24263 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.426122 24263 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.429005 24263 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.431375 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:11.432246 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.413480 24090 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:11.433233 24090 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:11.434098 18138 leader_election.cc:400] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17.
W20260709 12:24:11.435321 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61748.7us
W20260709 12:24:11.390002 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80625.8us
I20260709 12:24:11.357931 24226 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.393846 24047 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.394411 24018 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.403760 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:11.404888 24007 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.407389 23972 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.361248 23664 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.438925 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:11.443925 18165 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:11.371682 24114 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:11.444640 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:11.457943 18173 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:11.458582 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.458719 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.454787 18171 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:11.464979 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.462754 18095 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
W20260709 12:24:11.453121 23952 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.632s	user 0.009s	sys 0.006s
I20260709 12:24:11.370918 24118 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:11.473320 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.481688 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:11.482602 18168 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:11.483187 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:11.490114 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.487257 18099 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:11.488977 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:11.496171 24114 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.498804 24007 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:11.502485 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.503577 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.472746 24226 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:11.463536 23972 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.513999 18171 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:24:11.522411 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60239.4us
I20260709 12:24:11.523453 24118 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.506409 24191 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.524799 24191 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.508767 24215 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.525939 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1286) took 2301 ms. Trace:
I20260709 12:24:11.526437 18810 rpcz_store.cc:276] 0709 12:24:09.224677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.224848 (+   171us) service_pool.cc:224] Handling call
0709 12:24:09.225232 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.716686 (+1491454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.525564 (+808878us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54006f2cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:11.525844 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3258315520}
I20260709 12:24:11.527468 24191 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.522462 18092 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:24:11.530807 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.533327 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:11.515405 23921 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.471726 23664 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.544348 24087 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:11.546625 23855 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.547881 18018 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:24:11.548913 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2712) took 2067 ms. Trace:
I20260709 12:24:11.544625 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:11.549377 18169 rpcz_store.cc:276] 0709 12:24:09.481176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.542165 (+ 60989us) service_pool.cc:224] Handling call
0709 12:24:10.099574 (+557409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.157558 (+1057984us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.548667 (+391109us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b5400383454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:11.548850 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2218519296}
I20260709 12:24:11.550216 18017 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:11.551831 24264 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
I20260709 12:24:11.552124 24269 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:11.553469 23969 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:11.553977 23969 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:11.555137 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 151511us
I20260709 12:24:11.563583 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:11.551838 24092 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.574628 24092 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.539124 24191 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:24:11.579474 24176 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.579717 24268 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:11.580299 24268 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.580716 24268 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:11.561146 24114 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.583387 23972 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.580575 24191 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:11.584640 24191 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:11.565202 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:11.589463 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3465) took 3061 ms (client timeout 1795 ms). Trace:
W20260709 12:24:11.590106 18024 rpcz_store.cc:269] 0709 12:24:08.528157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.536650 (+  8493us) service_pool.cc:224] Handling call
0709 12:24:11.586986 (+3050336us) spinlock_profiling.cc:243] Waited 2.57 s on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:11.589334 (+  2348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5917699456}
I20260709 12:24:11.554560 24269 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:11.592489 24269 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.593930 18094 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:11.597597 24176 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:11.589540 24116 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:11.601220 24116 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.601646 24116 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.526458 24215 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.611354 23969 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.613108 23969 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:11.616837 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:11.617386 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.617960 18808 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:11.618305 18020 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:11.618559 18019 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:11.619351 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:11.619849 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.619917 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.620446 18022 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:11.622376 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.621582 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.635627 24007 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.635809 18019 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:11.636547 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:11.601577 24141 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.643436 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.644757 18023 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:24:11.597656 23931 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.687s	user 0.014s	sys 0.002s
I20260709 12:24:11.645637 23931 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.646875 18023 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:24:11.647738 18015 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:11.648604 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:11.621650 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:11.650022 18022 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.629482 24018 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.639659 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:11.637127 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:11.622407 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:11.657719 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:11.658526 18806 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.664058 18168 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:11.629187 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.677389 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:11.671252 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.630630 24201 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:11.654937 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:11.673314 18021 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.673522 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.682122 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.680207 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:11.686173 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.681408 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.628101 23664 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:11.656102 23931 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.687732 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.693264 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.698930 18170 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:11.695782 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.704367 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.705986 23855 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:11.722357 23855 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.722862 23855 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.723806 24116 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.717805 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1297) took 1083 ms. Trace:
I20260709 12:24:11.724891 18815 rpcz_store.cc:276] 0709 12:24:10.634942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.661291 (+ 26349us) service_pool.cc:224] Handling call
0709 12:24:10.661627 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.958608 (+296981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.706351 (+747743us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54006153d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:11.717702 (+ 11351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":541699328}
I20260709 12:24:11.726739 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.718323 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:11.727954 24270 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.174s	user 0.002s	sys 0.000s
I20260709 12:24:11.728381 18168 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 12:24:11.729456 18164 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:11.730675 23931 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:11.731732 18091 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:24:11.732534 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.732612 23931 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.734444 18098 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:11.733213 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.737536 18091 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:11.738747 18095 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:11.677076 24226 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 12:24:11.729471 18023 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:11.586103 24191 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.741832 23854 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.742847 24191 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:11.728464 18019 pending_rounds.cc:85] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 7
I20260709 12:24:11.743675 18019 pending_rounds.cc:107] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:24:11.743953 18019 raft_consensus.cc:2889] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.742444 23854 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.750603 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.740135 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.753201 18017 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:11.728024 18094 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:11.673661 23457 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:24:11.683215 24201 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:11.753568 18095 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260709 12:24:11.707027 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:11.679047 24155 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:11.757109 18021 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:11.749342 18165 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:24:11.758149 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:11.759179 18173 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:11.706341 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:11.757905 23457 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:11.685783 24141 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:11.758908 18169 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:11.683235 24176 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 12:24:11.760771 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:11.781859 18016 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:11.789386 24263 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.733279 23931 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.775398 24116 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:11.790759 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93967.5us
I20260709 12:24:11.791345 24226 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:11.796976 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.798620 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:11.808022 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.805799 18022 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:11.817201 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.821791 23457 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.772858 24155 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.854238 24226 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:11.855959 24226 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.860244 23931 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.862213 24268 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:11.863505 23854 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:11.823132 24141 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.864539 24141 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:11.867041 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:11.868336 24158 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:11.869009 24158 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:11.864617 24268 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:11.870723 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:11.898156 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:11.906813 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 365379us
I20260709 12:24:11.873077 24155 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.912609 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:11.913470 18094 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:11.913802 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:11.829923 18172 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:11.932739 18806 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:11.829933 24263 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:11.938452 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67828.7us
I20260709 12:24:11.940107 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:11.942175 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:11.917856 23457 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:24:11.954154 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.830655 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 348165us
W20260709 12:24:11.935412 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.952504 23664 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:11.957036 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.870311 24158 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:11.961165 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:11.977804 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 300118us
W20260709 12:24:11.983309 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.955149 23931 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:11.959712 23887 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:11.963495 24033 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:11.994848 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.972761 24216 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.950354 24086 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
W20260709 12:24:11.987098 24090 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.002s
I20260709 12:24:12.025246 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.027359 18090 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:24:11.993932 24114 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.029598 18099 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:12.030978 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.033954 18092 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:12.034343 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 283096us
I20260709 12:24:12.035436 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:11.933014 24201 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.038460 24201 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:12.043326 24007 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:12.043931 24007 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:12.008343 24194 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.002784 24273 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:12.045091 24007 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.056061 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.074250 18099 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:12.031877 23921 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:12.076676 18166 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:12.008289 23931 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:12.098074 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 158039us
I20260709 12:24:12.028512 24216 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.022070 24090 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.058552 24087 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.115314 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.116257 18099 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:12.090317 24273 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:12.124872 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 163598us
W20260709 12:24:12.127103 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.129143 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:12.090261 24263 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:12.053849 24194 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.050022 24114 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:12.047099 23972 consensus_queue.cc:1175] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ae9641d5c3184572a102d51a75f670ee" responder_term: 14 status { last_received { term: 14 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 14 index: 12 } }
I20260709 12:24:12.140210 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.142596 18806 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:24:12.146085 18808 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:12.106014 24284 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:12.147390 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 192001us
I20260709 12:24:12.118052 23664 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:12.151341 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.124960 24047 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:12.066648 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:12.172008 18172 raft_consensus.cc:606] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.137895 24281 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:12.144928 24007 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:12.153410 24284 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.183072 18807 pending_rounds.cc:85] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 10
I20260709 12:24:12.197062 18099 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:12.196864 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:12.194087 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.907s (SENT)
I20260709 12:24:12.205336 18807 pending_rounds.cc:107] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20260709 12:24:12.171561 24047 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:12.206925 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.207070 18807 raft_consensus.cc:2889] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:12.235432 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97531.9us
I20260709 12:24:12.221798 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.203153 24086 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.201148 24284 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:12.203752 24292 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:12.183497 23855 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.174183 24087 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:12.246490 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 226733us
I20260709 12:24:12.243160 18099 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:24:12.217254 23969 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.604s	user 0.016s	sys 0.004s
I20260709 12:24:12.215425 24047 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:12.251001 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:12.163802 24273 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.262408 23969 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.262553 18096 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:12.259284 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.265112 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:12.272406 24280 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
W20260709 12:24:12.273689 24289 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:24:12.199064 24281 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.256285 24040 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:12.281401 18097 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:12.284276 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.290987 24307 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:12.291082 24296 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:24:12.291808 23855 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:12.292163 24285 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:24:12.297256 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.299757 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:12.307757 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.271272 24086 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.241655 24298 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
W20260709 12:24:12.326735 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53807.2us
W20260709 12:24:12.313257 24276 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 12:24:12.254146 23664 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.362212 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:12.339566 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.280419 24040 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.345715 18812 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260709 12:24:12.346276 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:12.365195 18022 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
W20260709 12:24:12.346573 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.353750 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:12.266414 24302 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:24:12.264408 24292 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.341332 23931 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:12.365269 24291 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.221s	user 0.000s	sys 0.002s
I20260709 12:24:12.348488 24216 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:12.345786 24090 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.353557 24194 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:12.376454 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101508us
I20260709 12:24:12.374225 18808 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:24:12.384061 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.392266 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.390376 18022 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:12.458268 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.458675 18094 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.521s	user 0.016s	sys 0.001s
I20260709 12:24:12.459270 18094 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:12.464319 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.375505 24033 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:12.468531 18806 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:12.282826 23969 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:12.471009 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99679.8us
I20260709 12:24:12.390993 23931 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:12.485236 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:12.482700 24092 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.494786 24281 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.488540 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:12.496505 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:12.507189 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.509986 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:12.478659 23931 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.521883 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:12.522921 18022 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:12.487185 24114 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.531389 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.539943 18815 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:12.542291 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.471503 24297 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.129s
I20260709 12:24:12.520183 24283 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.522352 23931 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:12.542924 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.559342 24310 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 12:24:12.561662 24295 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.562386 24295 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.563047 24158 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:24:12.565980 24288 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.255s	user 0.000s	sys 0.002s
I20260709 12:24:12.566774 24040 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:12.559803 24283 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.514079 24307 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:24:12.522173 24298 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:24:12.543762 24281 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:24:12.591722 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48909.2us
I20260709 12:24:12.561192 24312 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:12.543540 24226 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:24:12.602734 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.531236 24116 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:24:12.582506 24092 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:24:12.619563 24287 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.290s	user 0.002s	sys 0.000s
I20260709 12:24:12.580206 24033 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.632561 24295 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:12.620116 24215 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:12.640110 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111188us
I20260709 12:24:12.661157 24087 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.662588 18022 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:24:12.663910 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 103817us
W20260709 12:24:12.664237 24201 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.625s	user 0.018s	sys 0.000s
I20260709 12:24:12.665697 24273 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.557633 24313 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.080s
I20260709 12:24:12.666431 24215 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:12.665537 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72418us
I20260709 12:24:12.668183 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.675343 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:12.665601 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:12.682551 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98956.9us
I20260709 12:24:12.684813 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:12.687865 18809 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.023s	sys 0.035s
W20260709 12:24:12.688973 18809 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.024s	sys 0.035s
I20260709 12:24:12.689265 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.698705 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:12.680841 24215 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:12.683954 24292 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:12.703404 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:12.704998 24273 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.705226 24302 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.696915 23886 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:12.692844 24087 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:12.685954 24086 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:12.698208 24307 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:12.721128 24307 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.721652 24307 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:12.712684 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9566837520001172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
I20260709 12:24:12.695101 23664 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:12.699597 24092 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:24:12.700018 24116 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:24:12.713773 24273 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.720052 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.712914 24318 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:12.723026 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1325119720004295s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260709 12:24:12.723383 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:12.891690 24277 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.859s	user 0.004s	sys 0.000s
W20260709 12:24:12.817457 23457 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.856s	user 0.011s	sys 0.006s
I20260709 12:24:12.861892 24301 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.887192 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3573) took 1089 ms. Trace:
I20260709 12:24:12.898321 24018 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:12.897902 18097 rpcz_store.cc:276] 0709 12:24:11.797935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.798201 (+   266us) service_pool.cc:224] Handling call
0709 12:24:12.886878 (+1088677us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54000523d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:12.887069 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1000806784}
W20260709 12:24:12.902097 24007 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.704s	user 0.012s	sys 0.004s
I20260709 12:24:12.897150 24301 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:12.905325 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.858249 24155 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.917s	user 0.014s	sys 0.013s
W20260709 12:24:12.861552 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.089s (SENT) [suppressed 23 similar messages]
W20260709 12:24:12.907446 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45932us
I20260709 12:24:12.908565 24283 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:12.827656 24141 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.962s	user 0.017s	sys 0.002s
I20260709 12:24:12.909525 24141 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.725277 24302 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.912197 24141 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.717435 24087 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.913172 24087 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:12.914640 23969 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:12.737999 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.728389 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.921238 18094 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:12.728684 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.829895 18811 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:12.712330 24296 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:12.803864 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100560us
I20260709 12:24:12.919751 18093 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:12.922922 18098 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 9.
I20260709 12:24:12.930751 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.932462 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.934942 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:12.932633 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.724190 24215 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:12.898041 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174773us
I20260709 12:24:12.907147 24155 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.924712 24318 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.949141 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.951095 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.954730 18809 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:24:12.936309 24092 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:12.956848 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:12.957767 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:12.938287 24047 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.663s	user 0.017s	sys 0.000s
I20260709 12:24:12.958063 18167 raft_consensus.cc:2376] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17. Re-sending same reply.
I20260709 12:24:12.964603 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:12.965294 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:12.959203 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:12.957863 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:12.966248 18172 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:12.960804 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:12.961318 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.929661 24301 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.970144 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2778) took 1238 ms. Trace:
I20260709 12:24:12.967072 18813 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:12.957110 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:12.966274 18173 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:12.965385 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:12.929965 24176 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:12.963613 24309 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.408s	user 0.009s	sys 0.021s
I20260709 12:24:12.978142 18099 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:24:12.970623 18096 rpcz_store.cc:276] 0709 12:24:11.731389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.739783 (+  8394us) service_pool.cc:224] Handling call
0709 12:24:12.969902 (+1230119us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400235d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:12.970068 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1173418880}
I20260709 12:24:12.935968 24297 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.978374 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:12.943998 24215 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:12.982366 24092 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:12.990311 24092 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.990952 24092 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.991500 24092 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:12.992141 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.993498 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:12.981065 24176 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.995882 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:12.996721 24273 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:12.997589 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:12.998714 18173 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:12.999511 18170 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260709 12:24:13.001166 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.000888 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:13.006865 18166 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:12.933775 24296 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.009658 24114 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:13.010393 24114 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:13.011811 24114 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.014560 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:12.959088 24318 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:12.997442 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:13.031191 18169 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:24:12.978267 23969 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.026602 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:13.055331 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71578.4us
I20260709 12:24:13.054890 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:13.057399 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.010427 24296 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:12.983915 24215 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:13.055495 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:12.987408 24141 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:12.963095 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.040059 18168 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:12.972318 24047 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:12.997480 24176 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:13.082561 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:13.075028 18093 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:13.085170 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.061208 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.035388 24116 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:13.019739 24297 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.062762 24302 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:13.095659 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:13.098694 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.100495 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.101240 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:13.109934 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:13.117430 24305 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.824s	user 0.004s	sys 0.000s
I20260709 12:24:13.117888 24155 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:13.087978 24274 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.124104 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.097860 24116 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:13.125633 24194 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.122507 24018 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.099925 24176 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:24:13.166488 24281 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:13.173355 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77789.3us
I20260709 12:24:13.120015 24141 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.045s
W20260709 12:24:13.164582 24284 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.895s	user 0.008s	sys 0.012s
I20260709 12:24:13.193320 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:13.199383 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85134.6us
I20260709 12:24:13.206076 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.206349 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.207444 18021 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:13.169147 24328 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:13.209035 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:13.209740 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.215117 18022 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:13.210901 18812 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:13.212723 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:13.193259 24311 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.213953 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:13.214226 18017 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:13.219372 18020 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:13.219290 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:13.224018 18137 connection.cc:256] RPC call timeout handler was delayed by -1.0062724409999646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260709 12:24:13.160213 24307 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.229087 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:13.193557 24313 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.230597 24018 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.244115 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:13.245749 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75091.2us
I20260709 12:24:13.246402 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:13.267355 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67052.9us
I20260709 12:24:13.268819 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:13.254282 18813 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:13.220824 24155 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:13.237377 24311 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.172200 24194 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.221839 24328 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.275250 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.275468 18017 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:13.276002 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.276103 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.276861 24298 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:13.278748 24324 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:24:13.279155 24311 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:13.282312 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.332953 18021 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:13.333220 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.284884 24194 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:13.286098 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:13.287628 24325 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:24:13.302661 24047 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:13.303576 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:13.337255 18018 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:13.304966 23931 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.721s	user 0.009s	sys 0.006s
I20260709 12:24:13.314597 18024 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:13.315851 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.939s (SENT)
I20260709 12:24:13.315347 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:13.324661 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49490.4us
I20260709 12:24:13.338449 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:13.326568 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.201787 24284 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.329499 18020 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:13.330459 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 103271us
I20260709 12:24:13.348361 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.349416 18015 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:13.349938 24311 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:13.350684 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:13.351719 24326 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.121s	user 0.001s	sys 0.001s
I20260709 12:24:13.351737 18015 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:13.354822 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.357666 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:13.358657 24332 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:24:13.327817 24155 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:13.336634 24298 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:13.363392 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.370457 24311 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:13.370848 24215 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.372153 18164 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:13.362998 24298 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.375931 18022 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
I20260709 12:24:13.378124 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.347395 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:13.391752 18017 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:13.386462 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.286163 24297 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:13.374158 23931 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.316738 24328 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:13.441306 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 96580.7us
I20260709 12:24:13.456177 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:13.452029 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 342148us
W20260709 12:24:13.468761 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 152928us
I20260709 12:24:13.372332 24341 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:13.557677 18016 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:13.563886 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.397392 24338 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:13.487622 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.487632 18017 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:13.489171 18020 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:13.399407 24114 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:13.504347 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 224237us
I20260709 12:24:13.489643 18095 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:24:13.508594 24194 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:13.509224 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:13.507596 24087 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.594s	user 0.011s	sys 0.007s
I20260709 12:24:13.273442 24116 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.489720 24118 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:13.630501 24118 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.273067 24313 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.638779 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:13.640796 24313 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:13.388731 24302 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.117s
I20260709 12:24:13.348475 24334 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:24:13.515239 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146150us
I20260709 12:24:13.649292 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:13.569337 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:13.650341 18165 raft_consensus.cc:606] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:13.650772 18165 raft_consensus.cc:614] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:13.570856 24215 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:13.652211 24334 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:13.571002 23931 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:13.653278 23931 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:13.653424 24334 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.656097 24092 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:13.657166 23886 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:13.657663 23886 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:13.489723 24047 raft_consensus.cc:2764] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:24:13.360183 24155 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.611536 24338 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.664772 18020 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:24:13.507550 24295 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:13.666392 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.577438 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.667157 18015 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:13.668399 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:13.671005 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:13.582453 24284 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:13.584343 18019 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:13.586777 18090 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:13.561578 24341 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:13.673857 24341 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.512001 24339 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:13.674705 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.674979 24299 raft_consensus.cc:2764] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:24:13.600656 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:13.607692 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:13.676815 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69265.6us
I20260709 12:24:13.676401 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:13.677800 18090 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:13.677955 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:13.614272 18018 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:13.671783 18165 raft_consensus.cc:606] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:13.678905 18165 raft_consensus.cc:614] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:13.676393 18094 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:13.620884 24194 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.623783 24336 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:13.624027 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114936us
I20260709 12:24:13.683763 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:13.624567 24087 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:13.625733 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.695377 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.627863 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:13.663053 23886 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.676385 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:13.610468 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.703840 18099 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:13.576761 24318 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.560s	user 0.013s	sys 0.000s
I20260709 12:24:13.697909 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:13.700326 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.701859 18022 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:13.675355 24339 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.715966 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.716688 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.719434 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:13.719532 18015 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:13.724409 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:13.729524 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.729907 18808 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:24:13.729390 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:13.665711 24338 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:13.691761 24295 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:13.755995 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72385us
I20260709 12:24:13.754534 18017 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:13.754773 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.731381 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.773537 18094 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
W20260709 12:24:13.737558 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.774724 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:13.730659 18021 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:13.777752 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.675904 24299 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:24:13.786149 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:13.703467 24215 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:13.789799 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:13.792512 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:13.707314 24092 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:13.796562 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:13.723150 24333 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.711892 24346 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.752803 23886 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:13.758104 24313 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:13.786243 24332 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:24:13.823412 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:13.824316 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.824501 18099 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:13.825841 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:13.714795 24297 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.115s
I20260709 12:24:13.755230 24307 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:13.795288 24215 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.787447 24281 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:13.827761 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:13.795436 23664 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:13.712998 24318 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.816319 24141 raft_consensus.cc:2764] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:24:13.867285 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:13.881004 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:13.834477 24333 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:13.881546 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:13.881994 18165 raft_consensus.cc:606] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:24:13.882663 18165 raft_consensus.cc:614] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:13.788326 24295 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.886893 24092 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:13.857239 24313 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:13.859398 24018 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.915198 18023 raft_consensus.cc:2368] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15.
I20260709 12:24:13.877115 24215 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:13.881004 24176 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:13.891851 24336 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:13.929924 24040 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:13.930574 24040 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:13.931666 24040 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:13.932915 24040 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:13.875753 24033 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:24:13.944147 24313 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:13.952939 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:13.945087 24328 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:13.978626 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.507s (SENT)
I20260709 12:24:13.979470 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:14.008430 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.002s	sys 0.019s
W20260709 12:24:14.008910 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.002s	sys 0.019s
W20260709 12:24:14.009219 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 695170us
I20260709 12:24:14.010020 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:14.011848 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:14.019972 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.021267 18090 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
W20260709 12:24:14.022559 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69515.7us
I20260709 12:24:14.023223 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:14.020979 18093 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:13.915638 24018 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:14.042580 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.006429 24344 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.497s	user 0.006s	sys 0.001s
W20260709 12:24:14.026783 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 299796us
I20260709 12:24:14.059489 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.056787 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:13.990032 24118 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:14.061040 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:14.066358 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2796) took 1348 ms. Trace:
I20260709 12:24:14.066567 18092 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 16. Ops: [14.12-14.12]
I20260709 12:24:14.068737 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:14.070320 18165 raft_consensus.cc:606] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.070879 18165 raft_consensus.cc:614] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:14.073225 18093 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:14.067111 18095 rpcz_store.cc:276] 0709 12:24:12.718024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.718154 (+   130us) service_pool.cc:224] Handling call
0709 12:24:12.718491 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.605794 (+887303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.065844 (+460050us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54002c9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:14.066244 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553186688}
I20260709 12:24:14.085592 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.087384 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:14.083519 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:24:14.091463 18810 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1107) took 1971 ms (client timeout 1906 ms). Trace:
I20260709 12:24:14.056265 24007 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:14.104048 18810 rpcz_store.cc:269] 0709 12:24:12.123240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.128787 (+  5547us) service_pool.cc:224] Handling call
0709 12:24:14.083607 (+1954820us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54000e2ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:14.083783 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3192889728}
I20260709 12:24:14.052534 24299 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:24:14.104143 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.025095 24311 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:14.107185 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:14.108258 18090 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:14.111346 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:14.067297 24281 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:14.067682 24087 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:14.114519 18093 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:14.089903 24118 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.126343 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:14.134492 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:14.135669 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 247310us
I20260709 12:24:14.105772 24007 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.112033 24299 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:14.142513 18090 raft_consensus.cc:2376] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16. Re-sending same reply.
I20260709 12:24:14.142212 18097 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:14.113600 24311 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:14.085242 24141 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:24:14.116042 24194 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.158663 18098 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 9.
I20260709 12:24:14.166231 18097 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:14.191524 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.193156 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:14.193464 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:14.166790 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.168859 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:14.174928 24090 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:14.209856 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 234690us
I20260709 12:24:14.164685 24356 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:14.167649 24299 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:24:14.205013 24348 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.298s	user 0.000s	sys 0.002s
I20260709 12:24:14.203338 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:14.179418 24311 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:14.214203 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.223455 18097 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:14.294355 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.254825 24366 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:14.256162 24359 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:14.257287 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2639) took 1803 ms (client timeout 2247 ms). Trace:
W20260709 12:24:14.297638 18019 rpcz_store.cc:269] 0709 12:24:12.453719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.468343 (+ 14624us) service_pool.cc:224] Handling call
0709 12:24:14.256971 (+1788628us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:14.257202 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2525106560}
W20260709 12:24:14.258140 24347 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.337s	user 0.001s	sys 0.000s
I20260709 12:24:14.258363 18024 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:24:14.259335 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2800) took 1533 ms. Trace:
I20260709 12:24:14.158785 24007 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:14.263056 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 94349.6us
I20260709 12:24:14.306437 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:14.309362 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.301316 18091 rpcz_store.cc:276] 0709 12:24:12.726277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.726425 (+   148us) service_pool.cc:224] Handling call
0709 12:24:12.728247 (+  1822us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.025295 (+297048us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:13.127460 (+102165us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.755986 (+628526us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b1800025480. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:24:13.756252 (+   266us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.756837 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.757007 (+   170us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:13.757037 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.918566 (+161529us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:13.935726 (+ 17160us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400214b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:13.935797 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.258998 (+323201us) raft_consensus.cc:1685] finished
0709 12:24:14.259027 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.259248 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11303,"prepare.queue_time_us":216010,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":20008,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":27503,"spinlock_wait_cycles":671351680,"thread_start_us":129045,"threads_started":2,"wal-append.queue_time_us":225730}
W20260709 12:24:14.238677 24349 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:24:14.264060 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 285489us
W20260709 12:24:14.265671 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 204720us
I20260709 12:24:14.312433 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.313285 18168 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20. Re-sending same reply.
W20260709 12:24:14.270989 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.239670 24353 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:24:14.283247 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.315102 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.315989 18168 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:14.287716 24342 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:14.317202 24342 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:14.318320 24342 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:14.316478 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:14.319609 18016 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:14.325959 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.290285 18093 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:14.221742 24318 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:14.295178 18097 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:14.149850 24118 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.300181 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.342501 18022 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:14.300530 18017 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:14.314006 18169 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: [18.17-18.17]
I20260709 12:24:14.291015 24295 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:14.351975 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.351102 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.340123 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.338256 18165 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:14.356076 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.348879 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.297159 24176 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.335988 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.274118 24364 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:14.358603 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.349381 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.354841 18170 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
W20260709 12:24:14.359496 18099 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.517s	user 0.015s	sys 0.003s
I20260709 12:24:14.361503 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.297271 24366 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:14.339059 24306 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:14.369956 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.370324 18815 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:14.371194 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:14.320423 24333 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.367092 18099 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:14.368013 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.364993 18015 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:14.357161 18165 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:14.369863 18172 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:14.351086 24359 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:14.304080 23664 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:24:14.396461 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.293871 24356 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.389729 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.398972 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.414014 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.416429 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:14.402168 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.398181 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.452533 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.454110 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:14.401854 18164 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:14.448819 18015 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:24:14.455972 18091 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:14.359467 24328 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:14.458482 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 145361us
I20260709 12:24:14.472250 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:14.453682 18165 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:14.466080 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.468323 18098 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:14.374310 24306 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:14.485786 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.397130 24342 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.366480 24215 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:14.514981 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.874s (SENT)
I20260709 12:24:14.530850 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.530939 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:14.518853 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.531898 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:14.398576 24116 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:14.546353 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.468989 24018 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.366515 24176 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.533237 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.559654 18165 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:14.394904 24359 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.461161 24364 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:14.528729 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.570362 18168 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 12:24:14.555640 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 184545us
I20260709 12:24:14.573150 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.526981 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.524263 24306 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:14.575531 18172 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:14.576681 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61762.8us
I20260709 12:24:14.564745 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.582418 18165 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.427520 24333 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.591557 18169 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:14.595994 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.598359 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.545336 24295 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
W20260709 12:24:14.588515 24040 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.655s	user 0.013s	sys 0.005s
I20260709 12:24:14.569446 24364 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.609138 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.608177 18168 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.590826 24328 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:14.616827 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.571012 24359 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:14.619133 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46356.2us
W20260709 12:24:14.614769 24355 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.340s	user 0.000s	sys 0.002s
I20260709 12:24:14.594061 24372 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:14.625481 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 151106us
I20260709 12:24:14.631342 18165 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:14.638921 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.640148 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:14.622246 18171 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:24:14.611181 24047 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.619184 24364 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:14.657485 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.582270 24361 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:24:14.654884 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:14.660605 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78754.2us
I20260709 12:24:14.759413 18171 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.760005 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.663188 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:14.645495 24372 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:14.668706 24365 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
W20260709 12:24:14.669638 24362 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:24:14.655431 24295 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:14.651103 24379 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:14.682068 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61885.4us
I20260709 12:24:14.778853 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:14.590436 24366 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:14.784395 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121327us
I20260709 12:24:14.786288 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:14.665855 24118 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:14.655239 24047 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:14.693194 18168 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:14.697327 24333 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.698263 24215 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:14.702373 24358 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
W20260709 12:24:14.710927 24114 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.100s	user 0.017s	sys 0.000s
I20260709 12:24:14.806613 24377 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:14.714408 24386 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:14.718077 23886 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.884s	user 0.014s	sys 0.003s
I20260709 12:24:14.807905 24368 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:14.724395 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.808418 24386 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.809042 18167 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:14.809959 24338 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.810459 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.810634 24338 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:14.812132 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.813131 18167 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:14.814371 18172 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:14.814693 24090 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.815260 24090 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:14.816864 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.728248 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.798243 24318 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:14.818264 18017 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:14.818472 24318 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.818950 24318 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:14.819303 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.819522 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.821199 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.821524 24341 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:14.763744 24340 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:14.824234 24366 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.732811 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.734001 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:14.823169 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.826171 18020 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:14.750677 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:14.754150 18170 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:14.644902 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:14.827349 18097 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:14.827726 18165 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:14.765973 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127161us
I20260709 12:24:14.769232 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:14.770443 18780 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.660s (SENT)
I20260709 12:24:14.829464 18171 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:14.779870 18090 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.764153 24372 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:14.718150 24382 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.069s	user 0.002s	sys 0.000s
I20260709 12:24:14.841854 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:14.842684 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73538.1us
W20260709 12:24:14.843456 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.779309 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:14.847116 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67912.3us
I20260709 12:24:14.766999 24328 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.730191 24176 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.847681 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.768589 24295 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:14.830777 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:14.847623 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:14.850175 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:14.853243 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.850526 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:14.850589 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:14.854475 18167 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:14.857384 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:14.858974 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.860375 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.865109 18165 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:14.866257 18093 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:14.877979 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:14.837765 24356 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:14.852602 24299 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.551s	user 0.010s	sys 0.003s
I20260709 12:24:14.850901 24379 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:24:14.851352 23886 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.852981 24118 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:14.887682 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:14.861676 23931 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.013s	sys 0.010s
I20260709 12:24:14.892052 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.894680 18172 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 12:24:14.894569 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:14.895422 18172 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 7
I20260709 12:24:14.895840 18172 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:24:14.896181 18172 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:14.898069 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 104245us
I20260709 12:24:14.898774 18167 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:14.899364 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:14.888484 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:14.903206 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:14.901253 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:14.874754 24176 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:14.848868 24340 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
W20260709 12:24:14.907397 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57357.1us
I20260709 12:24:14.911372 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:14.892096 23931 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.069s	user 0.014s	sys 0.010s
I20260709 12:24:14.853579 24377 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:24:14.921186 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:14.855618 24328 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.923305 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:14.923954 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153555us
I20260709 12:24:14.904903 23664 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:14.903414 24114 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:14.905064 24333 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:14.905401 24215 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:14.907971 24047 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:14.906766 24381 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:14.921922 24361 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:14.922669 24274 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:14.910421 24176 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:14.922602 24155 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:14.934160 24114 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:14.942328 24373 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.259s	user 0.001s	sys 0.001s
I20260709 12:24:14.942680 18167 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 12:24:14.942788 24389 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:14.906947 24387 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:14.944664 18090 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:14.926920 24194 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:14.952005 24311 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.598s	user 0.005s	sys 0.008s
I20260709 12:24:14.909590 24297 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:24:14.902933 23886 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:14.930733 24352 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:14.943224 23931 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:15.008296 23931 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.960091 24388 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.070s
I20260709 12:24:15.011097 23931 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.961046 24360 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:15.011693 24388 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:15.012050 24360 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:15.012104 24388 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:14.982741 24361 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:15.013603 24361 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:15.015511 24395 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:14.970943 24350 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.858s	user 0.003s	sys 0.000s
I20260709 12:24:14.971479 24215 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:14.971587 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3498) took 1715 ms (client timeout 1795 ms). Trace:
W20260709 12:24:14.974450 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2644) took 1699 ms (client timeout 2200 ms). Trace:
I20260709 12:24:15.018199 24395 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:15.021514 18021 rpcz_store.cc:269] 0709 12:24:13.274649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.346424 (+ 71775us) service_pool.cc:224] Handling call
0709 12:24:14.974156 (+1627732us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54001d1354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:14.974360 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":842560256}
I20260709 12:24:15.023082 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:15.023254 24395 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:15.024693 24007 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.004s
W20260709 12:24:15.021102 18023 rpcz_store.cc:269] 0709 12:24:13.256418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.256722 (+   304us) service_pool.cc:224] Handling call
0709 12:24:14.971317 (+1714595us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:14.971516 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308867456}
I20260709 12:24:15.013098 24388 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 18.17, Last appended by leader: 17, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.026171 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1315) took 1225 ms. Trace:
I20260709 12:24:15.026485 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:15.026700 24388 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:15.026576 18809 rpcz_store.cc:276] 0709 12:24:13.801270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.801468 (+   198us) service_pool.cc:224] Handling call
0709 12:24:13.835161 (+ 33693us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.084778 (+249617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.084820 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.091823 (+  7003us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.362719 (+270896us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:14.362786 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.362909 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.025905 (+662996us) raft_consensus.cc:1685] finished
0709 12:24:15.025931 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.026098 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":418748,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":148,"spinlock_wait_cycles":1474304,"thread_start_us":425066,"threads_started":2,"wal-append.queue_time_us":508425}
I20260709 12:24:15.028314 18023 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:14.999316 24387 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:15.029260 24387 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:15.016530 24306 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.980286 24318 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.988003 24386 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:15.016310 24281 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:24:15.032433 24318 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.990306 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3502) took 1732 ms. Trace:
I20260709 12:24:14.974669 24090 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.992388 18016 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:24:15.000116 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.991355 18167 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 10
I20260709 12:24:15.034405 18167 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:24:15.034684 18167 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.997648 24176 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:15.013223 24360 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:14.919965 24295 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.898384 24299 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:14.990981 24114 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:15.030094 18023 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:15.001794 24194 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.033331 18024 rpcz_store.cc:276] 0709 12:24:13.258134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.373580 (+115446us) service_pool.cc:224] Handling call
0709 12:24:14.258012 (+884432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.301296 (+ 43284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.990071 (+688775us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b540024e850. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:14.990239 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1881103616}
W20260709 12:24:15.003309 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75587.4us
I20260709 12:24:14.989054 24328 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:14.990857 24155 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:15.025235 24007 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.024462 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:15.045607 24215 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.050422 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:15.050444 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:15.003643 24389 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:15.020318 23886 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.020005 24352 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:15.059194 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114750us
I20260709 12:24:15.063872 18167 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:15.054904 24194 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:15.070399 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.072695 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.075348 18024 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:15.021338 24372 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.058467 24155 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:15.080283 18021 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:15.005921 24311 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.064286 24295 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:15.099028 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44807.8us
I20260709 12:24:15.099583 18023 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:24:15.093883 18138 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.795s (SENT)
I20260709 12:24:15.099934 18017 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:15.100983 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:15.099717 24338 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.110158 24338 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:15.094058 24306 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.123474 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:15.082657 24090 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.125095 24399 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:24:15.128751 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:24:15.131714 24007 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.129354 24386 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:15.153167 24386 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:15.153609 24386 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:15.150496 24118 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 12:24:15.130270 24352 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.150010 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:15.160156 18811 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:15.158882 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:15.162021 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.162842 18167 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:24:15.162751 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68951.3us
W20260709 12:24:15.164690 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 83295.7us
I20260709 12:24:15.164956 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:15.165762 18809 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:15.169605 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.169993 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:15.170440 18813 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:24:15.170979 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.173087 18167 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:15.173693 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.170624 18810 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:15.130621 24215 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.131472 24299 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.181540 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.187099 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.187899 18167 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:15.188906 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.190227 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:15.196079 18024 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:15.129856 24389 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.194255 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:15.196556 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.196208 18167 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:15.206992 24359 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.002s
I20260709 12:24:15.207592 24176 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.208982 24311 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.216926 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:15.197865 24372 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:15.218439 24342 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.688s	user 0.014s	sys 0.001s
I20260709 12:24:15.222959 18813 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:24:15.224187 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.230094 24407 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:15.230496 18020 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260709 12:24:15.229418 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63187.4us
I20260709 12:24:15.231213 24407 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:15.232293 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.136674 24364 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.169927 24399 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:24:15.235306 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.238488 18022 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:24:15.239158 18167 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:15.241595 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.242334 18167 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:15.226925 24311 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:15.245672 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.246855 18022 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:15.246043 24090 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:15.249147 24090 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:15.250272 24339 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.250224 24090 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.207628 24359 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.256129 24359 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:15.257331 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.695s (SENT)
I20260709 12:24:15.259312 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.261121 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.261906 18095 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:15.260707 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.267695 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.268524 18095 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:15.268424 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.276839 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.235069 24194 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.288013 18097 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:15.269143 24352 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:15.301205 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.280180 24215 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:15.266393 24402 raft_consensus.cc:2764] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:24:15.258509 24311 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.232390 24407 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:15.325273 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.346127 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.347069 18096 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:15.315634 24215 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:15.358460 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57353.5us
I20260709 12:24:15.359575 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.336177 24399 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:15.363081 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.344698 24402 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:15.345292 24176 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.372769 24334 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:15.375777 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.381228 24334 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:15.382040 24334 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.369171 24411 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:15.382639 24334 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:15.382892 24411 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:15.384025 24411 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.368371 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:15.398350 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.831s (SENT)
I20260709 12:24:15.358366 24215 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:15.406373 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:15.391505 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.389631 18170 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:15.312903 24387 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.416368 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.410322 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.409044 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:15.423048 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.423985 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:15.426692 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64042.8us
I20260709 12:24:15.428659 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 890) took 1117 ms. Trace:
I20260709 12:24:15.326944 24352 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.429302 18164 rpcz_store.cc:276] 0709 12:24:14.311677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.311821 (+   144us) service_pool.cc:224] Handling call
0709 12:24:15.428366 (+1116545us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b5400383454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.428565 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":112459136}
I20260709 12:24:15.366147 24399 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:15.435971 18170 pending_rounds.cc:85] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 8
I20260709 12:24:15.436506 18170 pending_rounds.cc:107] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:24:15.436957 18170 raft_consensus.cc:2889] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:15.440557 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.415969 24388 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:15.414330 24364 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.415769 24090 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:15.449858 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.453771 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.389488 24155 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.461186 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.462674 18812 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:15.436491 24399 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:15.465637 18810 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:15.426131 24334 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.491528 18095 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:15.495704 18090 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:15.498270 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 110894us
I20260709 12:24:15.500221 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.462831 24340 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.466585 24194 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.503265 18808 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:15.506059 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44976.5us
I20260709 12:24:15.507469 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:15.504108 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:15.467080 24218 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.510972 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.453656 24402 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
W20260709 12:24:15.518759 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2657) took 1183 ms (client timeout 1573 ms). Trace:
I20260709 12:24:15.518918 18171 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:15.524261 18169 rpcz_store.cc:269] 0709 12:24:14.335418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.345962 (+ 10544us) service_pool.cc:224] Handling call
0709 12:24:15.518450 (+1172488us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400031bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.518658 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547778944}
I20260709 12:24:15.528457 24218 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.530269 24311 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.532786 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:15.533205 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2737) took 1073 ms. Trace:
I20260709 12:24:15.528980 24155 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:15.533740 18167 rpcz_store.cc:276] 0709 12:24:14.460201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.243024 (+782823us) service_pool.cc:224] Handling call
0709 12:24:15.243315 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.243730 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.243752 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.246581 (+  2829us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.331961 (+ 85380us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:15.363563 (+ 31602us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.363813 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.363904 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.532919 (+169015us) raft_consensus.cc:1685] finished
0709 12:24:15.532943 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.533132 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1215,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":5221,"thread_start_us":31880,"threads_started":1,"wal-append.queue_time_us":129717}
I20260709 12:24:15.497288 24414 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:15.478168 24386 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.548703 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:15.503324 24090 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:24:15.512503 24346 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:24:15.574203 18812 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:15.509938 24340 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.542474 24342 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:15.586059 18809 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:24:15.564241 24155 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:15.582021 24007 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.571794 24311 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:15.596026 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.615732 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.612937 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.626973 18807 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:15.608521 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:15.600955 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 984) took 1495 ms. Trace:
I20260709 12:24:15.585701 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 902) took 1257 ms. Trace:
I20260709 12:24:15.578981 24405 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:15.620451 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:15.630295 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 119435us
I20260709 12:24:15.620939 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.623268 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:15.631467 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:15.631984 18165 raft_consensus.cc:2376] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18. Re-sending same reply.
I20260709 12:24:15.633145 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.635018 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:15.628103 18098 rpcz_store.cc:276] 0709 12:24:14.105401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.193526 (+ 88125us) service_pool.cc:224] Handling call
0709 12:24:14.457618 (+264092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.473970 (+ 16352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.600648 (+1126678us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54002977d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.600854 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606509824}
W20260709 12:24:15.572204 24366 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.747s	user 0.016s	sys 0.000s
I20260709 12:24:15.640341 18095 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:24:15.585583 24090 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:15.628170 18171 rpcz_store.cc:276] 0709 12:24:14.328499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.852521 (+524022us) service_pool.cc:224] Handling call
0709 12:24:15.518652 (+666131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:15.524450 (+  5798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.585468 (+ 61018us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400031bd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.585630 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527746432}
I20260709 12:24:15.644636 24194 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:15.645450 24092 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:15.595763 24141 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:15.646494 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.699014 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.700472 18091 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:15.622442 24155 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:15.693075 24092 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.704676 24092 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:15.705785 24386 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:15.705966 24340 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.707978 24334 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:15.589166 24361 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.575s	user 0.003s	sys 0.009s
I20260709 12:24:15.656724 18096 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:24:15.655606 18095 pending_rounds.cc:85] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:24:15.715643 18095 pending_rounds.cc:107] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:24:15.715972 18095 raft_consensus.cc:2889] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:15.645963 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:15.691349 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.641728 24387 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:15.568861 24299 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.123s
I20260709 12:24:15.656023 24364 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:15.710590 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:15.640707 24342 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:15.713117 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.657151 24313 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:15.642057 24007 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.689950 24090 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:15.721729 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.735574 18096 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:15.724591 18171 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:15.695429 24141 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:15.650081 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.729434 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:15.727520 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 81689.6us
I20260709 12:24:15.739853 18095 raft_consensus.cc:2368] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16.
I20260709 12:24:15.730144 18091 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 9: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:15.709662 24155 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:15.744644 18171 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:24:15.745626 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.746308 18171 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:15.742551 24090 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:15.747056 24415 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.290s	user 0.002s	sys 0.000s
I20260709 12:24:15.738432 24313 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:15.754492 24141 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.755419 24313 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.757279 24007 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:15.757943 18171 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:15.763458 24092 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:15.766192 24092 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.766381 24215 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:15.767156 24371 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.218s	user 0.005s	sys 0.000s
I20260709 12:24:15.768016 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2653) took 1033 ms. Trace:
I20260709 12:24:15.781647 18021 rpcz_store.cc:276] 0709 12:24:14.734934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.024145 (+289211us) service_pool.cc:224] Handling call
0709 12:24:15.767821 (+743676us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b54003bb5d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.767944 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47011200}
I20260709 12:24:15.782269 18169 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:15.769320 24383 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.222s	user 0.003s	sys 0.000s
I20260709 12:24:15.768016 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2745) took 1244 ms. Trace:
I20260709 12:24:15.790114 18168 rpcz_store.cc:276] 0709 12:24:14.523717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.579845 (+ 56128us) service_pool.cc:224] Handling call
0709 12:24:15.767801 (+1187956us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.767939 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25956224}
I20260709 12:24:15.790663 18167 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [12.10-14.11]
I20260709 12:24:15.791337 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:15.792004 18168 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:15.792397 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.775354 24405 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:15.784505 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2657) took 1048 ms. Trace:
I20260709 12:24:15.803130 18017 rpcz_store.cc:276] 0709 12:24:14.736409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.842206 (+105797us) service_pool.cc:224] Handling call
0709 12:24:15.784334 (+942128us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54001095d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.784440 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":552171264}
I20260709 12:24:15.784668 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 899) took 1459 ms. Trace:
I20260709 12:24:15.804399 18173 rpcz_store.cc:276] 0709 12:24:14.325324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.325539 (+   215us) service_pool.cc:224] Handling call
0709 12:24:14.326000 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.326468 (+   468us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.326503 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.346992 (+ 20489us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.487135 (+140143us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:14.487210 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.487334 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.768690 (+1281356us) spinlock_profiling.cc:243] Waited 3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:15.784326 (+ 15636us) raft_consensus.cc:1685] finished
0709 12:24:15.784351 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.784598 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12655,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":12239,"spinlock_wait_cycles":6909312,"thread_start_us":92865,"threads_started":1,"wal-append.queue_time_us":183095}
I20260709 12:24:15.784514 24215 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.787715 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.807879 18094 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 15
W20260709 12:24:15.788801 24114 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.730s	user 0.014s	sys 0.000s
I20260709 12:24:15.721323 24361 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.798437 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:15.810457 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:15.817319 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.825302 18098 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:24:15.734725 24336 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:15.843968 18096 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:15.830945 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 192670us
W20260709 12:24:15.860222 18780 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.907s (SENT)
I20260709 12:24:15.860818 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.803043 24194 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.822317 24364 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:15.879623 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 965) took 2230 ms. Trace:
I20260709 12:24:15.901245 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.880184 18092 rpcz_store.cc:276] 0709 12:24:13.649123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.796467 (+147344us) service_pool.cc:224] Handling call
0709 12:24:14.066186 (+269719us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.069508 (+  3322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.877480 (+1807972us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54002c9f50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:15.879543 (+  2063us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560109312}
I20260709 12:24:15.906634 18094 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:15.910145 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:15.929735 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.010s (SENT)
W20260709 12:24:15.930462 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 65118.1us
I20260709 12:24:15.930526 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:15.823719 24007 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:24:15.813342 24334 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
W20260709 12:24:15.953874 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46096.7us
I20260709 12:24:15.823060 24346 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:24:15.863860 24336 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:15.951287 18098 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:15.965157 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.965749 24378 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:15.969995 24342 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.825469 24114 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.962136 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:15.899873 24372 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:15.988619 24218 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:15.954514 24306 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:15.956974 24295 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:15.990155 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:15.991142 18098 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:15.943349 24422 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:24:15.993191 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:15.952306 24370 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:16.013311 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2666) took 1403 ms. Trace:
I20260709 12:24:16.008236 18807 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:16.025995 18169 rpcz_store.cc:276] 0709 12:24:14.609754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.525684 (+915930us) service_pool.cc:224] Handling call
0709 12:24:15.526198 (+   514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:15.782769 (+256571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.012885 (+230116us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:16.013201 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":588385664}
W20260709 12:24:16.007642 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.878986 24194 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.034477 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:16.048545 24346 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.049232 24428 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:16.049319 24346 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.050202 24428 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:16.050450 24194 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:16.051306 24428 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.052280 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:16.054067 24423 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:24:16.055009 24114 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.055752 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2662) took 1574 ms. Trace:
I20260709 12:24:16.059466 18170 rpcz_store.cc:276] 0709 12:24:14.481785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.426702 (+944917us) service_pool.cc:224] Handling call
0709 12:24:15.435695 (+  8993us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.447609 (+ 11914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.447664 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.448388 (+   724us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.654975 (+206587us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:15.655038 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.655154 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.055488 (+400334us) raft_consensus.cc:1685] finished
0709 12:24:16.055516 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.055680 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":168032,"threads_started":1,"wal-append.queue_time_us":367369}
I20260709 12:24:16.027912 24295 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:16.046157 24352 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.047231 24336 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:16.034833 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 400756us
I20260709 12:24:16.029065 24340 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:16.027654 24372 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:15.952121 24364 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:15.990327 24306 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:16.022063 24311 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:16.005405 24092 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:16.022671 24370 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.075683 24340 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.076471 24372 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:16.091200 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.117333 18016 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.092022 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.112327 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:16.094189 18164 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.011s	sys 0.035s
I20260709 12:24:16.119285 18017 raft_consensus.cc:2376] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16. Re-sending same reply.
I20260709 12:24:16.081432 24295 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:16.113747 24299 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:16.137917 24407 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:16.119410 18164 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.011s	sys 0.035s
I20260709 12:24:16.122131 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.141269 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.142277 18023 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.110562 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:16.145654 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:16.124253 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.147979 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.126050 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.148975 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.149403 18017 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:16.150007 18020 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:16.150887 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.151814 18017 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:16.153388 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.155272 18017 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:16.156620 24414 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:16.157178 24414 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:16.157562 24414 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:16.161896 24434 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:16.164599 24434 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:16.087420 24284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.481s	user 0.013s	sys 0.025s
I20260709 12:24:16.168147 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:16.124778 18022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.044s	sys 0.031s
I20260709 12:24:16.138923 24407 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.119170 24343 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:16.124753 24399 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.636s	user 0.016s	sys 0.005s
W20260709 12:24:16.170920 18022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.044s	sys 0.031s
I20260709 12:24:16.120111 24092 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:16.110466 24340 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:16.174299 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.094559 24370 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.088615 24352 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:16.171928 24284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.571s	user 0.014s	sys 0.026s
I20260709 12:24:16.189862 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.194844 18020 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:16.198424 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3507) took 1917 ms. Trace:
I20260709 12:24:16.180112 24346 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:16.212807 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.222571 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.210541 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.171430 24366 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:16.182983 24424 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:16.225240 18018 rpcz_store.cc:276] 0709 12:24:14.280581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.293462 (+ 12881us) service_pool.cc:224] Handling call
0709 12:24:14.342068 (+ 48606us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:14.355781 (+ 13713us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.355829 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:14.360680 (+  4851us) log.cc:844] Serialized 52 byte log entry
0709 12:24:15.577502 (+1216822us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b1800414a00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:15.837352 (+259850us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:15.874782 (+ 37430us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.875216 (+   434us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.114954 (+239738us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b1800414a00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:16.197690 (+ 82736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.197907 (+   217us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.198063 (+   156us) raft_consensus.cc:1685] finished
0709 12:24:16.198087 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.198325 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":153882,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":114,"raft.queue_time_us":284007,"raft.run_cpu_time_us":544,"raft.run_wall_time_us":25655,"spinlock_wait_cycles":38742144,"thread_start_us":581723,"threads_started":2,"wal-append.queue_time_us":1661182}
I20260709 12:24:16.196820 24439 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.174288 24299 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.197347 24361 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:16.258221 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47850us
I20260709 12:24:16.260854 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.249238 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.203794 24313 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:16.186841 24343 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.266230 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.267036 18017 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:16.270668 18018 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:16.272222 24443 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.272578 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.273546 18018 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:16.275354 24313 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:16.276283 24432 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
W20260709 12:24:16.278218 24430 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:24:16.278275 24343 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.281270 18023 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:16.282680 18020 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 12.
I20260709 12:24:16.283041 18023 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:24:16.232985 24436 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.290500 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.222895 24438 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:16.296808 18017 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:16.187952 24399 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.225606 24352 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.263890 24176 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:16.291117 24386 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.268680 24405 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.196540 24311 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.219617 24372 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.249748 24366 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.228647 24306 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:16.359527 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.360641 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 398624us
I20260709 12:24:16.309166 24424 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.371176 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:16.248318 24295 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.373390 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.257563 24439 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.374666 24364 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.290017 24092 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:16.378638 24155 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.623s	user 0.011s	sys 0.004s
I20260709 12:24:16.356030 24114 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:16.411150 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.411549 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2669) took 1787 ms. Trace:
I20260709 12:24:16.412036 18165 rpcz_store.cc:276] 0709 12:24:14.624489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.651231 (+1026742us) service_pool.cc:224] Handling call
0709 12:24:15.651661 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:15.791657 (+139996us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.791691 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:15.791739 (+    48us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:15.791783 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.411247 (+619464us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:16.411310 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.411476 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321413504}
I20260709 12:24:16.373250 24284 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:24:16.305685 24436 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.386482 24405 raft_consensus.cc:2764] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:24:16.355180 24176 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.406690 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:16.358795 24386 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:16.459918 24431 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:24:16.460968 24386 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:16.394688 24372 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:16.376171 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:16.465545 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89541.9us
I20260709 12:24:16.357491 24438 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:24:16.441533 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:16.377995 24402 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:24:16.442934 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.478587 18023 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:24:16.492964 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.447099 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:16.447808 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.451310 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.396133 24378 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:16.453730 18096 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:16.514603 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2667) took 1898 ms. Trace:
I20260709 12:24:16.455461 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.456686 18095 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:16.515379 18167 rpcz_store.cc:276] 0709 12:24:14.616036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.544661 (+928625us) service_pool.cc:224] Handling call
0709 12:24:15.790358 (+245697us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:15.791118 (+   760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.514340 (+723222us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54001b5150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:16.514517 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":563647232}
I20260709 12:24:16.458307 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.517007 18017 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:16.415630 24346 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:16.507138 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.454622 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.529345 18092 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:16.529439 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:16.519258 24437 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:16.523592 18098 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:16.443850 18018 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:16.416998 24155 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:16.535765 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88785us
I20260709 12:24:16.556286 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.533913 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:16.475097 24452 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.478708 24407 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.474748 24424 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:16.561548 18016 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:16.526278 24361 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:16.427450 24114 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.584530 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:16.590909 18094 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:16.594326 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:16.594257 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60493.7us
I20260709 12:24:16.595692 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.596457 18018 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.591837 18811 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:24:16.599941 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:16.512633 24274 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:16.534246 24444 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:16.607803 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.564354 24364 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:16.611754 18016 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:16.608842 24424 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:16.613297 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.613515 24424 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.614014 24424 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:16.615316 18023 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [14.7-15.8]
I20260709 12:24:16.589844 24299 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.562780 24346 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.592703 24361 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:16.617210 18023 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:16.623462 18811 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:16.623809 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.648960 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.647017 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2738) took 2186 ms. Trace:
W20260709 12:24:16.663110 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.626258 24399 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.620534 24114 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.626503 24352 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.664791 24114 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:16.634327 24366 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:16.586141 23886 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:16.667646 18015 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:16.630281 24155 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.659430 18172 rpcz_store.cc:276] 0709 12:24:14.460584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.371776 (+911192us) service_pool.cc:224] Handling call
0709 12:24:15.430090 (+ 58314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.430701 (+   611us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.430750 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.437487 (+  6737us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.508966 (+ 71479us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:15.803410 (+294444us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.805485 (+  2075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.805594 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.805643 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:15.805657 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.646818 (+841161us) spinlock_profiling.cc:243] Waited 57.8 ms on lock 0x7b5400383450. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:16.646944 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":521432,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":89,"raft.queue_time_us":9203,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":13812,"spinlock_wait_cycles":132999040,"thread_start_us":306726,"threads_started":2,"wal-append.queue_time_us":115047}
I20260709 12:24:16.634804 24361 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:16.619220 24447 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:24:16.618391 24440 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:24:16.535095 24378 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.665525 24114 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:16.754297 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.610553 24018 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:16.635535 24364 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:16.681619 18018 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:16.771301 18018 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [14.8-15.9]
I20260709 12:24:16.646436 24462 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.108s
I20260709 12:24:16.772591 24114 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:16.682672 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 276059us
I20260709 12:24:16.773967 24462 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:16.774662 24462 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.775419 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.776943 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.777269 24462 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.778108 24378 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:16.774020 24364 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.668910 24463 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.127s
I20260709 12:24:16.771302 24343 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:16.783980 24343 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:16.784160 24460 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:24:16.785320 24090 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.038s	user 0.014s	sys 0.002s
I20260709 12:24:16.785795 24090 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.786772 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.787158 24388 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:16.787635 24388 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:16.788033 24388 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.788224 24090 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.790205 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.791115 18021 raft_consensus.cc:2376] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13. Re-sending same reply.
I20260709 12:24:16.792930 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:16.793192 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:16.793746 18811 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:16.795698 24386 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:16.796356 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:16.795696 24366 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:16.800161 24366 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.800634 24366 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:16.625126 24424 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.724942 24451 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:16.822688 18090 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:16.741096 18020 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:24:16.745177 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.750954 17915 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 14, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 14 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:16.752391 24436 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.826969 18015 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 9.
I20260709 12:24:16.752614 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:16.765102 24439 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.648015 24455 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260709 12:24:16.579934 24452 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.682502 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:16.785020 24343 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.700363 24313 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.758344 24338 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:16.846824 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 896) took 2527 ms. Trace:
I20260709 12:24:16.850664 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3613) took 1256 ms. Trace:
I20260709 12:24:16.849402 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:16.853500 18097 rpcz_store.cc:276] 0709 12:24:15.594214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.619001 (+ 24787us) service_pool.cc:224] Handling call
0709 12:24:15.619356 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.619824 (+   468us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.619849 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.651013 (+ 31164us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.111289 (+460276us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:16.111341 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.122899 (+ 11558us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.850395 (+727496us) raft_consensus.cc:1685] finished
0709 12:24:16.850425 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.850589 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113795,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":7746,"thread_start_us":402839,"threads_started":1,"wal-append.queue_time_us":637370}
I20260709 12:24:16.857152 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:16.859740 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.860831 18015 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:16.839535 24281 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:16.851833 24439 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:16.863119 24281 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.863938 24439 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:16.849762 24370 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.621s	user 0.015s	sys 0.000s
I20260709 12:24:16.866155 23886 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.866346 24370 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.866772 23886 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.868886 24370 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.869007 24218 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:16.849942 24414 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.692s	user 0.008s	sys 0.008s
I20260709 12:24:16.870034 24414 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.870702 24141 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.872438 24414 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.781606 24346 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.852645 18166 rpcz_store.cc:276] 0709 12:24:14.318992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.319303 (+   311us) service_pool.cc:224] Handling call
0709 12:24:14.361761 (+ 42458us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:14.366135 (+  4374us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:15.613289 (+1247154us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b180037cfc0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:24:15.827511 (+214222us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.236127 (+408616us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:16.258956 (+ 22829us) log.cc:844] Serialized 52 byte log entry
0709 12:24:16.259162 (+   206us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:16.259201 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.259284 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.815118 (+555834us) raft_consensus.cc:1685] finished
0709 12:24:16.846474 (+ 31356us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.846738 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":83685,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":1752,"raft.queue_time_us":8200,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":48764,"spinlock_wait_cycles":71374208,"thread_start_us":494432,"threads_started":2,"wal-append.queue_time_us":544774}
I20260709 12:24:16.880389 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:16.869740 24218 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:16.864570 24452 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:16.883437 24018 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:16.883996 24018 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.884373 24018 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:16.886268 24338 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:16.887434 24338 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.887900 18023 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:16.888409 18023 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:16.888569 24338 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:16.892258 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 844) took 1713 ms. Trace:
I20260709 12:24:16.893093 24439 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:16.896883 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.899762 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 217348us
I20260709 12:24:16.900486 18814 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:16.901508 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 307263us
I20260709 12:24:16.902474 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.904700 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:16.905184 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.892778 18022 rpcz_store.cc:276] 0709 12:24:15.178483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.265665 (+ 87182us) service_pool.cc:224] Handling call
0709 12:24:15.271079 (+  5414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:15.456740 (+185661us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:15.467197 (+ 10457us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.393415 (+926218us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.393790 (+   375us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.393884 (+    94us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:16.393979 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.890673 (+496694us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b180045ade0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:24:16.891895 (+  1222us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:16.891955 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.892175 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23234,"prepare.run_cpu_time_us":143,"prepare.run_wall_time_us":3829,"raft.queue_time_us":99891,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":8253,"spinlock_wait_cycles":457837056,"thread_start_us":925356,"threads_started":2,"wal-append.queue_time_us":801598}
I20260709 12:24:16.889333 18023 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.916154 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.917152 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:16.872449 24340 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.650s	user 0.014s	sys 0.000s
I20260709 12:24:16.918987 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:16.920166 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:16.928242 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:16.869036 23886 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:16.929771 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:16.931923 18023 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:16.936460 24370 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:16.937054 24370 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.937178 24436 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:16.937508 24370 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.915146 24462 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.942896 24194 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:16.943523 24414 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:16.953972 17987 leader_election.cc:400] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16.
I20260709 12:24:16.954557 17987 leader_election.cc:403] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:24:16.956527 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.946822 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:16.960309 18098 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:24:16.957031 18023 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:16.962740 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:16.963873 18098 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:16.966094 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:16.971241 18097 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:16.980077 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:16.930500 24218 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.994885 24436 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:16.995551 24436 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.995949 24436 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:16.997160 24421 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:16.940980 24424 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.998440 24281 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:16.934641 24336 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 12:24:16.944926 23886 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:17.003121 24343 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:17.004822 24434 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.840s	user 0.009s	sys 0.010s
I20260709 12:24:17.005447 24434 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:16.968766 24462 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.978067 24414 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.989157 24352 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16.
I20260709 12:24:16.989849 24457 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:16.997680 24421 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:16.969669 24302 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.985219 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:16.987323 24194 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:17.010453 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.011296 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:17.017088 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.924906 24340 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:16.934181 24306 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:17.042253 18094 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:17.020993 18093 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:17.022192 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:16.852151 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:17.030674 23886 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:17.040912 24302 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:17.050637 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65609.6us
I20260709 12:24:17.051697 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:17.051934 18092 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:17.055572 18167 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:24:17.057614 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.059535 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.060212 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:17.064432 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3622) took 1277 ms. Trace:
I20260709 12:24:17.064720 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.064921 18090 rpcz_store.cc:276] 0709 12:24:15.786680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.814720 (+ 28040us) service_pool.cc:224] Handling call
0709 12:24:15.836278 (+ 21558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.823233 (+986955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.064177 (+240944us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b5400411ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:17.064361 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2251133824}
I20260709 12:24:17.068332 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.069068 18018 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.113221 18173 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:17.113466 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:17.113284 18097 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:17.078603 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:17.099327 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.116217 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56072.9us
I20260709 12:24:17.070544 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.118770 18022 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [16.10-17.11]
W20260709 12:24:17.101751 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 332762us
I20260709 12:24:17.101678 18165 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:17.121536 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3523) took 1028 ms. Trace:
I20260709 12:24:17.123909 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.124702 18165 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:17.128283 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:17.129086 18807 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:24:17.133303 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54858.6us
I20260709 12:24:17.133960 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.134348 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:17.133373 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.062106 24424 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.165884 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.165009 18018 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:17.169885 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.175135 18168 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:17.025462 24455 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260709 12:24:17.183499 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.072501 24439 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:17.185595 18170 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:17.184332 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.113847 24386 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.209131 24386 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.167663 24444 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:17.189153 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54920.7us
I20260709 12:24:17.212157 18812 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:17.214195 24386 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.188860 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:17.195156 18169 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:17.234963 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.196094 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.119491 24334 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:17.184155 24306 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.198035 24407 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:17.198824 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.201288 24364 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:17.240281 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.119184 24443 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20260709 12:24:17.202133 23931 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:17.126945 24378 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.130493 24465 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:17.215319 24464 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
W20260709 12:24:17.207893 17916 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 294) took 42435 ms (client timeout 15000 ms). Trace:
W20260709 12:24:17.208564 24360 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:24:17.062306 24313 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:24:17.028214 24361 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.118686 24467 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.050714 23886 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.299417 18170 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:17.241884 18164 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.019s	sys 0.048s
I20260709 12:24:17.300453 18022 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:17.051661 24388 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.173465 24411 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:17.232734 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:17.169303 24399 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.330245 24399 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.113008 24466 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:17.051388 24194 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.333629 24466 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:17.333914 24414 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.334739 24414 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:17.335239 24414 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:17.334828 24466 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.335644 24194 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.071445 24299 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.163s
I20260709 12:24:17.336445 24414 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.275137 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:17.118849 24090 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 12:24:17.299010 24176 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.185205 24471 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.339669 24472 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:17.339370 24471 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.340209 24472 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:17.071388 24281 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.164s
I20260709 12:24:17.343429 18169 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:17.344244 24388 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:17.346630 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.237703 18807 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:24:17.235049 18173 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
W20260709 12:24:17.300684 18164 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.019s	sys 0.048s
I20260709 12:24:17.209008 17915 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 13, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 13 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:17.308904 18172 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:17.238909 24407 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:17.236027 24342 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
W20260709 12:24:17.362738 24366 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.561s	user 0.014s	sys 0.003s
I20260709 12:24:17.193693 24424 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.362365 17915 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:17.363255 24366 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.364421 17915 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:17.365271 24424 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.314266 18808 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:17.310814 24467 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.272148 24334 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.366268 17915 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:17.314975 24361 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.368332 24467 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.368518 24334 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.367954 17915 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:17.279224 24439 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:17.320377 24370 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.215126 24405 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:24:17.280000 24421 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.294327 24047 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:24:17.390722 24366 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.391395 24471 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:17.276883 17916 rpcz_store.cc:269] 0709 12:23:34.772245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.772552 (+   307us) service_pool.cc:224] Handling call
0709 12:24:17.207511 (+42434959us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:05.520125 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:05.902782 (+382657us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:05.902850 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:24:05.902864 (+    14us) tablet.cc:662] Decoding operations
0709 12:24:06.141297 (+238433us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:06.141402 (+   105us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:06.141440 (+    38us) tablet.cc:685] Acquiring locks for 29 operations
0709 12:24:06.373237 (+231797us) tablet.cc:701] Row locks acquired
0709 12:24:06.373306 (+    69us) write_op.cc:260] PREPARE: finished
0709 12:24:06.373533 (+   227us) write_op.cc:270] Start()
0709 12:24:06.373734 (+   201us) write_op.cc:276] Timestamp: P: 1783599846373447 usec, L: 0
0709 12:24:06.373760 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:24:06.551885 (+178125us) log.cc:844] Serialized 11282 byte log entry
0709 12:24:07.761257 (+1209372us) op_driver.cc:464] REPLICATION: finished
0709 12:24:08.354667 (+593410us) write_op.cc:301] APPLY: starting
0709 12:24:08.365173 (+ 10506us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:09.061159 (+695986us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:09.061303 (+   144us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:09.659235 (+597932us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:09.659831 (+   596us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=58,key_file_lookups=58,delta_file_lookups=58,mrs_lookups=0
0709 12:24:09.659885 (+    54us) write_op.cc:312] APPLY: finished
0709 12:24:09.660528 (+   643us) log.cc:844] Serialized 252 byte log entry
0709 12:24:09.851585 (+191057us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:09.868519 (+ 16934us) write_op.cc:454] Released schema lock
0709 12:24:09.875451 (+  6932us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:09.899136 (+ 23685us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:09.924215 (+ 25079us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34196,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":473969,"apply.run_cpu_time_us":88850,"apply.run_wall_time_us":1556112,"cfile_cache_hit":118,"cfile_cache_hit_bytes":125521,"num_ops":29,"prepare.queue_time_us":487342,"prepare.run_cpu_time_us":37985,"prepare.run_wall_time_us":1494280,"raft.queue_time_us":531048,"raft.run_cpu_time_us":51704,"raft.run_wall_time_us":303075,"replication_time_us":1387235,"thread_start_us":1271459,"threads_started":5,"wal-append.queue_time_us":704988,"wal-append.run_cpu_time_us":23225,"wal-append.run_wall_time_us":2578850}]]}
I20260709 12:24:17.122022 18024 rpcz_store.cc:276] 0709 12:24:16.092769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.108412 (+ 15643us) service_pool.cc:224] Handling call
0709 12:24:16.110401 (+  1989us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:16.378207 (+267806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.378258 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:16.392731 (+ 14473us) log.cc:844] Serialized 52 byte log entry
0709 12:24:16.670415 (+277684us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:16.682123 (+ 11708us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.873314 (+191191us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b1800436ae0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:17.046251 (+172937us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b1800436ae0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:24:17.053853 (+  7602us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.064026 (+ 10173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.121167 (+ 57141us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54003476d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:17.121256 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.121312 (+    56us) raft_consensus.cc:1685] finished
0709 12:24:17.121325 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.121472 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":259508,"prepare.queue_time_us":393063,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":105,"raft.queue_time_us":21909,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":7369,"spinlock_wait_cycles":739865856,"thread_start_us":249723,"threads_started":1,"wal-append.queue_time_us":362856}
I20260709 12:24:17.279392 24465 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.326531 24360 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:17.333032 24018 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.272331 24306 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.277889 24378 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.370579 18173 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:24:17.339373 24340 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:17.369781 24407 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:17.381188 18169 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
W20260709 12:24:17.404932 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 185412us
I20260709 12:24:17.411443 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:17.341341 24472 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.389362 24421 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:17.420835 18021 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:17.417934 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 220535us
I20260709 12:24:17.371420 24414 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:17.415897 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 201231us
I20260709 12:24:17.407433 24366 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.415490 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:17.426949 18167 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:17.402746 24360 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:17.417380 24141 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.003s
W20260709 12:24:17.441254 18138 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.020s	sys 0.012s
W20260709 12:24:17.442087 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 508717us
I20260709 12:24:17.451956 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:17.457685 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:17.426503 24471 raft_consensus.cc:2764] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:17.459162 18812 raft_consensus.cc:2368] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16.
W20260709 12:24:17.460646 24479 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:24:17.460427 24194 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:17.463120 18016 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:17.455487 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:17.437731 24281 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.267549 18168 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:24:17.445613 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.459183 24018 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.462213 24467 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:17.471843 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:17.479285 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 353003us
I20260709 12:24:17.507490 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:17.501978 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:17.570787 18812 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:17.466488 24194 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.453325 24176 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:24:17.461872 24334 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260709 12:24:17.496264 24338 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.607s	user 0.019s	sys 0.001s
I20260709 12:24:17.460863 24439 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.409916 18022 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 10.
I20260709 12:24:17.503895 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:17.502413 18138 leader_election.cc:336] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.035s (SENT)
W20260709 12:24:17.600720 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98361.9us
I20260709 12:24:17.500177 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:17.584144 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:17.496675 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 283159us
W20260709 12:24:17.495069 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 369110us
W20260709 12:24:17.592051 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90241.3us
I20260709 12:24:17.455258 24366 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.491735 24378 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
W20260709 12:24:17.612980 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 109172us
I20260709 12:24:17.614378 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:17.590901 24194 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:17.598616 24487 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:24:17.503731 24372 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.108s
I20260709 12:24:17.478492 24436 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.503454 24388 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.111s
I20260709 12:24:17.608563 24092 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.638015 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.509057 24386 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.641217 18063 leader_election.cc:400] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16.
I20260709 12:24:17.657198 18063 leader_election.cc:403] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:17.613296 24295 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:17.644701 18166 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:17.659260 18815 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 12:24:17.491756 24281 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.659724 18815 pending_rounds.cc:85] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 12
I20260709 12:24:17.660055 18815 pending_rounds.cc:107] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:24:17.660328 18815 raft_consensus.cc:2889] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:17.614768 24338 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:17.647100 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 175428us
I20260709 12:24:17.662853 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:17.650214 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:17.621047 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:17.664710 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43759.6us
I20260709 12:24:17.647640 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2663) took 3096 ms. Trace:
I20260709 12:24:17.616039 24424 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:24:17.679646 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.689759 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.690516 18023 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:17.682647 24388 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.686487 18807 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:17.688722 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:17.680752 18814 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:24:17.691892 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.667150 18164 rpcz_store.cc:276] 0709 12:24:14.551196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.449552 (+898356us) service_pool.cc:224] Handling call
0709 12:24:15.462192 (+ 12640us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.488349 (+ 26157us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:16.139913 (+651564us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.390779 (+250866us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.707113 (+316334us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b18002e3840. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:16.894154 (+187041us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b18002e3840. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:17.348218 (+454064us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:17.348969 (+   751us) log.cc:844] Serialized 26 byte log entry
0709 12:24:17.349202 (+   233us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:17.628272 (+279070us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.647146 (+ 18874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.647338 (+   192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.647387 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:17.647400 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.647567 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19643,"prepare.queue_time_us":503217,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":14125,"raft.queue_time_us":14153,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":17359,"spinlock_wait_cycles":39081216,"thread_start_us":1814688,"threads_started":4,"wal-append.queue_time_us":475852}
I20260709 12:24:17.650547 24370 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:17.573004 24360 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.624902 24428 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:24:17.651006 24218 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.654485 24487 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:17.654837 24473 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:17.745623 18023 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:17.747946 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:17.685662 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.749887 18023 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:17.698567 17915 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 17, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 17 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:17.711069 24486 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:24:17.703040 24388 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.655045 24467 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:17.725957 24366 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.713562 24306 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.697376 24465 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.729357 24482 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:17.730767 24472 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:17.733474 24281 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:17.688308 24090 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:17.736901 24481 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:17.738440 24352 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:17.738844 24338 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:17.740466 24462 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:17.740742 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.742084 24480 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:17.743448 24411 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:17.746330 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.663561 24378 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:17.761119 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.830487 18098 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:17.763721 18090 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:24:17.720733 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:17.832998 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112391us
I20260709 12:24:17.782487 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:17.783514 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:17.833184 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 376894us
I20260709 12:24:17.785429 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.786710 18814 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 12:24:17.838212 18814 pending_rounds.cc:85] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 14
I20260709 12:24:17.838562 18814 pending_rounds.cc:107] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:24:17.838862 18814 raft_consensus.cc:2889] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:17.666922 24092 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.787817 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.789530 18023 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [14.8-15.9]
W20260709 12:24:17.869599 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.679090 24386 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.794749 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.788051 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:17.779229 24040 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.780886 24424 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:17.825383 17915 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:17.826387 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.826996 24483 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:17.821723 24480 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:17.866523 24370 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:17.829166 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.836691 18093 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:17.895203 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.784392 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.896376 18096 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:17.895990 24480 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.897446 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:17.898698 24444 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.688s	user 0.014s	sys 0.002s
I20260709 12:24:17.898926 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.899228 24444 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:17.899760 18096 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:17.900048 18018 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:17.900666 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.840226 18019 raft_consensus.cc:2368] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16.
I20260709 12:24:17.904670 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.906226 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.907092 18019 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:24:17.911635 24483 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.914324 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.915838 24436 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:17.915948 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.916705 24370 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:17.850905 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 209881us
I20260709 12:24:17.917907 24370 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.918701 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:17.919217 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:17.919884 18022 raft_consensus.cc:2376] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16. Re-sending same reply.
I20260709 12:24:17.754204 24218 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.795481 24493 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.926787 24493 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.930629 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:17.797329 24467 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:17.874416 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.875936 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:17.877038 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94677.5us
I20260709 12:24:17.877909 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:17.885064 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:17.885070 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.822585 24411 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:17.888782 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.936219 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:17.897405 24480 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.789280 24306 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.936205 18090 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:17.939433 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:17.940635 18165 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:17.944957 18169 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:17.946753 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:17.888151 24336 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.949189 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:17.937660 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59921.1us
I20260709 12:24:17.955533 18023 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:17.955305 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:17.993932 18096 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:17.952958 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:17.881909 24428 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260709 12:24:17.884891 24040 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.885393 24424 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.881173 24487 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:17.994867 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:18.401412 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.980894 24493 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.403448 24493 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:18.404042 24493 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.024035 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.404600 18094 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:18.405061 18172 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:18.406432 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.407100 24040 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.408167 24040 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.409121 18814 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:18.408774 24040 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.409523 18810 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:24:18.105544 17915 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 14. New cstate: current_term: 14 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:17.976465 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.412082 24424 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.411579 17915 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:24:18.203871 24414 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.754s	user 0.008s	sys 0.006s
I20260709 12:24:18.412799 24281 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:18.413235 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:18.112615 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 237935us
I20260709 12:24:18.198235 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.414177 18165 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:18.414786 18092 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 17 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:18.414636 24281 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:18.415401 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:18.373654 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.039757 24444 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:18.439788 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:18.416256 24306 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.064014 24501 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:18.417215 18092 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:17.952339 24455 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.465s
W20260709 12:24:18.350893 24503 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
I20260709 12:24:18.417920 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.161474 24411 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.061947 24338 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:18.418521 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:18.202987 24018 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:18.024447 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93897.5us
W20260709 12:24:18.265759 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.404156 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.436961 18171 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:24:18.423100 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.062655 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 121179us
W20260709 12:24:18.114127 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 433863us
W20260709 12:24:17.954735 24343 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.951s	user 0.012s	sys 0.008s
I20260709 12:24:18.403331 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.454110 18096 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:17.975716 24090 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:18.469895 18062 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.004s	sys 0.018s
W20260709 12:24:18.534752 18062 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.004s	sys 0.018s
W20260709 12:24:18.535364 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.473481 18165 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:18.475577 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:18.536243 18024 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:18.404685 24493 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:18.456017 24428 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:18.541872 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.472662 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:18.063158 24302 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.488119 24092 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:17.977773 24218 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.474931 24457 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:17.971135 24194 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.451877 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.495234 24040 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.496237 17915 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 13, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 13 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:18.500308 24486 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.729s	user 0.011s	sys 0.001s
I20260709 12:24:18.500844 18090 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:18.444410 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:18.441195 24405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.005s	sys 0.011s
I20260709 12:24:18.519201 18023 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:18.522230 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:18.527736 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:18.591012 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63445.9us
I20260709 12:24:18.292776 24500 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:18.539558 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:18.603288 18171 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:18.541620 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:18.604502 18810 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
W20260709 12:24:18.487097 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.077696 24482 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:18.449877 24414 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:18.554783 24517 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.555361 24512 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:18.516412 24141 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.024098 24336 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:17.971302 24467 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.521246 24018 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:18.563225 24386 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:18.562885 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:18.569402 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.572722 18022 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:18.520249 24434 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
W20260709 12:24:18.561098 24472 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.750s	user 0.009s	sys 0.005s
I20260709 12:24:18.594761 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.131047 24483 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.599596 18023 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:18.550817 24457 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:18.673322 24457 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.674595 24457 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:18.674281 17915 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 17, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 17 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:18.588357 24511 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.114s
I20260709 12:24:18.547411 24428 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:18.679697 24407 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.257s	user 0.013s	sys 0.004s
I20260709 12:24:18.680302 24407 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.680779 24428 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.586769 24486 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16.
I20260709 12:24:18.682138 24018 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.657748 24141 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.683552 24194 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.683720 24018 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:18.684981 24141 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.684489 24090 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.652944 24500 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.686059 24500 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:18.687489 24336 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:18.660833 24455 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:18.652101 24510 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:24:18.692396 18173 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260709 12:24:18.653232 24489 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.829s	user 0.004s	sys 0.001s
I20260709 12:24:18.693688 24472 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.694325 24472 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.695046 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2680) took 1632 ms. Trace:
I20260709 12:24:18.695175 18168 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:18.695501 18015 rpcz_store.cc:276] 0709 12:24:17.062660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.062808 (+   148us) service_pool.cc:224] Handling call
0709 12:24:17.063272 (+   464us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:17.063737 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.063771 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:17.071986 (+  8215us) log.cc:844] Serialized 52 byte log entry
0709 12:24:17.605488 (+533502us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:17.605538 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.605631 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.393831 (+788200us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:18.694633 (+300802us) raft_consensus.cc:1685] finished
0709 12:24:18.694663 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.694972 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38577,"raft.queue_time_us":96,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":46014,"spinlock_wait_cycles":14594304,"thread_start_us":371435,"threads_started":1,"wal-append.queue_time_us":542674}
I20260709 12:24:18.664189 24512 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.655377 24414 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.699826 24424 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:18.700342 24424 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:18.702802 24194 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.703450 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.703380 24194 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.704463 18164 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:18.705713 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.706749 24455 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.707371 24455 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.707631 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.649420 24334 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:18.709745 24334 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.710309 24334 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:18.703282 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 855) took 1800 ms (client timeout 1916 ms). Trace:
I20260709 12:24:18.712019 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 858) took 1807 ms. Trace:
I20260709 12:24:18.659597 24517 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.670940 18172 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:18.701553 24414 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.721036 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.722035 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.767823 18098 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260709 12:24:18.722983 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:18.724571 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2747) took 1681 ms. Trace:
I20260709 12:24:18.728313 24194 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:18.729980 24455 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.730647 24472 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:18.736670 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 292394us
I20260709 12:24:18.737430 24512 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.642683 24482 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:18.737573 24405 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:18.739239 24407 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.740187 24513 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:24:18.739279 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 144664us
W20260709 12:24:18.800345 24516 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:24:18.800879 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.801762 24518 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.802435 24518 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.803747 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.742063 24510 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:18.804754 24510 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:18.804930 24518 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.562126 24509 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.111s
I20260709 12:24:18.805846 24510 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.806579 24194 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.807060 24509 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:18.807354 24510 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:18.807127 24194 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.807715 24509 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.742149 24517 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:18.743835 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.813386 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.744630 24444 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:18.747190 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.758563 24386 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:18.816301 18090 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:18.755483 18171 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:18.596230 24343 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:18.712102 18016 rpcz_store.cc:269] 0709 12:24:16.903348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.903508 (+   160us) service_pool.cc:224] Handling call
0709 12:24:18.703085 (+1799577us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:18.703217 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1245703424}
I20260709 12:24:18.712518 18021 rpcz_store.cc:276] 0709 12:24:16.904625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.928383 (+ 23758us) service_pool.cc:224] Handling call
0709 12:24:16.928890 (+   507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.421502 (+492612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.711692 (+1290190us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:18.711933 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130772864}
I20260709 12:24:18.764225 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.769377 18166 rpcz_store.cc:276] 0709 12:24:17.043392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.043538 (+   146us) service_pool.cc:224] Handling call
0709 12:24:17.049943 (+  6405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.659411 (+609468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.724117 (+1064706us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:18.724454 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1367254272}
W20260709 12:24:18.790577 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 268396us
I20260709 12:24:18.791672 18169 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
W20260709 12:24:18.792337 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 319693us
I20260709 12:24:18.701321 24424 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.795279 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.798281 24514 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.810633 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.827705 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.828771 18090 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:18.829691 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.810930 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:18.821923 18164 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:18.817261 24518 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:18.831354 18020 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.013s	sys 0.005s
W20260709 12:24:18.839210 18020 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.013s	sys 0.006s
W20260709 12:24:18.833282 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 304589us
I20260709 12:24:18.815500 24486 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:18.834738 18064 leader_election.cc:400] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16.
I20260709 12:24:18.840692 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.838191 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:18.851132 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:18.832850 24302 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:18.826133 24343 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.840924 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:18.865406 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.815194 24405 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:24:18.878364 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:18.855844 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.841583 24414 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:18.866751 24018 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:18.839210 24428 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:18.869974 24527 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:18.872788 24509 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.875465 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:18.908336 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.875589 23886 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:18.864992 24486 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:18.876554 24518 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:18.879375 18164 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:18.879626 18095 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:18.881124 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.832063 24444 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:24:18.920509 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1007) took 1154 ms. Trace:
W20260709 12:24:18.883087 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.921057 18095 rpcz_store.cc:276] 0709 12:24:17.766028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.766169 (+   141us) service_pool.cc:224] Handling call
0709 12:24:18.722744 (+956575us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:24:18.920430 (+197686us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10137472}
W20260709 12:24:18.873672 24480 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.849s	user 0.011s	sys 0.002s
W20260709 12:24:18.886595 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45790.5us
I20260709 12:24:18.888347 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:18.850389 24485 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.007s	sys 0.012s
I20260709 12:24:18.833350 24407 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:24:18.676950 17915 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 7 to 10, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 10 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:18.877440 24500 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.911219 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2835) took 1107 ms. Trace:
I20260709 12:24:18.855156 24114 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:18.882190 24471 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:18.882169 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:18.926604 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:18.927088 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:18.932144 18016 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:18.844609 24455 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:24:18.897442 24428 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:18.897948 24527 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.843686 24194 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.939527 18164 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:18.886940 24414 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.899375 24302 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:18.912928 24306 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.899955 24018 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:18.912779 23886 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:18.854970 24514 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.839645 24482 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:18.939215 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58250.1us
I20260709 12:24:19.001003 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:19.001257 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.002388 18015 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:19.002713 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:19.002985 24467 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:18.972988 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.004273 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.004347 24518 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.930406 24457 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.929781 24281 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.005002 24518 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.957460 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:19.005196 24457 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:18.935412 24512 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:19.006692 24512 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.007676 24388 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:19.008309 24457 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.008390 24518 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:18.947708 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.009421 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2688) took 1060 ms. Trace:
I20260709 12:24:19.009899 18019 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:19.009939 18164 rpcz_store.cc:276] 0709 12:24:17.948667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.920684 (+972017us) service_pool.cc:224] Handling call
0709 12:24:18.939328 (+ 18644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.009098 (+ 69770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.009343 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4352}
I20260709 12:24:18.964581 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1001) took 1202 ms. Trace:
I20260709 12:24:18.974905 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2694) took 1163 ms. Trace:
W20260709 12:24:18.939631 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51432.7us
I20260709 12:24:19.003877 23886 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.974843 24486 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.045393 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:19.046380 24452 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:19.046921 24452 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:19.047111 17915 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 5 to 13, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 13 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:19.049022 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:18.941056 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:19.009315 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.013078 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:18.939324 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:18.981593 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1150) took 2334 ms. Trace:
I20260709 12:24:19.050814 18171 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:19.010746 18097 rpcz_store.cc:276] 0709 12:24:17.803450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.402399 (+598949us) service_pool.cc:224] Handling call
0709 12:24:18.402820 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.403118 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:18.642165 (+239047us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.910738 (+268573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.910824 (+    86us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:18.910890 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.910973 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.911142 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23550,"prepare.queue_time_us":394661,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":23961,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":9426,"thread_start_us":444676,"threads_started":2,"wal-append.queue_time_us":324776}
W20260709 12:24:18.968827 24526 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:19.015657 24539 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:18.953590 24523 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:18.959196 24480 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:19.016585 24114 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:18.934199 24155 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.017657 24472 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:19.018981 24340 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:19.005302 24281 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:19.011981 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 129897us
I20260709 12:24:19.021909 24527 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:19.023329 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 917) took 1853 ms (client timeout 1508 ms). Trace:
I20260709 12:24:19.024824 18021 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:19.170326 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.026942 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2692) took 1070 ms. Trace:
I20260709 12:24:19.028008 24483 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:19.077991 24480 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.030351 24524 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:19.030818 24302 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.179181 24399 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.173154 24483 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:19.032770 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.178294 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:19.033669 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.034164 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.033937 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.182788 18168 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:19.037477 24471 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:19.037211 24428 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:19.015511 24343 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.029598 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.034946 24018 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.032783 24360 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:18.936372 24485 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.008s	sys 0.013s
I20260709 12:24:18.984835 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3549) took 1184 ms. Trace:
I20260709 12:24:19.041882 24455 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:19.042837 23886 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.045544 24411 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:19.047834 24452 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.050107 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 863) took 1431 ms. Trace:
I20260709 12:24:18.921191 24517 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:19.012181 18024 rpcz_store.cc:276] 0709 12:24:17.811405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.561610 (+750205us) service_pool.cc:224] Handling call
0709 12:24:18.587382 (+ 25772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.587803 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.587835 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.642985 (+ 55150us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.898581 (+255596us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:18.898649 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.898749 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.974618 (+ 75869us) raft_consensus.cc:1685] finished
0709 12:24:18.974644 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.974834 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14561,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":13471,"thread_start_us":150261,"threads_started":1,"wal-append.queue_time_us":256942}
I20260709 12:24:18.948215 18166 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:19.011832 18091 rpcz_store.cc:276] 0709 12:24:17.762295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.762476 (+   181us) service_pool.cc:224] Handling call
0709 12:24:18.714629 (+952153us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:18.714889 (+   260us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:18.774897 (+ 60008us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.774948 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:18.813196 (+ 38248us) log.cc:844] Serialized 52 byte log entry
0709 12:24:18.921412 (+108216us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:18.962897 (+ 41485us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.963239 (+   342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.963340 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.963404 (+    64us) raft_consensus.cc:1685] finished
0709 12:24:18.963528 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.964488 (+   960us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86010,"prepare.queue_time_us":38197,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":1589,"raft.queue_time_us":61567,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":32743,"spinlock_wait_cycles":10416512,"thread_start_us":101423,"threads_started":1,"wal-append.queue_time_us":97981}
I20260709 12:24:19.051168 18808 rpcz_store.cc:276] 0709 12:24:16.646841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.647159 (+   318us) service_pool.cc:224] Handling call
0709 12:24:16.647566 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.366089 (+718523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.981158 (+1615069us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54004a60d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:18.981472 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1532642048}
I20260709 12:24:19.064369 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.066064 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:19.222344 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 156404us
W20260709 12:24:19.070433 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.198279 24472 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.078490 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.225435 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.069895 24090 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:19.079571 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3509968739999749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
I20260709 12:24:19.226199 18017 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:19.087570 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.093179 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:19.220036 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2689) took 1269 ms. Trace:
I20260709 12:24:19.226967 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:19.227170 18171 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:19.087738 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.227276 18166 rpcz_store.cc:276] 0709 12:24:17.950402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.930321 (+979919us) service_pool.cc:224] Handling call
0709 12:24:18.944099 (+ 13778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.219585 (+275486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.219937 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:19.228250 24471 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.228945 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:19.155652 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67877.8us
I20260709 12:24:19.229668 18807 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:19.230911 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.158458 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.158459 18164 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:24:19.232394 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.210335 24372 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.544s	user 0.021s	sys 0.011s
I20260709 12:24:19.069485 24518 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.171779 18173 rpcz_store.cc:276] 0709 12:24:17.956286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.417613 (+461327us) service_pool.cc:224] Handling call
0709 12:24:19.026609 (+608996us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54000505d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:19.026837 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407729920}
I20260709 12:24:19.029531 24525 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:19.170318 18167 rpcz_store.cc:269] 0709 12:24:17.170225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.184029 (+ 13804us) service_pool.cc:224] Handling call
0709 12:24:19.023019 (+1838990us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:19.023231 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527491840}
I20260709 12:24:19.244311 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.171312 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:19.082595 24512 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:18.995002 24334 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.087287 24514 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:19.082736 24539 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:18.970609 24531 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:19.035348 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.163640 24340 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:19.181079 24483 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.187350 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.187433 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.188525 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.283373 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.284415 18097 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:24:19.188542 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:19.285574 24542 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 12:24:19.287549 24541 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:24:19.192421 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.194398 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.283756 18172 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:19.290182 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2846) took 1367 ms. Trace:
I20260709 12:24:19.290278 18168 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:19.290653 18090 rpcz_store.cc:276] 0709 12:24:17.922368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.833717 (+911349us) service_pool.cc:224] Handling call
0709 12:24:18.855814 (+ 22097us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.877774 (+ 21960us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.877850 (+    76us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.898171 (+ 20321us) log.cc:844] Serialized 26 byte log entry
0709 12:24:19.081552 (+183381us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:19.081618 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.081703 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:19.289867 (+208164us) raft_consensus.cc:1685] finished
0709 12:24:19.289892 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.290108 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30943,"raft.queue_time_us":41452,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":11324,"thread_start_us":138368,"threads_started":1,"wal-append.queue_time_us":241072}
I20260709 12:24:19.291738 18168 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:19.176625 24480 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:19.164987 24155 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.199726 18020 rpcz_store.cc:276] 0709 12:24:17.800703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.070462 (+269759us) service_pool.cc:224] Handling call
0709 12:24:18.120815 (+ 50353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.121425 (+   610us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.121460 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.135030 (+ 13570us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.839867 (+704837us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:18.839922 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.840002 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.984494 (+144492us) raft_consensus.cc:1685] finished
0709 12:24:18.984529 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.984737 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":151,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":23475,"thread_start_us":556596,"threads_started":1,"wal-append.queue_time_us":735677}
I20260709 12:24:19.217486 18018 rpcz_store.cc:276] 0709 12:24:17.619010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.619262 (+   252us) service_pool.cc:224] Handling call
0709 12:24:17.619665 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.900425 (+280760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.049805 (+1149380us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:19.050036 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":644329216}
W20260709 12:24:19.070159 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 207270us
W20260709 12:24:19.298591 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.222054 18806 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:19.062332 24493 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.001s
I20260709 12:24:19.232805 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2691) took 1277 ms. Trace:
I20260709 12:24:19.303615 18164 rpcz_store.cc:276] 0709 12:24:17.955867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.062506 (+1106639us) service_pool.cc:224] Handling call
0709 12:24:19.102986 (+ 40480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.232505 (+129519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.232743 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:19.167008 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:19.233265 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.233573 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:19.307385 18808 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:24:19.229009 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.308905 18166 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:19.229669 18019 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:19.240489 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.202773 24455 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:19.245332 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.243772 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.208037 24524 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:19.266417 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.264256 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.273567 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.391847 18017 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:19.270684 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:19.242432 24372 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.574s	user 0.025s	sys 0.011s
I20260709 12:24:19.219066 24517 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.280350 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:19.405457 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.165392 24500 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:19.232260 24481 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.414955 24481 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.415836 18166 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:19.244797 24510 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:19.418628 24510 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:19.418831 24543 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:24:19.419193 24510 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:19.272476 24526 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:19.088140 17915 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:19.190047 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.262235 23931 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.294754 18172 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:19.282680 18815 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:19.296473 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.342049 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:19.448748 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106891us
W20260709 12:24:19.400431 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 129863us
W20260709 12:24:19.401957 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 235057us
W20260709 12:24:19.450265 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.451067 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:19.402247 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.402354 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.388845 18173 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:24:19.406459 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 107111us
I20260709 12:24:19.278144 24525 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:19.407753 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.471266 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2699) took 1129 ms. Trace:
I20260709 12:24:19.471947 18172 rpcz_store.cc:276] 0709 12:24:18.342332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.290752 (+948420us) service_pool.cc:224] Handling call
0709 12:24:19.291120 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.470858 (+179738us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.471174 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:19.408164 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.270714 24313 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.124s
I20260709 12:24:19.481770 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:19.413713 18164 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:19.413712 18167 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:19.262624 24343 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:24:19.246492 24047 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.155s
I20260709 12:24:19.238806 24090 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.501459 24090 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.503139 24155 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.503855 24155 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:19.504273 24155 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.480552 24525 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:19.504699 24155 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.505625 24334 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:19.506968 24340 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.508174 24340 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.414258 24517 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:19.302389 24421 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.271414 24518 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.532725 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:19.534596 18097 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:24:19.245348 24472 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.302340 24523 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:19.431833 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 206061us
I20260709 12:24:19.540578 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:19.447742 18023 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:19.474584 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:19.405745 24311 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:19.412966 24473 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:19.496310 24485 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.406877 24509 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:19.422410 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.510862 18091 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:19.420598 24483 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:19.512787 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.515765 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.532691 24539 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:19.482566 24531 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:24:19.403570 24493 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.495890 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2702) took 1005 ms. Trace:
I20260709 12:24:19.494800 24306 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:19.481604 24500 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:19.419947 24194 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.116s
I20260709 12:24:19.533497 24421 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.523502 24114 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:19.537338 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:19.628564 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91465.1us
I20260709 12:24:19.583241 18165 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:19.499179 24436 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:19.581733 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:19.631220 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49658.2us
I20260709 12:24:19.587710 18171 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:24:19.602995 18164 rpcz_store.cc:276] 0709 12:24:18.490375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.304837 (+814462us) service_pool.cc:224] Handling call
0709 12:24:19.413520 (+108683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.495569 (+ 82049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.495810 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23808}
I20260709 12:24:19.571285 24311 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:19.481434 23931 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.523713 24462 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:24:19.630167 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2704) took 1139 ms. Trace:
I20260709 12:24:19.595458 24512 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:19.577406 24509 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:19.635564 18165 rpcz_store.cc:276] 0709 12:24:18.491179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.306861 (+815682us) service_pool.cc:224] Handling call
0709 12:24:19.582938 (+276077us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.629665 (+ 46727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.630066 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:19.633479 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:19.610105 24007 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.571789 24334 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:24:19.658385 18098 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:19.659966 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.614890 24306 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.610525 24281 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:19.664836 24466 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:19.644496 24311 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:19.674824 23931 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.642957 24512 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:19.657490 24114 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:19.671402 24553 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:19.605422 24471 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:24:19.656713 24509 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:19.688793 24531 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:24:19.725106 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.574295 24523 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.733415 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:19.700232 24306 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:19.722306 24360 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.516s	user 0.013s	sys 0.001s
I20260709 12:24:19.629693 24500 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.585724 24485 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.656026 24436 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.711527 24466 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:19.766045 24538 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:19.767645 24548 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:24:19.749379 24553 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.763976 24114 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.800388 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:19.585323 24524 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.773965 24457 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.783864 18017 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260709 12:24:19.825909 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.845598 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.778522 24518 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:19.788816 24466 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:19.858044 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 406993us
I20260709 12:24:19.866791 24547 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:19.871608 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3646) took 1202 ms. Trace:
I20260709 12:24:19.872448 18091 rpcz_store.cc:276] 0709 12:24:18.669788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.578868 (+909080us) service_pool.cc:224] Handling call
0709 12:24:19.585508 (+  6640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.585781 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:19.622944 (+ 37163us) log.cc:844] Serialized 14 byte log entry
0709 12:24:19.870995 (+248051us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:19.871079 (+    84us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:19.871166 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.871277 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.871490 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31896,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"thread_start_us":247491,"threads_started":1}
W20260709 12:24:19.879870 18062 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.015s	sys 0.011s
W20260709 12:24:19.896761 18062 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.015s	sys 0.012s
W20260709 12:24:19.888166 18780 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.508s (SENT) [suppressed 16 similar messages]
I20260709 12:24:19.893781 24547 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.906266 24547 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.778577 24372 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:24:19.835119 24553 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:19.911636 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 371426us
I20260709 12:24:19.837334 24538 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:19.845131 24552 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:19.699373 17915 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:20.048517 18779 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.000s	sys 0.017s
W20260709 12:24:20.050226 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84739.2us
I20260709 12:24:19.851461 24457 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:19.980350 24557 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.985673 18024 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:19.942484 24538 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:19.986373 24562 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.947240 24472 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:19.973567 24414 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.991537 24485 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:19.994185 24510 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.574s	user 0.012s	sys 0.003s
I20260709 12:24:19.941620 24313 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:19.996172 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.996628 24558 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:19.999969 23886 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.783s	user 0.012s	sys 0.004s
I20260709 12:24:20.003793 24486 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:20.049933 24411 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.833s	user 0.009s	sys 0.005s
W20260709 12:24:20.055219 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.931842 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 270057us
I20260709 12:24:19.822096 24525 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.117419 24510 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:19.851586 24114 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.109004 24561 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260709 12:24:20.116920 24500 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:19.945025 24372 consensus_queue.cc:1243] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 15.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:20.125423 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:20.115485 24436 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:20.163620 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.000s	sys 0.018s
W20260709 12:24:20.063021 18779 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.000s	sys 0.018s
I20260709 12:24:20.187983 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:20.121627 24527 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.947s	user 0.008s	sys 0.004s
I20260709 12:24:20.067736 24557 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.200608 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:20.470834 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 930281us
I20260709 12:24:20.471868 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:20.355459 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 331923us
I20260709 12:24:20.474189 18808 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 14 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:24:20.474434 24574 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.475179 24574 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.224962 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.478919 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.357796 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.290330 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 402190us
W20260709 12:24:20.482008 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.906s	user 0.000s	sys 0.023s
I20260709 12:24:20.482460 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:20.482741 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.00821e+06us
I20260709 12:24:20.483671 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:20.469380 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2682) took 2530 ms. Trace:
I20260709 12:24:20.122037 24472 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:20.494122 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.376405 24473 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.001s
I20260709 12:24:20.486855 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:20.487905 24299 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:20.487689 24552 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:20.493148 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:20.499466 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:20.468154 24480 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.067s	user 0.013s	sys 0.002s
I20260709 12:24:20.531591 18017 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:20.531688 24480 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.532742 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:20.533456 18017 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:20.534549 24480 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.534228 24299 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.535523 18812 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:20.541329 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:20.362326 24018 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.201392 24313 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.514644 24559 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.324s	user 0.000s	sys 0.001s
I20260709 12:24:20.514075 24472 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.499655 24218 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:20.515394 24523 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.517602 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2843) took 2701 ms. Trace:
I20260709 12:24:20.383701 23886 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:20.518719 24566 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.452s	user 0.000s	sys 0.001s
W20260709 12:24:20.519249 24483 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.915s	user 0.016s	sys 0.004s
I20260709 12:24:20.484023 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:20.543164 24473 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:20.537325 24299 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.531095 24574 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.476936 24539 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.552265 18095 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:20.349054 24340 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.240360 24558 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:20.523069 24311 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.768s	user 0.012s	sys 0.005s
I20260709 12:24:20.518622 24532 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:20.653019 24532 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:20.653856 24532 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.500474 24421 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.569346 18096 rpcz_store.cc:276] 0709 12:24:17.816237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.588852 (+772615us) service_pool.cc:224] Handling call
0709 12:24:18.589305 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.737704 (+148399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.737744 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:19.287628 (+549884us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:19.288229 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:24:19.784684 (+496455us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:19.784745 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.784859 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.517213 (+732354us) raft_consensus.cc:1685] finished
0709 12:24:20.517242 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.517517 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":147815,"raft.queue_time_us":148416,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"spinlock_wait_cycles":63028480,"thread_start_us":495913,"threads_started":1,"wal-append.queue_time_us":1092978}
W20260709 12:24:20.561146 24452 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.018s	sys 0.047s
I20260709 12:24:20.525794 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:20.588411 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 104467us
I20260709 12:24:20.659209 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:20.663057 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:20.656214 24299 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.666415 24568 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.666651 24299 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.620827 24483 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:20.668344 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 142627us
I20260709 12:24:20.665859 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:20.669672 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:20.595229 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111638us
I20260709 12:24:20.670724 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:20.596138 24500 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:20.551198 24405 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:20.240094 24486 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.070379 24562 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.475502 24527 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.575716 24428 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.385s	user 0.010s	sys 0.006s
I20260709 12:24:20.560216 24573 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.585953 24547 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:20.653023 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1355) took 1604 ms. Trace:
I20260709 12:24:20.699901 18811 rpcz_store.cc:276] 0709 12:24:19.049116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.049267 (+   151us) service_pool.cc:224] Handling call
0709 12:24:19.049627 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.169431 (+119804us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:19.169477 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:19.171981 (+  2504us) log.cc:844] Serialized 26 byte log entry
0709 12:24:19.636204 (+464223us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:19.636244 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.636325 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.652673 (+1016348us) raft_consensus.cc:1685] finished
0709 12:24:20.652703 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.652927 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":32475,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":53,"thread_start_us":583000,"threads_started":2,"wal-append.queue_time_us":769429}
I20260709 12:24:20.663261 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:20.663981 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:20.614961 24480 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:20.671902 24574 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:20.720129 18809 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:20.655757 24580 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:20.567785 24018 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.661696 24452 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.019s	sys 0.047s
I20260709 12:24:20.647895 24558 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.677408 24505 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:20.674046 24517 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.107s	user 0.016s	sys 0.001s
I20260709 12:24:20.588443 23886 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.682893 24586 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 12:24:20.677006 24485 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:20.587985 24218 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.699082 24466 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.003s
I20260709 12:24:20.682523 24360 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:20.646477 24340 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.656038 24311 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.656944 24539 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.667109 24568 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.684468 24483 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.541339 24552 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:20.688175 24405 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.726164 24555 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.716403 24428 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:20.807499 24509 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.046s	user 0.018s	sys 0.000s
I20260709 12:24:20.715528 24573 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.814474 24510 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.816413 24572 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:20.808888 24557 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.800820 24517 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:20.822275 24525 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.825407 24555 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.871080 24578 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.004s
W20260709 12:24:20.869264 24570 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.076s	user 0.002s	sys 0.008s
I20260709 12:24:20.871809 24568 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.872653 24573 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:20.540632 17915 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 15, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:20.671078 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:20.928406 18015 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:24:20.664649 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.530756 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:20.932323 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2708) took 1845 ms. Trace:
I20260709 12:24:20.676008 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:20.934466 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 382131us
I20260709 12:24:20.935503 18096 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:20.677645 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:20.725775 18810 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:20.943785 18815 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:20.932691 18024 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:20.705950 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:20.949752 24114 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.954854 24313 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.956037 24313 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.933043 18019 rpcz_store.cc:276] 0709 12:24:19.086518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.777997 (+691479us) service_pool.cc:224] Handling call
0709 12:24:19.834991 (+ 56994us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.835432 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:19.835466 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:19.836017 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 12:24:20.672231 (+836214us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:20.672277 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.672357 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.931534 (+259177us) raft_consensus.cc:1685] finished
0709 12:24:20.931588 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.932029 (+   441us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52744,"raft.queue_time_us":7280,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":1791,"spinlock_wait_cycles":6912,"thread_start_us":203032,"threads_started":1,"wal-append.queue_time_us":898424}
I20260709 12:24:20.954959 24114 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:20.958956 24311 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.957201 24313 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:20.961088 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 289896us
I20260709 12:24:20.963688 24313 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.964102 24299 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.964663 24313 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:20.966030 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.807713 24360 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:20.970651 24360 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:20.971262 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 247910us
I20260709 12:24:20.928808 17915 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:20.991255 24572 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:20.991626 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:20.992153 24486 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.996208 24517 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.999500 24527 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:20.812287 24466 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:21.003446 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.001362 24527 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:21.014420 24472 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.016585 18018 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:24:21.019238 24424 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:20.953143 24018 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:20.957680 23886 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.025861 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:20.988967 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.008066 24405 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.039422 18020 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
W20260709 12:24:21.039893 18093 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2836) took 3235 ms (client timeout 1871 ms). Trace:
W20260709 12:24:21.043035 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2849) took 2181 ms (client timeout 2009 ms). Trace:
W20260709 12:24:21.043756 18095 rpcz_store.cc:269] 0709 12:24:18.861202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.173477 (+312275us) service_pool.cc:224] Handling call
0709 12:24:21.042377 (+1868900us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400214654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.042715 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2894472064}
I20260709 12:24:21.046025 24340 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:21.046387 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55117.7us
W20260709 12:24:21.040158 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.040210 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2707) took 1954 ms. Trace:
I20260709 12:24:21.048173 18020 rpcz_store.cc:276] 0709 12:24:19.086220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.405150 (+318930us) service_pool.cc:224] Handling call
0709 12:24:21.001422 (+1596272us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:24:21.040104 (+ 38682us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5255424}
I20260709 12:24:21.052596 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:21.040548 18093 rpcz_store.cc:269] 0709 12:24:17.803830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.894830 (+ 91000us) service_pool.cc:224] Handling call
0709 12:24:21.039295 (+3144465us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400411ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.039620 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2637331712}
I20260709 12:24:21.059712 18811 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:21.063179 18094 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3652) took 2195 ms (client timeout 1575 ms). Trace:
W20260709 12:24:21.063838 18094 rpcz_store.cc:269] 0709 12:24:18.867649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.875154 (+  7505us) service_pool.cc:224] Handling call
0709 12:24:21.062570 (+2187416us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.062907 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":860948224}
I20260709 12:24:21.064303 24428 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.068104 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2851) took 2206 ms. Trace:
I20260709 12:24:21.068858 18097 rpcz_store.cc:276] 0709 12:24:18.863116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.471453 (+608337us) service_pool.cc:224] Handling call
0709 12:24:21.067440 (+1595987us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b5400052654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.067797 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118018304}
W20260709 12:24:21.074177 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 396447us
W20260709 12:24:21.074455 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87621.9us
I20260709 12:24:21.076211 24340 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:21.077785 24340 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:21.078282 24340 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:21.077335 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:21.079620 24340 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.086490 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.087419 24340 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:20.484757 18170 rpcz_store.cc:276] 0709 12:24:17.939288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.939445 (+   157us) service_pool.cc:224] Handling call
0709 12:24:17.952100 (+ 12655us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.540464 (+588364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.540496 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.541072 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.793054 (+251982us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:18.836350 (+ 43296us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.836788 (+   438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.005663 (+168875us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b1800357ca0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:19.015223 (+  9560us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b1800357ca0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:24:19.971067 (+955844us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b54003ba1d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:19.971135 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:19.971197 (+    62us) raft_consensus.cc:1685] finished
0709 12:24:19.971212 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.469294 (+498082us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39882,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":43,"raft.queue_time_us":5112,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":2961,"spinlock_wait_cycles":1367702272,"thread_start_us":251609,"threads_started":1,"wal-append.queue_time_us":282964,"wal-append.run_cpu_time_us":8196,"wal-append.run_wall_time_us":1657775}
W20260709 12:24:21.102458 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.103281 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.105576 24466 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.108601 17915 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:21.110859 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1141) took 2063 ms (client timeout 2696 ms). Trace:
W20260709 12:24:21.111469 18809 rpcz_store.cc:269] 0709 12:24:19.047520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.048666 (+  1146us) service_pool.cc:224] Handling call
0709 12:24:21.110263 (+2061597us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540009bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.110583 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3841003776}
I20260709 12:24:21.112047 17915 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 12 to 14, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 14 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:21.125494 24552 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.126609 18096 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:21.128167 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:20.822187 24509 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.125636 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.114374 24428 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.133711 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1137) took 2100 ms. Trace:
I20260709 12:24:20.798794 24505 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.134506 18809 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:24:21.135967 24007 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.356s	user 0.013s	sys 0.008s
I20260709 12:24:21.147075 24592 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.142310 24472 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:21.142191 24428 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.148407 24483 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:21.148633 24538 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.079s	user 0.015s	sys 0.001s
I20260709 12:24:21.145066 18812 rpcz_store.cc:276] 0709 12:24:19.032589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.033299 (+   710us) service_pool.cc:224] Handling call
0709 12:24:19.033682 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.536050 (+1502368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.132818 (+596768us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54005ad554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:21.133345 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3453347712}
I20260709 12:24:21.162194 18090 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:21.132570 24509 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.135833 24574 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.170400 24574 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.171191 24505 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.172420 24370 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:21.173067 24370 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:21.137547 24573 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:21.174358 24007 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:21.180759 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 103700us
I20260709 12:24:21.182596 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:21.183892 18808 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:21.185101 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.185768 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:21.186671 18813 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:21.175772 24218 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:21.187502 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.175905 24592 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.176131 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.191167 18807 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:21.192860 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:21.176299 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:21.178836 17989 connection.cc:256] RPC call timeout handler was delayed by -1.8344035989998702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26070000000000004s for a response
I20260709 12:24:21.173779 24574 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.218847 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.219012 24370 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.220404 24444 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:21.224622 24455 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.176149 24538 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.177428 24555 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.246104 18812 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:21.248397 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.248553 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.248857 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.234561 24444 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:21.258919 18809 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
W20260709 12:24:21.233085 24155 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.625s	user 0.013s	sys 0.017s
I20260709 12:24:21.269869 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.270895 18813 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 12.
W20260709 12:24:21.275629 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99711.9us
I20260709 12:24:21.275988 24343 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:21.274405 18138 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.012s	sys 0.013s
I20260709 12:24:21.278314 24343 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.278837 24343 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:21.278694 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 711604us
I20260709 12:24:21.278421 24155 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.281422 24155 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.282341 24421 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:24:21.283908 24547 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:21.286892 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.289261 18806 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:21.203630 24592 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.296134 24434 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.292649 18814 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:24:21.299144 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.276635 24370 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.302150 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:21.305636 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.179819 24573 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.282205 24455 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:21.307999 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:21.311030 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.312147 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.311705 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.309589 18808 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:21.314108 18811 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:21.303566 24018 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.332674 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:21.337807 24434 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:21.346344 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.360692 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:21.368018 24421 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:21.368925 24421 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:21.369444 24421 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:21.370170 24553 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.425s	user 0.015s	sys 0.007s
I20260709 12:24:21.370793 24553 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:21.368235 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3637) took 4309 ms (client timeout 3425 ms). Trace:
W20260709 12:24:21.374621 18099 rpcz_store.cc:269] 0709 12:24:17.058958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.059237 (+   279us) service_pool.cc:224] Handling call
0709 12:24:21.365090 (+4305853us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.367916 (+  2826us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4646050176}
I20260709 12:24:21.376714 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:21.327185 24444 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.317075 24547 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.373492 24553 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.385192 24574 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:21.385280 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3642) took 2789 ms (client timeout 2696 ms). Trace:
I20260709 12:24:21.390532 24574 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.390733 24428 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.391113 24574 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:21.377796 24018 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:21.390604 18098 rpcz_store.cc:269] 0709 12:24:18.595645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.792960 (+197315us) service_pool.cc:224] Handling call
0709 12:24:21.377744 (+2584784us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b54002959d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.384952 (+  7208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1849185536}
I20260709 12:24:21.397972 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.398411 24547 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.393496 24018 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:21.406179 24306 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.627s	user 0.018s	sys 0.013s
I20260709 12:24:21.407282 24306 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.409485 24434 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.416055 24436 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:21.417472 24274 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:21.418047 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:21.418105 24274 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:21.419899 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:21.420287 24274 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.421128 24428 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:21.421666 24428 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.421214 24481 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.422536 24428 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:21.426661 24274 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:21.427860 24457 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.429085 24444 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.438900 24493 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:21.439348 24558 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.633s	user 0.012s	sys 0.005s
I20260709 12:24:21.440132 24558 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.442284 18091 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:21.446861 24573 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.434700 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:21.451941 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.454020 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:21.454563 18096 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:24:21.454715 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:21.455307 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55384.8us
I20260709 12:24:21.455627 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:21.456619 24471 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:21.457028 24471 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.457397 24471 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.436079 24411 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:21.458149 24411 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:21.458649 24411 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.459031 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.459849 18016 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:24:21.461908 18016 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:21.463706 24514 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.463932 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:21.464498 24514 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.465438 24194 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:21.466084 24194 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:21.463770 24336 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:24:21.466599 24194 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:21.452229 24532 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.798s	user 0.012s	sys 0.008s
I20260709 12:24:21.458273 18098 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
W20260709 12:24:21.469645 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49881.5us
I20260709 12:24:21.471027 18099 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 12:24:21.478587 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63867us
I20260709 12:24:21.480413 24306 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.480928 24514 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.455439 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1009) took 2623 ms. Trace:
I20260709 12:24:21.489032 24539 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.483748 18096 rpcz_store.cc:276] 0709 12:24:18.832502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.453626 (+2621124us) service_pool.cc:224] Handling call
0709 12:24:21.454255 (+   629us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.455072 (+   817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.455350 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:21.494472 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:21.498960 18136 connection.cc:256] RPC call timeout handler was delayed by -1.0874509259997467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3425s for a response
I20260709 12:24:21.503315 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:21.503604 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.509949 18806 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:21.510447 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.513767 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:21.520841 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.513661 24336 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.528821 24306 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:21.546401 24336 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.546487 24306 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.558056 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3580) took 1041 ms. Trace:
I20260709 12:24:21.558925 18019 rpcz_store.cc:276] 0709 12:24:20.516374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.958750 (+442376us) service_pool.cc:224] Handling call
0709 12:24:20.959765 (+  1015us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:20.960689 (+   924us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:20.960750 (+    61us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.022252 (+ 61502us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.267766 (+245514us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:21.295083 (+ 27317us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.297913 (+  2830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.298022 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.557658 (+259636us) raft_consensus.cc:1685] finished
0709 12:24:21.557692 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.557952 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3808,"prepare.queue_time_us":28099,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":83,"raft.queue_time_us":6955,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":4655,"thread_start_us":125913,"threads_started":1,"wal-append.queue_time_us":296387}
I20260709 12:24:21.561606 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3579) took 1062 ms. Trace:
I20260709 12:24:21.531428 24539 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.563187 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.563728 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.563359 24539 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.498293 24360 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.562680 18812 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:24:21.566159 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 927) took 3040 ms (client timeout 1634 ms). Trace:
W20260709 12:24:21.566671 18169 rpcz_store.cc:269] 0709 12:24:18.492646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.550895 (+ 58249us) service_pool.cc:224] Handling call
0709 12:24:21.528388 (+2977493us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.532951 (+  4563us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":304717056}
I20260709 12:24:21.490072 24532 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:21.568292 24576 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.072s	user 0.000s	sys 0.004s
I20260709 12:24:21.570271 24532 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.547153 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3649) took 2715 ms. Trace:
W20260709 12:24:21.572733 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.562341 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.572006 18090 rpcz_store.cc:276] 0709 12:24:18.831308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.121514 (+1290206us) service_pool.cc:224] Handling call
0709 12:24:21.116906 (+995392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.162897 (+ 45991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.546701 (+383804us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b5400214650. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.546930 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1853492480}
W20260709 12:24:21.574395 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44484.8us
I20260709 12:24:21.580734 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2842) took 3763 ms. Trace:
I20260709 12:24:21.562201 18024 rpcz_store.cc:276] 0709 12:24:20.501349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.950395 (+449046us) service_pool.cc:224] Handling call
0709 12:24:21.014565 (+ 64170us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.078171 (+ 63606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.078207 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.079077 (+   870us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.202586 (+123509us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:21.275808 (+ 73222us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.276090 (+   282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.276178 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.561270 (+285092us) raft_consensus.cc:1685] finished
0709 12:24:21.561297 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.561495 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110386,"prepare.queue_time_us":37601,"prepare.run_cpu_time_us":145,"prepare.run_wall_time_us":7931,"raft.queue_time_us":99734,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":28138,"thread_start_us":158175,"threads_started":2,"wal-append.queue_time_us":426605}
I20260709 12:24:21.572081 24336 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.586743 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.587978 18173 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:21.588625 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:21.593772 18137 connection.cc:256] RPC call timeout handler was delayed by -2.1834031479997975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
W20260709 12:24:21.597159 18137 connection.cc:256] RPC call timeout handler was delayed by -1.9449031479998666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20260709 12:24:21.597308 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:21.598240 18137 connection.cc:256] RPC call timeout handler was delayed by -1.8369031480001468s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20830000000000004s for a response
W20260709 12:24:21.598696 18137 connection.cc:256] RPC call timeout handler was delayed by -1.7982031480000842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2126s for a response
W20260709 12:24:21.599020 18137 connection.cc:256] RPC call timeout handler was delayed by -1.7910031480000725s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
I20260709 12:24:21.599716 24336 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:21.600600 24340 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.600397 24336 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:21.599784 18137 connection.cc:256] RPC call timeout handler was delayed by -1.2852031480001642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2696s for a response
I20260709 12:24:21.598263 18165 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:21.603387 24336 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.603224 24512 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.604847 24512 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.605297 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.606422 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.606830 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.603425 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:21.601991 18137 connection.cc:256] RPC call timeout handler was delayed by -1.2081734519997553s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18610000000000002s for a response
I20260709 12:24:21.614095 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:21.582636 23886 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.581293 18092 rpcz_store.cc:276] 0709 12:24:17.812391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.544156 (+731765us) service_pool.cc:224] Handling call
0709 12:24:18.544591 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.739536 (+194945us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.739585 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.060131 (+2320546us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:21.061050 (+   919us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.119127 (+ 58077us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:21.119223 (+    96us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.573328 (+454105us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400280354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:21.573397 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.573446 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:21.573459 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.575177 (+  1718us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57655,"raft.queue_time_us":59409,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":122,"spinlock_wait_cycles":373842560,"thread_start_us":57431,"threads_started":1,"wal-append.queue_time_us":261163}
I20260709 12:24:21.617952 24558 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:21.599114 24194 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.601285 24340 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.622691 24340 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.622813 24194 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.618660 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.624421 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.623821 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:21.597460 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.629186 18812 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:24:21.630151 18169 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:24:21.633162 18092 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:21.631389 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.585460 24421 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.636232 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:21.637133 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.594484 24539 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.637058 18165 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:21.640617 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:21.643241 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 197081us
I20260709 12:24:21.578035 24343 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.652827 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.658721 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.659632 18017 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 12:24:21.658747 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.667095 24421 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.646689 24405 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:21.668085 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.668622 24463 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:21.643935 23886 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.670060 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.670832 24482 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:21.671088 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.629588 24360 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:21.673569 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.674835 18095 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:21.676366 18092 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:21.674147 18017 pending_rounds.cc:85] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 12
I20260709 12:24:21.676790 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:21.679003 24606 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:24:21.679158 24598 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.405s	user 0.003s	sys 0.000s
I20260709 12:24:21.679898 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:21.636394 24336 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.735257 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.680680 24565 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.623344 24574 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:21.683640 24556 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.673s	user 0.000s	sys 0.003s
I20260709 12:24:21.742780 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.743746 18015 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:24:21.743991 24589 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.960s	user 0.007s	sys 0.006s
I20260709 12:24:21.745214 18018 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:21.712196 18017 pending_rounds.cc:107] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 12:24:21.746162 18017 raft_consensus.cc:2889] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:21.746649 24336 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.684643 18024 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:21.687029 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.687978 18016 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
W20260709 12:24:21.749917 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.751703 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.705659 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.722766 18169 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:21.723474 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:21.752799 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:21.724341 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.725358 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.727108 18019 raft_consensus.cc:2368] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17.
I20260709 12:24:21.727609 23886 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.728018 24424 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:21.697590 24463 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:21.733139 24306 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:21.734109 24343 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.710305 18098 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:21.734839 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3648) took 2925 ms. Trace:
I20260709 12:24:21.740998 18166 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:21.755076 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.754675 18020 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:21.729157 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.761673 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:21.716883 24405 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:21.771432 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.771452 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.767370 18091 rpcz_store.cc:276] 0709 12:24:18.809582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.895707 (+1086125us) service_pool.cc:224] Handling call
0709 12:24:19.896205 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.442914 (+1546709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.734506 (+291592us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540011dd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:21.734762 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3554406784}
I20260709 12:24:21.772724 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.774439 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.771850 18165 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
W20260709 12:24:21.777712 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.778831 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.782737 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.784673 18169 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:21.786454 18092 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.787366 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.788023 18098 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:21.789522 18173 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:21.789875 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.781106 18172 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:21.785202 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.793182 18165 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:21.794251 18021 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:21.792443 18018 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:21.795235 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.797247 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.798549 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.798434 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.799628 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.758072 24411 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.757606 24565 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:21.808429 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.814599 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.809197 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.813331 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.815663 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:21.815984 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.813419 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.815299 18169 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:21.794221 24343 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:21.821310 18173 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:21.818233 18093 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:21.822027 18165 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:21.824788 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:21.825121 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.799136 24336 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.826397 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:21.828560 18165 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:21.830498 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.831143 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.830276 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.832365 18024 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:21.834874 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.835884 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.836117 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.837925 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.838899 18165 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:21.855116 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.860683 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.855618 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1010) took 3001 ms. Trace:
I20260709 12:24:21.854629 18090 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:21.862547 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3650) took 2998 ms. Trace:
I20260709 12:24:21.862733 18091 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:24:21.863122 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.863662 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1013) took 3029 ms. Trace:
I20260709 12:24:21.863808 24141 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:21.864114 18091 rpcz_store.cc:276] 0709 12:24:18.834270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.811082 (+2976812us) service_pool.cc:224] Handling call
0709 12:24:21.862438 (+ 51356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.863327 (+   889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.863595 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:21.862931 18090 rpcz_store.cc:276] 0709 12:24:18.864697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.834089 (+2969392us) service_pool.cc:224] Handling call
0709 12:24:21.841255 (+  7166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.862212 (+ 20957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.862476 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:21.862392 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:21.859333 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2850) took 2997 ms. Trace:
I20260709 12:24:21.866132 18093 rpcz_store.cc:276] 0709 12:24:18.861617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.823711 (+2962094us) service_pool.cc:224] Handling call
0709 12:24:21.840952 (+ 17241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.858751 (+ 17799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.858802 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:21.858879 (+    77us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:21.858935 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.859038 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.859236 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:21.868064 18172 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:21.861686 18098 rpcz_store.cc:276] 0709 12:24:18.833270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.793034 (+2959764us) service_pool.cc:224] Handling call
0709 12:24:21.823506 (+ 30472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.828733 (+  5227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.828770 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:21.834353 (+  5583us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:21.834419 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.834487 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.834633 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:21.828230 24574 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:21.870738 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.866878 18015 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:21.873040 18165 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:24:21.876434 18021 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:21.878695 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:21.876345 18812 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:21.885764 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.885171 24405 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.886636 18169 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:21.887912 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.809437 24463 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.891341 24463 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:21.872428 24343 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:21.918582 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:22.071241 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.050576 18098 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
W20260709 12:24:22.016361 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 264585us
I20260709 12:24:21.912593 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.900241 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.103801 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:21.904330 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:21.914855 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:22.063145 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:22.050951 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 54839.6us
I20260709 12:24:21.903829 18019 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:21.896711 18169 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:21.915565 18813 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:22.123116 18091 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:21.918202 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:21.914759 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2701) took 3438 ms. Trace:
I20260709 12:24:21.900030 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:21.926383 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.129789 18814 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:21.911643 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.986953 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:22.000170 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:22.132814 18165 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.104921 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.996582 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:22.058362 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.134981 18166 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:22.016853 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:22.017429 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:22.138773 18164 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:21.896220 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:22.141147 18092 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:24:22.106963 18098 pending_rounds.cc:85] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 10
W20260709 12:24:22.141271 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.142019 18098 pending_rounds.cc:107] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:21.915627 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:22.142485 18098 raft_consensus.cc:2889] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.133662 18809 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:22.147898 18808 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:24:22.125936 18168 rpcz_store.cc:276] 0709 12:24:18.476559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.292923 (+816364us) service_pool.cc:224] Handling call
0709 12:24:21.548716 (+2255793us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.549291 (+   575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.549327 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:21.549388 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:21.549476 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.549572 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.914370 (+364798us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b5400273dd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:21.914508 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5151415680}
I20260709 12:24:21.925388 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:22.151799 18166 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:21.918463 24560 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.937s	user 0.006s	sys 0.000s
I20260709 12:24:22.157510 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3651) took 3274 ms. Trace:
I20260709 12:24:22.158236 18091 rpcz_store.cc:276] 0709 12:24:18.867299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.865075 (+2997776us) service_pool.cc:224] Handling call
0709 12:24:22.109187 (+244112us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.128689 (+ 19502us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.141604 (+ 12915us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.154891 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2857) took 1606 ms. Trace:
I20260709 12:24:22.156041 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2865) took 1053 ms. Trace:
I20260709 12:24:22.052142 24411 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.172930 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3655) took 3304 ms. Trace:
I20260709 12:24:22.173703 18098 rpcz_store.cc:276] 0709 12:24:18.868485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.870085 (+3001600us) service_pool.cc:224] Handling call
0709 12:24:22.049989 (+179904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.172514 (+122525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.172833 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.174821 18166 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:24:22.175674 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.176252 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.176784 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.162403 18094 rpcz_store.cc:276] 0709 12:24:21.103038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.915297 (+812259us) service_pool.cc:224] Handling call
0709 12:24:22.155931 (+240634us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.163223 18093 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:21.897972 24538 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.178241 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:21.922812 24565 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:22.161208 18024 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.017614 24572 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.169633 18172 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.160871 18097 rpcz_store.cc:276] 0709 12:24:20.548904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.912092 (+1363188us) service_pool.cc:224] Handling call
0709 12:24:22.154778 (+242686us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.181725 18092 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:22.185922 18166 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:22.222674 18166 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:22.186501 24092 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:22.188423 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.190763 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:22.162577 24141 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:22.225075 18814 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:22.199149 18169 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:22.183796 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3654) took 3315 ms. Trace:
I20260709 12:24:22.227612 18093 rpcz_store.cc:276] 0709 12:24:18.868225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.868225 (+3000000us) service_pool.cc:224] Handling call
0709 12:24:22.095420 (+227195us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.177638 (+ 82218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.183708 (+  6070us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.200079 24444 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:22.231570 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3657) took 3352 ms. Trace:
I20260709 12:24:22.232172 18092 rpcz_store.cc:276] 0709 12:24:18.869068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.155812 (+3286744us) service_pool.cc:224] Handling call
0709 12:24:22.181477 (+ 25665us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.221328 (+ 39851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.221600 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.200253 18165 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:21.898972 24428 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:22.202517 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 337688us
W20260709 12:24:22.203685 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 278406us
I20260709 12:24:22.206795 18098 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:22.210889 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54074us
W20260709 12:24:22.214342 24610 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.171s	user 0.001s	sys 0.003s
W20260709 12:24:22.215698 24549 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 2.992s	user 0.008s	sys 0.007s
W20260709 12:24:22.216753 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2710) took 3182 ms (client timeout 2998 ms). Trace:
I20260709 12:24:21.904917 24483 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.313s
I20260709 12:24:22.218463 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3661) took 1032 ms. Trace:
I20260709 12:24:21.917598 24471 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.255053 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2855) took 2955 ms. Trace:
I20260709 12:24:22.255548 18099 rpcz_store.cc:276] 0709 12:24:21.186373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.376218 (+189845us) service_pool.cc:224] Handling call
0709 12:24:22.218152 (+841934us) spinlock_profiling.cc:243] Waited 90.3 ms on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:22.218350 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207575808}
I20260709 12:24:22.239125 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.161937 24562 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.230860 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:22.238546 18168 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:22.255750 18098 rpcz_store.cc:276] 0709 12:24:19.300127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.177212 (+2877085us) service_pool.cc:224] Handling call
0709 12:24:22.206213 (+ 29001us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.254534 (+ 48321us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.254946 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:22.242019 24512 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.011s	sys 0.004s
W20260709 12:24:22.264834 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.265138 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 365291us
I20260709 12:24:22.266309 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.266494 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:22.247579 18171 rpcz_store.cc:269] 0709 12:24:19.034619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.246281 (+211662us) service_pool.cc:224] Handling call
0709 12:24:22.216240 (+2969959us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:22.216495 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":416616704}
I20260709 12:24:22.268419 18096 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:22.279150 18093 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:22.282557 18091 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:24:22.284739 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2861) took 1694 ms. Trace:
I20260709 12:24:22.284956 24568 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:22.285470 18095 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:22.285746 24568 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:22.286551 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2854) took 2986 ms. Trace:
I20260709 12:24:22.286954 18095 rpcz_store.cc:276] 0709 12:24:19.299755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.171001 (+2871246us) service_pool.cc:224] Handling call
0709 12:24:22.260600 (+ 89599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.286196 (+ 25596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.286469 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.287025 24568 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.288869 18095 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260709 12:24:22.289328 18095 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:24:22.289580 18095 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:24:22.290030 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1015) took 3224 ms. Trace:
I20260709 12:24:22.290215 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:22.261135 24411 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:22.290776 24092 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:22.291145 18095 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.285305 18093 rpcz_store.cc:276] 0709 12:24:20.590525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.239197 (+1648672us) service_pool.cc:224] Handling call
0709 12:24:22.272442 (+ 33245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.284404 (+ 11962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.284662 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.291671 24092 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:22.291821 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.292193 24092 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.292585 24092 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:22.254145 24444 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.284775 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.300242 24444 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:22.302800 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.304852 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.305612 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:22.305955 24343 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.306622 24343 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:22.307088 24343 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.307440 24343 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.307898 18095 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:24:22.310825 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.311666 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3660) took 1127 ms. Trace:
I20260709 12:24:22.229640 24572 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.305619 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.313376 24338 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:22.313810 24338 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.314338 24338 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.314406 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:22.312114 18092 rpcz_store.cc:276] 0709 12:24:21.184526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.243292 (+1058766us) service_pool.cc:224] Handling call
0709 12:24:22.267290 (+ 23998us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.311224 (+ 43934us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.311260 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.311340 (+    80us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:22.311386 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.311462 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.311599 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.318192 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:22.320163 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:22.290550 18091 rpcz_store.cc:276] 0709 12:24:19.065560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.170665 (+3105105us) service_pool.cc:224] Handling call
0709 12:24:22.260066 (+ 89401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.283105 (+ 23039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.289951 (+  6846us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.323601 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:22.324685 18018 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:22.325016 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:22.326033 24610 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:24:22.328213 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:22.330180 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.328286 24471 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:22.331650 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.332376 24471 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:22.332597 24572 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:22.332880 24471 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.333384 24471 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.335988 18097 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:22.337237 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:22.337769 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64942.7us
I20260709 12:24:22.311611 18095 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:22.339114 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.346004 24411 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:22.347971 24411 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:22.336288 18020 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 10.
I20260709 12:24:22.351924 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.352946 18023 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:22.351924 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:22.355512 18020 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:22.355974 18173 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.527s	user 0.007s	sys 0.006s
I20260709 12:24:22.356451 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2856) took 3055 ms. Trace:
I20260709 12:24:22.356891 18097 rpcz_store.cc:276] 0709 12:24:19.301249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.232806 (+2931557us) service_pool.cc:224] Handling call
0709 12:24:22.272535 (+ 39729us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.356123 (+ 83588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.356388 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:22.359061 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75382.5us
I20260709 12:24:22.361311 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:22.363267 18020 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:22.356501 18173 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:22.365231 18023 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:22.369941 18018 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:22.370558 18090 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:22.369941 18164 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:22.366895 18021 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:24:22.379554 24482 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.660s	user 0.007s	sys 0.008s
I20260709 12:24:22.380131 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:22.356626 24411 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.382673 24411 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.379359 18096 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:22.381983 24555 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.364553 24428 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.401062 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:22.404213 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:22.409315 18020 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:24:22.414582 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:22.420032 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:22.423036 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:22.429754 18811 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:24:22.431397 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:22.455745 18815 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.422544 24281 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:22.467806 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43939.4us
I20260709 12:24:22.469295 18806 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:22.470534 24482 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:22.474751 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60640.8us
I20260709 12:24:22.450947 18811 pending_rounds.cc:85] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 9
I20260709 12:24:22.476693 18811 pending_rounds.cc:107] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:24:22.477207 18811 raft_consensus.cc:2889] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.416652 24405 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:24:22.478440 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:22.484570 18809 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.487391 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:22.395995 24555 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.494591 18018 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:24:22.496153 18024 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:22.499255 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.036s	sys 0.048s
W20260709 12:24:22.499742 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.037s	sys 0.049s
I20260709 12:24:22.471522 24281 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.529835 24281 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.533661 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:22.534747 18093 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.483306 24306 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:22.536700 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.517460 24444 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:22.565007 18813 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260709 12:24:22.566623 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75333.4us
W20260709 12:24:22.598069 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 223328us
I20260709 12:24:22.600036 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.601069 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.601976 24483 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:22.602149 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:22.602837 24483 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.603820 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:22.597971 24141 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:22.605093 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114380us
I20260709 12:24:22.606040 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:22.607976 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.610605 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.562881 24586 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:22.601263 24299 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:22.615093 24574 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:22.613087 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.604516 24482 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:22.622318 24565 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.621898 24482 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.622975 24565 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.618348 24306 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:22.626147 24574 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:22.626415 24306 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:22.626842 24574 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.627347 24306 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:22.627411 24574 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.609809 24483 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:22.631103 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.561293 18024 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:24:22.643229 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.653307 18022 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:22.611421 24405 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:22.654692 24405 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.655141 24405 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.659462 24092 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.653617 24555 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.607625 24444 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.666321 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2860) took 2082 ms. Trace:
I20260709 12:24:22.666842 18094 rpcz_store.cc:276] 0709 12:24:20.584160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.240689 (+1656529us) service_pool.cc:224] Handling call
0709 12:24:22.664845 (+424156us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:22.665248 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.665776 (+   528us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.665810 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.665935 (+   125us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:22.665998 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.666072 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.666217 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34432768}
W20260709 12:24:22.668826 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.639324 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43732.3us
I20260709 12:24:22.683585 18172 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:22.683967 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:22.684516 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2790) took 1079 ms. Trace:
I20260709 12:24:22.684900 18813 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 18
W20260709 12:24:22.686772 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43681.6us
W20260709 12:24:22.692643 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.697139 18809 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
W20260709 12:24:22.698513 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53272.5us
I20260709 12:24:22.702970 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3653) took 3815 ms. Trace:
I20260709 12:24:22.703653 18090 rpcz_store.cc:276] 0709 12:24:18.867958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.865429 (+2997471us) service_pool.cc:224] Handling call
0709 12:24:21.889208 (+ 23779us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.378152 (+488944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.682981 (+304829us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b540011d854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:22.683229 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1106452352}
I20260709 12:24:22.703862 18093 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:22.685051 18172 rpcz_store.cc:276] 0709 12:24:21.611609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.875733 (+264124us) service_pool.cc:224] Handling call
0709 12:24:22.684449 (+808716us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:22.712628 24463 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:22.713785 24482 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:22.714800 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.713958 24302 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:22.712635 24092 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:22.704612 24558 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 12:24:22.725152 24439 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:24:22.727070 24439 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:22.727721 24439 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:22.729637 24439 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.737644 24439 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:22.738751 24090 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:22.739212 24444 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:22.740281 24565 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:22.741497 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 137823us
I20260709 12:24:22.748669 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:22.750787 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3565) took 3821 ms (client timeout 1760 ms). Trace:
W20260709 12:24:22.751330 18022 rpcz_store.cc:269] 0709 12:24:18.924536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.938894 (+ 14358us) service_pool.cc:224] Handling call
0709 12:24:22.715602 (+3776708us) spinlock_profiling.cc:243] Waited 3.53 s on lock 0x7b540010aed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:22.746192 (+ 30590us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8122836864}
I20260709 12:24:22.755935 18780 leader_election.cc:400] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17.
I20260709 12:24:22.756484 18780 leader_election.cc:403] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:24:22.718943 24302 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:22.775787 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.706177 24572 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.045s
W20260709 12:24:22.775635 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60225us
W20260709 12:24:22.779678 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57894.1us
I20260709 12:24:22.715299 24338 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.781044 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:22.787082 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.791695 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.793110 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2873) took 1305 ms. Trace:
I20260709 12:24:22.801892 18815 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:22.794710 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 482830us
I20260709 12:24:22.808316 18138 leader_election.cc:400] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:22.809161 18138 leader_election.cc:403] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 12:24:22.790158 23886 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.043s	sys 0.066s
I20260709 12:24:22.785750 24306 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:22.838039 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 95493us
I20260709 12:24:22.838652 24411 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.839267 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.843245 24281 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:22.841336 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:22.815132 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 576093us
I20260709 12:24:22.841501 24411 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:22.820868 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.828943 24444 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.834890 24302 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:22.850636 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59105.4us
I20260709 12:24:22.804212 24463 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17.
W20260709 12:24:22.816169 18169 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.016s	sys 0.023s
I20260709 12:24:22.778908 24343 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:22.813007 23886 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.044s	sys 0.066s
I20260709 12:24:22.855530 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.795913 18099 rpcz_store.cc:276] 0709 12:24:21.487338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.268710 (+781372us) service_pool.cc:224] Handling call
0709 12:24:22.292878 (+ 24168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.293212 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:22.293918 (+   706us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.792477 (+498559us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.792575 (+    98us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:22.792664 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.792777 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.793007 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":188,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":5652,"thread_start_us":492214,"threads_started":1}
I20260709 12:24:22.899592 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:22.900797 18097 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:22.812750 24562 raft_consensus.cc:2764] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:24:22.855064 24306 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:22.883747 18169 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.016s	sys 0.023s
I20260709 12:24:22.850471 24621 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.889108 18813 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:22.876641 24572 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:22.904533 18019 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:22.877060 24444 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:22.912698 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.919032 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:22.921967 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.924698 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:22.923803 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:22.927822 18094 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:22.931315 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90075.1us
I20260709 12:24:22.933936 24501 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.918887 24338 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.938768 24338 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:22.941177 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:22.938930 24482 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:22.941321 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:22.944803 18090 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.945088 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2717) took 1184 ms. Trace:
W20260709 12:24:22.928378 24471 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.006s
I20260709 12:24:22.918574 24572 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:22.951627 18091 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:24:22.932818 24302 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:22.972584 18807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.015s	sys 0.007s
I20260709 12:24:22.943179 24338 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.940301 24281 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:22.977399 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:22.930088 23886 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:24:22.948573 18017 rpcz_store.cc:276] 0709 12:24:21.761010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.761191 (+   181us) service_pool.cc:224] Handling call
0709 12:24:21.781765 (+ 20574us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.802883 (+ 21118us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:21.839459 (+ 36576us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.943481 (+1104022us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.944608 (+  1127us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.944674 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:22.944741 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.944827 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.945010 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2079,"prepare.queue_time_us":19379,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":4919,"raft.queue_time_us":1841,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":13131,"thread_start_us":1042041,"threads_started":2,"wal-append.queue_time_us":732897}
I20260709 12:24:22.953536 24572 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:22.973939 24336 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.040s	sys 0.024s
I20260709 12:24:22.955168 24482 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:22.979558 24338 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.954973 24471 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:23.015533 24194 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.015s	sys 0.012s
W20260709 12:24:22.993492 24336 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.041s	sys 0.024s
W20260709 12:24:23.035996 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58755.3us
I20260709 12:24:22.993832 24411 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.062175 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:23.034536 24194 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.015s	sys 0.013s
I20260709 12:24:23.063032 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:23.069878 24618 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.070639 24618 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.071058 24618 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.075485 24471 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.076433 24471 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:23.077198 24336 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:23.085947 24388 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:24:23.086081 24607 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.358s	user 0.011s	sys 0.021s
I20260709 12:24:23.074481 24482 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:23.095283 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 494292us
I20260709 12:24:23.097111 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.100832 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:23.106499 18090 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:23.103253 24607 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.429s	user 0.012s	sys 0.021s
W20260709 12:24:23.112494 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50514.4us
I20260709 12:24:23.106092 24512 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:23.106593 24439 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.103991 24562 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:23.136520 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.139210 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.179741 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79010.8us
I20260709 12:24:23.183290 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:23.199647 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.201856 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 421066us
I20260709 12:24:23.207288 24634 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:23.203687 23886 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:23.206387 24632 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:23.175118 24439 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:23.212606 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 256035us
I20260709 12:24:23.213748 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:23.214311 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 361076us
I20260709 12:24:23.175891 24562 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:23.220718 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.126248 24405 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:23.223771 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 279555us
I20260709 12:24:23.188556 24512 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:23.228897 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3602) took 1027 ms. Trace:
W20260709 12:24:23.229796 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 376309us
I20260709 12:24:23.209620 24538 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:23.209408 24141 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:23.229264 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.232139 18024 rpcz_store.cc:276] 0709 12:24:22.201259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.641940 (+440681us) service_pool.cc:224] Handling call
0709 12:24:22.710102 (+ 68162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.710451 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:23.153195 (+442744us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.227466 (+ 74271us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:23.228237 (+   771us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:23.228368 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.228500 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.228769 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":364068,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":515796,"threads_started":2}
I20260709 12:24:23.232136 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:23.225376 24562 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.220613 24439 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.222942 24092 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:23.243289 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.227761 24624 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.248369 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3601) took 1047 ms. Trace:
I20260709 12:24:23.250084 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:23.203092 24631 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:24:23.222054 24634 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 12:24:23.221689 23886 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.255407 24565 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:24:23.255674 24623 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:24:23.262804 24512 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.267246 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:23.267693 24501 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:23.268013 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.314440 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 85297.3us
I20260709 12:24:23.235069 24538 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:23.316450 24538 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.254666 24141 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.249961 24194 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.249608 24624 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:23.252964 24092 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.346988 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.277099 24631 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:23.277776 24634 consensus_queue.cc:1243] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 12:24:23.285992 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62463.4us
W20260709 12:24:23.288172 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.251399 18016 rpcz_store.cc:276] 0709 12:24:22.200821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.501650 (+300829us) service_pool.cc:224] Handling call
0709 12:24:22.686884 (+185234us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:22.687119 (+   235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.687458 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:23.016180 (+328722us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.245804 (+229624us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:23.246182 (+   378us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:23.247233 (+  1051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.248129 (+   896us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.248292 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":386828,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":17064,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"spinlock_wait_cycles":2932992,"thread_start_us":524961,"threads_started":3,"wal-append.queue_time_us":216987}
I20260709 12:24:23.310693 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2719) took 1621 ms. Trace:
W20260709 12:24:23.319940 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.394281 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70702.6us
W20260709 12:24:23.323203 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.319149 24647 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:23.330823 24444 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.407752 18024 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:23.332217 24388 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:23.336102 23886 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.382277 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:23.425261 18019 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.392042 24299 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:23.401737 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.405246 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.434041 18090 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.406912 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:23.414089 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3592) took 1539 ms. Trace:
I20260709 12:24:23.414498 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:23.416360 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:23.385340 24631 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:23.420692 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.381202 24624 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.408957 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:23.403206 24501 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:23.393450 18169 rpcz_store.cc:276] 0709 12:24:21.664033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.231054 (+567021us) service_pool.cc:224] Handling call
0709 12:24:22.263469 (+ 32415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.930534 (+667065us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.930574 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.982282 (+ 51708us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.222054 (+239772us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:23.222105 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.222190 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.285276 (+ 63086us) raft_consensus.cc:1685] finished
0709 12:24:23.285306 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.285513 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2074,"raft.queue_time_us":30090,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":147,"thread_start_us":813104,"threads_started":2,"wal-append.queue_time_us":257276}
I20260709 12:24:23.453856 18020 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:23.346526 24141 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.450064 18017 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:23.450583 18023 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.424435 24388 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:23.430531 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 180560us
I20260709 12:24:23.337929 24618 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.452260 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:23.483593 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88466.3us
I20260709 12:24:23.451227 18015 rpcz_store.cc:276] 0709 12:24:21.874543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.882841 (+  8298us) service_pool.cc:224] Handling call
0709 12:24:21.894667 (+ 11826us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.895451 (+   784us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.895501 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:22.319952 (+424451us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:22.330986 (+ 11034us) log.cc:844] Serialized 52 byte log entry
0709 12:24:22.807132 (+476146us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:23.161991 (+354859us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.163852 (+  1861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.163977 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.413588 (+249611us) raft_consensus.cc:1685] finished
0709 12:24:23.413712 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.413996 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":267638,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":3628,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":13034,"spinlock_wait_cycles":71214464,"thread_start_us":785627,"threads_started":2,"wal-append.queue_time_us":698889}
I20260709 12:24:23.421358 24411 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:23.482342 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.491709 24618 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:23.493994 24654 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.494236 24636 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:23.494630 17989 leader_election.cc:336] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.901s (SENT)
I20260709 12:24:23.494732 24654 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.495337 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:23.496692 24652 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.483722 18016 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.497385 24654 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.380877 24194 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:23.501437 24302 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.000s
I20260709 12:24:23.460543 24463 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.512946 24463 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.514374 18815 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:23.463492 24299 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:23.430102 24538 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.501906 24090 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.516183 24090 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:23.515869 24299 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.516656 24090 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.502046 24281 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:23.520794 18016 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:23.502351 24194 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.503235 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:23.503520 24555 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:23.522353 24555 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:23.522456 18807 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:23.523499 24555 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.494937 24636 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.504446 24343 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.508809 24338 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.510648 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:23.477306 18780 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.508s (SENT)
W20260709 12:24:23.486574 24621 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.572s	user 0.016s	sys 0.000s
I20260709 12:24:23.502964 24631 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:23.525787 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 231040us
I20260709 12:24:23.526294 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:23.459244 24405 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:23.528081 24574 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.900s	user 0.017s	sys 0.003s
I20260709 12:24:23.537333 18020 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:24:23.541558 18091 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:23.458623 24501 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:23.515875 24463 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.528618 24555 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:23.515923 24620 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:23.531284 24565 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:23.577556 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.531117 24281 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:23.581440 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 226849us
I20260709 12:24:23.541138 24621 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:23.583554 18090 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:23.581622 24631 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:23.593211 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.594348 18172 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:23.570139 24501 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:23.568892 24618 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:23.597121 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:23.605404 18018 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:24:23.595844 17989 leader_election.cc:336] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.009s (SENT)
W20260709 12:24:23.598295 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 352531us
I20260709 12:24:23.589944 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.546464 24643 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.580870 24306 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:23.612797 24306 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:23.614428 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 152919us
I20260709 12:24:23.614580 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.600739 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:23.615423 18780 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.553s (SENT)
I20260709 12:24:23.602629 18098 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:23.582978 24620 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:23.615993 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:23.617255 24620 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.618130 24463 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:23.619040 24620 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.625149 18170 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:24:23.624912 24562 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.627452 24338 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:23.630285 24572 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.612s	user 0.013s	sys 0.001s
I20260709 12:24:23.569252 24647 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:24:23.638986 24439 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.642189 24657 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:23.607923 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:23.609330 24644 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:24:23.645841 18015 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:23.627514 18172 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:24:23.653574 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:23.617676 18780 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.634s (SENT)
I20260709 12:24:23.655941 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.658033 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:23.658393 18017 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:23.659797 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.604779 24618 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.571898 24574 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.666363 18164 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:23.670864 18097 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:23.677152 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.630874 24572 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.593988 24565 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.685349 24512 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:23.673840 24338 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:24:23.668339 24618 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:23.687494 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.688961 24388 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.689105 24281 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.660483 24621 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:24:23.691929 24306 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:23.699098 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47666.8us
W20260709 12:24:23.687752 24630 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.484s	user 0.004s	sys 0.005s
I20260709 12:24:23.710820 18815 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:23.705941 18164 raft_consensus.cc:2376] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17. Re-sending same reply.
I20260709 12:24:23.709420 18023 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:23.714083 18024 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 12:24:23.715638 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.718545 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.719378 18172 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:23.721689 24565 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.722348 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:23.722718 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105089us
I20260709 12:24:23.723258 18164 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.724485 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.725346 18173 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:23.726675 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.727605 18167 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.729185 18168 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:23.730499 24306 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:23.731468 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62899.9us
I20260709 12:24:23.734153 24471 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.734699 24657 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:23.734429 24658 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:24:23.735270 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.735456 24338 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:23.735857 24338 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:23.737141 24439 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:23.736781 24338 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.738862 24338 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:23.734138 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53688.8us
I20260709 12:24:23.740332 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.742180 24663 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:23.743119 24663 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:23.744401 24562 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.747361 24572 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.747869 24483 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:23.748520 24483 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:23.683223 24463 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.735152 24657 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:23.724735 18780 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.896s (SENT)
I20260709 12:24:23.698120 24538 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.701061 24512 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.765909 18168 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:23.765684 18807 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:23.744439 24663 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.781474 24636 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.754894 24483 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.780043 24574 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:23.772706 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.783023 24388 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.785044 24624 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.770110 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.787405 24620 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:23.788679 24624 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:23.763759 24562 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
W20260709 12:24:23.773059 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48383.5us
I20260709 12:24:23.753350 24657 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:23.799226 18780 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.650s (SENT)
I20260709 12:24:23.799968 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:23.783578 24090 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.763300 24538 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:23.766471 24512 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.807384 18168 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:23.817611 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.827616 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:23.828711 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.805152 24538 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.832185 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.833596 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.834712 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.835614 18093 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.814488 24636 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:23.837707 18099 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.844244 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.845736 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.846935 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.847370 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.847723 18099 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.848035 18097 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.849144 18099 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:23.843016 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.850276 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.851063 18092 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.838208 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.852914 18173 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:23.833662 18168 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:23.854210 24620 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.855070 24620 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.857479 24620 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:23.858265 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.859385 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.859768 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.860023 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.860780 18168 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:23.861756 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.862243 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.862346 24620 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.862497 18168 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:23.842521 18164 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:23.854350 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60295.9us
I20260709 12:24:23.864647 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:23.865957 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67656.6us
I20260709 12:24:23.866932 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:23.867312 24620 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:23.867820 24090 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:23.868759 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:23.870124 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:23.857501 24618 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.871809 24483 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:23.872864 24618 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:23.874703 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2874) took 2387 ms. Trace:
I20260709 12:24:23.875175 18098 rpcz_store.cc:276] 0709 12:24:21.487689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.264720 (+777031us) service_pool.cc:224] Handling call
0709 12:24:22.656063 (+391343us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:22.656264 (+   201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:23.616246 (+959982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.874431 (+258185us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:23.874637 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2200909952}
I20260709 12:24:23.876998 24302 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:23.878051 24302 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:23.878474 24558 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:23.881390 24388 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:23.882302 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:23.882786 24463 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:23.882925 24662 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:23.883268 18171 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:24:23.883208 24463 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:23.883497 24662 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:23.884510 24463 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.886212 24463 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.884533 24662 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:23.888134 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 298317us
I20260709 12:24:23.888552 24632 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:23.889178 24632 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:23.890007 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 294191us
W20260709 12:24:23.890327 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 174774us
I20260709 12:24:23.844244 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.870134 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.895946 18090 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.897037 18091 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:23.861414 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.898979 18166 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:23.818629 24624 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:23.916531 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:23.910260 18092 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:23.916069 18167 raft_consensus.cc:2368] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:24:23.922802 18172 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:23.927410 24654 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.928463 24654 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:23.931934 24654 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.918824 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:23.937623 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:23.939782 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:23.940384 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:23.946725 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.948678 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:23.857640 24141 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:23.985073 18096 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:23.907536 24657 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:23.907678 24444 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:23.984979 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:23.948733 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:23.987417 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 95627.3us
I20260709 12:24:24.001874 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:24.002725 18021 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.916719 24636 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:23.924125 24662 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:24.009545 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:24.002530 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54831.2us
I20260709 12:24:24.004658 18808 pending_rounds.cc:85] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 13
I20260709 12:24:24.011166 18808 pending_rounds.cc:107] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:24:24.011564 18808 raft_consensus.cc:2889] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.993928 18091 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: [16.8-16.8]
W20260709 12:24:24.008069 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47318.7us
I20260709 12:24:23.922765 24555 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:23.917963 24336 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 12:24:24.013203 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:24.010439 18020 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:23.948858 24306 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:24.016187 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.017349 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:23.992172 24618 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:24.030165 24306 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.030576 24618 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:24.030820 24306 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.031827 24618 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:23.992575 24090 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:23.985123 24659 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.022928 18815 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:24.021099 18807 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.017870 24338 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:24.016177 24555 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:24.076411 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53801.6us
I20260709 12:24:24.082854 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:23.933211 24624 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:24.085695 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:24.090821 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:24.091614 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.041692 24090 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:24.084935 24336 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.101197 18168 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:24.101228 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.096060 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.096899 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.103394 18170 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.085623 24555 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:24.081840 24663 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.118580 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:24.087718 24338 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:24.125313 24671 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:23.984222 24632 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.100270 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.102736 24090 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.151407 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.154094 18172 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:24.019526 24636 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.157881 18164 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:24.158375 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.189935 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:24.040663 24501 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.162536 24338 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:24.164372 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45943us
I20260709 12:24:24.157272 24663 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.170703 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:24.224392 18098 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:24.173786 24306 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.235255 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:24.181322 18807 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:24.182981 24092 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.190975 18167 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17. Re-sending same reply.
I20260709 12:24:24.238372 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.191679 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.214850 18815 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:24.193675 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:24.216830 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.242828 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.244135 18164 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:24.246547 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:24.247182 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.250160 18169 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:24.254402 18091 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:24.257402 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.257897 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.258226 18172 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:24.258682 18168 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.130646 24336 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.261011 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.263403 18164 raft_consensus.cc:2368] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:24:24.270582 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:24.273061 18171 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.276474 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:24.279258 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:24.280388 18807 raft_consensus.cc:2410] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:24:24.285969 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:24.287145 18815 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:24.302599 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.306775 18167 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:24.314405 24644 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.315112 24644 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.320466 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:24.321437 24655 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:24.321673 24674 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.203s	user 0.000s	sys 0.004s
I20260709 12:24:24.321921 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:24.322389 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:24.323915 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.319942 24644 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.322872 24565 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:24.326610 17989 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.073s (SENT)
I20260709 12:24:24.343175 24565 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.340693 18022 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:24.344919 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:24.321470 18015 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:24.322677 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.322432 24299 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:24.346722 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:24.347993 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.331631 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:24.322074 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 337193us
I20260709 12:24:24.337751 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:24.322131 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.346843 24299 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.362504 18022 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:24.337872 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:24.371868 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:24.377362 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:24.378216 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:24.379623 18170 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:24.388365 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.396392 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.400209 18024 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:24.400241 18169 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:24.401748 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56943.8us
I20260709 12:24:24.405906 24141 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.406857 24302 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:24.407334 24302 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:24.382216 24644 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.408334 24302 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:24.410526 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 175387us
I20260709 12:24:24.394925 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:24.415005 18811 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
W20260709 12:24:24.416898 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.406564 24141 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.332777 24621 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:24.422729 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:24.425418 18017 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:24.419289 18021 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:24.358026 24653 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:24.365748 24299 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:24.414143 18168 raft_consensus.cc:2376] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16. Re-sending same reply.
I20260709 12:24:24.431780 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.382511 24194 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.446390 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.447069 18092 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:24:24.447510 18092 pending_rounds.cc:85] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:24:24.447810 18092 pending_rounds.cc:107] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:24:24.447726 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.448333 18092 raft_consensus.cc:2889] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:24.450412 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:24.349634 24565 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:24.357755 24680 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.444566 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.422942 24620 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.457247 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:24.489553 18095 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 12:24:24.491434 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.458146 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:24.397490 24624 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.396697 24306 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.395313 24663 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.448555 18168 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:24.444883 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:24.371829 24655 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.423962 24302 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:24.424286 24659 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:24.513911 24501 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:24.516938 24512 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.692s	user 0.011s	sys 0.000s
I20260709 12:24:24.523290 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.527447 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:24.561887 18813 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
W20260709 12:24:24.528264 24538 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.644s	user 0.012s	sys 0.000s
I20260709 12:24:24.374658 24586 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 12:24:24.529122 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:24.530987 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:24.531244 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:24.534658 24679 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:24.536592 24684 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:24.538084 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:24.634553 18164 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 13: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.538268 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.538790 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:24.635759 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 258295us
I20260709 12:24:24.425040 24621 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:24.488284 24565 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:24.558801 18015 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:24.556393 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:24.563367 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:24.642323 18809 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:24.635385 18168 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:24.564147 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:24.560897 18167 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:24.509615 24141 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:24.583645 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 181255us
I20260709 12:24:24.585919 18020 raft_consensus.cc:2376] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19. Re-sending same reply.
I20260709 12:24:24.542802 24659 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.617983 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:24.656764 18813 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:24.588070 18807 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:24.555902 24624 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.506426 24558 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.081s
I20260709 12:24:24.628750 18814 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:24.689096 18170 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:24.661764 24555 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.001s
I20260709 12:24:24.639653 24636 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:24.665194 24483 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.793s	user 0.009s	sys 0.005s
I20260709 12:24:24.589205 24687 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.098s
I20260709 12:24:24.567516 24512 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.665235 24655 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:24.661764 24657 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.664s	user 0.008s	sys 0.005s
W20260709 12:24:24.666569 24685 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 12:24:24.668342 24336 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.670746 18812 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:24.670990 24568 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.671468 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:24.617394 24092 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.673720 24090 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.653956 24679 consensus_queue.cc:1243] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 14.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:24.678210 24654 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:24.677969 24621 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:24.679431 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 123163us
W20260709 12:24:24.680970 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.686128 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:24.679488 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:24.687813 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.708130 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:24.636940 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.586643 24538 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:24.713308 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 300408us
I20260709 12:24:24.785967 18811 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:24.693257 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.706434 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:24.789973 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:24.790727 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.710368 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:24.793510 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 105639us
I20260709 12:24:24.655185 24562 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:24.818389 18173 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.818197 18165 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:24.688297 24659 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:24.789780 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.789240 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:24.800496 18171 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.824671 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:24.839910 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50749.6us
I20260709 12:24:24.692476 24684 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:24.820801 18806 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:24.794792 18170 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:24.719259 24558 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:24.789443 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 152637us
I20260709 12:24:24.797235 24439 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:24:24.840853 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.799587 24306 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:24.846767 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.799842 24680 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:24.855531 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.859886 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:24.785637 24654 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:24.852843 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:24.860769 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:24.856112 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 405777us
I20260709 12:24:24.851912 24695 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
W20260709 12:24:24.857791 24688 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:24:24.858822 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3690) took 1017 ms. Trace:
I20260709 12:24:24.827417 18091 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:24.711547 24555 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:24.851774 24463 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
W20260709 12:24:24.846412 24668 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:24.860260 18092 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:24.845921 24634 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:24.845321 24636 raft_consensus.cc:2764] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:24.868240 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.798099 24568 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.826191 24621 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:24.869326 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:24.882968 18015 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.822144 24653 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:24.861804 18093 rpcz_store.cc:276] 0709 12:24:23.841526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.850802 (+  9276us) service_pool.cc:224] Handling call
0709 12:24:23.851198 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:23.851712 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:23.943574 (+ 91862us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.222377 (+278803us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.223322 (+   945us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.223531 (+   209us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:24.223930 (+   399us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.339819 (+115889us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b18003a5be0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:24.398082 (+ 58263us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b18003a5be0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:24.646069 (+247987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.647628 (+  1559us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.858560 (+210932us) raft_consensus.cc:1685] finished
0709 12:24:24.858587 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.858749 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86076,"prepare.queue_time_us":74054,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":109,"raft.queue_time_us":189216,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":352,"spinlock_wait_cycles":51770112,"thread_start_us":535763,"threads_started":3,"wal-append.queue_time_us":339137}
I20260709 12:24:24.874368 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.890232 18021 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:24.844158 24684 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:24.879573 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.866354 24336 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:24.855232 24306 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.873083 24634 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.855016 24586 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:24.898264 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.884905 18780 leader_election.cc:400] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
I20260709 12:24:24.858075 24411 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 12:24:24.800288 24092 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.912223 24092 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:24.887111 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:24.913151 18097 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:24.865166 24680 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.914376 24680 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.882119 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:24.918643 24538 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:24.918984 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.845746 24141 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:24:24.896330 18098 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:24.919520 24634 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:24.921409 24586 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.921977 24586 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.801357 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:24.924455 24568 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:24.901571 24306 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.926744 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:24.927531 18021 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:24.928454 24194 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.843763 24558 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:24.822592 24693 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:24:24.894711 18015 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:24.891736 18019 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:24.942214 18168 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:24.942404 24336 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:24.943521 24336 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:24.878105 24657 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.880393 24655 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:24.892601 18096 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:24.891532 24684 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:24.900218 24699 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:24.948289 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 954) took 1089 ms. Trace:
I20260709 12:24:24.933434 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.948771 18165 rpcz_store.cc:276] 0709 12:24:23.856441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.861950 (+  5509us) service_pool.cc:224] Handling call
0709 12:24:24.945766 (+1083816us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:24.945905 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196510464}
I20260709 12:24:24.903278 18780 leader_election.cc:403] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:24.940472 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.544s (SENT)
I20260709 12:24:24.965118 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:24.947654 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43928.5us
W20260709 12:24:24.965206 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80467.6us
I20260709 12:24:24.974416 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:24.940058 24663 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:25.018427 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44125.3us
I20260709 12:24:24.933106 24512 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:24.945947 24697 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:25.040598 24697 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.043736 24697 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:25.030865 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 90443us
W20260709 12:24:25.045127 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.554s (SENT)
I20260709 12:24:24.930011 24565 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.108s
I20260709 12:24:25.027407 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.029435 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:25.015195 24688 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:24:25.007085 24644 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:25.055437 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:24.950155 24655 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:25.036128 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:25.066061 18021 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.960286 24699 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:25.067822 24708 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:25.069200 24708 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:25.087390 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58093us
I20260709 12:24:25.088675 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.093978 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:24.949122 24691 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.174s
I20260709 12:24:25.014945 24636 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18.
W20260709 12:24:25.100895 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 446839us
I20260709 12:24:25.033601 24439 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:24.978770 24657 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.031059 24692 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.025004 24663 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.107765 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:25.114058 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68977.5us
I20260709 12:24:25.070456 24699 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.111505 24463 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:25.082007 24655 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:25.014716 24141 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 12:24:25.107722 18016 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:25.131247 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:25.099160 24501 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:25.148963 18096 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:25.079607 24644 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.107406 24618 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:25.148612 24092 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:24:25.152810 24692 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.181352 24696 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:25.182261 24696 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:25.183529 24696 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.058681 24483 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.090s
W20260709 12:24:25.168427 24689 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260709 12:24:25.121233 24680 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.168066 24644 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:25.191732 24680 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:25.121752 24299 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.642s	user 0.013s	sys 0.000s
I20260709 12:24:24.939260 24654 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.043856 24306 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.148550 24663 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:25.150715 24463 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:25.208937 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 193722us
W20260709 12:24:25.156448 24620 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.649s	user 0.010s	sys 0.005s
I20260709 12:24:25.227622 18021 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:25.229290 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.154503 24090 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:25.163041 24632 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:25.223572 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.777s (SENT)
I20260709 12:24:25.210427 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:25.229902 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:25.230558 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.228562 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:25.196231 24439 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.169579 24618 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:25.248809 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:25.258989 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:25.203229 24471 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:25.263952 18172 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:25.268463 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174578us
I20260709 12:24:25.269469 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:25.274418 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.266345 18137 leader_election.cc:400] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:25.228730 24092 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:25.282290 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2735) took 1199 ms. Trace:
I20260709 12:24:25.229241 24555 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:25.246891 24632 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:25.115473 24653 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.263994 24703 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.281283 24538 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:25.261387 24618 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:25.206014 24299 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.250298 24692 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:25.282743 18166 rpcz_store.cc:276] 0709 12:24:24.082710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.087826 (+  5116us) service_pool.cc:224] Handling call
0709 12:24:24.088227 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.090750 (+  2523us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:24.118910 (+ 28160us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.171337 (+ 52427us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b180034e340. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9
0709 12:24:24.314699 (+143362us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b180034e340. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:24:24.325773 (+ 11074us) spinlock_profiling.cc:243] Waited 6.62 ms on lock 0x7b180034e340. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:24.605448 (+279675us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.606442 (+   994us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.626965 (+ 20523us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.627266 (+   301us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:24.627617 (+   351us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.627952 (+   335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.628072 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.674763 (+ 46691us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b180034e340. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:25.282013 (+607250us) raft_consensus.cc:1685] finished
0709 12:24:25.282047 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.282221 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3822,"prepare.queue_time_us":81691,"prepare.run_cpu_time_us":212,"prepare.run_wall_time_us":6777,"raft.queue_time_us":18121,"raft.run_cpu_time_us":717,"raft.run_wall_time_us":47376,"spinlock_wait_cycles":82199680,"thread_start_us":288994,"threads_started":1,"wal-append.queue_time_us":588486}
I20260709 12:24:25.247423 24620 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:25.278333 24691 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:25.321048 18167 raft_consensus.cc:606] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:25.321558 18167 raft_consensus.cc:614] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:25.288035 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.322618 18169 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:25.277627 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:25.310806 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:25.327335 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 103830us
W20260709 12:24:25.312815 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 423881us
W20260709 12:24:25.334570 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46637.5us
I20260709 12:24:25.334482 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:25.335870 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.285867 24092 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:25.289928 24483 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.289716 24444 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:25.296990 24680 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:25.347579 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.271306 24636 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:24:25.297952 24471 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:25.310716 24194 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.342308 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:25.337267 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59727.5us
I20260709 12:24:25.367383 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:25.313174 24501 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:25.358180 18020 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:25.382417 24194 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.385185 24194 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:25.370862 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:25.389372 18166 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:25.371348 24679 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:25.390908 24306 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:25.396073 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2760) took 1280 ms. Trace:
I20260709 12:24:25.396749 18169 rpcz_store.cc:276] 0709 12:24:24.115573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.401329 (+285756us) service_pool.cc:224] Handling call
0709 12:24:25.395765 (+994436us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:25.395963 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2016056704}
I20260709 12:24:25.391234 24679 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:25.405619 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63361.8us
I20260709 12:24:25.378100 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:25.380930 24501 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.409992 18808 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.409204 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:25.353472 24538 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:25.348527 24555 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:25.346397 24712 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:25.428640 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:25.435880 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:25.360253 24693 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:25.438882 18814 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:25.440850 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 309657us
I20260709 12:24:25.442265 18165 raft_consensus.cc:1217] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Deduplicated request from leader. Original: 11.12->[12.13-16.15]   Dedup: 12.13->[15.14-16.15]
I20260709 12:24:25.449275 18165 pending_rounds.cc:85] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 13
I20260709 12:24:25.449533 18165 pending_rounds.cc:107] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 14.14
I20260709 12:24:25.449836 18165 raft_consensus.cc:2889] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:25.443602 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:25.446472 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 311220us
I20260709 12:24:25.450695 18016 raft_consensus.cc:2368] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:24:25.379885 24620 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:25.314144 24463 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.409627 24636 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:25.312573 24703 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.394738 24657 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:25.353094 24483 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.460419 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:25.459672 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.452445 24471 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.437247 24299 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:25.469257 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60221.1us
I20260709 12:24:25.429584 24715 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.470358 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:25.470726 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.471405 18092 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:25.370824 24710 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.164s
I20260709 12:24:25.475069 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:25.475337 18091 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:25.476500 24703 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:25.480660 24690 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.693s	user 0.002s	sys 0.002s
I20260709 12:24:25.480403 24655 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.349222 24632 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.482349 24632 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.482203 24679 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.484163 24679 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.460903 24306 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:24:25.489208 24710 consensus_queue.cc:1243] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:25.466565 24657 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
W20260709 12:24:25.489576 24659 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.011s	sys 0.002s
I20260709 12:24:25.491961 24659 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.494411 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.495193 18098 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:25.496100 24586 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.574s	user 0.010s	sys 0.005s
I20260709 12:24:25.496755 24586 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:25.491443 24662 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.475s	user 0.017s	sys 0.000s
I20260709 12:24:25.456812 24693 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:25.499922 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.506256 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.511003 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.464736 24555 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:25.498436 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.623s (SENT) [suppressed 47 similar messages]
I20260709 12:24:25.370573 24092 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.522934 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.529552 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:25.530424 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:25.524302 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.473647 24463 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:25.534967 18019 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:25.558405 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.568354 18091 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 20. Ops: []
I20260709 12:24:25.570477 18090 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:25.572268 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.573287 18090 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.577261 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.505303 24680 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:25.577327 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.498298 24720 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:25.520571 24636 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:25.517599 24555 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.474211 24471 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:25.600885 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97839.7us
W20260709 12:24:25.601419 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.529074 24624 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:25.531630 24687 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:25.613930 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.618221 24687 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.619243 24712 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:25.603386 24723 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:25.622231 24624 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:25.604846 24721 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:24:25.623436 24624 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.602962 24714 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:25.625349 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.600400 24306 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.626153 24306 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.603087 24338 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.627296 24655 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:25.627740 24338 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.603821 24680 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:25.600451 24636 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:25.629913 24680 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.630136 24636 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.624949 24714 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.516368 24618 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.596926 24090 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:25.601431 24621 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.701s	user 0.014s	sys 0.001s
I20260709 12:24:25.604089 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.656667 18091 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:25.657176 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.658186 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:25.672837 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 422496us
W20260709 12:24:25.678048 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102954us
I20260709 12:24:25.604995 24555 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:25.681871 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69194.7us
I20260709 12:24:25.693005 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.696305 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.696444 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:25.697846 18093 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:24:25.714754 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 216329us
W20260709 12:24:25.717406 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.936s (SENT)
I20260709 12:24:25.722163 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:25.731320 18092 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:25.735948 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.656059 24720 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:25.757851 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.753340 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.760201 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:25.729206 24644 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.519s	user 0.018s	sys 0.005s
I20260709 12:24:25.677657 24306 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:25.762245 18091 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:25.766405 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73516.1us
I20260709 12:24:25.755728 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.768337 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.768710 24722 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.769165 18092 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:25.769472 24338 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:25.769925 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:25.770208 24338 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.771008 24338 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.778530 24538 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:25.779289 24471 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:25.780670 24726 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:25.781219 24726 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.781587 24726 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:25.782940 24721 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:24:25.764232 24708 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.785645 24620 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:25.768519 24725 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.786306 24620 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.786998 24620 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:25.685803 24090 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:25.768414 24709 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.088s	user 0.002s	sys 0.000s
I20260709 12:24:25.770731 18024 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:25.766594 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.799674 18093 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:25.757078 24655 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:25.809093 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.809897 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:25.763273 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 257090us
I20260709 12:24:25.823282 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:25.821635 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.765722 18090 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:25.812403 18814 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:25.810782 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52105.3us
I20260709 12:24:25.835530 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.843402 18808 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:24:25.844967 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127699us
I20260709 12:24:25.846921 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.847754 18093 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:25.848898 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.849597 18093 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:25.852728 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.854245 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.857087 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:25.858034 18021 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:25.858963 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.855957 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.865581 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.861383 18091 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:25.853487 24730 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:24:25.887354 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.890071 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:25.833621 24586 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:25.869136 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.876658 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:25.876868 24338 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:25.723021 24693 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.17, Last appended by leader: 17, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.786638 24725 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.785434 24708 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.882444 24718 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:25.883046 24724 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:25.833247 24483 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:25.887403 18090 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:25.946720 18090 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:25.893791 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:25.899588 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:25.905670 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:25.765985 24720 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:25.915215 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61097.6us
I20260709 12:24:25.951062 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.840402 24655 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:25.919422 18093 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:25.954713 18093 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:25.822265 24644 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:25.922492 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:25.860046 24565 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:25.769322 24722 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:25.916957 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:25.937803 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.202s (SENT)
W20260709 12:24:25.914100 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.930547 24444 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:25.991518 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:25.887518 24687 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:25.958282 18095 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 12:24:25.982409 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76922.9us
I20260709 12:24:25.915529 24501 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:25.941963 24711 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:24:25.999151 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:25.985970 18091 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:25.994594 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:25.999575 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.002375 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.002722 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 115520us
W20260709 12:24:26.007994 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91191.1us
I20260709 12:24:26.013365 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:26.008937 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:26.030444 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.008045 18093 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.027273 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:26.036164 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.016599 24657 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:26.042443 24657 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.042052 24555 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.042899 24657 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:26.016750 24501 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.046926 24725 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:26.047294 24501 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:26.047830 24725 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:26.048446 24725 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:26.016017 24696 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:25.987339 24483 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:26.051090 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47398.2us
I20260709 12:24:26.099702 18093 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:26.055693 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.994354 24338 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:26.056384 18091 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:26.061309 18098 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.014s
I20260709 12:24:26.102878 18098 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:26.103958 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.104739 18098 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:25.999106 24565 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:26.015166 24684 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.001s	user 0.008s	sys 0.020s
W20260709 12:24:26.029685 24634 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.109s	user 0.014s	sys 0.011s
I20260709 12:24:26.099730 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:26.115373 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 177634us
I20260709 12:24:26.102185 18091 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:26.004727 24692 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:26.119704 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 284266us
I20260709 12:24:26.116499 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.016515 24659 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:26.186849 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159707us
W20260709 12:24:26.187516 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64959.8us
I20260709 12:24:26.190400 24750 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:26.197386 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 96927.9us
I20260709 12:24:26.202092 18098 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:26.199245 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:26.184217 18091 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:26.273128 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73313.1us
I20260709 12:24:26.103696 24586 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:26.273093 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 173479us
I20260709 12:24:26.210498 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:26.280588 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.219024 18780 leader_election.cc:400] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:24:26.281199 18780 leader_election.cc:403] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:26.281777 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63004.7us
I20260709 12:24:26.104142 24338 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.283916 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:26.284648 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:26.285645 18815 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 18 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 16.
I20260709 12:24:26.098450 24696 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.098912 24444 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:26.287812 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:26.290112 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.288326 18091 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:26.326668 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.327592 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:26.184238 24090 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:26.053181 24734 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:26.119660 24565 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:26.321220 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 122116us
I20260709 12:24:26.323556 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:26.272625 24663 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.964s	user 0.017s	sys 0.009s
I20260709 12:24:26.324106 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.049548 24725 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.207751 24299 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.205000 24718 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:26.343786 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:26.354598 18093 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:24:26.355396 18806 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:26.358383 24463 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.782s	user 0.014s	sys 0.003s
I20260709 12:24:26.316924 24696 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.210517 24750 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:26.289018 24338 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.287827 24738 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:26.353907 18098 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:26.377043 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.315223 24724 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:26.327102 24444 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:26.518114 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:26.527154 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2764) took 2376 ms (client timeout 3012 ms). Trace:
W20260709 12:24:26.503005 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.618871 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:26.619567 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.456898 24618 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:26.496760 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.811s (SENT)
I20260709 12:24:26.406697 24483 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:26.440958 24710 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
W20260709 12:24:26.473546 24620 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.686s	user 0.009s	sys 0.005s
I20260709 12:24:26.185350 24634 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.408324 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:26.542300 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:24:26.476760 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.631198 18165 raft_consensus.cc:606] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:26.474500 24714 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:26.631443 24463 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:26.456064 24644 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:26.452356 24636 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.357429 24663 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.633680 24644 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:26.634298 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.635560 24663 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:26.637010 24620 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.639611 24620 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:26.612233 24738 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.646256 24634 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:26.646833 24738 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:26.648773 24724 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.543327 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:26.651190 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.458592 24388 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:26.651247 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:26.264052 24692 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.388192 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:26.536696 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.653467 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.654131 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.454327 24092 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:26.503010 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.457631 24739 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:24:26.652832 24663 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:26.464882 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 276892us
I20260709 12:24:26.666944 24663 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.667737 24692 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:26.668210 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:26.620523 24750 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.669662 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2907) took 1211 ms. Trace:
I20260709 12:24:26.651845 24388 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.670279 24388 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:26.196923 24655 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.547961 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:26.616519 24708 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:26.454687 24722 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:26.432883 24744 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 12:24:26.480434 24718 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.417284 24336 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:26.431653 24742 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.421s	user 0.000s	sys 0.001s
W20260709 12:24:26.438506 24731 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:24:26.389187 18815 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:24:26.687153 18815 pending_rounds.cc:85] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 11
I20260709 12:24:26.687511 18815 pending_rounds.cc:107] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:24:26.687796 18815 raft_consensus.cc:2889] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:26.438621 24737 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:26.379954 18167 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:26.538396 18019 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:24:26.445807 24736 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:24:26.484163 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.526953 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.525871 24632 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.043s	user 0.010s	sys 0.014s
I20260709 12:24:26.129103 24684 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:26.528467 24538 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.749s	user 0.015s	sys 0.002s
I20260709 12:24:26.722374 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:26.518436 24501 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:26.530021 24439 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.272s	user 0.025s	sys 0.001s
I20260709 12:24:26.536741 24555 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:26.604048 24712 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.984s	user 0.004s	sys 0.015s
I20260709 12:24:26.612794 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.618121 18093 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
W20260709 12:24:26.619174 18172 rpcz_store.cc:269] 0709 12:24:24.151020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.706042 (+555022us) service_pool.cc:224] Handling call
0709 12:24:26.526770 (+1820728us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54000320d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:26.527018 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1088986496}
W20260709 12:24:26.438407 24743 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.488s	user 0.001s	sys 0.000s
W20260709 12:24:26.623553 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126890us
I20260709 12:24:26.631699 18165 raft_consensus.cc:614] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:26.435853 24729 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:26.651446 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:26.459442 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.653483 18095 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:24:26.631595 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 223389us
I20260709 12:24:26.674460 18023 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.763342 18170 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:24:26.386312 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.622903 24444 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:26.670131 18096 rpcz_store.cc:276] 0709 12:24:25.458621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.458829 (+   208us) service_pool.cc:224] Handling call
0709 12:24:25.473082 (+ 14253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.533912 (+ 60830us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:25.533962 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:25.576599 (+ 42637us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.865424 (+288825us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:25.865473 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.865531 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:26.669416 (+803885us) raft_consensus.cc:1685] finished
0709 12:24:26.669439 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.669592 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47560,"raft.queue_time_us":93778,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":19676,"thread_start_us":255554,"threads_started":1,"wal-append.queue_time_us":675146}
I20260709 12:24:26.670691 24750 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
W20260709 12:24:26.673094 24679 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.188s	user 0.011s	sys 0.017s
W20260709 12:24:26.758142 24754 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:24:26.676470 24740 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.478569 24558 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.671731 24634 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:26.699811 24692 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.686654 24720 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:26.702710 24715 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.216s	user 0.015s	sys 0.008s
W20260709 12:24:26.703591 24726 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.921s	user 0.009s	sys 0.005s
W20260709 12:24:26.706913 24657 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.663s	user 0.008s	sys 0.005s
I20260709 12:24:26.796490 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:26.713770 18166 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:26.723455 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:26.679653 24723 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.735257 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.735313 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:26.818254 18098 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:26.634330 24644 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.835821 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:26.735854 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.758283 18093 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:24:26.836858 18093 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:26.837733 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3706) took 1329 ms. Trace:
I20260709 12:24:26.760093 18167 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 15. Ops: []
I20260709 12:24:26.761076 18097 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:24:26.763547 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:26.840032 18172 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:26.764024 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:26.764536 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.762513 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:26.769675 18810 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:26.770237 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:26.838250 18093 rpcz_store.cc:276] 0709 12:24:25.508674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.918844 (+410170us) service_pool.cc:224] Handling call
0709 12:24:25.919269 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.837442 (+918173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.837670 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:26.842145 18173 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:26.842988 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:26.771512 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:26.777232 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.920s (SENT)
I20260709 12:24:26.792058 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:26.844872 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:26.845069 18019 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:26.686375 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.732702 24722 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:26.762599 24663 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:26.759397 24439 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.732932 24618 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:26.747035 24538 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:26.743537 24734 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:26.816419 24715 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.779359 24712 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:26.800503 24720 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.820499 18023 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:26.900480 18017 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:26.818215 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:26.815968 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:26.815375 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:26.815667 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.902251 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:26.902541 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.903077 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.840520 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.759621 24501 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:26.908232 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.795357 24745 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:24:26.920018 18021 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:26.837777 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.921348 18022 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.853302 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:26.856047 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92106.9us
I20260709 12:24:26.687235 24336 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.795598 24654 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.923622 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.924546 18093 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:26.860667 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.925066 18091 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.862361 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.863036 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:26.863678 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:26.927052 18172 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:26.864527 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:26.918887 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:26.833979 24558 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.814574 18165 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:26.795113 24586 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:26.800468 24759 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.902958 18098 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:26.969209 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:26.969923 18023 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:26.845397 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68239.1us
I20260709 12:24:26.904067 18170 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:26.908062 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1399) took 1527 ms. Trace:
I20260709 12:24:26.970320 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:26.908818 18064 connection.cc:256] RPC call timeout handler was delayed by -3.6875511070002176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
I20260709 12:24:26.899699 24725 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:26.835615 24726 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.796561 24555 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.108s
I20260709 12:24:26.802332 24634 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:26.834869 24657 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.005612 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.006631 18094 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:27.007593 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:26.943874 18812 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:26.839049 24723 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.008941 18166 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:26.971020 18811 rpcz_store.cc:276] 0709 12:24:25.374390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.375729 (+  1339us) service_pool.cc:224] Handling call
0709 12:24:25.376220 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.492822 (+116602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:25.492858 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:25.493514 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.813993 (+320479us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:25.814065 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.417726 (+603661us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:26.417797 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:26.435069 (+ 17272us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:26.901450 (+466381us) raft_consensus.cc:1685] finished
0709 12:24:26.901476 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.901639 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2512,"raft.queue_time_us":40750,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":142,"spinlock_wait_cycles":275539200,"thread_start_us":319908,"threads_started":1,"wal-append.queue_time_us":692934}
I20260709 12:24:26.945837 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:27.012234 18093 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:26.972025 18064 connection.cc:256] RPC call timeout handler was delayed by -3.041351107000537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2658s for a response
I20260709 12:24:26.872821 18173 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:26.976604 18022 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:26.901199 24632 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.811192 24692 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.940361 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:26.860667 24194 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
W20260709 12:24:26.939858 24736 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:24:27.007654 24657 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:27.015146 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74941us
W20260709 12:24:26.998466 24743 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:24:27.017560 18094 raft_consensus.cc:2410] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:24:27.022269 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.061133 18808 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:27.027927 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.035848 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:27.064682 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:26.945541 24538 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:27.049495 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:27.055222 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:26.855594 24722 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.986369 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:27.064684 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48871.5us
I20260709 12:24:27.085233 18813 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:27.017863 18164 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:26.907953 24720 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.861106 24618 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:26.982610 24725 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.082289 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:27.098762 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.083986 18170 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:27.088797 18091 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:27.101019 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.093590 18092 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:27.093145 18093 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 11.
I20260709 12:24:27.102762 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.103488 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:27.103557 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.104586 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.094257 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.090214 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:27.106539 18098 raft_consensus.cc:2368] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:24:27.107223 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:27.104611 18091 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:27.111718 18093 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:27.116868 18165 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:27.117177 18093 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:27.032300 24733 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:26.998159 24759 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.129921 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2906) took 1671 ms. Trace:
W20260709 12:24:27.126447 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77123.6us
I20260709 12:24:27.133598 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:27.130694 18099 rpcz_store.cc:276] 0709 12:24:25.458212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.463301 (+  5089us) service_pool.cc:224] Handling call
0709 12:24:25.463749 (+   448us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:25.498548 (+ 34799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:25.498582 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:25.506220 (+  7638us) log.cc:844] Serialized 52 byte log entry
0709 12:24:25.768154 (+261934us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:25.840247 (+ 72093us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.841933 (+  1686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.842024 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.129444 (+1287420us) raft_consensus.cc:1685] finished
0709 12:24:27.129484 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.129787 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44448,"prepare.queue_time_us":26887,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":40153,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":23433,"thread_start_us":162763,"threads_started":1,"wal-append.queue_time_us":541005}
I20260709 12:24:27.134495 18094 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
W20260709 12:24:27.135869 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.136610 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:27.139117 18098 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:27.139640 18098 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:27.076925 24723 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.118266 24697 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.032210 24632 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.119822 24725 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:27.162573 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.152431 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.158758 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:27.125672 24684 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.166810 18093 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.161752 24697 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.142347 24693 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:27.179642 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 124532us
I20260709 12:24:27.125587 24538 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:27.148409 24643 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:27.182281 18093 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:27.185869 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.186206 18813 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 17. Preceding OpId from leader: term: 20 index: 18. (index mismatch)
I20260709 12:24:27.173063 18098 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.059725 24734 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:27.196798 24760 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:24:27.199126 24692 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.200593 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:27.236579 18813 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:27.206396 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:27.206737 24768 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.072s	user 0.001s	sys 0.001s
W20260709 12:24:27.209779 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.210933 24618 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.144197 24733 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:27.218523 24761 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.319s	user 0.000s	sys 0.000s
I20260709 12:24:27.220244 24501 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:27.221230 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3635) took 1760 ms. Trace:
I20260709 12:24:27.255399 24726 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.254364 18016 rpcz_store.cc:276] 0709 12:24:25.460602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.470914 (+ 10312us) service_pool.cc:224] Handling call
0709 12:24:25.471265 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.716405 (+245140us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:25.717157 (+   752us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.780977 (+ 63820us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.781781 (+   804us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.782212 (+   431us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:25.782714 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.782872 (+   158us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:25.782920 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.783006 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.220972 (+1437966us) raft_consensus.cc:1685] finished
0709 12:24:27.220999 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.221158 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45490,"prepare.queue_time_us":899,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":100,"raft.queue_time_us":48478,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":15818,"thread_start_us":262236,"threads_started":2,"wal-append.queue_time_us":123108}
W20260709 12:24:27.256078 24338 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.000s
I20260709 12:24:27.256577 24338 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.256981 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.222203 24644 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:27.222512 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3637) took 1750 ms (client timeout 2059 ms). Trace:
W20260709 12:24:27.258627 18017 rpcz_store.cc:269] 0709 12:24:25.472145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.472278 (+   133us) service_pool.cc:224] Handling call
0709 12:24:27.222296 (+1750018us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400347bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:27.222428 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3276386048}
I20260709 12:24:27.260144 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.222940 24679 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:27.224045 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:27.264108 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:27.227321 24720 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.180343 24771 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:27.185719 24687 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:27.229195 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.196918 24657 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
W20260709 12:24:27.230926 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.231554 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:27.270126 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:27.232825 18098 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:27.273499 24636 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.641s	user 0.008s	sys 0.001s
I20260709 12:24:27.141839 24586 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.234484 24749 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:27.184311 24643 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:27.242880 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50615.1us
W20260709 12:24:27.298389 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104381us
I20260709 12:24:27.248947 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:27.299504 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.241977 24693 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:27.250007 24336 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:27.248248 24750 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.251214 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3643) took 1386 ms. Trace:
I20260709 12:24:27.251861 18023 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:24:27.252820 24388 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.001s
I20260709 12:24:27.253641 24733 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.266422 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:27.274228 18811 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:27.340837 18806 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:27.274839 24538 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:27.275770 24565 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.896s	user 0.014s	sys 0.000s
I20260709 12:24:27.257766 18016 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:24:27.286434 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.290006 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:27.238137 18020 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.247396 18093 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:24:27.351771 18093 pending_rounds.cc:85] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 12
I20260709 12:24:27.352072 18093 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:24:27.305553 24643 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:27.307523 18807 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:24:27.361919 18807 pending_rounds.cc:85] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 11
I20260709 12:24:27.362267 18807 pending_rounds.cc:107] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:24:27.316612 24555 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.362558 18807 raft_consensus.cc:2889] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:27.362982 18807 pending_rounds.cc:107] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 12:24:27.363432 18807 raft_consensus.cc:2889] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:27.331472 24762 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:24:27.372522 24194 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:27.372834 24720 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:27.373359 24720 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.374675 18173 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:27.375334 24338 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:27.375748 24720 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.375991 24338 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.372604 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70691us
I20260709 12:24:27.376971 24501 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.377534 24501 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.378050 18166 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:27.377647 24693 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.378504 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:27.379350 24621 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:27.380362 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:27.313791 18022 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:27.343003 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:27.352406 18093 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:27.301641 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:27.296188 24759 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:27.406916 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.407809 18022 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:27.376591 24338 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.380550 24644 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:27.380789 24501 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.379665 18170 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:24:27.410017 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108519us
W20260709 12:24:27.410784 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 262718us
I20260709 12:24:27.412179 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.967887 18016 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:24:27.969344 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:27.316249 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:27.380766 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:28.192579 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 811835us
I20260709 12:24:27.399173 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.196219 24772 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:28.196465 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 797363us
I20260709 12:24:28.198525 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.199538 24772 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:28.200074 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:28.201088 24772 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:28.201262 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:27.330894 18098 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:28.211088 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2919) took 1516 ms. Trace:
I20260709 12:24:28.211578 18098 rpcz_store.cc:276] 0709 12:24:26.694727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.107337 (+412610us) service_pool.cc:224] Handling call
0709 12:24:27.138946 (+ 31609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.210638 (+1071692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.210995 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:27.376099 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:27.385207 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1209) took 2242 ms. Trace:
I20260709 12:24:28.218986 18810 rpcz_store.cc:276] 0709 12:24:25.142454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.142768 (+   314us) service_pool.cc:224] Handling call
0709 12:24:25.169175 (+ 26407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.841666 (+1672491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.384898 (+543232us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:27.385131 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3662136064}
I20260709 12:24:27.386466 18814 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 12:24:28.221555 18814 pending_rounds.cc:85] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 12
I20260709 12:24:28.223366 18814 pending_rounds.cc:107] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 12:24:28.223686 18814 raft_consensus.cc:2889] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:27.876333 18778 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.022s	sys 0.023s
I20260709 12:24:27.891065 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2771) took 1177 ms. Trace:
I20260709 12:24:27.431352 24483 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:27.362916 24555 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.361304 24565 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:27.803119 24632 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.252374 18024 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:28.255033 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3644) took 2369 ms (client timeout 1621 ms). Trace:
W20260709 12:24:28.255544 18024 rpcz_store.cc:269] 0709 12:24:25.885684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.889612 (+  3928us) service_pool.cc:224] Handling call
0709 12:24:28.254688 (+2365076us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.254922 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5430119808}
I20260709 12:24:28.256992 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:27.383340 24634 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.675673 24643 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.526110 24338 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.249552 24565 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.675928 24697 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:27.670462 24725 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.264773 24725 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.430883 24654 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.430509 24679 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.266412 24654 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.316155 24687 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:27.439347 24733 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:27.337304 24750 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:27.392356 24738 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.745s	user 0.009s	sys 0.003s
W20260709 12:24:27.684446 24755 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.974s	user 0.004s	sys 0.008s
W20260709 12:24:27.382588 24092 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.711s	user 0.013s	sys 0.001s
I20260709 12:24:28.292747 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3713) took 2718 ms. Trace:
I20260709 12:24:28.293159 24624 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.293877 24624 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.293310 18090 rpcz_store.cc:276] 0709 12:24:25.574330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.947704 (+373374us) service_pool.cc:224] Handling call
0709 12:24:25.948076 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.948576 (+   500us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:25.993446 (+ 44870us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.404947 (+411501us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.405585 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.405775 (+   190us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:26.430046 (+ 24271us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.430500 (+   454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.430633 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.213529 (+782896us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b180036ac60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:27.682959 (+469430us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:28.292420 (+609461us) raft_consensus.cc:1685] finished
0709 12:24:28.292459 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.292662 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86792,"prepare.queue_time_us":57315,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":11161,"raft.queue_time_us":77902,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":40333,"spinlock_wait_cycles":48476288,"thread_start_us":352695,"threads_started":1,"wal-append.queue_time_us":654308}
I20260709 12:24:27.346817 24388 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:27.334117 24439 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:24:28.305631 18812 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:28.307296 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1172) took 1043 ms. Trace:
I20260709 12:24:28.307678 18812 rpcz_store.cc:276] 0709 12:24:27.264017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.269782 (+  5765us) service_pool.cc:224] Handling call
0709 12:24:28.307035 (+1037253us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54005e01d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.307215 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379217280}
W20260709 12:24:27.385591 24444 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.594s	user 0.005s	sys 0.008s
I20260709 12:24:27.306097 24749 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.314821 24749 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:27.502046 24696 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.881s	user 0.014s	sys 0.001s
I20260709 12:24:27.331084 24771 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:28.316480 24696 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.318176 18017 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.317978 24771 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:28.319386 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2754) took 1362 ms (client timeout 1786 ms). Trace:
I20260709 12:24:27.409088 24776 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.145s
I20260709 12:24:27.970620 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:28.322749 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 352219us
I20260709 12:24:28.325217 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:27.316431 18021 rpcz_store.cc:276] 0709 12:24:25.864785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.865649 (+   864us) service_pool.cc:224] Handling call
0709 12:24:27.250973 (+1385324us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:27.251130 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2230693248}
I20260709 12:24:28.326939 24444 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.327996 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.328912 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 899) took 1032 ms. Trace:
I20260709 12:24:28.329242 18021 rpcz_store.cc:276] 0709 12:24:27.296433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.327598 (+1031165us) service_pool.cc:224] Handling call
0709 12:24:28.328837 (+  1239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.330201 18015 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.331387 24388 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:28.332165 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138781us
W20260709 12:24:28.333531 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2732) took 1613 ms (client timeout 1954 ms). Trace:
W20260709 12:24:28.334022 18015 rpcz_store.cc:269] 0709 12:24:26.720133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.723105 (+  2972us) service_pool.cc:224] Handling call
0709 12:24:28.333300 (+1610195us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540004c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.333463 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3483458432}
W20260709 12:24:27.972687 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.335311 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.336213 18015 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.336650 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2734) took 1615 ms. Trace:
I20260709 12:24:28.336938 18015 rpcz_store.cc:276] 0709 12:24:26.720884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.334982 (+1614098us) service_pool.cc:224] Handling call
0709 12:24:28.336585 (+  1603us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:27.976181 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:28.334018 24691 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:28.338555 24691 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:28.338958 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 362868us
I20260709 12:24:28.339705 24691 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.340996 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:27.983922 18165 raft_consensus.cc:606] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:27.989470 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.357646 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.006563 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2736) took 1285 ms. Trace:
I20260709 12:24:28.034894 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2772) took 1320 ms. Trace:
I20260709 12:24:28.065305 24538 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:28.088912 18019 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.162s	user 0.017s	sys 0.002s
I20260709 12:24:28.096041 24653 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:28.109519 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2741) took 1329 ms (client timeout 1672 ms). Trace:
I20260709 12:24:28.112335 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1220) took 2099 ms. Trace:
W20260709 12:24:28.155301 24770 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.989s	user 0.002s	sys 0.002s
W20260709 12:24:28.160703 24758 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.060s	user 0.007s	sys 0.000s
W20260709 12:24:28.169765 18020 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.812s	user 0.016s	sys 0.000s
I20260709 12:24:28.183694 17919 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:28.186188 24693 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:28.187759 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:28.185729 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 869484us
I20260709 12:24:28.187754 24722 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.193914 24620 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:28.360186 18018 rpcz_store.cc:269] 0709 12:24:26.779987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.811775 (+ 31788us) service_pool.cc:224] Handling call
0709 12:24:28.109158 (+1297383us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001d1d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.109417 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379220224}
I20260709 12:24:28.363469 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:28.364389 18018 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:27.400274 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.201624 24772 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.205250 24719 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:28.373951 24772 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.374395 24719 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:28.374704 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.375798 24719 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.206532 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:28.377920 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 171497us
I20260709 12:24:28.378125 24722 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:28.378300 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 82) took 1244 ms. Trace:
I20260709 12:24:28.378636 24722 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:27.616698 24789 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:24:28.379163 24722 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:28.229401 18778 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.891s	user 0.023s	sys 0.024s
I20260709 12:24:28.379513 24338 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:28.367430 24388 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.234081 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1163) took 1951 ms. Trace:
I20260709 12:24:28.382133 18814 rpcz_store.cc:276] 0709 12:24:26.282742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.282906 (+   164us) service_pool.cc:224] Handling call
0709 12:24:26.323134 (+ 40228us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.224221 (+1901087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.233746 (+  9525us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:28.233997 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2444221824}
I20260709 12:24:28.224917 24757 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.225158 18171 rpcz_store.cc:276] 0709 12:24:26.713771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.714425 (+   654us) service_pool.cc:224] Handling call
0709 12:24:26.764677 (+ 50252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.765048 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:27.083080 (+318032us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.890252 (+807172us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b180042fb80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:24:27.890556 (+   304us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.890628 (+    72us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:27.890699 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.890800 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.890981 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":354085,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":1741,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":128,"spinlock_wait_cycles":5837568,"thread_start_us":345250,"threads_started":2,"wal-append.queue_time_us":184301}
I20260709 12:24:28.383584 24757 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.384485 24092 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.238863 24483 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.385082 24092 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.260556 24555 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.867738 24657 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.396991 24657 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.383544 24388 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.267143 24679 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.269285 24687 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.414597 24439 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.419467 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.419746 24439 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.276718 24654 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.412673 24555 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.277045 24643 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.281302 24733 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.421784 24388 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.424690 24439 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.423930 24657 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.425412 24757 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.288023 24738 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.289314 24750 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:27.373324 24194 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.298867 24624 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:28.319783 18017 rpcz_store.cc:269] 0709 12:24:26.956799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.259804 (+303005us) service_pool.cc:224] Handling call
0709 12:24:28.319113 (+1059309us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001d1354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.319319 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431364480}
I20260709 12:24:28.590588 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.592024 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.592703 18017 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:28.320586 18809 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:24:28.595186 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1171) took 1331 ms (client timeout 1758 ms). Trace:
W20260709 12:24:28.595583 18809 rpcz_store.cc:269] 0709 12:24:27.263625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.263772 (+   147us) service_pool.cc:224] Handling call
0709 12:24:28.594982 (+1331210us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54006f2054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.595124 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2427781376}
I20260709 12:24:28.342481 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.596952 18098 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:28.597168 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1416) took 1269 ms. Trace:
W20260709 12:24:27.979010 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 636138us
I20260709 12:24:28.597565 18811 rpcz_store.cc:276] 0709 12:24:27.328397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.341113 (+ 12716us) service_pool.cc:224] Handling call
0709 12:24:28.595135 (+1254022us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.595360 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.595386 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.595436 (+    50us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:28.595471 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.595527 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.596950 (+  1423us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54006f2050. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.597084 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2883071232}
I20260709 12:24:28.600675 24722 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.344416 24696 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.602309 18166 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.602646 24194 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.603513 24722 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:28.604209 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 966) took 1230 ms (client timeout 1600 ms). Trace:
W20260709 12:24:28.604669 18166 rpcz_store.cc:269] 0709 12:24:27.398651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.399919 (+  1268us) service_pool.cc:224] Handling call
0709 12:24:28.603970 (+1204051us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.604146 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508840704}
I20260709 12:24:28.348723 24697 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:28.329705 24444 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.351936 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.354122 24779 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:28.353224 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.606638 18094 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:28.354960 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:28.357271 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:28.097915 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.360909 18020 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
W20260709 12:24:28.612300 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.612418 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2749) took 1805 ms (client timeout 1770 ms). Trace:
W20260709 12:24:28.612805 18020 rpcz_store.cc:269] 0709 12:24:26.806960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.206051 (+399091us) service_pool.cc:224] Handling call
0709 12:24:28.612330 (+1406279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.604385 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.366173 24620 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:28.360257 18808 rpcz_store.cc:276] 0709 12:24:26.012844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.013050 (+   206us) service_pool.cc:224] Handling call
0709 12:24:28.111966 (+2098916us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54007823d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.112203 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2406921984}
I20260709 12:24:28.366698 24693 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:28.620525 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 265646us
I20260709 12:24:28.620445 24620 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.367089 18019 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:28.359067 18023 rpcz_store.cc:276] 0709 12:24:26.721630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.163884 (+442254us) service_pool.cc:224] Handling call
0709 12:24:28.006253 (+842369us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b54001d1854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.006476 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148351872}
W20260709 12:24:28.621996 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2750) took 1807 ms (client timeout 1578 ms). Trace:
I20260709 12:24:28.622491 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.610702 24779 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.362563 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 900) took 1065 ms. Trace:
I20260709 12:24:28.623659 24779 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:28.623823 18022 rpcz_store.cc:276] 0709 12:24:27.296871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.187395 (+890524us) service_pool.cc:224] Handling call
0709 12:24:28.362266 (+174871us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.624673 24722 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:28.624965 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.625370 24722 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.625849 24722 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.625939 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2740) took 1846 ms. Trace:
I20260709 12:24:28.610940 24696 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.360646 18164 rpcz_store.cc:276] 0709 12:24:26.714836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.715002 (+   166us) service_pool.cc:224] Handling call
0709 12:24:28.034619 (+1319617us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.034801 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42199296}
I20260709 12:24:28.626677 24696 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.627328 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.627940 18164 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.629069 24696 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:28.361445 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.371551 24653 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:28.630937 24653 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:28.631780 18023 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.631935 24653 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:28.379768 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.04211e+06us
I20260709 12:24:28.386893 24338 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:28.229867 24763 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.387493 24092 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:28.387765 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.634222 24338 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.634564 24763 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.635105 18810 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:28.365546 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.388023 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.389379 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.636579 18021 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:28.636646 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.636766 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.378863 18165 rpcz_store.cc:276] 0709 12:24:27.133686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.135603 (+  1917us) service_pool.cc:224] Handling call
0709 12:24:28.378100 (+1242497us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b540002add4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.378204 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1946701056}
I20260709 12:24:28.388612 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:28.634248 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.873s (SENT)
I20260709 12:24:28.391875 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:28.637562 18018 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:28.637892 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.638293 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.638459 18170 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:28.638531 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.639394 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2737) took 1902 ms. Trace:
I20260709 12:24:28.639652 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.639952 24757 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:28.633566 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.386951 24483 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.394603 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.637589 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:28.641852 18021 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.642220 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.642285 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.642964 18095 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.643592 24092 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.644187 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.644356 24092 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.645031 18095 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:28.639735 18018 rpcz_store.cc:276] 0709 12:24:26.736576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.638270 (+1901694us) service_pool.cc:224] Handling call
0709 12:24:28.639334 (+  1064us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.407670 24803 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:28.269697 18024 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:24:28.646510 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2759) took 1411 ms. Trace:
I20260709 12:24:28.646184 24803 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.646718 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.646896 18024 rpcz_store.cc:276] 0709 12:24:27.235079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.256626 (+1021547us) service_pool.cc:224] Handling call
0709 12:24:28.646424 (+389798us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.647403 18094 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:28.408900 24806 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.647986 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.648430 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.648233 24806 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.649071 18094 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:28.649287 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.411136 18172 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:28.650274 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.609272 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:28.650803 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.417536 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.638309 18813 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:28.652371 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:28.651970 24624 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:28.653442 24624 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:28.653887 24624 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:28.654301 24624 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:28.430413 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.656221 18091 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:28.657404 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.490376 24808 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.490775 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.659564 24808 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.660151 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.660283 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.491714 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.662384 18092 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:28.438277 24659 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.662767 24808 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.662957 24659 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.663552 18095 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.15-16.15]
I20260709 12:24:28.664736 18092 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:28.665113 18092 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:28.665401 24659 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:28.492530 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60924.7us
I20260709 12:24:28.666129 18098 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:28.666756 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:28.428815 17919 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 9 to 11, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 11 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:28.502053 18016 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.661612 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.668843 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3659) took 1648 ms. Trace:
I20260709 12:24:28.493999 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:28.507439 24802 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:24:28.456612 24657 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 12:24:28.670264 18167 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:28.669870 24338 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.492905 24388 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:28.671478 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.671512 24388 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.672484 18167 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.673195 24803 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:28.422928 24555 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.516132 24679 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:28.517627 24723 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.675647 24555 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.676323 24723 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.668881 18099 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.661432 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.678746 24388 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.518092 24740 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.483385 24439 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:24:28.421923 24565 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.422179 24632 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.546886 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:28.551898 18064 leader_election.cc:336] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.548s (SENT)
I20260709 12:24:28.704223 18096 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.704366 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.525045 24750 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.569922 24772 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.573359 24776 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:28.583459 18136 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50510 dropped due to backpressure: service queue is full with 50 items
I20260709 12:24:28.345145 24631 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:28.593669 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:28.723183 18017 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.723838 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2756) took 1622 ms. Trace:
I20260709 12:24:28.724205 18017 rpcz_store.cc:276] 0709 12:24:27.101365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.593395 (+1492030us) service_pool.cc:224] Handling call
0709 12:24:28.723764 (+130369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.678965 24723 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.609549 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:28.731329 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 121882us
I20260709 12:24:28.732921 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.354499 18093 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:28.734719 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.360574 24538 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.610409 24697 consensus_queue.cc:1243] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:28.620805 24693 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.392648 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.747156 18807 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:28.717527 18136 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50510 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2784)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 979)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2785)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2811)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2825)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 976)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2794)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 977)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2789)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 975)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2798)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 83)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2782)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2786)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2787)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2790)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2793)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2803)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2815)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2818)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2821)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2822)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2824)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2826)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2827)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2831)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2834)
W20260709 12:24:28.622403 18019 rpcz_store.cc:269] 0709 12:24:26.814408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.845736 (+ 31328us) service_pool.cc:224] Handling call
0709 12:24:28.621909 (+1776173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.748689 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:28.749578 18019 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.750572 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.396064 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.416345 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.751451 18090 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:28.751788 18173 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.752563 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.752974 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.753324 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.653323 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.753755 18173 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:28.754848 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.756234 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.757090 18168 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:24:28.423445 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.758304 18169 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.432416 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.759230 24772 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.759447 18806 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.658075 18018 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:28.742106 24723 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.760803 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2733) took 2040 ms. Trace:
I20260709 12:24:28.742959 24757 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 12:24:28.761384 18018 rpcz_store.cc:276] 0709 12:24:26.720515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.657459 (+1936944us) service_pool.cc:224] Handling call
0709 12:24:28.657839 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.760503 (+102664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.760738 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.762007 24757 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:28.652014 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.762460 24757 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:28.762861 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:28.763478 24757 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.422016 24634 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:28.666602 24483 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:28.669267 18016 rpcz_store.cc:276] 0709 12:24:27.020931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.969000 (+948069us) service_pool.cc:224] Handling call
0709 12:24:28.668624 (+699624us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b54001fca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:28.668773 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1209815424}
I20260709 12:24:28.422979 24733 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.672006 24806 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:28.517328 24655 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:28.675664 24092 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.676810 24555 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:28.519917 24738 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:28.679673 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45469.5us
I20260709 12:24:28.797345 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:28.673478 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.684170 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.798936 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.798837 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:28.799938 18094 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:24:28.685700 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:28.682122 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:28.803639 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.623s (SENT)
W20260709 12:24:28.805084 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0841622130001269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
I20260709 12:24:28.693291 18165 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:28.620553 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.705380 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.806401 18097 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:28.708671 18170 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:28.807458 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.791270 24737 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.810298 24737 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:28.811321 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 129381us
I20260709 12:24:28.812767 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.812950 24737 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.709285 18021 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:28.720258 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.814365 18097 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:28.815052 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.815488 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.816246 18172 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:28.698947 18092 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.817510 18092 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.607376 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.710443 24632 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.716744 24750 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.706998 24565 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.707224 24740 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:28.726717 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 175154us
I20260709 12:24:28.731400 24388 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:28.613790 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.854779 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.862419 18020 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.734282 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:28.735662 18812 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:28.863886 18814 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:28.736294 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.737408 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.684065 24763 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.754925 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.727299 24776 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.626442 18022 rpcz_store.cc:276] 0709 12:24:26.779677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.624673 (+1844996us) service_pool.cc:224] Handling call
0709 12:24:28.625867 (+  1194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.870666 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.764473 24781 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:24:28.869135 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.760823 24723 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.662217 18024 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.787467 24634 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.766860 24538 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:28.884689 18092 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:28.767315 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.775748 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:28.785378 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.787982 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.888835 18016 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:28.791011 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.790277 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.681531 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.889603 18093 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:28.889878 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 208446us
I20260709 12:24:28.890635 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.876243 24439 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:28.891665 24439 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.891930 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.891973 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.809580 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.893716 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.893891 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.894397 24776 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.894589 18169 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.865747 24194 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:28.895159 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.895545 24194 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.895606 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.795127 24655 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:28.897133 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.725302 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.898180 18017 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:28.898224 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.899035 18168 raft_consensus.cc:2376] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20. Re-sending same reply.
I20260709 12:24:28.899610 18017 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:28.899950 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.900292 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2744) took 2113 ms. Trace:
I20260709 12:24:28.900739 18017 rpcz_store.cc:276] 0709 12:24:26.787169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.899058 (+2111889us) service_pool.cc:224] Handling call
0709 12:24:28.899410 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.900023 (+   613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.900231 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.901206 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.902036 18023 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:28.818481 18166 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.902431 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.902868 18022 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:24:28.903288 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.903554 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2743) took 2116 ms. Trace:
I20260709 12:24:28.903718 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.903934 18022 rpcz_store.cc:276] 0709 12:24:26.786828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.902203 (+2115375us) service_pool.cc:224] Handling call
0709 12:24:28.902589 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.903272 (+   683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.903494 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.904630 18169 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:28.904539 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.905311 18022 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:28.725988 24808 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:28.823167 18813 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:28.835395 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.906930 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.907335 18097 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:28.907374 24657 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:28.907862 18097 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:28.836804 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:28.905591 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.896087 18094 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:28.813649 24737 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:28.849570 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.908982 18097 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.836483 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.909916 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.910754 18172 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:28.795727 24757 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:28.911556 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:28.912189 18172 raft_consensus.cc:606] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:28.851379 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.909773 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.914321 18170 raft_consensus.cc:2368] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16.
I20260709 12:24:28.908974 24737 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.915031 24737 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:28.915706 24750 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:28.910781 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.919054 18165 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:28.919584 18016 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:28.920317 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2748) took 2121 ms. Trace:
I20260709 12:24:28.920675 18016 rpcz_store.cc:276] 0709 12:24:26.798808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.919102 (+2120294us) service_pool.cc:224] Handling call
0709 12:24:28.919405 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.920067 (+   662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.920259 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.922005 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.923635 24538 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:28.924036 24538 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.924377 24538 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.914228 18095 pending_rounds.cc:85] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
I20260709 12:24:28.927268 18095 pending_rounds.cc:107] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 15.8
W20260709 12:24:28.857321 18064 leader_election.cc:336] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.581s (SENT)
I20260709 12:24:28.927966 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:28.872512 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 135193us
I20260709 12:24:28.874454 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:28.930368 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3657) took 2070 ms. Trace:
I20260709 12:24:28.930745 18020 rpcz_store.cc:276] 0709 12:24:26.860413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.929395 (+2068982us) service_pool.cc:224] Handling call
0709 12:24:28.929753 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.930009 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.930036 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.930083 (+    47us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:28.930124 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.930186 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.930305 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.931444 24740 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.932184 24763 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:28.841463 24697 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:28.932621 24763 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.932847 24697 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:28.933077 24763 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:28.933351 24697 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:28.936818 24632 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:28.766376 24696 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.938230 24696 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:28.852280 24738 raft_consensus.cc:2764] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:24:28.939252 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:28.928364 18097 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:28.762147 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.940606 24723 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:28.940838 18098 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:28.942657 24565 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:28.805586 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0094622130004609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20260709 12:24:28.947029 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:28.948129 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.949092 17915 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 295) took 26059 ms (client timeout 15000 ms). Trace:
I20260709 12:24:28.895995 24194 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.905252 18166 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:28.932634 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3658) took 2065 ms. Trace:
I20260709 12:24:28.708810 18164 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:28.906028 18168 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:28.907732 18092 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:28.842720 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:28.843494 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157953us
I20260709 12:24:28.951776 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:28.964915 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:28.962502 18020 rpcz_store.cc:276] 0709 12:24:26.866991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.931742 (+2064751us) service_pool.cc:224] Handling call
0709 12:24:28.932084 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.932295 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.932323 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.932376 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:28.932413 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.932468 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.932575 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.889240 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.968767 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:28.890694 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:28.922791 24657 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.927563 18095 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:28.885192 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:28.970055 18093 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.892102 24439 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:28.952318 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 95046.2us
I20260709 12:24:28.983464 18024 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:24:28.984733 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2742) took 2198 ms. Trace:
I20260709 12:24:28.951702 17919 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 9 to 11, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 11 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:28.952869 24808 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:28.954900 24634 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:28.955451 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08003184c0 after lost signal to thread 16795
I20260709 12:24:28.956225 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2746) took 2168 ms. Trace:
I20260709 12:24:29.005388 18022 rpcz_store.cc:276] 0709 12:24:26.787811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.904824 (+2117013us) service_pool.cc:224] Handling call
0709 12:24:28.905135 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.918407 (+ 13272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.956154 (+ 37747us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.957409 18016 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:24:28.853600 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56351.8us
I20260709 12:24:28.960633 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:29.010037 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800334260 after lost signal to thread 17956
I20260709 12:24:29.010529 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:29.011832 18836 debug-util.cc:398] Leaking SignalData structure 0x7b080017c540 after lost signal to thread 18048
I20260709 12:24:29.012125 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:29.013350 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002e3540 after lost signal to thread 18123
I20260709 12:24:28.854653 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:28.969506 18167 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:29.015883 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.007155 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2752) took 2151 ms. Trace:
I20260709 12:24:29.018100 18016 rpcz_store.cc:276] 0709 12:24:26.855973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.921500 (+2065527us) service_pool.cc:224] Handling call
0709 12:24:28.921808 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.006775 (+ 84967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.007083 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.849789 24687 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.107s
I20260709 12:24:28.913607 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:28.884095 24772 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:28.985236 18024 rpcz_store.cc:276] 0709 12:24:26.786442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.958618 (+2172176us) service_pool.cc:224] Handling call
0709 12:24:28.959170 (+   552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.984281 (+ 25111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.984642 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.989749 18808 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:28.990119 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:28.994925 18020 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:28.995905 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:28.996824 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 160136us
I20260709 12:24:28.998636 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:28.999904 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.022405 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:29.026808 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 137659us
I20260709 12:24:28.858021 24555 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.133s
W20260709 12:24:29.044034 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08003f2f40 after lost signal to thread 18197
W20260709 12:24:29.044736 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800416880 after lost signal to thread 18198
I20260709 12:24:29.030452 18172 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:29.038456 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.045701 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:29.046271 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08004100c0 after lost signal to thread 18839
I20260709 12:24:29.031913 18809 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:24:29.047034 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6600 after lost signal to thread 18840
I20260709 12:24:28.980240 18098 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
W20260709 12:24:29.048043 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79400us
I20260709 12:24:29.048920 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:29.048075 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.051100 18164 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:29.051481 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.047094 18811 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.052659 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:29.053754 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50718.2us
I20260709 12:24:29.055542 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:29.056098 18064 leader_election.cc:336] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.867s (SENT)
I20260709 12:24:29.056582 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.046411 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2757) took 1893 ms. Trace:
I20260709 12:24:29.025908 24776 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.057567 18020 rpcz_store.cc:276] 0709 12:24:27.153544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.994140 (+1840596us) service_pool.cc:224] Handling call
0709 12:24:28.994650 (+   510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.046110 (+ 51460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.046348 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.058183 24776 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.059679 18020 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:28.878230 24586 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.149s
I20260709 12:24:29.062055 24621 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.044127 24772 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.063930 24772 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:29.065910 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66168.6us
I20260709 12:24:29.066038 24691 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.066319 24807 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:29.067617 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:29.067925 24654 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:29.068676 24654 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:29.068934 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:29.045192 18097 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 12:24:28.993716 24806 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:28.993952 24657 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:29.070343 18097 pending_rounds.cc:85] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 9
I20260709 12:24:29.071044 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:29.085274 18097 pending_rounds.cc:107] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:24:29.071705 18093 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 10
I20260709 12:24:29.071957 18020 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:29.086051 18093 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:24:29.086369 18093 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:29.087571 18021 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:29.087845 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:29.077977 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800409560 after lost signal to thread 24778
I20260709 12:24:29.088423 18020 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:24:29.031451 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.022598 24750 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:28.993511 24655 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.080729 24634 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:29.080689 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.171690 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:29.162209 24807 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:29.021121 24444 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:29.048815 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.096457 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.183000 18167 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:29.184047 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:29.184666 18167 raft_consensus.cc:606] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:24:29.185038 18167 raft_consensus.cc:614] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:29.028682 24092 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:29.094213 17919 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 2 to 10, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 10 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:29.100533 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.188349 18173 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:29.026621 24620 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:29.108609 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.007099 24388 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:29.127314 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51447us
I20260709 12:24:28.995846 24679 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:29.049273 24740 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:29.131906 18814 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:29.132267 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:29.132434 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 289833us
I20260709 12:24:29.198818 18812 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:29.199419 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.200570 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.201100 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.075338 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:29.164577 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115745us
I20260709 12:24:29.083426 24808 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:29.170476 24634 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.173066 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.085613 18097 raft_consensus.cc:2889] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:29.173785 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.174088 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.220798 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.221290 18021 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:29.221642 18098 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:29.084208 24806 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:29.174923 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:29.177933 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121841us
I20260709 12:24:29.176811 24738 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:29.227362 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.228229 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 140486us
I20260709 12:24:29.180084 24803 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:29.170940 24779 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.002s
I20260709 12:24:29.210326 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.169709 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.212491 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:29.213495 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.222678 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:29.225605 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50845.6us
I20260709 12:24:29.239056 18092 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:29.091776 24586 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.099s
I20260709 12:24:29.220556 18171 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:29.192929 24620 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.233050 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.243559 18097 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:29.234941 17919 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:29.235417 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.246142 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.247074 18021 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:29.240763 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:29.248845 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:29.252611 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.253440 18097 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:29.252611 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.254863 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.256068 18173 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:29.257023 18097 raft_consensus.cc:2368] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:24:29.260802 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:29.259193 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46819.1us
I20260709 12:24:29.264174 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:29.234150 24624 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.003s
I20260709 12:24:29.265739 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:29.172461 24750 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.272032 18021 raft_consensus.cc:2376] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16. Re-sending same reply.
I20260709 12:24:29.266706 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:29.226364 24634 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.268226 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.283835 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.226776 24740 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:29.236662 24806 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:29.277150 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.286885 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:29.293013 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.243767 24781 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:29.253034 24620 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:29.309964 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51575.2us
I20260709 12:24:29.266093 24723 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:29.224035 24679 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.211884 24388 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.254668 24654 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:29.416947 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 368178us
W20260709 12:24:29.432152 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 150714us
W20260709 12:24:29.291378 24722 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.005s
I20260709 12:24:29.325466 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.276389 24738 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.241990 24779 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.307485 24781 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:29.414505 24631 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.421118 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:29.274035 24653 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.469682 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.416144 18020 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 12:24:29.331133 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.432708 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.286479 24624 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.288858 24776 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.291508 24655 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.418856 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:29.447616 17919 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:29.507063 18169 raft_consensus.cc:2368] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:24:29.458137 24828 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.477579 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:29.507170 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.474967 24807 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.325444 24829 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:29.433409 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:29.462719 24738 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.486673 24586 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:29.532795 18811 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:29.533299 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 458095us
I20260709 12:24:29.534024 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:29.543799 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.443647 24555 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
W20260709 12:24:29.477526 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56531us
I20260709 12:24:29.504957 18021 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:29.566155 18021 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:29.475896 24781 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:29.551059 18165 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:29.486822 24723 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:29.552208 18168 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.564s	user 0.012s	sys 0.000s
I20260709 12:24:29.455518 24722 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.557242 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:29.559998 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 126713us
I20260709 12:24:29.533525 24826 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.574795 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.547641 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:29.580832 18021 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:29.583973 24679 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.584316 24841 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:29.584792 24841 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:29.585182 24841 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:29.585389 24697 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.000s
I20260709 12:24:29.552840 24838 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:24:29.536000 24740 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:29.578790 24737 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.663s	user 0.009s	sys 0.004s
I20260709 12:24:29.571640 24829 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:29.570860 24723 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:29.582144 24586 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:29.571427 24808 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:29.580395 24654 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.536556 24836 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:29.611343 17919 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 3 to 9, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 9 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:29.582089 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:24:29.606564 24823 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:24:29.529670 24763 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.596s	user 0.007s	sys 0.003s
I20260709 12:24:29.559525 24691 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.590466 24838 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:29.637748 24806 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.661365 24838 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:29.662583 24838 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.663846 24806 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:29.635854 24733 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.842s	user 0.004s	sys 0.010s
I20260709 12:24:29.664232 24838 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:29.665354 18166 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:29.664939 24733 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.674911 18015 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:29.675664 24733 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.616648 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:29.590059 24825 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:24:29.608201 24738 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.623260 18091 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260709 12:24:29.626941 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.678691 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.679196 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.679432 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62884us
I20260709 12:24:29.680154 24634 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.681267 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:29.585768 24388 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.682210 24776 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:29.682601 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:29.683669 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:29.615484 24836 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:29.684830 24803 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:29.685171 24836 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:29.685428 24806 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:29.685563 24803 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.686223 24806 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:29.686409 24803 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.599015 24620 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.687337 24806 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.631390 24722 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:29.607900 24586 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.630401 17919 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 8, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:24:29.632563 24439 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.610s	user 0.012s	sys 0.000s
I20260709 12:24:29.629248 18098 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:24:29.691458 24439 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:29.634163 24772 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.004s
I20260709 12:24:29.692622 24763 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.553421 24828 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.684900 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:29.634231 24538 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.709s	user 0.012s	sys 0.001s
I20260709 12:24:29.634263 24659 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:29.603214 24829 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:29.635854 24757 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.724s	user 0.013s	sys 0.000s
I20260709 12:24:29.602144 24781 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.636859 24779 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:29.585968 24822 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.318s	user 0.001s	sys 0.000s
W20260709 12:24:29.591527 24818 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.427s	user 0.000s	sys 0.002s
I20260709 12:24:29.749652 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2782) took 1321 ms. Trace:
W20260709 12:24:29.590137 24816 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.421s	user 0.001s	sys 0.000s
W20260709 12:24:29.750705 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.751257 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.751746 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.623281 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:29.754504 24655 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:29.755070 24655 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:29.756024 24655 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.757474 24655 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:29.609856 24631 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.594331 24737 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:29.726215 24194 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.721s	user 0.011s	sys 0.002s
I20260709 12:24:29.591414 24624 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.760617 18021 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:24:29.760272 24194 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.761369 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2790) took 1276 ms. Trace:
I20260709 12:24:29.750159 18170 rpcz_store.cc:276] 0709 12:24:28.428386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.915096 (+486710us) service_pool.cc:224] Handling call
0709 12:24:28.915527 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.915946 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.915978 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:28.916581 (+   603us) log.cc:844] Serialized 26 byte log entry
0709 12:24:29.099144 (+182563us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:29.269636 (+170492us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.299205 (+ 29569us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.299336 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:29.749365 (+450029us) raft_consensus.cc:1685] finished
0709 12:24:29.749394 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.749583 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29062,"prepare.queue_time_us":357293,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":69,"raft.queue_time_us":187066,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":18926,"thread_start_us":272201,"threads_started":2,"wal-append.queue_time_us":277239}
I20260709 12:24:29.761839 18167 rpcz_store.cc:276] 0709 12:24:28.485081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.197250 (+712169us) service_pool.cc:224] Handling call
0709 12:24:29.197736 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.198280 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.198319 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.225013 (+ 26694us) log.cc:844] Serialized 26 byte log entry
0709 12:24:29.474546 (+249533us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:29.638206 (+163660us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.638453 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.638545 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:29.692370 (+ 53825us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b18002d9f40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:29.761066 (+ 68696us) raft_consensus.cc:1685] finished
0709 12:24:29.761194 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.761309 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":138021,"prepare.queue_time_us":65176,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":55,"raft.queue_time_us":89689,"raft.run_cpu_time_us":529,"raft.run_wall_time_us":89922,"spinlock_wait_cycles":61008128,"thread_start_us":285323,"threads_started":2,"wal-append.queue_time_us":330852}
I20260709 12:24:29.763522 24194 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:29.764066 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.764925 24620 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.765161 18019 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:29.765321 24737 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.685552 24776 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.586018 24697 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.767290 24634 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.767866 24697 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.589473 24839 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.769204 18023 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:29.633008 24819 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.371s	user 0.001s	sys 0.001s
I20260709 12:24:29.770622 24781 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.561017 24807 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.771904 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3740) took 1047 ms. Trace:
I20260709 12:24:29.693373 24772 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.694060 18018 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:29.698818 18017 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 22. Ops: [20.13-20.13]
I20260709 12:24:29.701587 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:28.961778 17915 rpcz_store.cc:269] 0709 12:24:02.889318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.893148 (+  3830us) service_pool.cc:224] Handling call
0709 12:24:04.378708 (+1485560us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5400002490. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c135930b2 00007f6c1c50c759 00007f6c1c4a4a76 00007f6c1c475d58 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc
0709 12:24:28.948948 (+24570240us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:21.648246 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:21.658259 (+ 10013us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:21.658337 (+    78us) write_op.cc:435] Acquired schema lock
0709 12:24:21.658372 (+    35us) tablet.cc:662] Decoding operations
0709 12:24:21.661034 (+  2662us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:21.661192 (+   158us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:21.661227 (+    35us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:24:21.663334 (+  2107us) tablet.cc:701] Row locks acquired
0709 12:24:21.663407 (+    73us) write_op.cc:260] PREPARE: finished
0709 12:24:21.663648 (+   241us) write_op.cc:270] Start()
0709 12:24:21.663852 (+   204us) write_op.cc:276] Timestamp: P: 1783599861663561 usec, L: 0
0709 12:24:21.663880 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:24:21.664941 (+  1061us) log.cc:844] Serialized 7795 byte log entry
0709 12:24:22.621748 (+956807us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b
0709 12:24:23.727118 (+1105370us) op_driver.cc:464] REPLICATION: finished
0709 12:24:23.820507 (+ 93389us) write_op.cc:301] APPLY: starting
0709 12:24:23.820672 (+   165us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:24.403494 (+582822us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:24.423631 (+ 20137us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:24.914930 (+491299us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b180049efa0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca557 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:26.324257 (+1409327us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:26.436536 (+112279us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=40,mrs_lookups=0
0709 12:24:26.436604 (+    68us) write_op.cc:312] APPLY: finished
0709 12:24:26.437264 (+   660us) log.cc:844] Serialized 180 byte log entry
0709 12:24:26.775632 (+338368us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:26.807427 (+ 31795us) write_op.cc:454] Released schema lock
0709 12:24:26.833240 (+ 25813us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:26.937547 (+104307us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:26.951467 (+ 13920us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2202197,"reported_tablets":49,"spinlock_wait_cycles":23400832,"child_traces":[["op",{"apply.queue_time_us":92963,"apply.run_cpu_time_us":81897,"apply.run_wall_time_us":3046839,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87847,"num_ops":20,"prepare.queue_time_us":145287,"prepare.run_cpu_time_us":39376,"prepare.run_wall_time_us":1565964,"raft.queue_time_us":209414,"raft.run_cpu_time_us":60815,"raft.run_wall_time_us":113329,"replication_time_us":2063028,"spinlock_wait_cycles":54251136,"thread_start_us":2031631,"threads_started":5,"wal-append.queue_time_us":2028294,"wal-append.run_cpu_time_us":23025,"wal-append.run_wall_time_us":2702078}]]}
I20260709 12:24:29.695958 24828 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.711028 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.721854 18093 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:24:29.804101 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93243.1us
W20260709 12:24:29.725664 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 293017us
I20260709 12:24:29.635810 24723 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:29.808709 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.005s	sys 0.019s
I20260709 12:24:29.743763 24696 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:29.744177 24657 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.616s	user 0.013s	sys 0.000s
I20260709 12:24:29.744788 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2816) took 1190 ms. Trace:
I20260709 12:24:29.822955 18169 rpcz_store.cc:276] 0709 12:24:28.553906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.909329 (+355423us) service_pool.cc:224] Handling call
0709 12:24:29.744535 (+835206us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b540029e0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:29.744712 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1362287872}
I20260709 12:24:29.758239 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1182) took 1024 ms. Trace:
W20260709 12:24:29.824246 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.591238 24750 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.776660 18094 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:29.760110 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.831691 18097 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:29.830247 18016 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.002s	sys 0.026s
W20260709 12:24:29.832432 18016 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.003s	sys 0.026s
I20260709 12:24:29.764935 18167 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:29.693751 17919 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 7 to 8, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 8 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:29.763514 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.834481 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:29.697598 24691 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.091s
I20260709 12:24:29.772356 18090 rpcz_store.cc:276] 0709 12:24:28.724431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.770185 (+ 45754us) service_pool.cc:224] Handling call
0709 12:24:28.770603 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.771031 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.771065 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:28.781974 (+ 10909us) log.cc:844] Serialized 26 byte log entry
0709 12:24:29.029566 (+247592us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:29.083116 (+ 53550us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.083521 (+   405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.083646 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:29.626290 (+542644us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b18003c1e40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:29.632373 (+  6083us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:29.771600 (+139227us) raft_consensus.cc:1685] finished
0709 12:24:29.771652 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.771836 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":54344,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":107111,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":13607,"spinlock_wait_cycles":23837952,"thread_start_us":211883,"threads_started":1,"wal-append.queue_time_us":394041}
I20260709 12:24:29.694902 24538 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.780397 24841 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.791042 24807 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.696326 24722 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:29.570883 18168 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:29.798169 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2803) took 1251 ms. Trace:
W20260709 12:24:29.859243 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.790s	user 0.015s	sys 0.018s
I20260709 12:24:29.859823 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 977) took 1441 ms. Trace:
I20260709 12:24:29.808041 24815 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:29.697216 24693 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.908s	user 0.012s	sys 0.001s
I20260709 12:24:29.809444 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3669) took 1579 ms. Trace:
W20260709 12:24:29.811939 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.005s	sys 0.019s
I20260709 12:24:29.811810 18091 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:29.816543 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:29.720503 24659 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:29.835289 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:29.748558 24836 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.835335 18169 raft_consensus.cc:2368] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:29.839282 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3676) took 1491 ms. Trace:
I20260709 12:24:29.841653 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.824473 18810 rpcz_store.cc:276] 0709 12:24:28.733349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.733479 (+   130us) service_pool.cc:224] Handling call
0709 12:24:28.733871 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.734297 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.734324 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:28.734863 (+   539us) log.cc:844] Serialized 26 byte log entry
0709 12:24:28.859111 (+124248us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:29.003687 (+144576us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.015453 (+ 11766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.015579 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:29.757901 (+742322us) raft_consensus.cc:1685] finished
0709 12:24:29.757930 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.758144 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11419,"prepare.queue_time_us":144570,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":59289,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":4283,"thread_start_us":244297,"threads_started":2,"wal-append.queue_time_us":285357}
I20260709 12:24:29.845662 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3661) took 1621 ms. Trace:
W20260709 12:24:29.847345 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.875514 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.875020 18019 rpcz_store.cc:276] 0709 12:24:28.223798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.025252 (+801454us) service_pool.cc:224] Handling call
0709 12:24:29.074394 (+ 49142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.765642 (+691248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.845313 (+ 79671us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54003476d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:29.845585 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588048128}
I20260709 12:24:29.849660 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:29.879433 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2787) took 1395 ms. Trace:
I20260709 12:24:29.852473 18022 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:24:29.854277 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:29.852488 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2747) took 3055 ms (client timeout 3049 ms). Trace:
I20260709 12:24:29.847450 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:29.858888 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.860313 18168 rpcz_store.cc:276] 0709 12:24:28.418224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.900946 (+482722us) service_pool.cc:224] Handling call
0709 12:24:29.859747 (+958801us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.859388 18093 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:24:29.861091 18813 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.011s	sys 0.008s
W20260709 12:24:29.859864 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.015s	sys 0.018s
I20260709 12:24:29.862672 18171 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:29.863407 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.859748 18173 rpcz_store.cc:276] 0709 12:24:28.546834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.208229 (+661395us) service_pool.cc:224] Handling call
0709 12:24:29.208744 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.256637 (+ 47893us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.797724 (+541087us) spinlock_profiling.cc:243] Waited 47 ms on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:29.798073 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108141952}
I20260709 12:24:29.867182 18023 rpcz_store.cc:276] 0709 12:24:28.230064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.032422 (+802358us) service_pool.cc:224] Handling call
0709 12:24:29.089539 (+ 57117us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.769670 (+680131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.809070 (+ 39400us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:29.809340 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1562581888}
I20260709 12:24:29.865552 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:29.868942 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52577.3us
I20260709 12:24:29.870442 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.871654 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:29.872722 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:29.874339 18016 rpcz_store.cc:276] 0709 12:24:28.347925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.052898 (+704973us) service_pool.cc:224] Handling call
0709 12:24:29.053244 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.053478 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:29.275898 (+222420us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.838687 (+562789us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.838776 (+    89us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:29.838865 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.838975 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.839183 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":377964,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":7733,"raft.queue_time_us":1939,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8707,"thread_start_us":710854,"threads_started":2}
I20260709 12:24:29.851567 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2745) took 3064 ms. Trace:
I20260709 12:24:29.830133 24757 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
W20260709 12:24:29.931337 18813 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.014s	sys 0.010s
I20260709 12:24:29.931550 18098 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:29.882330 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.883426 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:29.937498 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:29.937568 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 102426us
I20260709 12:24:29.845645 17919 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 13 to 12, leader changed from <none> to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 12 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:29.929922 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.932333 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.933545 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.939847 18094 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.899093 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:29.940383 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:29.942188 18095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.005s	sys 0.012s
W20260709 12:24:29.942624 18095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.005s	sys 0.012s
I20260709 12:24:29.941963 17919 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:29.832921 24657 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.947294 18173 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:29.938380 18810 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:29.943255 18017 rpcz_store.cc:276] 0709 12:24:26.787486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.901691 (+2114205us) service_pool.cc:224] Handling call
0709 12:24:28.946272 (+ 44581us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.792751 (+846479us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.851129 (+ 58378us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b54003485d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:29.851475 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1730021248}
I20260709 12:24:29.910035 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:29.949921 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2795) took 1396 ms. Trace:
I20260709 12:24:29.950364 18173 rpcz_store.cc:276] 0709 12:24:28.553757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.946362 (+1392605us) service_pool.cc:224] Handling call
0709 12:24:29.946798 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.949464 (+  2666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.949858 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.947554 18166 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
W20260709 12:24:29.941776 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 872846us
I20260709 12:24:29.947563 24679 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.876319 24693 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:29.954092 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:29.948652 18015 rpcz_store.cc:269] 0709 12:24:26.796938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.337658 (+1540720us) service_pool.cc:224] Handling call
0709 12:24:29.852238 (+1514580us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:29.852405 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443909504}
I20260709 12:24:29.942914 18164 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:29.956202 24693 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.958062 24828 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:29.946686 24737 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.958734 24828 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:29.960206 24696 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:29.961321 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:29.962041 24781 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.875298 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.963191 24841 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.873791 24839 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:29.963994 24839 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.964027 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.964541 24839 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:29.869941 24722 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:29.965389 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2822) took 1386 ms. Trace:
I20260709 12:24:29.965806 18166 rpcz_store.cc:276] 0709 12:24:28.578676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.946754 (+1368078us) service_pool.cc:224] Handling call
0709 12:24:29.947216 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.965058 (+ 17842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.965316 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.966241 24722 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.966956 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.970073 18166 raft_consensus.cc:1217] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Deduplicated request from leader. Original: 13.8->[14.9-14.9]   Dedup: 14.9->[]
I20260709 12:24:29.930233 24338 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:29.970628 18166 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:24:29.971432 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2827) took 1390 ms. Trace:
I20260709 12:24:29.961217 24631 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:29.962793 18099 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:29.972688 24092 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.982342 24092 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:29.983558 24723 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:29.959995 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 110424us
I20260709 12:24:29.984712 24839 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:29.988726 24781 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:29.990835 24781 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:24:29.992738 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60524.8us
I20260709 12:24:29.993646 24808 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.994673 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.973974 18167 raft_consensus.cc:1217] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Deduplicated request from leader. Original: 14.7->[15.8-15.8]   Dedup: 15.8->[]
I20260709 12:24:29.975569 24763 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.995115 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:29.995922 18167 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:24:29.996614 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2829) took 1415 ms. Trace:
I20260709 12:24:29.997290 24831 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:29.990873 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50962.4us
W20260709 12:24:29.999599 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.959995 24828 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.002274 18807 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:30.002415 24687 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:30.004027 24836 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:29.974822 24772 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:29.978211 17919 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:29.995951 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:29.977087 18094 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:30.006244 18091 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:30.006538 18172 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:30.007314 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2818) took 1452 ms. Trace:
I20260709 12:24:29.879940 18165 rpcz_store.cc:276] 0709 12:24:28.484027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.919932 (+435905us) service_pool.cc:224] Handling call
0709 12:24:28.920242 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.567609 (+647367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.871883 (+304274us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b540007c454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:29.879348 (+  7465us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1450113280}
I20260709 12:24:29.880721 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:30.007642 18172 rpcz_store.cc:276] 0709 12:24:28.554480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.947883 (+1393403us) service_pool.cc:224] Handling call
0709 12:24:29.948226 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.007070 (+ 58844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.007257 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.961261 24657 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:30.014007 18169 raft_consensus.cc:2368] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18.
I20260709 12:24:30.013378 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:29.948879 18168 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: [12.10-15.11]
I20260709 12:24:30.015237 18173 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 17. Ops: []
I20260709 12:24:30.016041 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2821) took 1439 ms. Trace:
I20260709 12:24:29.979745 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:29.996953 18167 rpcz_store.cc:276] 0709 12:24:28.581095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.973421 (+1392326us) service_pool.cc:224] Handling call
0709 12:24:29.973835 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.996329 (+ 22494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.996551 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.016571 18168 rpcz_store.cc:276] 0709 12:24:28.576797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.947941 (+1371144us) service_pool.cc:224] Handling call
0709 12:24:29.948570 (+   629us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:30.015573 (+ 67003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.015961 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":615296}
I20260709 12:24:30.008806 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:30.019245 18168 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:30.019790 18172 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:30.020361 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2831) took 1438 ms. Trace:
I20260709 12:24:30.020731 18168 rpcz_store.cc:276] 0709 12:24:28.581695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.018693 (+1436998us) service_pool.cc:224] Handling call
0709 12:24:30.019032 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.020066 (+  1034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.020302 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.016129 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2824) took 1436 ms. Trace:
I20260709 12:24:29.935534 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:29.994376 24092 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:30.021220 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:30.028760 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 300273us
W20260709 12:24:30.030251 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 473092us
I20260709 12:24:30.022150 18168 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:30.021562 18173 rpcz_store.cc:276] 0709 12:24:28.579669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.951575 (+1371906us) service_pool.cc:224] Handling call
0709 12:24:29.951990 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.015868 (+ 63878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.016068 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.033439 17919 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 4 to 7, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 7 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:30.022189 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112372us
I20260709 12:24:30.038899 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:30.008565 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:30.032404 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.023674 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:30.035146 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:30.042124 18165 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:30.035951 18166 rpcz_store.cc:276] 0709 12:24:28.580542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.969516 (+1388974us) service_pool.cc:224] Handling call
0709 12:24:29.969933 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.971133 (+  1200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.971371 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.049274 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2834) took 1466 ms. Trace:
I20260709 12:24:30.053045 18808 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:30.053418 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.055503 18173 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:30.054257 18168 rpcz_store.cc:276] 0709 12:24:28.582618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.021576 (+1438958us) service_pool.cc:224] Handling call
0709 12:24:30.021949 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.034105 (+ 12156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.049187 (+ 15082us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.051153 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:30.057981 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.059136 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.061081 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.074244 18166 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:30.075374 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2838) took 1326 ms. Trace:
I20260709 12:24:30.076504 18166 rpcz_store.cc:276] 0709 12:24:28.749174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.059808 (+1310634us) service_pool.cc:224] Handling call
0709 12:24:30.060199 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.074898 (+ 14699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.075280 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.965392 24538 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:30.081118 18814 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:30.089027 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.081300 18168 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:24:30.092423 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2801) took 1119 ms. Trace:
I20260709 12:24:30.082119 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2836) took 1333 ms. Trace:
I20260709 12:24:30.094281 18173 rpcz_store.cc:276] 0709 12:24:28.748504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.054859 (+1306355us) service_pool.cc:224] Handling call
0709 12:24:30.055272 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.064016 (+  8744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.082011 (+ 17995us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.096458 18172 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:30.082962 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.083439 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.084882 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:30.097689 18813 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:30.097165 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:30.099133 18814 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:30.093003 18168 rpcz_store.cc:276] 0709 12:24:28.973009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.080462 (+1107453us) service_pool.cc:224] Handling call
0709 12:24:30.080977 (+   515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.091990 (+ 11013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.092333 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.100507 18172 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:30.104010 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.078328 24657 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:24:30.110162 24439 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:30.103204 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.079155 24808 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:24:30.047662 24834 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:30.112851 24834 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:30.113126 24092 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:30.113392 24834 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:30.116264 24653 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:30.117535 24653 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:30.091830 24827 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:24:29.964180 24659 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.121272 24659 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:30.123162 24654 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:30.103279 24831 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 12:24:30.124912 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.032457 24843 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:30.054520 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3678) took 1432 ms. Trace:
I20260709 12:24:30.013305 24693 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:30.142383 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.060341 18098 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:30.078940 24723 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:30.131291 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77965.7us
I20260709 12:24:30.060551 24757 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:30.018514 24838 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.085s
I20260709 12:24:30.112874 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.131927 18024 rpcz_store.cc:276] 0709 12:24:28.622083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.046848 (+424765us) service_pool.cc:224] Handling call
0709 12:24:29.084200 (+ 37352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.084539 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:29.415251 (+330712us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.054089 (+638838us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.054157 (+    68us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:30.054213 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.054299 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.054446 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":211318,"prepare.queue_time_us":414563,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":210053,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":16440,"thread_start_us":766943,"threads_started":2,"wal-append.queue_time_us":694161}
W20260709 12:24:30.171244 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58475.6us
I20260709 12:24:30.173197 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:30.148108 24723 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.150576 18172 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:30.158824 18806 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:24:30.049768 24836 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:30.137535 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.102408 24820 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:24:30.062016 24776 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.106s
I20260709 12:24:30.208809 24776 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:30.209457 24776 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:30.145963 24841 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:30.141446 24693 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:30.210727 24776 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:30.212555 24776 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.055696 24740 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.110s
I20260709 12:24:30.141417 24843 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.149490 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.086898 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.165143 24757 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.203912 18806 pending_rounds.cc:85] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 13
I20260709 12:24:30.214258 24763 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:30.214946 24763 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:30.215755 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128989us
I20260709 12:24:30.216255 24763 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:30.206511 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.214502 18806 pending_rounds.cc:107] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:24:30.218175 18806 raft_consensus.cc:2889] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.214304 24757 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:30.207026 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 69650.6us
I20260709 12:24:30.223008 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.223356 24839 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:30.197132 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.212126 24693 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.227109 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:30.213469 24843 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.216818 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.192078 24655 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:30.231314 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:30.233332 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.230765 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3738) took 1522 ms. Trace:
I20260709 12:24:30.205706 24723 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:30.236545 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 185523us
I20260709 12:24:30.234069 18095 rpcz_store.cc:276] 0709 12:24:28.708334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.708673 (+   339us) service_pool.cc:224] Handling call
0709 12:24:28.709087 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.994086 (+284999us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.994140 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.020975 (+ 26835us) log.cc:844] Serialized 26 byte log entry
0709 12:24:29.943160 (+922185us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:29.943204 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.126322 (+183118us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b5400411ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:30.126392 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.225131 (+ 98739us) raft_consensus.cc:1685] finished
0709 12:24:30.225164 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.230672 (+  5508us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471031168,"thread_start_us":797685,"threads_started":1,"wal-append.queue_time_us":1133744}
I20260709 12:24:30.149108 24803 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.205207 24439 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:30.196178 24538 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:24:30.241767 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.228017 24738 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:30.227805 24737 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.243889 18813 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
W20260709 12:24:30.246100 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 96697.7us
I20260709 12:24:30.282795 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:30.247159 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50224.6us
I20260709 12:24:30.233012 24388 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:30.234157 24565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.002s	sys 0.010s
I20260709 12:24:30.191939 24657 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:24:30.293040 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:30.260532 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3662) took 2034 ms. Trace:
I20260709 12:24:30.295364 18022 rpcz_store.cc:276] 0709 12:24:28.226588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.025420 (+798832us) service_pool.cc:224] Handling call
0709 12:24:29.852086 (+826666us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.939748 (+ 87662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.260202 (+320454us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54003485d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:30.260435 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1900086016}
I20260709 12:24:30.261873 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3663) took 2033 ms. Trace:
I20260709 12:24:30.297139 18018 rpcz_store.cc:276] 0709 12:24:28.228672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.027514 (+798842us) service_pool.cc:224] Handling call
0709 12:24:29.069897 (+ 42383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.793717 (+723820us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.261451 (+467734us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54000ff854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:30.261749 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434907904}
I20260709 12:24:30.263163 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2815) took 1709 ms. Trace:
I20260709 12:24:30.256369 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:30.298753 18171 rpcz_store.cc:276] 0709 12:24:28.553621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.242373 (+688752us) service_pool.cc:224] Handling call
0709 12:24:29.860730 (+618357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.898524 (+ 37794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.262866 (+364342us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b54001b3d50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:30.263070 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421025536}
I20260709 12:24:30.267797 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3737) took 1563 ms. Trace:
W20260709 12:24:30.270691 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.277022 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:30.336521 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59723.6us
I20260709 12:24:30.231913 24634 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:30.280313 17989 leader_election.cc:400] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18.
I20260709 12:24:30.359244 17989 leader_election.cc:403] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
W20260709 12:24:30.360873 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101710us
I20260709 12:24:30.284688 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.240038 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:30.362475 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 122511us
W20260709 12:24:30.362783 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78193us
W20260709 12:24:30.286173 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44526us
W20260709 12:24:30.278517 24829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.003s	sys 0.011s
I20260709 12:24:30.244532 24439 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.233119 24194 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:24:30.238261 24849 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:30.282397 24655 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:30.280192 24750 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:30.259399 24827 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:30.318396 24388 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:30.280057 24620 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:30.400532 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.401407 24841 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:30.316855 24835 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:30.404146 24835 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:30.404711 24835 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.327673 24693 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.287777 24847 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:30.288923 24565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.002s	sys 0.010s
I20260709 12:24:30.284580 24836 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:30.287156 24737 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.411525 24737 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:30.335529 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.413645 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.414319 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:30.415076 24763 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:30.367316 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.301658 18099 rpcz_store.cc:276] 0709 12:24:28.704005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.705051 (+  1046us) service_pool.cc:224] Handling call
0709 12:24:30.267409 (+1562358us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:30.267684 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2657659264}
I20260709 12:24:30.415934 18098 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:30.416997 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.369480 18164 raft_consensus.cc:1217] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Deduplicated request from leader. Original: 19.12->[20.13-20.13]   Dedup: 20.13->[]
I20260709 12:24:30.417804 18099 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:30.418164 18164 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:24:30.370481 24092 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:30.371465 24779 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:30.420396 24779 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:30.421540 24779 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:30.382567 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46276us
I20260709 12:24:30.422395 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.370671 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.423684 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.424003 24847 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:24:30.425679 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55109.7us
I20260709 12:24:30.426258 24621 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:30.414620 18093 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:30.427273 24092 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:30.337589 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:30.456619 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 119175us
I20260709 12:24:30.389181 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.415184 18090 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:24:30.434159 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.434099 24092 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:30.428169 24538 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:24:30.446867 18098 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:30.428081 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.497915 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:30.513365 18093 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:30.499312 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.514494 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:30.514799 18020 raft_consensus.cc:2376] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14. Re-sending same reply.
I20260709 12:24:30.512072 18024 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:30.500550 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:30.453052 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.428917 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:30.511776 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:30.519248 18095 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:30.504630 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:30.462919 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 144726us
I20260709 12:24:30.471056 18064 leader_election.cc:400] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:24:30.521479 18064 leader_election.cc:403] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:24:30.375593 24740 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:30.374099 24820 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:30.485384 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 131870us
I20260709 12:24:30.375278 24697 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:30.491739 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.375468 24439 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:30.380698 24831 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:30.494730 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:30.552415 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 125421us
I20260709 12:24:30.382117 24750 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:30.465154 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:30.521157 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:30.571362 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:30.571862 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.572314 18019 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:30.393657 24849 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:30.495321 18099 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:30.496935 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.578680 18017 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:24:30.448875 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2826) took 1868 ms. Trace:
W20260709 12:24:30.525251 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54402.1us
I20260709 12:24:30.579248 18164 rpcz_store.cc:276] 0709 12:24:28.580251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.956808 (+1376557us) service_pool.cc:224] Handling call
0709 12:24:29.957208 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.418632 (+461424us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.448472 (+ 29840us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:30.448780 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":947792128}
I20260709 12:24:30.579958 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.518056 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:30.496100 24621 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:30.512364 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:30.599218 18018 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 11.
I20260709 12:24:30.531028 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:30.515789 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:30.600327 18090 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.15-16.15]
W20260709 12:24:30.538480 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113132us
W20260709 12:24:30.544359 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.573s (SENT) [suppressed 31 similar messages]
W20260709 12:24:30.601686 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57469us
I20260709 12:24:30.392730 24781 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.132s
I20260709 12:24:30.572902 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:30.521752 24827 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:30.582811 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:30.583717 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:30.532518 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.534003 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.611524 18020 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:30.605307 18806 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:30.608400 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:30.540649 24839 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:24:30.613648 18814 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:30.604449 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.571385 24538 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:30.577950 24631 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:24:30.635004 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.661530 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.662479 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.598383 24776 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:30.598662 24831 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:30.685734 18093 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:30.568683 24820 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:30.687144 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 81917.1us
I20260709 12:24:30.647002 24538 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:30.699939 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146211us
I20260709 12:24:30.598824 24737 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260709 12:24:30.669884 24839 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:30.613652 24657 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
W20260709 12:24:30.683933 24834 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.005s
I20260709 12:24:30.715086 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:30.722252 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:30.723546 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 139909us
I20260709 12:24:30.724063 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:30.729595 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 156789us
I20260709 12:24:30.699857 24631 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:30.752126 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:30.758004 18018 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:24:30.760376 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.761475 18019 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:30.773509 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58518.1us
W20260709 12:24:30.775983 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.752321 18064 leader_election.cc:400] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:30.622444 24827 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.697901 24883 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:30.668030 24879 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.105s
I20260709 12:24:30.665364 24843 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:30.766390 24848 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.432s	user 0.000s	sys 0.001s
W20260709 12:24:30.830179 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.785547 24862 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.132s	user 0.001s	sys 0.001s
I20260709 12:24:30.851130 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:30.781109 24696 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:30.729250 24750 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:30.892453 24869 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.272s	user 0.000s	sys 0.000s
I20260709 12:24:30.894161 24806 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:30.894382 24880 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:30.850199 24538 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:30.895478 24868 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:24:30.896143 24836 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:30.898854 24697 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.821710 24849 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:30.901000 24820 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:30.865760 24887 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
W20260709 12:24:30.902220 24857 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.405s	user 0.002s	sys 0.000s
W20260709 12:24:31.051306 24835 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.646s	user 0.012s	sys 0.004s
W20260709 12:24:30.903621 24874 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:24:30.907598 24894 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:30.954324 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.981557 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 229537us
W20260709 12:24:30.986174 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.987617 18811 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:30.856138 24586 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:30.820883 24772 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.100s
I20260709 12:24:30.767865 24834 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:30.994696 24757 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.775s	user 0.011s	sys 0.004s
W20260709 12:24:31.030112 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.031673 18173 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.006s	sys 0.009s
I20260709 12:24:31.034313 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:31.004340 24860 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.476s	user 0.002s	sys 0.000s
W20260709 12:24:30.998724 24858 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.379s	user 0.001s	sys 0.001s
I20260709 12:24:31.041527 18019 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:31.114910 24868 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:31.053061 18810 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:30.996613 24621 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:31.054281 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.170456 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.171402 18171 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:31.055094 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.055500 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.172585 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.054970 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.056968 18169 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:31.174088 18166 raft_consensus.cc:2368] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:24:31.174212 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:31.058851 24697 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.061311 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:30.873586 24877 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.168s	user 0.002s	sys 0.000s
W20260709 12:24:31.054859 24659 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.927s	user 0.011s	sys 0.005s
W20260709 12:24:31.169361 24851 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:24:31.068378 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 278832us
I20260709 12:24:31.075519 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:31.050081 24836 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:31.076686 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.204605 18090 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:24:31.205052 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.205981 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 130618us
I20260709 12:24:30.908461 24831 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:30.919922 24631 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.057690 24750 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:31.093716 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.215701 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.132056 18099 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:31.219118 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:31.220031 18099 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.130153 18017 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:31.122789 18173 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.007s	sys 0.009s
W20260709 12:24:31.071993 24092 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.002s	sys 0.015s
I20260709 12:24:31.132431 18093 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260709 12:24:31.049575 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:31.225464 18093 pending_rounds.cc:85] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 7
W20260709 12:24:31.142858 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.225860 18018 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.146595 24904 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.147423 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2822) took 1123 ms. Trace:
I20260709 12:24:31.149250 24889 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.137764 24586 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.150789 24834 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:31.151962 24882 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:24:31.152644 24827 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:31.152638 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:31.157701 24910 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:31.252614 24895 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:24:31.164654 24843 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.167461 24738 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.053916 24806 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.174798 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.036031 24815 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:31.175716 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1429) took 1279 ms. Trace:
I20260709 12:24:31.180989 18168 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:31.182219 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.234556 24338 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:31.184686 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.050912 24883 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:31.296277 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143735us
I20260709 12:24:31.143157 24722 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.206939 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:31.181542 24621 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.279271 24910 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:31.144533 24757 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.306672 24910 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.308118 24910 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:31.308629 24757 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:31.233402 24092 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.003s	sys 0.015s
I20260709 12:24:31.296214 24865 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:24:31.273869 24889 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.312458 24889 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.072376 24835 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.301525 24828 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:31.313447 24828 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.313802 24828 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:31.314090 18813 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:31.211835 24820 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:31.315042 24883 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:31.296838 24776 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 12:24:31.316516 24776 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:31.316222 24883 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.316915 24776 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:31.262295 18167 rpcz_store.cc:276] 0709 12:24:30.024171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.123884 (+ 99713us) service_pool.cc:224] Handling call
0709 12:24:30.274863 (+150979us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:30.275255 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.375439 (+100184us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:30.399498 (+ 24059us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.686844 (+287346us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.747121 (+ 60277us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.748664 (+  1543us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.749769 (+  1105us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:30.749839 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.749920 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.147157 (+397237us) raft_consensus.cc:1685] finished
0709 12:24:31.147184 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.147351 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":152081,"prepare.queue_time_us":24244,"raft.queue_time_us":584211,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":116952,"spinlock_wait_cycles":275076096,"thread_start_us":144965,"threads_started":2,"wal-append.queue_time_us":198075}
I20260709 12:24:31.183954 18165 raft_consensus.cc:2368] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
W20260709 12:24:31.324337 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 719953us
I20260709 12:24:31.324656 18165 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 20 index: 18. (index mismatch)
I20260709 12:24:31.325467 18778 leader_election.cc:400] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16.
I20260709 12:24:31.326160 24875 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:31.290397 18166 raft_consensus.cc:2368] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:24:31.328804 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.329278 18166 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:31.329625 18164 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:31.291000 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.289849 18812 rpcz_store.cc:276] 0709 12:24:29.896152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.896386 (+   234us) service_pool.cc:224] Handling call
0709 12:24:29.896961 (+   575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.897530 (+   569us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.897575 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.930319 (+ 32744us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.234462 (+304143us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:30.234523 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.234605 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.175415 (+940810us) raft_consensus.cc:1685] finished
0709 12:24:31.175450 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.175636 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33239,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":32535,"thread_start_us":238268,"threads_started":1,"wal-append.queue_time_us":526698}
I20260709 12:24:31.332559 18169 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:31.292330 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2803) took 2318 ms. Trace:
I20260709 12:24:31.333559 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1433) took 1306 ms. Trace:
I20260709 12:24:31.334161 18810 rpcz_store.cc:276] 0709 12:24:30.027261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.027451 (+   190us) service_pool.cc:224] Handling call
0709 12:24:31.333338 (+1305887us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b5400494154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:31.333491 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2240547584}
I20260709 12:24:31.333652 18170 rpcz_store.cc:276] 0709 12:24:28.973482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.087993 (+1114511us) service_pool.cc:224] Handling call
0709 12:24:30.319412 (+231419us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:30.319671 (+   259us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.372191 (+ 52520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.372228 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.372759 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.451608 (+ 78849us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:30.451736 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.451946 (+   210us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.286843 (+834897us) raft_consensus.cc:1685] finished
0709 12:24:31.286878 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.292230 (+  5352us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70224,"raft.queue_time_us":54966,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":8565,"spinlock_wait_cycles":23863680,"thread_start_us":78181,"threads_started":1,"wal-append.queue_time_us":332585}
I20260709 12:24:31.336820 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.289708 24815 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:31.192003 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.297608 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:31.353294 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55773.7us
I20260709 12:24:31.354965 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:31.228314 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2810) took 1664 ms. Trace:
I20260709 12:24:31.303073 24890 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:24:31.359282 18173 rpcz_store.cc:276] 0709 12:24:29.564212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.095603 (+531391us) service_pool.cc:224] Handling call
0709 12:24:30.096198 (+   595us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:30.132097 (+ 35899us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.132133 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:30.147257 (+ 15124us) log.cc:844] Serialized 78 byte log entry
0709 12:24:30.390888 (+243631us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:31.227147 (+836259us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.227696 (+   549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.227879 (+   183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.227949 (+    70us) raft_consensus.cc:1685] finished
0709 12:24:31.227968 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.228219 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":171018,"prepare.queue_time_us":691010,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":27252,"raft.queue_time_us":49093,"raft.run_cpu_time_us":435,"raft.run_wall_time_us":26999,"thread_start_us":904882,"threads_started":2,"wal-append.queue_time_us":432522}
I20260709 12:24:31.255241 24893 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:24:31.317957 24776 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.321137 24624 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:31.260588 24902 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.321810 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1430) took 1389 ms. Trace:
I20260709 12:24:31.335628 24586 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.363219 24902 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.363328 24776 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.363165 18809 rpcz_store.cc:276] 0709 12:24:29.932724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.932897 (+   173us) service_pool.cc:224] Handling call
0709 12:24:29.966282 (+ 33385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.966546 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:30.312057 (+345511us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.321345 (+1009288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:31.321409 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:31.321487 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.321588 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.321739 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":778799,"prepare.queue_time_us":521491,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":779834,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":126,"thread_start_us":575147,"threads_started":2,"wal-append.queue_time_us":360772}
I20260709 12:24:31.366161 24902 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:31.334026 24697 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.273566 24892 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.121s
I20260709 12:24:31.289471 24659 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:24:31.273597 24904 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.293287 24738 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.375097 24827 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:24:31.382742 24657 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:31.409889 24904 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.291018 24836 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.339016 24722 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.583498 24691 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:31.584401 24691 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:31.587666 24723 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.328s	user 0.012s	sys 0.005s
I20260709 12:24:31.587476 24691 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.590925 24691 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:31.592621 24738 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:31.486681 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.599340 24722 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:31.588820 24723 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.602654 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:31.489185 18015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.001s	sys 0.010s
I20260709 12:24:31.605176 24723 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:31.605731 18015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s	user 0.001s	sys 0.010s
I20260709 12:24:31.606098 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.607620 18172 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:31.607451 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 915) took 1074 ms. Trace:
I20260709 12:24:31.192023 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:31.609117 24740 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.058s	user 0.009s	sys 0.009s
I20260709 12:24:31.609576 18019 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:31.609776 24740 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18.
I20260709 12:24:31.611886 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:31.502226 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2777) took 1002 ms. Trace:
I20260709 12:24:31.613173 24803 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.612982 18024 rpcz_store.cc:276] 0709 12:24:30.500006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.535973 (+ 35967us) service_pool.cc:224] Handling call
0709 12:24:30.576623 (+ 40650us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:30.606890 (+ 30267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.606941 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:30.607822 (+   881us) log.cc:844] Serialized 52 byte log entry
0709 12:24:30.791800 (+183978us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:30.860879 (+ 69079us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.864598 (+  3719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.864702 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.418950 (+554248us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b180004af80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1
0709 12:24:31.419526 (+   576us) spinlock_profiling.cc:243] Waited 7.52 ms on lock 0x7b180004af80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:31.501221 (+ 81695us) raft_consensus.cc:1685] finished
0709 12:24:31.501325 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.501919 (+   594us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37803,"prepare.queue_time_us":69095,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":80282,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":17510,"spinlock_wait_cycles":85632256,"thread_start_us":118445,"threads_started":1,"wal-append.queue_time_us":405748}
I20260709 12:24:31.616566 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2772) took 1118 ms. Trace:
I20260709 12:24:31.617081 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:31.292384 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:31.619750 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 327258us
I20260709 12:24:31.225865 18093 pending_rounds.cc:107] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:24:31.521613 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3775) took 1047 ms. Trace:
I20260709 12:24:31.522439 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 2971) took 1154 ms. Trace:
I20260709 12:24:31.522754 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1066) took 1052 ms. Trace:
I20260709 12:24:31.281013 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:31.321097 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114245us
I20260709 12:24:31.332095 18164 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:31.340198 18165 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:31.330664 18166 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:31.293771 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:31.337723 18170 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:31.294683 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.524071 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3771) took 1108 ms. Trace:
I20260709 12:24:31.527700 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:31.529727 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.527835 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 172803us
I20260709 12:24:31.538842 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3766) took 1125 ms. Trace:
W20260709 12:24:31.542635 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 201486us
I20260709 12:24:31.541699 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:31.556952 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.565763 24565 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:31.570037 24654 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.570792 24586 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.571388 24659 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.572911 24631 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.573899 24679 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:31.419835 24835 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.579023 24697 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.579365 24757 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:31.634578 24565 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:31.624289 24657 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:31.582412 24693 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.175s	user 0.015s	sys 0.000s
I20260709 12:24:31.634397 24092 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:31.628254 18024 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:31.435503 24909 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.287s	user 0.003s	sys 0.000s
I20260709 12:24:31.630497 18097 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:31.635262 24586 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.632431 18173 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:31.623904 18810 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:31.630893 18091 rpcz_store.cc:276] 0709 12:24:30.413505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.421978 (+  8473us) service_pool.cc:224] Handling call
0709 12:24:31.538283 (+1116305us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400052154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:31.538556 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2523547904}
I20260709 12:24:31.631256 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1245) took 1111 ms. Trace:
I20260709 12:24:31.638768 18020 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:31.637741 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:31.617112 18023 rpcz_store.cc:276] 0709 12:24:30.498282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.498415 (+   133us) service_pool.cc:224] Handling call
0709 12:24:31.616041 (+1117626us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b540024e354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:31.616312 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2229408512}
I20260709 12:24:31.624600 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:31.632911 24722 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.620734 24538 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.746330 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:31.747648 18091 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:31.749416 24839 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:31.750521 18814 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:31.750859 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:31.752017 24722 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.752749 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:31.754211 24763 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.338s	user 0.019s	sys 0.001s
I20260709 12:24:31.622833 18090 rpcz_store.cc:276] 0709 12:24:30.470596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.601749 (+131153us) service_pool.cc:224] Handling call
0709 12:24:30.616551 (+ 14802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.205194 (+588643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.522110 (+316916us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b5400291b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:31.522508 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1322966528}
I20260709 12:24:31.755880 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1432) took 1740 ms. Trace:
I20260709 12:24:31.756219 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:31.737780 18813 rpcz_store.cc:276] 0709 12:24:30.519557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.520159 (+   602us) service_pool.cc:224] Handling call
0709 12:24:30.520603 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.314585 (+793982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.627799 (+313214us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:31.630942 (+  3143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1824232832}
I20260709 12:24:31.759824 24634 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:31.624142 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:31.760848 24634 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:31.762903 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 138949us
I20260709 12:24:31.763840 24890 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:31.764602 24890 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:31.765015 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:31.640482 24655 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.622445 18093 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:31.639760 24806 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.626209 24620 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:31.577209 18018 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:31.622685 18094 rpcz_store.cc:276] 0709 12:24:30.367708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.367850 (+   142us) service_pool.cc:224] Handling call
0709 12:24:30.436765 (+ 68915us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:30.437058 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.437660 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.437706 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.455543 (+ 17837us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.549663 (+ 94120us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:30.549719 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.549806 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.521648 (+971842us) raft_consensus.cc:1685] finished
0709 12:24:31.521699 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.522138 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":462,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"spinlock_wait_cycles":4987008,"thread_start_us":93513,"threads_started":1,"wal-append.queue_time_us":417657}
I20260709 12:24:31.633930 24826 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:31.608158 18015 rpcz_store.cc:276] 0709 12:24:30.533462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.583446 (+ 49984us) service_pool.cc:224] Handling call
0709 12:24:30.583917 (+   471us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:30.646424 (+ 62507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.646464 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:30.664141 (+ 17677us) log.cc:844] Serialized 52 byte log entry
0709 12:24:31.606707 (+942566us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:31.606813 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.606965 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.607049 (+    84us) raft_consensus.cc:1685] finished
0709 12:24:31.607070 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.607368 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35105,"raft.queue_time_us":88064,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":13917,"thread_start_us":861796,"threads_started":1,"wal-append.queue_time_us":626797}
I20260709 12:24:31.638226 24624 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.634172 24772 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:24:31.636448 24631 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.635125 24654 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.633657 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:31.794261 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:31.731959 24697 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.732689 24565 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.735105 24693 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.798904 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:31.629356 18092 rpcz_store.cc:276] 0709 12:24:30.415072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.536515 (+121443us) service_pool.cc:224] Handling call
0709 12:24:30.537032 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.765999 (+228967us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.766034 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.766683 (+   649us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.048978 (+282295us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:31.049279 (+   301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.522938 (+473659us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b540025e254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:31.523187 (+   249us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.523345 (+   158us) raft_consensus.cc:1685] finished
0709 12:24:31.523368 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.523866 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17971,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":10183,"spinlock_wait_cycles":487734528,"thread_start_us":281905,"threads_started":1,"wal-append.queue_time_us":555597}
I20260709 12:24:31.635825 24659 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.746570 24831 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:31.765128 24917 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:31.406888 24815 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 20.18, Last appended by leader: 18, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.631865 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:31.757459 18090 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:31.766134 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75042.7us
I20260709 12:24:31.749418 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:31.770026 24655 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:31.774623 24864 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.608s	user 0.002s	sys 0.002s
I20260709 12:24:31.622604 18098 rpcz_store.cc:276] 0709 12:24:30.474373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.603122 (+128749us) service_pool.cc:224] Handling call
0709 12:24:30.603467 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.652193 (+ 48726us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:30.990317 (+338124us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.257690 (+267373us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.274184 (+ 16494us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.274353 (+   169us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:31.274390 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.274478 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.520781 (+246303us) raft_consensus.cc:1685] finished
0709 12:24:31.520840 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.521279 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26508,"prepare.queue_time_us":111039,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":1147,"raft.queue_time_us":19960,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":30315,"thread_start_us":505710,"threads_started":2,"wal-append.queue_time_us":304012}
I20260709 12:24:31.789049 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:31.777060 24439 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.194s	user 0.014s	sys 0.005s
I20260709 12:24:31.799731 18019 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:31.808395 18097 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:31.810298 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3695) took 1077 ms. Trace:
I20260709 12:24:31.777432 24620 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:31.831132 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:31.802271 24693 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.794317 24654 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.797279 24697 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.816085 24831 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.805238 24917 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.817411 24659 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.817101 24826 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.817579 24439 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.894526 18091 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:31.818430 18024 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:31.896752 18099 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:31.821430 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3761) took 2121 ms. Trace:
I20260709 12:24:31.770515 24890 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:31.822501 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.822841 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.824072 24631 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:31.816939 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.826771 24835 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:31.830437 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3692) took 1106 ms. Trace:
I20260709 12:24:31.830206 24624 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:31.813702 18018 rpcz_store.cc:276] 0709 12:24:30.725430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.037349 (+311919us) service_pool.cc:224] Handling call
0709 12:24:31.790326 (+752977us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:31.798934 24565 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.840950 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:31.756470 18807 rpcz_store.cc:276] 0709 12:24:30.015844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.016203 (+   359us) service_pool.cc:224] Handling call
0709 12:24:30.016699 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.105553 (+ 88854us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:30.427127 (+321574us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.314898 (+887771us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18004b6ac0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:31.315066 (+   168us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.557950 (+242884us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b2c000fafd8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:31.559106 (+  1156us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.559442 (+   336us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:31.560256 (+   814us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.560655 (+   399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.560870 (+   215us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.755353 (+194483us) raft_consensus.cc:1685] finished
0709 12:24:31.755387 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.755617 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60616,"prepare.queue_time_us":519089,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":134,"raft.queue_time_us":916452,"raft.run_cpu_time_us":561,"raft.run_wall_time_us":561,"spinlock_wait_cycles":84799104,"thread_start_us":330208,"threads_started":2,"wal-append.queue_time_us":296938}
W20260709 12:24:31.816895 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 185380us
W20260709 12:24:31.854147 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66833.7us
I20260709 12:24:31.855832 24620 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.856726 24831 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.860512 24586 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.865787 24892 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:31.867064 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 102313us
I20260709 12:24:31.868434 24836 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:31.802546 24092 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:31.870055 24865 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:31.870460 24657 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:31.871832 24917 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:31.875849 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:31.880576 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2779) took 1379 ms. Trace:
I20260709 12:24:31.896543 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:31.820307 24655 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:31.911947 18173 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
W20260709 12:24:31.915617 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64014.6us
I20260709 12:24:31.915215 18020 rpcz_store.cc:276] 0709 12:24:30.723659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.723791 (+   132us) service_pool.cc:224] Handling call
0709 12:24:31.812169 (+1088378us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:31.830360 (+ 18191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1980523776}
I20260709 12:24:31.896307 24693 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:31.943519 24750 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.726s	user 0.012s	sys 0.010s
I20260709 12:24:31.952621 18019 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:31.897708 18096 rpcz_store.cc:276] 0709 12:24:29.699990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.795847 (+ 95857us) service_pool.cc:224] Handling call
0709 12:24:29.796311 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.798286 (+  1975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.798335 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.801109 (+  2774us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.021111 (+220002us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800086860. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:30.289148 (+268037us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:30.289191 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.289267 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.773253 (+1483986us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:31.820836 (+ 47583us) raft_consensus.cc:1685] finished
0709 12:24:31.820869 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.821170 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3378,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":23134,"spinlock_wait_cycles":91075840,"thread_start_us":267790,"threads_started":1,"wal-append.queue_time_us":1364622}
I20260709 12:24:31.952029 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:31.996362 18018 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:31.999279 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3694) took 1274 ms. Trace:
I20260709 12:24:31.998649 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.003958 18024 rpcz_store.cc:276] 0709 12:24:30.725179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.896164 (+1170985us) service_pool.cc:224] Handling call
0709 12:24:31.999179 (+103015us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:31.999100 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3693) took 1274 ms. Trace:
I20260709 12:24:32.006278 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:32.007398 18020 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:32.009049 24872 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.863s	user 0.005s	sys 0.000s
W20260709 12:24:32.009985 18169 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.010s
I20260709 12:24:32.010536 18169 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:24:32.017982 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71717.9us
I20260709 12:24:32.021338 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.022358 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.023316 18813 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:32.023669 24624 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.025271 18093 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:32.034036 18015 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:32.035352 24826 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:32.037860 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:32.039556 24722 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.970845 24620 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:32.042161 24439 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:32.046892 24776 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.682s	user 0.015s	sys 0.005s
I20260709 12:24:32.047317 24815 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:31.953100 18021 rpcz_store.cc:276] 0709 12:24:30.500694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.542350 (+ 41656us) service_pool.cc:224] Handling call
0709 12:24:30.542909 (+   559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.047520 (+504611us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.047564 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.515546 (+467982us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:31.516861 (+  1315us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.702884 (+186023us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:31.769311 (+ 66427us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.809926 (+ 40615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.834537 (+ 24611us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.879862 (+ 45325us) raft_consensus.cc:1685] finished
0709 12:24:31.880013 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.880221 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50755,"prepare.queue_time_us":86451,"prepare.run_wall_time_us":75,"raft.queue_time_us":108841,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":53023,"spinlock_wait_cycles":119876480,"thread_start_us":657590,"threads_started":2,"wal-append.queue_time_us":322827}
I20260709 12:24:32.047591 24092 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:32.050205 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47294.1us
I20260709 12:24:32.054347 24586 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:32.058694 24722 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.059202 24883 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:32.059978 24883 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.060237 24772 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:32.059819 24722 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.062719 18016 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:24:32.063279 24722 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.063364 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.064301 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3690) took 1342 ms. Trace:
I20260709 12:24:32.064673 18016 rpcz_store.cc:276] 0709 12:24:30.721763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.721937 (+   174us) service_pool.cc:224] Handling call
0709 12:24:32.064022 (+1342085us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:32.064204 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2321785216}
I20260709 12:24:32.066015 24693 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.066223 24750 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:32.066605 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.066802 24693 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.066988 24750 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:32.071426 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111534us
I20260709 12:24:32.071630 24835 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:32.072177 24835 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:32.072330 24654 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:32.072711 24835 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:32.073097 24654 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.071475 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.018024 24892 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.074939 18813 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:32.075775 24654 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:32.076768 18807 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:32.077432 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.009080 18020 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:32.078490 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.078783 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.079541 18813 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:24:32.080037 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3702) took 1064 ms. Trace:
I20260709 12:24:32.069792 24750 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:32.080708 18024 rpcz_store.cc:276] 0709 12:24:31.015380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.078906 (+1063526us) service_pool.cc:224] Handling call
0709 12:24:32.079314 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.079641 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.079672 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:32.079743 (+    71us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:32.079823 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.079878 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.079971 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:32.082979 18024 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
W20260709 12:24:32.083814 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 208018us
I20260709 12:24:32.080156 24892 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.097872 24820 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:32.098476 24820 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:32.098567 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.099913 24865 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.101080 24654 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:32.005685 18018 rpcz_store.cc:276] 0709 12:24:30.724912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.948160 (+1223248us) service_pool.cc:224] Handling call
0709 12:24:31.999010 (+ 50850us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:32.087416 18017 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.780s	user 0.011s	sys 0.019s
W20260709 12:24:32.088093 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.078272 18814 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:32.095233 24883 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:32.100694 24865 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.107072 18092 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:24:32.078799 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3704) took 1021 ms. Trace:
I20260709 12:24:32.100523 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.108243 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:32.111981 18020 rpcz_store.cc:276] 0709 12:24:31.057936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.008211 (+950275us) service_pool.cc:224] Handling call
0709 12:24:32.008755 (+   544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.078482 (+ 69727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.078711 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:32.120131 18016 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:24:32.132953 18173 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:32.112169 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58567.6us
I20260709 12:24:32.117695 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:32.142589 18017 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:31.998871 24634 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:31.949589 24659 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.172s
I20260709 12:24:32.160611 24883 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:32.161281 24883 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:32.161980 24883 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:32.162354 24893 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.158926 24696 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:24:32.160467 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.166327 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.166723 24654 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.168620 24831 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.113699 24750 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
W20260709 12:24:32.161916 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 228411us
I20260709 12:24:32.171605 24875 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:32.171995 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3698) took 1412 ms (client timeout 1639 ms). Trace:
I20260709 12:24:32.172322 24875 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:32.172479 18017 rpcz_store.cc:269] 0709 12:24:30.760455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.033844 (+273389us) service_pool.cc:224] Handling call
0709 12:24:32.171875 (+1138031us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:32.113463 24826 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.174391 24659 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:32.174300 24826 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.174904 24659 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:32.176154 24659 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:32.157382 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71187us
W20260709 12:24:32.159751 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59413.9us
I20260709 12:24:32.177481 24826 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:32.186910 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.151886 24693 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.198122 18016 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:32.169405 24831 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.203828 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.194329 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:32.206235 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:32.207130 18099 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:32.209218 24893 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:32.209985 24828 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.866s	user 0.020s	sys 0.000s
I20260709 12:24:32.210608 24828 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.157385 24776 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:32.218278 18017 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:32.201617 24831 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:32.222854 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62592.1us
I20260709 12:24:32.188521 24931 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:32.228502 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:32.233546 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:32.236199 18018 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:32.173542 24875 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.233683 18173 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:32.245257 18096 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:32.243301 24893 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.247004 24931 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.232553 24538 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.248867 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3710) took 1037 ms. Trace:
W20260709 12:24:32.256562 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62351.2us
I20260709 12:24:32.241875 24634 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:32.267024 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:32.258279 18064 leader_election.cc:400] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:24:32.259370 24654 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:32.274667 24841 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:24:32.259112 24865 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.249881 24904 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.286715 24904 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.288331 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.256047 24931 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:32.253216 24893 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:32.251266 24910 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.862s	user 0.010s	sys 0.011s
I20260709 12:24:32.265704 24439 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:32.277889 18064 leader_election.cc:403] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:24:32.248940 18171 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:32.290688 24828 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:32.274704 24925 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:24:32.291900 24893 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:24:32.292068 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.293174 24931 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.290778 24439 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.294066 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.294660 24750 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:32.296268 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.296505 24439 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.297359 24865 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.289350 24904 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:32.298072 24931 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.293252 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 916) took 1759 ms. Trace:
I20260709 12:24:32.298893 18022 rpcz_store.cc:276] 0709 12:24:30.533885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.583361 (+ 49476us) service_pool.cc:224] Handling call
0709 12:24:30.619168 (+ 35807us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.647543 (+ 28375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.647579 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.664620 (+ 17041us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.507649 (+843029us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b1800254380. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:31.719023 (+211374us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:31.722421 (+  3398us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.738660 (+ 16239us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.252974 (+514314us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:32.292791 (+ 39817us) raft_consensus.cc:1685] finished
0709 12:24:32.292814 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.293016 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54105,"raft.queue_time_us":69623,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":13485,"spinlock_wait_cycles":89430784,"thread_start_us":176954,"threads_started":1,"wal-append.queue_time_us":1396133}
I20260709 12:24:32.296666 24750 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.271975 24826 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:32.305341 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.306526 24826 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:32.306571 18064 leader_election.cc:400] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:32.307215 18064 leader_election.cc:403] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:24:32.308099 24893 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:32.307955 24826 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.18, Last appended by leader: 18, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.286239 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1435) took 2121 ms. Trace:
W20260709 12:24:32.326483 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54144.9us
I20260709 12:24:32.327322 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:32.273574 24659 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:32.270243 18018 rpcz_store.cc:276] 0709 12:24:31.205170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.110755 (+905585us) service_pool.cc:224] Handling call
0709 12:24:32.189142 (+ 78387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.242203 (+ 53061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.242472 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:32.280941 24634 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:32.328333 18092 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:32.349572 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:32.348889 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.352749 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.355624 18173 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:32.327629 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:32.359031 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52704.6us
I20260709 12:24:32.315579 18808 rpcz_store.cc:276] 0709 12:24:30.105134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.119638 (+ 14504us) service_pool.cc:224] Handling call
0709 12:24:30.120099 (+   461us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:30.120612 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.120644 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:30.121339 (+   695us) log.cc:844] Serialized 52 byte log entry
0709 12:24:30.502447 (+381108us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:30.629808 (+127361us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.378099 (+748291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.378240 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.007774 (+629534us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:32.178366 (+170592us) raft_consensus.cc:1685] finished
0709 12:24:32.189757 (+ 11391us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.226370 (+ 36613us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":747772,"prepare.queue_time_us":95190,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":9018,"raft.queue_time_us":752019,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":9627,"spinlock_wait_cycles":75514112,"thread_start_us":506493,"threads_started":2,"wal-append.queue_time_us":939444}
I20260709 12:24:32.363219 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.357070 24722 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:24:32.367096 24696 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.367671 24696 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.368084 24696 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.372658 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:32.373880 24697 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:32.374486 24697 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:32.374711 24928 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.369s	user 0.003s	sys 0.001s
I20260709 12:24:32.374961 24697 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:32.378576 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3706) took 1263 ms. Trace:
I20260709 12:24:32.381474 18170 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 12:24:32.379159 18021 rpcz_store.cc:276] 0709 12:24:31.115663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.048797 (+933134us) service_pool.cc:224] Handling call
0709 12:24:32.049367 (+   570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.050014 (+   647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.050056 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.050655 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.218483 (+167828us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:32.218531 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.218601 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.378255 (+159654us) raft_consensus.cc:1685] finished
0709 12:24:32.378282 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.378461 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":841,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":139,"thread_start_us":167255,"threads_started":1,"wal-append.queue_time_us":325275}
I20260709 12:24:32.390913 24914 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:32.391701 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44679.8us
I20260709 12:24:32.392068 24914 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.385047 18166 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:32.394369 24828 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.394517 24624 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.394948 24828 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.395466 24828 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.395257 24624 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.398233 24624 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:32.399745 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:32.400182 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 133307us
I20260709 12:24:32.408717 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:32.357702 24904 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:32.384088 24910 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:32.409198 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.418054 18166 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:32.390918 24924 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.017s	sys 0.050s
I20260709 12:24:32.418630 18164 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:32.422497 18023 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:32.423910 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:32.421171 24924 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.018s	sys 0.051s
I20260709 12:24:32.430838 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:32.432044 24922 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.337s	user 0.000s	sys 0.004s
I20260709 12:24:32.411041 24914 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.441165 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:32.448390 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:32.434074 24910 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:32.455287 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:32.458418 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:32.460615 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.437258 24631 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:32.461544 18064 leader_election.cc:400] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:24:32.467037 18064 leader_election.cc:403] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:24:32.470721 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:32.479822 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:32.487262 24921 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.416s	user 0.000s	sys 0.003s
I20260709 12:24:32.440363 24904 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:32.494838 24927 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.460s	user 0.004s	sys 0.000s
I20260709 12:24:32.502688 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3778) took 1366 ms. Trace:
W20260709 12:24:32.506975 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87961.6us
I20260709 12:24:32.508445 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:32.503326 18095 rpcz_store.cc:276] 0709 12:24:31.135773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.137054 (+  1281us) service_pool.cc:224] Handling call
0709 12:24:31.138222 (+  1168us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.138678 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.138706 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.146584 (+  7878us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.524145 (+377561us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b18002b36c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:31.774541 (+250396us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:31.774623 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.774741 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.310680 (+535939us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:32.502052 (+191372us) raft_consensus.cc:1685] finished
0709 12:24:32.502090 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.502392 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3723,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"spinlock_wait_cycles":244661376,"thread_start_us":163449,"threads_started":1,"wal-append.queue_time_us":769898}
I20260709 12:24:32.516587 24631 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.517802 24634 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:32.538968 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 233912us
I20260709 12:24:32.521237 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3700) took 1671 ms. Trace:
W20260709 12:24:32.523226 24923 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.495s	user 0.003s	sys 0.000s
I20260709 12:24:32.455153 24621 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:32.525033 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:32.526983 24696 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.528051 24815 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:24:32.530421 24910 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:32.531704 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:32.538153 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:32.539961 18023 rpcz_store.cc:276] 0709 12:24:30.849790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.779073 (+929283us) service_pool.cc:224] Handling call
0709 12:24:32.520804 (+741731us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:32.521121 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1438972800}
I20260709 12:24:32.524713 24624 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:32.501382 24890 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:32.553889 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.559800 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.538923 24655 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:32.561203 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 200017us
I20260709 12:24:32.562121 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.558835 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:32.567688 18018 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:32.575407 18813 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:24:32.576134 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124369us
I20260709 12:24:32.575527 24624 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.586635 24624 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:32.586971 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:32.566751 24634 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:32.580233 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:32.584131 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:32.601966 18810 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:32.604022 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.604918 18166 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:32.606078 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.606729 18166 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:32.610373 24815 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.611719 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:32.612984 24815 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.616604 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:32.618129 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:32.613498 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:32.626461 24815 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.627863 18017 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:32.573871 24565 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:32.640000 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.588018 24655 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:32.628921 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:32.663339 24655 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:32.665433 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.666397 18807 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:32.624788 24893 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:32.627146 24917 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.601s	user 0.016s	sys 0.000s
I20260709 12:24:32.627120 24815 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.574049 24621 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.695076 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:32.702003 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:32.702241 18017 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:32.680029 24893 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:32.685042 24815 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.711479 24865 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.706230 24904 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:32.737361 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:32.738052 24904 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.738607 18023 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:32.737392 24865 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.740535 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:32.747125 24940 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:24:32.741652 18022 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:32.755134 18168 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:32.693670 24931 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:32.749583 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.686480 24917 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:32.761003 18099 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.553s	user 0.015s	sys 0.002s
I20260709 12:24:32.762739 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.714789 24893 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:32.768240 18019 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.035s	sys 0.069s
W20260709 12:24:32.800225 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 260541us
I20260709 12:24:32.739704 24865 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.698161 24565 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.770183 18167 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:24:32.781718 24941 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:24:32.805222 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:32.726663 24836 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.776s	user 0.013s	sys 0.003s
I20260709 12:24:32.824757 18097 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:32.825274 18064 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.818s (SENT)
I20260709 12:24:32.829118 24696 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:32.708223 24835 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.635s	user 0.016s	sys 0.007s
I20260709 12:24:32.807005 24931 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:32.952011 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:32.951234 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:32.964591 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:32.969396 18023 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:32.964416 18019 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.042s	sys 0.076s
I20260709 12:24:32.970244 18099 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
W20260709 12:24:32.950429 24916 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.020s	sys 0.072s
W20260709 12:24:32.854229 24657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s	user 0.010s	sys 0.004s
W20260709 12:24:32.971662 24883 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.809s	user 0.019s	sys 0.000s
I20260709 12:24:32.950717 24948 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.994952 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:32.997177 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 171938us
W20260709 12:24:32.984421 24916 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.928s	user 0.023s	sys 0.074s
I20260709 12:24:33.002804 18099 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:33.003361 24654 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:33.003468 24776 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:33.006400 18091 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
W20260709 12:24:32.986589 24657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.959s	user 0.011s	sys 0.016s
W20260709 12:24:33.011273 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 346031us
I20260709 12:24:33.011662 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.016381 18168 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
W20260709 12:24:33.019845 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 403321us
I20260709 12:24:33.001473 24836 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:33.025300 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:32.967296 24835 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.029071 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.026243 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.031028 18095 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:33.035936 18092 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:33.014286 24722 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:24:33.047377 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 418543us
W20260709 12:24:33.033304 24935 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.018s	sys 0.010s
I20260709 12:24:33.001067 24883 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.054570 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.028687 24836 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.009069 24654 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.062821 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:33.058418 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63635.4us
I20260709 12:24:33.061740 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3715) took 1421 ms. Trace:
I20260709 12:24:33.062459 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:33.052969 24697 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.677s	user 0.019s	sys 0.001s
I20260709 12:24:33.066174 18020 rpcz_store.cc:276] 0709 12:24:31.640776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.162697 (+521921us) service_pool.cc:224] Handling call
0709 12:24:32.163185 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.191634 (+ 28449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.191668 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.206219 (+ 14551us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.437053 (+230834us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:32.437101 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.438032 (+   931us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.061430 (+623398us) raft_consensus.cc:1685] finished
0709 12:24:33.061453 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.061660 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46816,"raft.queue_time_us":21349,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":7025,"thread_start_us":230026,"threads_started":1,"wal-append.queue_time_us":424960}
I20260709 12:24:33.067534 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.037631 24621 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:33.024926 24620 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:24:33.064532 24836 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:33.057220 24917 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:33.056480 24935 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.020s	sys 0.010s
W20260709 12:24:33.080449 24781 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.018s	sys 0.048s
I20260709 12:24:33.085636 18092 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:33.089905 24781 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.019s	sys 0.048s
I20260709 12:24:33.090060 24948 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:33.090198 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.091248 18095 raft_consensus.cc:2410] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:24:33.085944 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.096616 18096 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.078110 24916 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:33.104221 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.106065 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:33.108330 18094 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:33.113399 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2827) took 1908 ms. Trace:
I20260709 12:24:33.116676 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.114035 18171 rpcz_store.cc:276] 0709 12:24:31.199390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.294369 (+ 94979us) service_pool.cc:224] Handling call
0709 12:24:33.098482 (+1804113us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b54003011d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:33.107363 (+  8881us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1344837504}
I20260709 12:24:33.123493 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:33.130227 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.083509 24697 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:33.142151 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61401.1us
I20260709 12:24:33.148483 24697 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:33.143471 24565 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:33.146136 24950 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:24:33.146239 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.146551 24654 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.121582 24826 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:33.139133 24820 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.152228 24820 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:33.152426 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.161985 18166 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: [19.12-19.12]
W20260709 12:24:33.170338 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48725.2us
I20260709 12:24:33.173504 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.176687 18090 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:33.166294 24697 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:24:33.175443 24655 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.509s	user 0.016s	sys 0.000s
I20260709 12:24:33.200217 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.172271 24931 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:33.194836 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78373.1us
I20260709 12:24:33.206399 18096 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.176440 24957 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:33.208638 24697 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:33.220376 24697 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:33.227355 24946 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.190s	user 0.001s	sys 0.001s
I20260709 12:24:33.222177 18094 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:24:33.238056 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.250624 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.236923 24945 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:33.220496 24972 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.260421 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.217859 24835 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.196079 24655 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:33.266283 24949 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.168s	user 0.001s	sys 0.001s
I20260709 12:24:33.279605 18165 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:33.285658 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:33.255417 24815 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:33.289958 24932 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.206s	user 0.002s	sys 0.003s
I20260709 12:24:33.290665 24655 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.291736 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:33.293475 18098 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
W20260709 12:24:33.297979 24962 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.184s	user 0.002s	sys 0.000s
W20260709 12:24:33.283252 24875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.983s	user 0.011s	sys 0.005s
I20260709 12:24:33.307411 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:33.301887 24624 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.700s	user 0.010s	sys 0.006s
I20260709 12:24:33.305616 18811 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:33.308244 18098 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:33.309305 24959 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:24:33.309722 24917 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:33.311034 24828 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.915s	user 0.018s	sys 0.002s
I20260709 12:24:33.270078 24972 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.275753 24945 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.313992 24815 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.314720 18093 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:33.307732 24875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s	user 0.012s	sys 0.005s
W20260709 12:24:33.291781 24659 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.944s	user 0.017s	sys 0.000s
I20260709 12:24:33.285859 24883 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:33.322837 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:33.326942 24974 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:24:33.431520 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97014.8us
I20260709 12:24:33.328006 24836 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.329144 24631 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.329805 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.432438 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.435175 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:33.452451 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:33.352121 24967 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:24:33.368232 24971 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260709 12:24:33.451094 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 422202us
I20260709 12:24:33.462527 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.330842 24917 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:33.437140 24634 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.823s	user 0.012s	sys 0.006s
I20260709 12:24:33.370294 24820 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.323446 24624 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.494643 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1197) took 1735 ms. Trace:
I20260709 12:24:33.440703 24781 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:24:33.501447 24655 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:33.534153 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62706.3us
I20260709 12:24:33.534605 18098 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:33.472126 18092 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:33.353706 24776 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:33.540902 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:33.375510 24828 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:33.575824 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 279693us
I20260709 12:24:33.577535 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.598383 18813 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:33.605370 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.584903 18091 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:33.597522 18169 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:24:33.546967 24892 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:33.623183 18811 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:33.570670 18812 rpcz_store.cc:276] 0709 12:24:31.636528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.663947 (+ 27419us) service_pool.cc:224] Handling call
0709 12:24:31.681947 (+ 18000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.686405 (+  4458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.686455 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.734748 (+ 48293us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.843090 (+108342us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:31.843153 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.843270 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.279720 (+1436450us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:33.371550 (+ 91830us) raft_consensus.cc:1685] finished
0709 12:24:33.371574 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.371846 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56935680,"thread_start_us":107740,"threads_started":1,"wal-append.queue_time_us":348523}
I20260709 12:24:33.599807 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:33.604823 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:33.567983 24972 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.584882 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:33.599334 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:33.585184 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:33.494763 24975 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
W20260709 12:24:33.624864 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.602167 24828 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:33.629829 24865 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.005s
I20260709 12:24:33.600773 24917 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.573992 24659 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:33.603204 24983 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:33.649302 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44540.7us
I20260709 12:24:33.651963 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:33.671749 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.627169 18017 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:24:33.597949 24776 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.676494 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.677770 18166 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.678406 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.675586 18022 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:33.687177 24620 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.687759 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2846) took 1112 ms. Trace:
I20260709 12:24:33.687883 24620 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.689273 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.690689 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.690064 18815 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:33.692701 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:33.581063 24655 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.700459 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:33.688217 18173 rpcz_store.cc:276] 0709 12:24:32.576874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.580266 (+  3392us) service_pool.cc:224] Handling call
0709 12:24:32.614154 (+ 33888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.663108 (+ 48954us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:32.703164 (+ 40056us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.628459 (+925295us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.687328 (+ 58869us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.687392 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:33.687459 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.687546 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.687664 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":462330,"prepare.queue_time_us":43062,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":124,"raft.queue_time_us":423546,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":23941,"thread_start_us":465705,"threads_started":2,"wal-append.queue_time_us":474842}
I20260709 12:24:33.703948 18813 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.622918 24820 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.703449 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:33.691560 24890 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.096s	user 0.016s	sys 0.004s
I20260709 12:24:33.722604 18166 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:33.717666 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.708956 24620 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:33.712080 24988 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.713624 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2782) took 1269 ms. Trace:
I20260709 12:24:33.743229 18018 rpcz_store.cc:276] 0709 12:24:32.443790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.443945 (+   155us) service_pool.cc:224] Handling call
0709 12:24:33.713400 (+1269455us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:33.713553 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254512512}
W20260709 12:24:33.715016 18015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.677s	user 0.021s	sys 0.017s
W20260709 12:24:33.745090 18015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.708s	user 0.021s	sys 0.017s
I20260709 12:24:33.729982 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:33.722262 18167 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:33.736321 18095 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:33.746093 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3705) took 2675 ms. Trace:
I20260709 12:24:33.748275 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:33.749496 18171 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:33.682435 24634 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:33.683424 24659 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:33.748385 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:33.751030 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.755299 18166 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.758134 24687 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:33.755542 24659 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:33.758859 24687 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.761330 24687 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.762287 24781 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:33.764133 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.764972 24781 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:33.766624 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:33.767550 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64229.8us
I20260709 12:24:33.767805 18092 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
W20260709 12:24:33.737185 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 404869us
I20260709 12:24:33.717600 24986 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:33.770895 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:33.749105 24973 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.434s	user 0.000s	sys 0.004s
W20260709 12:24:33.775102 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45274.9us
W20260709 12:24:33.776839 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.685345 24865 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.779084 24958 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.787416 24986 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.788566 24836 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.787503 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1204) took 1201 ms. Trace:
I20260709 12:24:33.789340 18809 rpcz_store.cc:276] 0709 12:24:32.540260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.544557 (+  4297us) service_pool.cc:224] Handling call
0709 12:24:32.556667 (+ 12110us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:33.049233 (+492566us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.049284 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:33.080686 (+ 31402us) log.cc:844] Serialized 52 byte log entry
0709 12:24:33.264843 (+184157us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:33.264902 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.265013 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.741293 (+476280us) raft_consensus.cc:1685] finished
0709 12:24:33.741326 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.741491 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":373177,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":24434,"thread_start_us":638265,"threads_started":2,"wal-append.queue_time_us":222393}
I20260709 12:24:33.748190 24988 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.792369 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.792992 24988 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.796180 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.797215 18092 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 12: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:33.782830 24722 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:33.798794 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.800704 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:33.803272 24776 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.803551 24691 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:33.805374 24655 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.805807 24691 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:33.806313 24691 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.756320 24634 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.806779 24691 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.806912 24634 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.720535 24092 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:24:33.808631 24092 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:33.809113 24092 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:33.806457 18168 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.693007 24983 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.810184 24092 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.811758 24092 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:33.812162 18092 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:33.788242 24958 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.755092 24815 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.826473 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:33.827426 18171 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:33.827889 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:33.815917 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:33.765993 24916 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:33.719014 24953 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:24:33.834466 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.835623 18136 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:33.771081 24917 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:33.778200 24781 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:33.845589 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:33.720789 24985 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.108s
I20260709 12:24:33.862742 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.867503 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.864467 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.878029 18096 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:33.868584 18092 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:24:33.878944 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.556s (SENT)
I20260709 12:24:33.862337 24586 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:33.869566 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:33.880712 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45266.5us
I20260709 12:24:33.879842 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:33.854395 24988 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.852499 24820 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.832588 24841 raft_consensus.cc:2764] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:33.798797 24722 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.898082 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.900725 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.865566 24781 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.871457 24931 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:33.897686 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2861) took 1547 ms. Trace:
I20260709 12:24:33.908215 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.843847 24815 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:33.899334 24893 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.908s	user 0.012s	sys 0.005s
I20260709 12:24:33.915073 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.914465 18165 rpcz_store.cc:276] 0709 12:24:32.350664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.350841 (+   177us) service_pool.cc:224] Handling call
0709 12:24:32.362790 (+ 11949us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.282717 (+919927us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.897356 (+614639us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b5400382f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:33.897612 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2082208000}
I20260709 12:24:33.915892 18166 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:33.917237 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:33.885623 24586 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.893949 24657 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:33.940860 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:33.943029 18171 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:33.907011 24654 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:33.922214 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 2997) took 1718 ms (client timeout 2181 ms). Trace:
I20260709 12:24:33.938778 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:33.867561 24990 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:24:33.945047 18094 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 12:24:33.901898 24865 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:33.947711 18097 rpcz_store.cc:269] 0709 12:24:32.203978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.213508 (+  9530us) service_pool.cc:224] Handling call
0709 12:24:33.921878 (+1708370us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400259754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:33.922119 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1361248640}
I20260709 12:24:33.913566 24781 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.917770 24828 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:33.914417 24931 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:33.959661 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.893031 24917 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.942870 24586 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:33.937624 24820 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:33.921068 24986 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.966636 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:33.970357 18098 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:33.905918 24988 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:33.951207 24883 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.979506 24988 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:33.977373 24984 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.129s
I20260709 12:24:33.983157 24659 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.984354 24958 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:33.954617 24657 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:33.987785 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:34.006551 18167 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:34.008939 24659 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:33.999478 24993 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
W20260709 12:24:34.010448 24977 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.657s	user 0.000s	sys 0.003s
I20260709 12:24:34.011833 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:34.012284 18814 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:33.942652 24893 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.012778 18806 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:34.014834 24893 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.019723 24691 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.999671 24988 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.022209 24988 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.022604 24657 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.024829 24988 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.006309 24883 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:33.964831 24931 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:34.042768 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 184429us
I20260709 12:24:34.043170 18098 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:33.987234 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:34.044524 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57402.6us
I20260709 12:24:33.998944 24092 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:33.988695 18171 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 21 index: 19. (index mismatch)
I20260709 12:24:34.048110 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:34.053349 24092 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:33.990059 24951 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.899s	user 0.004s	sys 0.000s
I20260709 12:24:34.057303 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2848) took 1454 ms. Trace:
W20260709 12:24:34.064056 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.573s (SENT)
I20260709 12:24:33.985751 24983 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.028213 24883 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:34.070348 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:34.070772 24092 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:34.071970 18810 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 18
W20260709 12:24:34.071466 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 226858us
I20260709 12:24:33.989140 24722 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:34.078723 24980 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.533s	user 0.002s	sys 0.000s
I20260709 12:24:33.991564 24917 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.076092 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.074223 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.060810 24984 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:33.988968 24945 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:34.094614 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:33.988963 24836 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.081s
I20260709 12:24:33.994012 24820 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.109915 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.106330 24092 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.064244 18164 rpcz_store.cc:276] 0709 12:24:32.602820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.605953 (+  3133us) service_pool.cc:224] Handling call
0709 12:24:32.606306 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.719202 (+112896us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:33.063603 (+344401us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.542951 (+479348us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.543014 (+    63us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.543066 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.056978 (+513912us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54003025d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:34.057052 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.057224 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":346938,"prepare.queue_time_us":61692,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":37,"raft.queue_time_us":381855,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":17774,"spinlock_wait_cycles":266784384,"thread_start_us":445027,"threads_started":2,"wal-append.queue_time_us":234842}
W20260709 12:24:34.135640 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 92000.4us
W20260709 12:24:34.136595 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.600s (SENT)
I20260709 12:24:34.137756 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:34.139166 24092 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.102775 24984 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:34.150408 18098 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:24:34.145310 18090 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:34.114027 24883 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:34.090507 24586 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.097676 24948 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:24:34.212947 24654 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:34.223783 24586 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.224773 24092 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.157271 24984 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:34.202332 24974 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:24:34.153287 24691 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:34.236701 18810 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:34.249627 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:34.251428 24983 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:34.252297 24983 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:34.253947 24983 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.267005 18098 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:34.275583 18098 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:34.242972 24917 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.305693 24917 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.312561 24820 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.245936 24985 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.059s
W20260709 12:24:34.314716 24996 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 12:24:34.316408 25005 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:24:34.326213 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 189648us
I20260709 12:24:34.312071 25004 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:24:34.328158 18098 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:34.339527 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 263637us
I20260709 12:24:34.383579 18098 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:34.386637 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.390488 24092 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.391382 24092 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:34.393970 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.395071 25004 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:34.395527 25004 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:34.395336 24931 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.396399 25004 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.397789 25004 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:34.399792 24092 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.334427 24917 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:34.381165 24634 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.574s	user 0.010s	sys 0.005s
I20260709 12:24:34.413249 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3701) took 3540 ms. Trace:
I20260709 12:24:34.413898 18019 rpcz_store.cc:276] 0709 12:24:30.873360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.110404 (+1237044us) service_pool.cc:224] Handling call
0709 12:24:32.164057 (+ 53653us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.197881 (+ 33824us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:32.197921 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.224347 (+ 26426us) log.cc:844] Serialized 26 byte log entry
0709 12:24:34.411760 (+2187413us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b180040b880. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1
0709 12:24:34.412038 (+   278us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:34.412101 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.412222 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.412293 (+    71us) raft_consensus.cc:1685] finished
0709 12:24:34.412323 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.413140 (+   817us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37040,"raft.queue_time_us":70299,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":5130,"spinlock_wait_cycles":5771008,"thread_start_us":724721,"threads_started":1,"wal-append.queue_time_us":733428,"wal-append.run_cpu_time_us":4315,"wal-append.run_wall_time_us":1257602}
I20260709 12:24:34.380832 24776 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.385526 24691 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:34.419145 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.397698 24931 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.434172 18098 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:24:34.437981 18098 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:24:34.420538 24691 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:34.441087 18098 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:24:34.417137 24634 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.429939 25015 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:34.447151 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 120051us
I20260709 12:24:34.444000 18098 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:34.453244 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.647s (SENT)
I20260709 12:24:34.454016 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:34.420416 24781 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.437510 25010 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:34.435755 24776 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.467250 25010 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.475212 24953 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:34.476775 25010 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:33.748653 18015 rpcz_store.cc:276] 0709 12:24:31.070492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.035208 (+964716us) service_pool.cc:224] Handling call
0709 12:24:32.035776 (+   568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.036426 (+   650us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.036458 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.037104 (+   646us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.745558 (+1708454us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:33.745629 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.745748 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.745797 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:33.745809 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.746007 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1869,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":199,"thread_start_us":1707933,"threads_started":1,"wal-append.queue_time_us":474986,"wal-append.run_cpu_time_us":4039,"wal-append.run_wall_time_us":1236031}
I20260709 12:24:34.480307 24781 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.480975 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.481874 18811 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:34.482477 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.483162 18810 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:34.481669 25009 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:24:34.472221 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.463022 24948 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:34.478556 24984 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:34.510052 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60447.1us
I20260709 12:24:34.453318 25015 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:34.517206 24978 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.975s	user 0.003s	sys 0.000s
I20260709 12:24:34.483573 25019 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:24:34.528362 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:34.520761 24984 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.528470 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:34.531880 18809 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:34.525450 24983 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.536528 24983 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.541177 24634 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.522046 25001 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:34.542268 25001 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:34.525082 24931 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.543428 25001 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.544343 24931 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.545334 25001 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:34.503228 24691 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.549311 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.550451 18810 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:34.554554 24953 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.555608 24953 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.559520 24776 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.560281 25011 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:34.560787 25011 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.561159 25011 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:34.564016 24953 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.565809 24983 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.567456 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.568450 18812 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:34.569263 25015 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:34.509299 25004 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:24:34.571018 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:34.572613 25004 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:34.573130 25004 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.573522 25004 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.574602 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1074) took 1364 ms. Trace:
I20260709 12:24:34.575126 18095 rpcz_store.cc:276] 0709 12:24:33.210418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.210650 (+   232us) service_pool.cc:224] Handling call
0709 12:24:34.574409 (+1363759us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400410fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:34.574535 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":446871296}
I20260709 12:24:34.561075 24931 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:34.577096 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50939.5us
I20260709 12:24:34.582419 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:34.582777 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:34.470985 24883 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:34.622773 24883 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:34.635843 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:34.636876 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 204433us
I20260709 12:24:34.645942 18809 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:24:34.647989 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.651041 25015 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:34.654064 25015 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:34.667939 24948 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:34.696375 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 113874us
I20260709 12:24:34.698330 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:34.699695 24910 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:34.700294 24910 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:34.701392 24910 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.748843 18815 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:34.760823 25004 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.761067 24953 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:34.765434 25004 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:34.768428 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3788) took 1765 ms. Trace:
I20260709 12:24:34.768954 18093 rpcz_store.cc:276] 0709 12:24:33.003220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.011413 (+  8193us) service_pool.cc:224] Handling call
0709 12:24:34.768197 (+1756784us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400296dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:34.768348 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649622400}
W20260709 12:24:34.778234 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 195531us
I20260709 12:24:34.778482 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:34.782224 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:34.787837 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:34.789072 18813 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.791582 24948 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:34.792125 24948 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:34.792219 24969 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.433s	user 0.000s	sys 0.004s
I20260709 12:24:34.792613 24948 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:34.789865 24841 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:34.787305 25036 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:34.794490 24841 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.794668 25036 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.795114 24841 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.799278 24624 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:34.802860 25036 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:34.819026 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:34.821060 18169 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:34.821712 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:34.825557 18170 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:34.831670 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49466.8us
I20260709 12:24:34.841033 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:34.841921 18095 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.829682 24972 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:34.844575 25037 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 12:24:34.845083 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.845870 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.846690 18093 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:24:34.850564 18098 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:34.850471 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:34.853801 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.868649 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.858464 24654 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:34.869601 18092 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:34.871032 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.871877 18092 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:34.870152 24654 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.880697 18095 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:34.880962 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.882023 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1075) took 1648 ms. Trace:
I20260709 12:24:34.882557 18094 rpcz_store.cc:276] 0709 12:24:33.250826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.256252 (+  5426us) service_pool.cc:224] Handling call
0709 12:24:33.922423 (+666171us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.951373 (+ 28950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.879425 (+928052us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b5400259750. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:34.881921 (+  2496us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1498999168}
I20260709 12:24:34.883579 24654 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:34.878614 25004 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.862950 25036 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:34.887228 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.888965 25036 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:34.890216 25036 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.891659 25036 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:34.894603 24691 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.902324 25011 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.905282 25011 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:34.911917 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.914395 18095 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:34.914705 18097 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:34.868880 24841 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.924299 18807 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:34.926447 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:34.889015 24890 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:24:34.853044 24972 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:34.951576 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:34.958431 18097 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:34.958670 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.1131e+06us
I20260709 12:24:34.958993 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 927) took 1453 ms. Trace:
I20260709 12:24:34.959964 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:34.963618 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:34.964439 18806 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:34.959488 18020 rpcz_store.cc:276] 0709 12:24:33.505940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.541540 (+ 35600us) service_pool.cc:224] Handling call
0709 12:24:33.541976 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.542233 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:33.669842 (+127609us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.851833 (+181991us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.011327 (+159494us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b1800305860. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:34.958356 (+947029us) spinlock_profiling.cc:243] Waited 7.78 ms on lock 0x7b1800305860. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:34.958551 (+   195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.958613 (+    62us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:34.958676 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.958755 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.958919 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2972,"prepare.queue_time_us":171152,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":113,"raft.queue_time_us":3329,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":11644,"spinlock_wait_cycles":25643648,"thread_start_us":269998,"threads_started":2,"wal-append.queue_time_us":278750}
I20260709 12:24:34.913478 25004 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:34.988009 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 928) took 1413 ms. Trace:
I20260709 12:24:34.988806 18023 rpcz_store.cc:276] 0709 12:24:33.575033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.575615 (+   582us) service_pool.cc:224] Handling call
0709 12:24:33.598364 (+ 22749us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.598658 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:33.776918 (+178260us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.987500 (+1210582us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.987592 (+    92us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:34.987651 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.987758 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.987920 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1024814,"prepare.queue_time_us":90832,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"thread_start_us":350746,"threads_started":2,"wal-append.queue_time_us":422528}
W20260709 12:24:34.990613 25045 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:24:34.983443 24890 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.011986 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.014544 18098 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.992892 24914 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:34.998589 24958 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:35.026741 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:35.031483 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71887.4us
I20260709 12:24:35.030591 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.035794 18093 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:35.020949 24776 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.020377 24914 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:35.029321 24916 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:35.017313 24890 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.056851 24653 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.073314 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.056995 25012 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.058080 24634 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:35.084241 25012 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.060622 24776 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.061262 24655 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.086109 24655 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.087016 25012 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:35.060503 24565 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:35.087381 24776 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.087881 24565 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.088389 24655 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.067533 25004 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:35.030470 24958 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.090273 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.090979 24958 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.091221 18172 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
W20260709 12:24:35.091972 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.071291 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.037427 25051 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.093573 25051 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.096575 25051 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.051326 24914 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:35.072937 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.069039 24972 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.074872 24841 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:35.075737 18017 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:35.106961 24565 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:35.076436 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.113345 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.077926 18018 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:35.114256 18815 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:24:35.078997 24916 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.117736 24654 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:35.118391 24654 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.118817 24654 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:35.119314 24916 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.032919 24948 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.121279 18807 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:35.122905 24948 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.082957 24653 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.125573 24653 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.074652 18022 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 20. Ops: []
I20260709 12:24:35.109315 24890 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:35.075877 24815 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.110193 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:35.139056 25047 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.089s	user 0.000s	sys 0.004s
W20260709 12:24:35.140038 25028 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.092s	user 0.005s	sys 0.000s
W20260709 12:24:35.142043 25048 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.091s	user 0.005s	sys 0.000s
I20260709 12:24:35.141997 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.144342 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.144937 18024 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.145206 18022 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.143301 25004 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:35.146920 25004 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:35.147888 25004 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:35.148264 25004 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:35.149179 25004 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.150877 25004 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:35.151569 18018 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:24:35.152215 18018 pending_rounds.cc:85] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 11
I20260709 12:24:35.152652 18018 pending_rounds.cc:107] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:24:35.153041 18018 raft_consensus.cc:2889] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:35.143092 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:35.158490 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.166742 18098 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:35.170145 18806 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:35.175647 24092 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.179296 18813 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:24:35.180915 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.176394 24092 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.183889 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.187280 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.187783 18093 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:35.199244 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.211158 24948 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.211953 24948 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.215198 24948 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:35.215914 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.217293 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.217507 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.218216 18021 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:35.218698 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.219331 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.220175 18022 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.221240 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.222784 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.223696 18023 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.224179 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.225091 18016 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 12:24:35.180737 25050 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:24:35.230520 24914 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.230684 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.232223 18015 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:24:35.233290 24914 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:35.233492 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 910f8c8d787e4abbb092413c1f90ebf3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 17 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:35.218228 18020 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
W20260709 12:24:35.249953 18138 leader_election.cc:336] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.639s (SENT)
I20260709 12:24:35.252864 24092 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:35.254305 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67118.2us
I20260709 12:24:35.206187 24624 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:35.257763 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:35.261941 18015 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:35.278038 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.280386 18024 raft_consensus.cc:2376] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23. Re-sending same reply.
I20260709 12:24:35.282205 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.285435 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.261335 24972 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.274866 24691 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:35.305438 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 273066us
W20260709 12:24:35.305263 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55371.5us
I20260709 12:24:35.310827 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:35.298458 18098 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:35.295418 24624 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:35.316396 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:35.322404 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:35.330551 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.349217 18093 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:35.333765 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:35.356060 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.316409 24691 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:35.317142 24815 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:35.363616 18098 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:35.357448 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.332983 24945 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.375197 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.366113 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:35.376331 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 351966us
I20260709 12:24:35.378085 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:35.350334 24890 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:35.379980 24092 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:35.350069 24883 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.309739 24972 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.383245 18806 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:35.384271 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.351686 25017 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:35.387791 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.361446 25001 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.389858 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.389389 18019 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:35.395557 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.373258 25051 raft_consensus.cc:2764] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:35.400854 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:35.400025 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82834.9us
I20260709 12:24:35.398439 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.402563 18018 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:35.396848 17916 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:35.407455 25053 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:24:35.372912 24815 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:35.388862 24841 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:35.383183 24983 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:35.396332 24691 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.422091 18097 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 18. Preceding OpId from leader: term: 21 index: 19. (index mismatch)
I20260709 12:24:35.409777 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.409498 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.416944 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.437672 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.415560 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.439072 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.364193 24892 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
W20260709 12:24:35.425572 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.389146 25017 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:35.426112 24624 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.426921 25001 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.426450 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.442178 18092 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:35.419591 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.443513 24892 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:35.444867 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.474906 18095 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:35.394408 24776 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:35.450297 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.476382 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.451203 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.452749 18019 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:35.454312 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.455435 18806 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:35.479957 18021 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.456288 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.480896 18091 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:35.385535 24945 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.410090 24841 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.408576 24815 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:35.458228 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:35.456184 18097 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:35.483807 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:35.484412 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55322.9us
I20260709 12:24:35.476039 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.499380 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:35.472049 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.460690 25017 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:35.489144 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:35.506788 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.388278 24883 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.501479 18018 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:35.486462 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67011.8us
I20260709 12:24:35.503939 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.517391 18019 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:35.428795 24983 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:35.509752 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:35.512552 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.512367 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.506335 24836 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:24:35.519723 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:35.557683 18170 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:35.522751 18096 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:35.558531 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.559310 18091 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:35.479204 24892 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:35.440444 24620 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:24:35.524325 25020 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:35.526204 18017 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.529073 24883 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:35.529893 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.580121 18021 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:24:35.514952 24634 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:24:35.506484 24657 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:35.531379 24781 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.598137 24657 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.598369 24781 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.535902 18165 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:35.600955 24781 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:35.533322 24945 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.545646 18093 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:24:35.523758 25052 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:35.500895 24841 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:35.602564 25052 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:35.602972 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.603761 25052 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.604329 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.605041 18093 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:24:35.517776 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4971397969998179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21080000000000002s for a response
I20260709 12:24:35.601583 24781 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.607919 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.590974 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.612702 18017 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:35.573071 17916 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 14 to 16, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 16 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:35.574429 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.574797 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:35.575445 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.584789 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:35.598920 24657 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:35.407738 24654 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.664633 24654 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.608860 24781 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:35.571271 17987 connection.cc:256] RPC call timeout handler was delayed by -1.5174740539996492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
I20260709 12:24:35.635293 18096 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:35.669690 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.637701 18173 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:35.640615 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.673197 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.674397 18093 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:35.641451 24985 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:35.675290 24985 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:35.676597 24985 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.641839 25001 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:35.642410 18166 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:35.681375 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:35.618512 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:35.684525 18091 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:24:35.558061 24983 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:35.609444 25054 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:35.715770 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.732154 18093 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:35.502840 25055 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.108s
W20260709 12:24:35.629416 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.541s (SENT) [suppressed 19 similar messages]
I20260709 12:24:35.680451 24985 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:35.608685 24890 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:35.611907 24836 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:35.583195 25020 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:35.758383 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.752154 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.768923 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.770053 18166 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:35.775259 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.779196 18022 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:35.796296 18099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.007s	sys 0.009s
W20260709 12:24:35.797078 18099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.008s	sys 0.009s
I20260709 12:24:35.799014 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:35.801946 24092 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:35.839118 18019 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:24:35.840674 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.843232 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:35.844434 18019 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:35.845695 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.846609 18019 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:35.847862 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.833796 24092 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.851606 18170 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:35.609107 25056 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.003840 25056 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:35.771718 25020 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:35.910364 24892 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.914367 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:35.918710 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:35.923828 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.012282 18019 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 19 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:24:35.931931 24910 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.938938 24781 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:35.764096 25054 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:35.778308 24890 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:35.992931 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.030714 18022 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:36.032522 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.042689 18170 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:36.049151 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:36.053658 18017 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:24:36.054781 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.037925 24910 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.040448 24892 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.079470 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.080458 18019 raft_consensus.cc:2368] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
W20260709 12:24:36.073858 18018 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.558s	user 0.013s	sys 0.001s
I20260709 12:24:36.083334 18018 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:24:36.086404 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:36.094888 18091 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:36.095950 18021 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:36.098506 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.100006 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:36.069486 25004 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.918s	user 0.014s	sys 0.001s
I20260709 12:24:36.110021 24565 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.111765 24815 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.006s
I20260709 12:24:36.119738 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:36.122774 24657 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:36.122313 24841 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.003s
W20260709 12:24:36.138085 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.011s	sys 0.035s
W20260709 12:24:36.136420 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.039s	sys 0.034s
W20260709 12:24:36.140398 18165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.008s	sys 0.038s
W20260709 12:24:36.184019 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.039s	sys 0.035s
W20260709 12:24:36.185348 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 675653us
I20260709 12:24:36.144812 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:36.147226 24624 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.028s	sys 0.019s
W20260709 12:24:36.151521 24586 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.024s	sys 0.052s
W20260709 12:24:36.155243 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.034s	sys 0.047s
I20260709 12:24:36.164933 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.193518 18019 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:36.165795 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:36.184103 18165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.009s	sys 0.038s
W20260709 12:24:36.145200 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.007s	sys 0.040s
I20260709 12:24:36.193542 25055 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:36.194739 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.034s	sys 0.047s
I20260709 12:24:36.196591 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.195829 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:36.187523 18018 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:24:36.197976 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.198326 24890 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.195334 25055 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.198611 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.198908 18023 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:36.199265 18018 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:36.194556 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.007s	sys 0.040s
I20260709 12:24:36.200421 24657 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:36.200577 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 673605us
I20260709 12:24:36.200940 25055 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.201380 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:36.201771 25017 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.003s
I20260709 12:24:36.198029 18017 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.202355 24657 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:36.202283 25017 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.204775 25017 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:36.205365 24657 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:24:36.196326 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 663547us
W20260709 12:24:36.188126 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.011s	sys 0.035s
W20260709 12:24:36.189399 24624 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.029s	sys 0.019s
I20260709 12:24:36.195536 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.229357 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.232558 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.218581 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.229100 18018 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:24:36.234939 18019 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:36.207958 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.213596 18017 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:36.232389 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.177650 24565 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:36.233012 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 603614us
W20260709 12:24:36.207273 24586 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.025s	sys 0.053s
I20260709 12:24:36.241716 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.245519 18018 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:36.248950 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.250109 18164 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:36.185350 24815 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.207052 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:36.270258 18021 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:36.267961 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.274935 24948 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:36.275491 24948 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:36.275761 24815 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:36.243019 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.276492 24948 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.278156 24948 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:36.278776 18094 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:24:36.283053 25054 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.285181 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:36.286098 18098 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
W20260709 12:24:36.286698 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43748.1us
I20260709 12:24:36.228533 25004 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:24:36.287582 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.288537 25004 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:36.288944 18093 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:24:36.290396 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:36.291366 18092 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:36.291709 24948 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:36.289039 25004 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:36.246855 24624 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:36.277995 24958 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:36.194430 24841 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.269529 25063 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.298055 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3836) took 1252 ms. Trace:
W20260709 12:24:36.299187 25052 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.021s	sys 0.038s
I20260709 12:24:36.299218 24836 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:36.308984 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:36.301106 24958 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:36.304520 24890 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.306033 25069 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:36.311973 18094 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:24:36.361743 18024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.020s	sys 0.005s
W20260709 12:24:36.362792 18024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.021s	sys 0.005s
I20260709 12:24:36.364140 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3746) took 1026 ms. Trace:
I20260709 12:24:36.364758 18024 rpcz_store.cc:276] 0709 12:24:35.337422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.337582 (+   160us) service_pool.cc:224] Handling call
0709 12:24:35.358371 (+ 20789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.358618 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:35.804659 (+446041us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.363492 (+558833us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.363599 (+   107us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:36.363696 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.363812 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.364049 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":708900,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"thread_start_us":977551,"threads_started":2}
W20260709 12:24:36.372924 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.322561 24910 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:36.384063 25017 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:36.323803 18099 rpcz_store.cc:276] 0709 12:24:35.045790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.046465 (+   675us) service_pool.cc:224] Handling call
0709 12:24:35.151981 (+105516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.152229 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:35.152936 (+   707us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.297294 (+1144358us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.297596 (+   302us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.297658 (+    62us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:36.297713 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.297788 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.297980 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":202,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":217,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":52522,"thread_start_us":1143267,"threads_started":2,"wal-append.queue_time_us":911208}
I20260709 12:24:36.385550 24890 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.389694 18814 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:24:36.404945 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:36.323088 25052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.021s	sys 0.038s
W20260709 12:24:36.394963 24691 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.019s	sys 0.007s
I20260709 12:24:36.384959 24815 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.399433 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:36.406706 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:36.321869 25063 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.357383 24836 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.397248 25061 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:24:36.435706 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:36.438140 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:36.422149 18780 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.013s	sys 0.033s
I20260709 12:24:36.377985 25051 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.431104 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:36.405913 24910 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.320168 24958 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.440502 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:36.417307 24691 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.019s	sys 0.007s
W20260709 12:24:36.442301 18780 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.013s	sys 0.033s
I20260709 12:24:36.443249 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:36.443430 24910 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:36.448406 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 545991us
I20260709 12:24:36.450572 24910 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:36.444145 25051 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.451162 24916 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.443251 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.436777 24836 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.432349 24776 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:36.451867 24916 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.452571 24776 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.446914 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.456542 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.436875 24972 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.458628 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.460131 18164 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:24:36.460384 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.461915 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.415618 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3745) took 1119 ms. Trace:
I20260709 12:24:36.414744 25072 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.463327 18164 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:36.461421 24890 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.466207 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:24:36.432565 25063 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.463248 18020 rpcz_store.cc:276] 0709 12:24:35.296732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.342501 (+ 45769us) service_pool.cc:224] Handling call
0709 12:24:35.362911 (+ 20410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.368191 (+  5280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.368227 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.386715 (+ 18488us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.836769 (+450054us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:36.407930 (+571161us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.415039 (+  7109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.415209 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.415277 (+    68us) raft_consensus.cc:1685] finished
0709 12:24:36.415296 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.415527 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11262,"prepare.queue_time_us":310389,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":2427,"raft.queue_time_us":55572,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":10221,"thread_start_us":716482,"threads_started":2,"wal-append.queue_time_us":625595}
I20260709 12:24:36.467541 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.436339 25054 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:24:36.488049 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.489019 18164 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:36.490568 24565 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.494477 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.495574 18164 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:36.495838 25051 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:36.434230 24815 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:36.442420 24983 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.678s	user 0.011s	sys 0.005s
I20260709 12:24:36.498260 24983 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.499402 24815 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:36.500676 25074 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:24:36.494280 25067 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:36.502630 18096 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:36.505476 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.510492 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:36.510663 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:36.511323 18813 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:36.513172 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:36.536631 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:36.541049 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.542038 18164 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:36.542701 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:36.543412 18813 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.501078 24983 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.550493 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:36.551330 18094 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.504297 24841 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:36.553324 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:36.553745 24916 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.437644 25020 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.556856 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.560621 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.562104 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.555727 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:36.565212 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1019) took 1423 ms (client timeout 1733 ms). Trace:
W20260709 12:24:36.565690 18172 rpcz_store.cc:269] 0709 12:24:35.142330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.142783 (+   453us) service_pool.cc:224] Handling call
0709 12:24:36.565003 (+1422220us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540007c454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:36.565136 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2315893632}
I20260709 12:24:36.583230 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3012) took 1072 ms. Trace:
I20260709 12:24:36.583750 18096 rpcz_store.cc:276] 0709 12:24:35.511082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.669224 (+158142us) service_pool.cc:224] Handling call
0709 12:24:36.581453 (+912229us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b54002ca954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:36.583141 (+  1688us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1913051904}
I20260709 12:24:36.565455 18097 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:36.591156 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:36.463377 25004 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.508497 25054 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:36.595033 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.596626 17916 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:36.534744 25051 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:36.598575 18171 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:36.599865 25020 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:36.600979 24985 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.841s	user 0.011s	sys 0.004s
I20260709 12:24:36.602293 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:36.501578 25072 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.605363 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:36.556701 24565 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.603987 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:36.612726 18099 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:36.610690 24916 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:36.582755 24841 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:36.606549 18780 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.504s (SENT)
I20260709 12:24:36.620806 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.603900 25036 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.629307 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.025s	sys 0.026s
I20260709 12:24:36.635815 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:36.636333 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.026s	sys 0.026s
I20260709 12:24:36.637514 18813 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:36.556879 25057 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:24:36.640926 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.607259 25051 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.612432 24565 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:36.650307 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 362792us
I20260709 12:24:36.650583 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:36.624686 24841 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:36.657630 18016 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.017s	sys 0.016s
W20260709 12:24:36.659003 18016 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.017s	sys 0.016s
I20260709 12:24:36.659126 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:36.659461 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.011s	sys 0.026s
I20260709 12:24:36.622179 24983 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:36.650889 24972 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.618630 24910 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:24:36.656584 24565 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.675371 18094 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:36.611601 25054 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:36.705958 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 487436us
W20260709 12:24:36.665102 24914 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.025s	sys 0.012s
W20260709 12:24:36.713383 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54359.5us
I20260709 12:24:36.667416 24697 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:36.714247 18173 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:36.646826 24985 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:24:36.653795 25051 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.708061 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.714881 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3744) took 1417 ms. Trace:
I20260709 12:24:36.656625 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.648993 24916 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:36.662112 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.011s	sys 0.026s
I20260709 12:24:36.661978 24815 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:36.663497 24841 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:36.732309 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:36.713382 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3747) took 1342 ms. Trace:
I20260709 12:24:36.756138 18016 rpcz_store.cc:276] 0709 12:24:35.371437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.371713 (+   276us) service_pool.cc:224] Handling call
0709 12:24:35.386995 (+ 15282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.387246 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:35.798596 (+411350us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.712798 (+914202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.712896 (+    98us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:36.712972 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.713091 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.713284 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53142,"prepare.queue_time_us":683834,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":60,"raft.queue_time_us":76756,"thread_start_us":1255388,"threads_started":2}
I20260709 12:24:36.675949 24883 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.726961 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.742072 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:36.742477 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85960.6us
I20260709 12:24:36.652393 24634 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
W20260709 12:24:36.745014 24092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.003s	sys 0.020s
W20260709 12:24:36.777601 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69733.3us
I20260709 12:24:36.776976 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:36.739851 18015 rpcz_store.cc:276] 0709 12:24:35.276508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.280525 (+  4017us) service_pool.cc:224] Handling call
0709 12:24:35.281046 (+   521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.282607 (+  1561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.282655 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.284760 (+  2105us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.685861 (+401101us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:36.188134 (+502273us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.678138 (+490004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.678264 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.678335 (+    71us) raft_consensus.cc:1685] finished
0709 12:24:36.678347 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.678532 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":457595,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":2189,"raft.queue_time_us":171683,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":33742,"thread_start_us":1295376,"threads_started":3,"wal-append.queue_time_us":650399}
I20260709 12:24:36.646570 25036 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:36.731537 24914 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.026s	sys 0.012s
I20260709 12:24:36.760833 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:36.783154 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.784641 18813 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
I20260709 12:24:36.765628 17916 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 12 to 18, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 18 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:36.766747 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:36.776546 18096 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.793205 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:36.777953 24092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.003s	sys 0.020s
W20260709 12:24:36.748514 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 626775us
I20260709 12:24:36.790120 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:36.781306 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54490.5us
I20260709 12:24:36.743112 25056 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:36.730161 24892 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:24:36.752115 24983 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:36.746163 24697 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:36.767803 24655 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:36.775178 24883 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.785167 24654 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:36.834569 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.778081 25001 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:36.779418 24836 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:36.824693 18165 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:36.826347 18813 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.857391 25052 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:24:36.825143 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:36.826936 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 172236us
I20260709 12:24:36.796845 24691 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:36.839138 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.810264 24815 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:36.803440 24948 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:36.830289 25056 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.825755 24983 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.862264 24691 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.823273 24776 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.884943 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:36.885464 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2864) took 1301 ms (client timeout 1646 ms). Trace:
W20260709 12:24:36.879987 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90022.4us
I20260709 12:24:36.840233 24883 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.902035 18170 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:36.889711 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:36.898792 18169 rpcz_store.cc:269] 0709 12:24:35.584064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.584394 (+   330us) service_pool.cc:224] Handling call
0709 12:24:36.870691 (+1286297us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54000505d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:36.885385 (+ 14694us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2725009664}
I20260709 12:24:36.878207 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:36.838629 24654 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.896513 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.901597 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:36.906952 18813 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:36.829272 24697 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:36.774863 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:36.880872 25020 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:36.911324 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:36.890221 25056 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:36.905418 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122370us
I20260709 12:24:36.889863 24890 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:36.893693 24983 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:36.895301 24948 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:36.937945 18198 heartbeater.cc:646] Failed to heartbeat to 127.16.102.62:42293 (5 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.102.62:42293 timed out after 15.000s (SENT)
I20260709 12:24:36.938555 24948 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:36.940588 25080 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:24:36.895911 24910 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.926936 25054 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:36.927523 25078 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 12:24:36.929000 25036 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:36.903753 25086 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:36.971637 25086 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.898358 24916 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:36.908540 24654 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.898506 24691 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.898729 25057 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:36.949760 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.936331 24890 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:36.998378 18172 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:36.951916 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59384.2us
I20260709 12:24:36.880589 24655 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:24:36.946534 18170 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:36.912495 24697 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:37.001291 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:36.911270 25069 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:24:36.935973 25020 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:36.969190 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:36.970055 24620 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:36.935298 24836 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.936524 25056 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.924683 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:36.934433 25072 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:37.018477 18171 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:37.019815 25090 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:24:37.007697 24890 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:37.042646 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 118070us
I20260709 12:24:36.974516 25086 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.037209 24781 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:37.048019 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 207500us
I20260709 12:24:36.960129 24910 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.059450 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:37.036832 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 235346us
I20260709 12:24:37.052183 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:37.003443 25057 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:37.050150 25054 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:37.029671 24836 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:37.088903 18023 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:37.089545 18023 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:37.037374 25082 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:37.038584 24620 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.096408 25020 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.096800 24620 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.080673 24815 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:37.097926 24815 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:37.098258 25101 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:24:37.098492 24815 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:37.059065 24781 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.100666 25082 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.091164 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:37.103853 25063 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.612s	user 0.008s	sys 0.006s
I20260709 12:24:37.104483 25063 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.105055 24958 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:37.050807 25096 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:37.068500 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.066246 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:37.059197 25087 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
W20260709 12:24:37.111881 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45802.3us
W20260709 12:24:37.113320 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45150.7us
I20260709 12:24:37.114593 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.083621 24586 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:37.040575 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:37.015057 25102 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.073s
I20260709 12:24:37.105763 18167 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:37.090663 18023 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.107059 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:37.118250 18023 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:37.116298 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:37.054150 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 256910us
W20260709 12:24:37.116258 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 202461us
I20260709 12:24:37.124060 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.093884 24836 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.121733 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.131283 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.118459 25070 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:37.126113 25106 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:37.130425 24914 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:37.135840 25109 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:37.140049 25070 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:37.176048 25070 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:37.142273 24776 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:37.178848 24776 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.151515 24654 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.112247 24653 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:37.191325 24910 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.194487 18165 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:37.205806 18096 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:37.133047 24781 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.134013 25088 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:24:37.215909 25096 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:37.199584 25099 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:24:37.130950 25054 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:37.159435 25095 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 12:24:37.232611 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 258318us
I20260709 12:24:37.167474 25063 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:37.130640 25113 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:37.237370 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:37.129868 25072 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.102s
W20260709 12:24:37.241739 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 212783us
I20260709 12:24:37.230578 24586 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:24:37.248549 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 117368us
I20260709 12:24:37.250818 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.254714 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:37.248778 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:37.262109 18023 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:37.263336 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3764) took 1759 ms. Trace:
I20260709 12:24:37.263799 18023 rpcz_store.cc:276] 0709 12:24:35.503632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.200736 (+697104us) service_pool.cc:224] Handling call
0709 12:24:36.201057 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.088747 (+887690us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b5400540720. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662edc0 00007f6c166b91c0 00007f6c166ba231 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c
0709 12:24:37.262734 (+173987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.263261 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1358963456}
I20260709 12:24:37.265509 25063 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:37.268669 18165 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:37.272033 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:37.272383 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:37.271013 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:37.282648 18169 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:37.273936 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.286177 25109 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:37.285755 24697 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.288229 25084 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:37.266371 24910 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:37.160795 24815 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.289733 24634 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:37.266412 25108 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:37.292136 24634 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:37.278224 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.296505 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.297498 18091 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:37.301081 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.302284 18092 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:37.250293 24776 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.310716 24634 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.312134 24634 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.312311 24815 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.264616 25004 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:24:37.314638 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 193047us
I20260709 12:24:37.315826 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.317267 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:37.318009 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.257735 25072 consensus_queue.cc:1243] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:37.252807 25107 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:37.319664 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47745.6us
I20260709 12:24:37.326911 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:37.330216 24776 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:37.332857 24565 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.586s	user 0.014s	sys 0.000s
I20260709 12:24:37.333425 24565 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.334566 17916 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 11 to 15, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:37.342262 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.341327 25055 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:37.353018 25103 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260709 12:24:37.354395 25104 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:24:37.356021 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2876) took 1130 ms. Trace:
I20260709 12:24:37.334048 25004 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:37.356580 18166 rpcz_store.cc:276] 0709 12:24:36.225443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.226870 (+  1427us) service_pool.cc:224] Handling call
0709 12:24:36.227416 (+   546us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:36.234825 (+  7409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.234861 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:36.242375 (+  7514us) log.cc:844] Serialized 52 byte log entry
0709 12:24:36.776052 (+533677us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:36.776098 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.776185 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.355689 (+579504us) raft_consensus.cc:1685] finished
0709 12:24:37.355714 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.355921 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":568,"raft.queue_time_us":1250,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7707,"thread_start_us":480472,"threads_started":1,"wal-append.queue_time_us":709664}
I20260709 12:24:37.218211 25057 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.306813 25082 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.331182 24776 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.294441 25086 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.092s
I20260709 12:24:37.277084 24985 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.122s
I20260709 12:24:37.343183 18093 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:24:37.591480 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 264622us
I20260709 12:24:37.592305 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:37.455287 18096 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:37.406306 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.333969 25118 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:37.602847 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:37.501026 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:37.499666 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:37.437907 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:37.590253 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 263495us
I20260709 12:24:37.602174 18165 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:37.433434 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.322422 24910 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:37.333448 24972 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.628s	user 0.015s	sys 0.000s
W20260709 12:24:37.564872 24841 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.787s	user 0.011s	sys 0.004s
W20260709 12:24:37.495105 24883 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.575s	user 0.012s	sys 0.000s
W20260709 12:24:37.487416 24983 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.533s	user 0.007s	sys 0.007s
I20260709 12:24:37.635001 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:37.433759 25111 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.335s	user 0.004s	sys 0.001s
I20260709 12:24:37.439844 24697 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.642640 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:37.644366 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.341799 25088 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:37.403384 25051 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.654s	user 0.014s	sys 0.000s
I20260709 12:24:37.335201 25113 raft_consensus.cc:2764] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:24:37.647253 18096 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:37.648725 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.437718 25120 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.678855 25120 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.681161 25122 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:37.496376 25072 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.603833 24948 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.400281 25054 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:37.654979 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:37.659345 25056 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.604s	user 0.013s	sys 0.000s
I20260709 12:24:37.690651 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:37.690804 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55903.9us
I20260709 12:24:37.691459 18093 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:37.432722 24691 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.401708 24890 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.710201 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.711898 24890 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.674935 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:37.642679 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.651405 24972 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:37.360741 24654 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:37.320004 24836 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.731516 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:37.695129 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:37.698598 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43732.8us
I20260709 12:24:37.699380 25063 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:37.683305 25054 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.681936 25122 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.701541 24691 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:37.756565 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:37.705008 25092 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:37.681777 25120 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.659550 24655 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.719508 18023 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:37.676868 25051 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.737368 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.736083 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.749642 25063 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:37.675364 25004 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:37.745712 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 103139us
I20260709 12:24:37.729836 25106 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:37.750154 18098 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:37.760434 25092 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:37.751297 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.701229 24910 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.757395 18806 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:37.653644 24841 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.662130 24983 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.761215 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:37.661156 24883 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.831135 24983 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.832144 18099 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:37.832300 24883 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:37.763120 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:37.637102 25118 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.831156 24841 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:37.765313 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146267us
I20260709 12:24:37.833959 18173 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:37.834990 25118 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.835723 24836 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:37.773144 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 168303us
I20260709 12:24:37.740708 25094 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.839072 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:37.839821 18170 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.839463 25094 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.845232 18168 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:37.784531 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.784914 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:37.682442 24948 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.742952 25089 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:37.719770 18093 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:37.682156 25072 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.757488 25113 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:37.715304 25056 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.855235 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:37.856077 25072 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.856978 25056 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:37.794464 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106852us
I20260709 12:24:37.802139 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:37.804111 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:37.750694 18807 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:37.798548 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:37.805562 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 169644us
I20260709 12:24:37.810973 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:37.860400 18098 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.807608 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:37.861492 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:37.791499 25105 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:37.816493 24776 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.817552 25106 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.798893 24565 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.818135 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:37.819175 25092 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.799045 24655 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:37.824597 25070 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.648s	user 0.013s	sys 0.000s
I20260709 12:24:37.827559 18806 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:24:37.730854 25012 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:37.846985 18810 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:24:37.857391 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.884266 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.884791 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.893504 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:37.897413 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98925us
I20260709 12:24:37.898365 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:37.944717 18090 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.943141 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:37.894083 24836 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:37.909842 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45815.4us
I20260709 12:24:37.891184 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:37.951776 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60749.9us
I20260709 12:24:37.907289 18170 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:37.919461 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:37.910573 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64769.7us
I20260709 12:24:37.926451 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:37.839126 25122 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:37.947366 18093 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:37.948192 18018 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:37.924921 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:37.910559 25105 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:37.951979 18807 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:37.953464 18022 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:37.959227 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:37.960232 25105 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.911006 25072 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.957247 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:37.921795 18169 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:37.960562 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:37.961356 25072 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:37.962062 18018 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.963852 25072 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:37.909497 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:37.966691 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:37.967993 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.968789 18091 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:37.971951 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.976358 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:37.977851 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:37.954562 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:37.969620 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:37.918169 24985 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:37.984534 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:37.922080 24972 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:37.980537 18091 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:37.982518 18018 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:37.991705 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:37.923923 24776 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:37.925623 25056 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:37.996035 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.996387 17916 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 14 to 16, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 16 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:37.996791 18098 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:38.000049 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:37.994499 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:38.002202 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:38.003769 18091 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:37.917749 24657 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:24:38.012023 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:38.015748 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:38.031739 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.991277 24697 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:37.983549 24836 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.041978 18091 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:37.992738 24972 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.900501 25089 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:38.048689 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54300us
W20260709 12:24:38.050668 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97612.1us
I20260709 12:24:37.995437 25092 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.999217 24985 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:37.944429 25070 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.010054 24691 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:38.076848 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.086488 18015 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:38.044493 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 95151.1us
I20260709 12:24:38.084415 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.089684 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:38.090484 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.091204 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.092391 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:38.093357 18172 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.109709 18815 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:38.072958 24972 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.064625 25105 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.111629 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:38.111661 18064 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.541s (SENT)
I20260709 12:24:38.122136 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:38.112468 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.108155 18814 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:38.119401 18020 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:38.013105 24776 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.075050 24697 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:38.085701 24691 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.042239 25056 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.110381 25147 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:38.155779 18093 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.144083 24691 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:38.171881 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.173271 18023 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:38.139451 25012 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:38.174758 18172 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:38.174664 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.008936 25063 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.177520 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.178674 18165 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:38.154208 25051 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:38.158638 25147 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.135269 25105 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:38.186820 25114 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.923s	user 0.000s	sys 0.014s
W20260709 12:24:38.183226 25137 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:24:38.189354 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:38.204558 18093 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:38.222096 24697 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.226046 24655 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:38.240290 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.196179 25147 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:38.195714 24948 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:38.251030 25142 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:24:38.259564 25130 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
W20260709 12:24:38.260511 25133 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.198s	user 0.000s	sys 0.002s
W20260709 12:24:38.306521 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.307482 25094 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:38.309443 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2883) took 1425 ms. Trace:
I20260709 12:24:38.309959 18173 rpcz_store.cc:276] 0709 12:24:36.883636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.885225 (+  1589us) service_pool.cc:224] Handling call
0709 12:24:38.309153 (+1423928us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b540007bf54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:38.309358 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2131467520}
I20260709 12:24:38.312974 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3025) took 1032 ms. Trace:
I20260709 12:24:38.314603 25083 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:38.315176 25083 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:38.315688 25115 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.050s	user 0.006s	sys 0.007s
I20260709 12:24:38.316293 24653 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:38.316747 24653 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:38.317255 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3862) took 1383 ms. Trace:
I20260709 12:24:38.318384 24836 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.317878 24653 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.318969 24836 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.319521 24653 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.319489 24836 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.322338 18023 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.322719 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:38.324170 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:38.326041 25084 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:38.326480 25084 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:38.326844 25084 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.328048 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.329485 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.332978 25055 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:38.242209 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:38.334379 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92389.9us
I20260709 12:24:38.334486 25108 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:38.335023 25108 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.187125 25140 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:24:38.335386 25108 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:38.286459 24565 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.336822 25140 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:38.337239 25140 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:38.337090 24565 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.288866 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:38.289968 18807 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:38.290966 25150 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:38.317752 18095 rpcz_store.cc:276] 0709 12:24:36.933774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.938300 (+  4526us) service_pool.cc:224] Handling call
0709 12:24:36.939818 (+  1518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.958354 (+ 18536us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:36.958410 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:37.010383 (+ 51973us) log.cc:844] Serialized 26 byte log entry
0709 12:24:37.130962 (+120579us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:37.131013 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.131101 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:38.314730 (+1183629us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:38.316884 (+  2154us) raft_consensus.cc:1685] finished
0709 12:24:38.316915 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.317180 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23731,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":27023,"spinlock_wait_cycles":20580736,"thread_start_us":72901,"threads_started":1,"wal-append.queue_time_us":225924}
I20260709 12:24:38.292781 25012 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.183722 24985 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.295787 25070 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.364120 18092 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:24:38.297415 25056 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.300242 25105 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.301306 25089 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.339799 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.313504 18099 rpcz_store.cc:276] 0709 12:24:37.281118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.281334 (+   216us) service_pool.cc:224] Handling call
0709 12:24:37.281876 (+   542us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.832679 (+550803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.312591 (+479912us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54004119d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:38.312880 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1264902528}
W20260709 12:24:38.343616 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 254982us
I20260709 12:24:38.294916 24776 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:38.244725 25148 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:38.347704 25083 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.405649 25148 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.406220 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:38.406600 25083 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.406981 18806 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:38.409011 25148 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.409253 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:38.351493 18090 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:38.341233 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:38.410149 25145 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.370535 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.410804 18808 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:38.410905 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69779.2us
I20260709 12:24:38.379755 24565 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:38.384423 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.384672 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.413259 18016 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.385412 25112 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:38.285775 24948 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:38.386938 17919 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 278) took 69291 ms (client timeout 15000 ms). Trace:
I20260709 12:24:38.390321 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:38.417104 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.401254 18015 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:38.429076 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:38.422998 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.424439 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.432844 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:38.433683 18017 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:24:38.431612 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.435855 17916 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:38.439092 18015 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:38.447602 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.444399 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.370074 24654 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:38.446697 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:38.451561 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:38.455590 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.455005 17916 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 20, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 20 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:38.418586 25140 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.467900 25140 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:38.461000 25055 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
W20260709 12:24:38.455139 25146 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:38.471603 25055 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:38.474714 17916 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 8 to 15. New cstate: current_term: 15 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:38.418459 25112 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:38.475814 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.421011 24836 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.434665 25109 raft_consensus.cc:2764] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 12:24:38.450770 25088 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.774s	user 0.012s	sys 0.002s
I20260709 12:24:38.435029 25089 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:38.478653 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:38.455073 24634 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.142s	user 0.015s	sys 0.003s
I20260709 12:24:38.441092 25102 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:38.481619 17916 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 11 to 15, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 15 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:38.440627 24948 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.469022 24972 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.455662 25113 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:38.478446 25151 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
W20260709 12:24:38.515766 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.541652 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.542309 18017 raft_consensus.cc:2368] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:38.543512 18170 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:38.523733 18016 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:38.475242 25094 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:38.555893 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104455us
I20260709 12:24:38.554359 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.478315 25055 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.550288 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:38.561802 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.557156 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:38.566895 17916 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 10 to 13, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 13 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:38.535653 25088 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:38.672835 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 197229us
W20260709 12:24:38.695115 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.786947 18016 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:38.787109 25112 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.789199 25055 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:38.790791 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 234067us
I20260709 12:24:38.841485 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.842614 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.842711 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.448267 25108 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.794931 25069 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:38.794720 25001 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:38.845701 25069 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.798931 18022 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:38.838871 25102 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:38.424557 17919 rpcz_store.cc:269] 0709 12:23:29.095712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.096155 (+   443us) service_pool.cc:224] Handling call
0709 12:24:38.386664 (+69290509us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:31.070097 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:31.154592 (+ 84495us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:31.154642 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:24:31.154663 (+    21us) tablet.cc:662] Decoding operations
0709 12:24:31.156027 (+  1364us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:31.156128 (+   101us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:31.156162 (+    34us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:24:31.377476 (+221314us) tablet.cc:701] Row locks acquired
0709 12:24:31.377510 (+    34us) write_op.cc:260] PREPARE: finished
0709 12:24:31.396078 (+ 18568us) write_op.cc:270] Start()
0709 12:24:31.403652 (+  7574us) write_op.cc:276] Timestamp: P: 1783599871392959 usec, L: 0
0709 12:24:31.403680 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:24:31.423050 (+ 19370us) log.cc:844] Serialized 5243 byte log entry
0709 12:24:32.497159 (+1074109us) op_driver.cc:464] REPLICATION: finished
0709 12:24:32.803421 (+306262us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1a0edef0
0709 12:24:33.226311 (+422890us) write_op.cc:301] APPLY: starting
0709 12:24:33.250018 (+ 23707us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:33.356381 (+106363us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:33.356448 (+    67us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:33.804780 (+448332us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b18002a7480. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c163f60ec 00007f6c16442c0a 00007f6c164424f7 00007f6c16442490 00007f6c163f9f48 00007f6c1a10e1bb 00007f6c1a1555d4 00007f6c1a0018da 00007f6c1a0043cb
0709 12:24:34.015588 (+210808us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:34.016095 (+   507us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 12:24:34.016175 (+    80us) write_op.cc:312] APPLY: finished
0709 12:24:34.016767 (+   592us) log.cc:844] Serialized 121 byte log entry
0709 12:24:34.153728 (+136961us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:34.218207 (+ 64479us) write_op.cc:454] Released schema lock
0709 12:24:34.218351 (+   144us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:34.219219 (+   868us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:34.219272 (+    53us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1204155,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":694449,"apply.run_cpu_time_us":54978,"apply.run_wall_time_us":992891,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58545,"num_ops":13,"prepare.queue_time_us":761578,"prepare.run_cpu_time_us":31411,"prepare.run_wall_time_us":716866,"raft.queue_time_us":676358,"raft.run_cpu_time_us":31935,"raft.run_wall_time_us":649456,"replication_time_us":1074137,"spinlock_wait_cycles":82516992,"thread_start_us":1798601,"threads_started":5,"wal-append.queue_time_us":529637,"wal-append.run_cpu_time_us":72608,"wal-append.run_wall_time_us":2678222}]]}
I20260709 12:24:38.800117 25147 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.847080 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.804140 25004 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:38.839437 24781 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:38.848326 25069 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:38.804908 25158 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.416s	user 0.001s	sys 0.000s
I20260709 12:24:38.860946 25004 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.483381 24841 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
I20260709 12:24:38.806533 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:38.863724 25004 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:38.807024 24948 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:38.810179 25138 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.739s	user 0.000s	sys 0.005s
I20260709 12:24:38.900243 25057 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.901844 25112 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:38.904250 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3781) took 1003 ms. Trace:
I20260709 12:24:38.904711 18022 rpcz_store.cc:276] 0709 12:24:37.900949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.960262 (+ 59313us) service_pool.cc:224] Handling call
0709 12:24:38.904012 (+943750us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:38.904172 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1328850688}
I20260709 12:24:38.905138 25147 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:38.905965 18019 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.014s	sys 0.018s
W20260709 12:24:38.906376 18019 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.014s	sys 0.018s
I20260709 12:24:38.906338 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.907306 18022 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:38.907574 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3777) took 1007 ms. Trace:
I20260709 12:24:38.908437 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.908008 18019 rpcz_store.cc:276] 0709 12:24:37.899796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.283126 (+383330us) service_pool.cc:224] Handling call
0709 12:24:38.283434 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.283684 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:38.338596 (+ 54912us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.907094 (+568498us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:38.907164 (+    70us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:38.907233 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.907318 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.907493 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":83866,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":1209,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":568093,"threads_started":1,"wal-append.queue_time_us":568211}
I20260709 12:24:38.897298 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.814494 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.912278 18024 raft_consensus.cc:2368] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:38.804920 18810 raft_consensus.cc:2368] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18.
W20260709 12:24:38.837000 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 280491us
I20260709 12:24:38.841662 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:38.791641 24972 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:38.845762 18015 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:38.916795 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.847368 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.848901 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2886) took 1964 ms. Trace:
I20260709 12:24:38.918516 18023 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:38.918203 18168 rpcz_store.cc:276] 0709 12:24:36.884875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.885582 (+   707us) service_pool.cc:224] Handling call
0709 12:24:36.886034 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:37.846362 (+960328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.848604 (+1002242us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:38.848829 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2204936448}
I20260709 12:24:38.849632 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1282) took 1006 ms. Trace:
I20260709 12:24:38.857602 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.919981 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.920828 18016 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:38.582350 25094 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.853171 18171 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:38.922037 25109 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:38.922219 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.922794 25109 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:38.923643 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.924846 18016 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:38.924180 25109 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:38.855971 18780 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.831s (SENT)
I20260709 12:24:38.926195 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:38.869076 24691 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.710s	user 0.007s	sys 0.009s
I20260709 12:24:38.868566 17916 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 9 to 14, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 14 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:38.927752 24972 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:38.928016 24691 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:38.846211 25136 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.851s	user 0.005s	sys 0.000s
W20260709 12:24:38.934796 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.935424 18164 raft_consensus.cc:2368] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
W20260709 12:24:38.870388 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63966us
I20260709 12:24:38.936378 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:38.936704 24691 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:38.937403 24892 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:38.871204 25096 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:38.871337 25152 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:38.938253 24691 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:38.938845 25152 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:38.939088 24892 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:38.938802 25096 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.939484 25152 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:38.940208 24892 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.874358 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.875392 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:38.942899 18165 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.876583 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:38.921216 18023 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:38.410802 25145 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:38.951848 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3877) took 1105 ms. Trace:
I20260709 12:24:38.952281 18094 rpcz_store.cc:276] 0709 12:24:37.847411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.065807 (+218396us) service_pool.cc:224] Handling call
0709 12:24:38.088693 (+ 22886us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.089057 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:38.089509 (+   452us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.951237 (+861728us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b1800323fe0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:24:38.951430 (+   193us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:38.951495 (+    65us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:38.951554 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.951630 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.951776 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12255,"prepare.queue_time_us":8400,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"spinlock_wait_cycles":19438848}
I20260709 12:24:38.878247 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:38.849591 25084 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.007s
W20260709 12:24:38.885010 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 539721us
I20260709 12:24:38.891470 25108 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:38.910590 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.913221 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.979015 18024 raft_consensus.cc:2376] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18. Re-sending same reply.
I20260709 12:24:38.920516 18815 rpcz_store.cc:276] 0709 12:24:37.842862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.843170 (+   308us) service_pool.cc:224] Handling call
0709 12:24:38.849368 (+1006198us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54005ad554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:38.849527 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381675392}
I20260709 12:24:38.852969 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.981704 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.983546 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.846374 25001 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:38.930648 18015 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:38.941252 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3037) took 1291 ms (client timeout 1659 ms). Trace:
W20260709 12:24:38.990300 18096 rpcz_store.cc:269] 0709 12:24:37.649916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.692464 (+ 42548us) service_pool.cc:224] Handling call
0709 12:24:38.941022 (+1248558us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b540025e254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:38.941182 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1770398976}
W20260709 12:24:38.950450 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 108933us
I20260709 12:24:38.930675 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.882359 18020 raft_consensus.cc:2376] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24. Re-sending same reply.
I20260709 12:24:38.958808 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:38.959688 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:38.960736 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:38.929090 17916 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 7 to 11, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 11 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:38.950472 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94527.9us
I20260709 12:24:38.973694 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:38.976876 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 865168us
I20260709 12:24:38.996742 18018 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:38.999293 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:39.000970 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:38.996728 18780 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.601s (SENT)
W20260709 12:24:39.003062 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44033.9us
I20260709 12:24:38.997179 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:39.007798 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:39.009271 18019 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:39.003865 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:39.010316 18023 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:39.012065 18810 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:39.014945 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.019794 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:39.019212 17916 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:39.020452 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:39.020725 18811 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:39.022843 18169 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:39.021276 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1093) took 1161 ms. Trace:
I20260709 12:24:39.025806 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:39.031450 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.029656 18165 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:39.034469 18021 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:39.035224 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3780) took 1134 ms. Trace:
I20260709 12:24:39.035692 18021 rpcz_store.cc:276] 0709 12:24:37.900658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.003943 (+1103285us) service_pool.cc:224] Handling call
0709 12:24:39.034066 (+ 30123us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.034940 (+   874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.035156 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:39.037423 17916 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 9 to 15. New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:39.043347 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:39.024343 18091 rpcz_store.cc:276] 0709 12:24:37.859427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.072930 (+213503us) service_pool.cc:224] Handling call
0709 12:24:38.073342 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:38.073951 (+   609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:38.073988 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:38.074554 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:24:38.289534 (+214980us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:38.292039 (+  2505us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.804123 (+512084us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:38.987576 (+183453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.987670 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:38.987719 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:39.021000 (+ 33281us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.021196 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1541635,"prepare.queue_time_us":229,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":715790,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":10130,"spinlock_wait_cycles":54373120,"thread_start_us":177376,"threads_started":1,"wal-append.queue_time_us":306890}
I20260709 12:24:39.050922 18810 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:39.008279 24972 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:39.007212 24958 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:39.055337 17916 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 11 to 16. New cstate: current_term: 16 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:39.057236 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1099) took 1144 ms. Trace:
W20260709 12:24:39.062439 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 63281.9us
I20260709 12:24:39.057332 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1094) took 1173 ms. Trace:
I20260709 12:24:39.062958 18095 rpcz_store.cc:276] 0709 12:24:37.913058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.343907 (+430849us) service_pool.cc:224] Handling call
0709 12:24:38.367931 (+ 24024us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.368218 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:38.420199 (+ 51981us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.056814 (+636615us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:39.056881 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:39.056946 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.057037 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.057169 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":281100,"prepare.queue_time_us":58921,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":251208,"thread_start_us":356112,"threads_started":1,"wal-append.queue_time_us":419613}
I20260709 12:24:39.060827 18166 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:39.057201 18808 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:39.064225 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:39.011793 24815 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:39.082015 18092 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:39.082785 17988 leader_election.cc:400] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18.
I20260709 12:24:39.013453 25096 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:39.087703 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.088529 18019 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:38.990058 25084 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.071228 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:39.084502 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57975.7us
I20260709 12:24:39.071960 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:39.007777 25083 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
W20260709 12:24:39.068302 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64511.5us
I20260709 12:24:39.092145 17916 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 8 to 18. New cstate: current_term: 18 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:39.094823 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97732.3us
I20260709 12:24:39.098361 18092 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:39.055269 25124 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:39.107563 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:39.108325 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:39.108760 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 101054us
I20260709 12:24:39.109488 17916 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:39.110675 18165 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:39.111450 18095 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:39.112268 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:39.121781 18164 raft_consensus.cc:2368] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17.
I20260709 12:24:39.124279 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:39.126474 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:39.068483 24914 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:39.115835 18091 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:39.081260 24655 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:39.135515 18016 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:39.136639 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:39.134191 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:39.096315 25092 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:39.146641 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:39.149309 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:39.150293 18810 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:39.070447 25094 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:39.107690 24781 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.153733 17916 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 9 to 16, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 16 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:39.146622 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:39.165094 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:39.154736 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.093627 25054 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:39.167543 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:39.165933 18090 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:39.055222 25165 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:24:39.130987 25106 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:39.173807 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.083032 24972 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.138327 24655 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:39.135903 25102 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:39.195253 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:39.195536 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:39.196108 18814 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:39.176189 17916 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 10 to 13, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 13 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:39.179534 18811 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:39.136212 25127 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:39.171643 24586 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:24:39.136746 25147 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:39.100816 25113 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:24:39.094012 24815 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.177021 25092 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:39.186309 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:39.178112 25012 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:39.107323 24958 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:24:39.188889 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:39.181640 25056 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:39.177292 25070 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:24:39.200120 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:39.231864 18165 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:39.232606 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.197042 24655 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.195664 25102 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.175796 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:39.177645 24776 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:39.187903 18780 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.733s (SENT)
I20260709 12:24:39.208520 25072 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:39.196619 25106 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:39.211675 25116 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:39.212092 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:39.305799 18165 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:39.210902 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:24:39.317097 18090 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:39.211925 17916 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:39.214869 25127 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:39.202668 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:39.324215 18093 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:39.213917 25147 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:39.205997 18806 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.243598 18023 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:39.285382 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97499.9us
I20260709 12:24:39.231106 25056 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:39.189944 25083 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:24:39.291779 25004 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.179154 24781 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.181841 25054 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.319345 24624 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:39.319280 17916 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:39.134124 25055 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.315846 18021 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.326851 25167 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.284092 18091 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:39.328917 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:39.336526 25128 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.000s	sys 0.036s
I20260709 12:24:39.292302 25102 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:39.344105 18090 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:39.346283 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2894) took 1508 ms. Trace:
I20260709 12:24:39.314821 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:39.359129 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 221135us
I20260709 12:24:39.064324 18097 rpcz_store.cc:276] 0709 12:24:37.884030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.315097 (+431067us) service_pool.cc:224] Handling call
0709 12:24:38.315563 (+   466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:38.315820 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:38.315850 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:38.316546 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:24:38.451321 (+134775us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:38.832358 (+381037us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.057005 (+224647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.057081 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:39.057124 (+    43us) raft_consensus.cc:1685] finished
0709 12:24:39.057136 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.057268 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":224283,"prepare.queue_time_us":85650,"prepare.run_cpu_time_us":12288,"prepare.run_wall_time_us":208950,"raft.queue_time_us":229144,"thread_start_us":436534,"threads_started":2,"wal-append.queue_time_us":234323}
I20260709 12:24:39.362560 18166 rpcz_store.cc:276] 0709 12:24:37.837579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.858179 (+ 20600us) service_pool.cc:224] Handling call
0709 12:24:37.908332 (+ 50153us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.065647 (+1157315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.345954 (+280307us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:39.346198 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2649944960}
I20260709 12:24:39.337174 25056 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.362102 18810 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
W20260709 12:24:39.369905 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.374419 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:39.365196 18064 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.628s (SENT)
W20260709 12:24:39.382622 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 175676us
W20260709 12:24:39.383349 18064 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.665s (SENT)
I20260709 12:24:39.342144 25001 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:39.316403 25072 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.390785 18814 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:39.387442 18015 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:39.321223 24841 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:24:39.389518 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:39.380234 18806 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:39.397259 25113 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:39.397990 18021 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:24:39.365609 25128 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.000s	sys 0.037s
I20260709 12:24:39.400117 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.403187 17989 leader_election.cc:400] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17.
I20260709 12:24:39.406075 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.414311 18021 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:24:39.364946 25094 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.416555 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.417161 17989 leader_election.cc:403] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:24:39.337488 25143 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:24:39.345865 25004 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.362203 25167 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.489643 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:39.618773 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 213094us
I20260709 12:24:39.618614 18021 raft_consensus.cc:2368] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:24:39.373775 25134 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:39.476905 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 64716.3us
W20260709 12:24:39.449980 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 261179us
I20260709 12:24:39.626163 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:39.626006 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:39.631256 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.632287 18021 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:39.633589 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.384174 24624 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:39.640251 18021 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 19 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:24:39.357684 25092 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.409348 25051 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.472970 25190 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.358s
I20260709 12:24:39.410435 25084 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:39.458915 25127 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.410252 25113 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:39.475455 25147 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:39.508764 25175 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.175s	user 0.001s	sys 0.001s
I20260709 12:24:39.409200 24697 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.447s
W20260709 12:24:39.509243 25177 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:24:39.458977 25106 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.410054 25145 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:39.668468 24972 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:39.658257 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:39.684856 24815 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:39.858929 18167 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:39.666208 24948 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.194s
I20260709 12:24:39.413159 25105 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:24:39.786310 18170 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.000s	sys 0.008s
I20260709 12:24:39.695470 25054 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:39.861491 18170 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.000s	sys 0.009s
I20260709 12:24:39.793848 18097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:24:39.803154 25134 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:39.799170 18099 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:39.690377 24776 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:39.490669 25191 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:39.803766 25166 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:39.795261 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 940) took 1508 ms. Trace:
I20260709 12:24:39.814009 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:39.824719 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:39.869946 24092 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:39.869735 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45107.3us
W20260709 12:24:39.870445 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 116216us
I20260709 12:24:39.872462 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:39.837450 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.875967 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:39.849973 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:39.878497 18023 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:39.878697 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:39.660005 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 257250us
I20260709 12:24:39.862800 18099 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:39.810408 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3783) took 1908 ms. Trace:
I20260709 12:24:39.815279 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:39.822798 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 192081us
I20260709 12:24:39.863795 18015 rpcz_store.cc:276] 0709 12:24:38.287266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.990081 (+702815us) service_pool.cc:224] Handling call
0709 12:24:39.794957 (+804876us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54003bb0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:39.795164 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":816356736}
I20260709 12:24:39.883502 18018 rpcz_store.cc:276] 0709 12:24:37.901484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.005528 (+1104044us) service_pool.cc:224] Handling call
0709 12:24:39.006003 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.033348 (+ 27345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.033383 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.045539 (+ 12156us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.244931 (+199392us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:39.244983 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.245094 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:39.810086 (+564992us) raft_consensus.cc:1685] finished
0709 12:24:39.810118 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.810314 (+   196us) inbound_call.cc:177] Queueing success response
0709 12:24:39.823369 (+ 13055us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
Metrics: {"raft.queue_time_us":24422,"raft.run_cpu_time_us":3108,"raft.run_wall_time_us":786317,"spinlock_wait_cycles":18095872,"thread_start_us":165599,"threads_started":1,"wal-append.queue_time_us":742166}
I20260709 12:24:39.883996 18810 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:39.886404 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:39.871019 24092 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:39.830328 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.888183 18021 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:39.809947 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:39.888291 24092 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:39.926481 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46540.7us
I20260709 12:24:39.931185 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.933676 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.934963 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:39.882200 18099 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.938952 18099 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:39.921036 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:39.689061 25128 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:39.955812 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:39.956210 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:39.690565 25180 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
W20260709 12:24:39.658641 25178 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
I20260709 12:24:39.828293 25167 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:39.844594 25201 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.150s	user 0.003s	sys 0.000s
I20260709 12:24:39.824931 24892 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.830312 25052 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:39.863363 25166 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:39.835180 25088 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:39.829806 25083 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:39.694698 24586 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17.
I20260709 12:24:39.842623 25070 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.863138 24776 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.851046 25004 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:39.843693 25084 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:39.837121 25113 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.969324 25116 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.104s
W20260709 12:24:39.999346 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 189537us
I20260709 12:24:40.092849 18099 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 12:24:40.093377 18099 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 9
I20260709 12:24:40.093616 18099 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 16.10
W20260709 12:24:40.000105 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 327903us
I20260709 12:24:40.015625 25209 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:40.020773 25169 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.978005 25128 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.095340 25169 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.000823 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.016s	sys 0.031s
W20260709 12:24:40.097774 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.017s	sys 0.031s
I20260709 12:24:39.832975 24624 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.097990 25169 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:40.098234 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 718883us
I20260709 12:24:40.095333 18099 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.099390 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3046) took 1023 ms. Trace:
I20260709 12:24:40.099340 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:39.810681 24634 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.100333 24634 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:39.805990 25055 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:39.863150 25191 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:39.957176 18021 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:40.026360 25167 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:40.011016 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3779) took 2110 ms. Trace:
W20260709 12:24:40.010841 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89908.2us
I20260709 12:24:39.938309 24781 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:39.966464 24883 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:40.016943 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.016134 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.083519 25084 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.016292 18023 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:39.954427 25072 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:40.065002 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 189115us
I20260709 12:24:40.133121 18064 leader_election.cc:400] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:40.133499 18064 leader_election.cc:403] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 12:24:40.068368 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66622.2us
I20260709 12:24:40.072403 25105 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:39.829398 25051 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.139117 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.140789 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.141109 25051 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.103531 25128 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:39.752359 25210 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.142437 25210 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.118808 25134 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
W20260709 12:24:40.078800 25183 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.294s	user 0.000s	sys 0.001s
W20260709 12:24:40.080565 25181 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.375s	user 0.001s	sys 0.000s
I20260709 12:24:40.081099 25189 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.140s
W20260709 12:24:40.082823 25198 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:24:40.021566 25204 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.146039 25210 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:39.645768 25184 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.429s
I20260709 12:24:40.146380 25204 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.147388 25216 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:40.147893 25216 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:40.148458 25112 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.246s	user 0.016s	sys 0.001s
I20260709 12:24:40.149247 25204 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:40.151173 25134 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:40.151664 25134 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:40.151788 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:40.152513 25152 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.212s	user 0.020s	sys 0.001s
I20260709 12:24:40.152792 25134 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.153090 25152 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.099822 18099 rpcz_store.cc:276] 0709 12:24:39.075570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.077455 (+  1885us) service_pool.cc:224] Handling call
0709 12:24:39.077837 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.799021 (+721184us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b540035afa0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662edc0 00007f6c166b91c0 00007f6c166ba231 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c
0709 12:24:40.099125 (+300104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.099326 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4989696}
I20260709 12:24:40.002243 25203 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.154688 25203 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.155084 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:40.155859 18099 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:40.157378 25203 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:39.940274 25139 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:40.158015 24781 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.158630 24781 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.158505 25191 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.145838 24914 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.998s	user 0.011s	sys 0.001s
I20260709 12:24:40.159874 25191 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:40.160112 24914 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:40.162650 25187 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.461s	user 0.000s	sys 0.001s
I20260709 12:24:40.158051 25139 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.164297 25139 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:40.164674 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3044) took 1090 ms. Trace:
I20260709 12:24:40.165081 18098 rpcz_store.cc:276] 0709 12:24:39.074771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.076483 (+  1712us) service_pool.cc:224] Handling call
0709 12:24:39.076987 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.100410 (+ 23423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.100477 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.114012 (+ 13535us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.497614 (+383602us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:39.699432 (+201818us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.699815 (+   383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.700369 (+   554us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.164418 (+464049us) raft_consensus.cc:1685] finished
0709 12:24:40.164445 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.164601 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22778,"prepare.queue_time_us":157977,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":84150,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":35306,"thread_start_us":365776,"threads_started":2,"wal-append.queue_time_us":495820}
I20260709 12:24:40.167716 18098 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:24:40.119469 18020 rpcz_store.cc:276] 0709 12:24:37.900379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.994379 (+1094000us) service_pool.cc:224] Handling call
0709 12:24:38.994732 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.998030 (+  3298us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:39.032440 (+ 34410us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.791967 (+759527us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b18002a35e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:24:40.010532 (+218565us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.010613 (+    81us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:40.010678 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.010760 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.010933 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48963,"prepare.queue_time_us":32710,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":17350,"raft.run_cpu_time_us":5892,"raft.run_wall_time_us":66485,"spinlock_wait_cycles":13959296,"thread_start_us":209108,"threads_started":1,"wal-append.queue_time_us":1028888}
I20260709 12:24:40.171010 25190 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.171456 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.171845 25190 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.141541 18090 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:40.016016 25052 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:40.022821 24892 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:39.979158 24691 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.098s
I20260709 12:24:40.186374 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:40.063879 25083 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:40.135075 18097 raft_consensus.cc:606] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:40.161516 24781 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.136400 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.136595 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.131798 24972 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:40.131124 25209 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:40.138778 25202 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:24:40.142904 24634 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:39.641360 25094 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.145500 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1251) took 2386 ms. Trace:
I20260709 12:24:40.206087 18098 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:40.207726 18812 rpcz_store.cc:276] 0709 12:24:37.760513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.760680 (+   167us) service_pool.cc:224] Handling call
0709 12:24:37.864495 (+103815us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b5800003998. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:37.864978 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:37.912288 (+ 47310us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:38.862083 (+949795us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b18002058a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:24:39.093709 (+231626us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.667747 (+574038us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.681566 (+ 13819us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.682436 (+   870us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:39.682488 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.682594 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.145235 (+462641us) raft_consensus.cc:1685] finished
0709 12:24:40.145268 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.145419 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18835,"prepare.queue_time_us":951422,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":277148,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":16320,"spinlock_wait_cycles":16284928,"thread_start_us":751566,"threads_started":3,"wal-append.queue_time_us":591952}
I20260709 12:24:40.148829 25216 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.118539 25169 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:24:40.154155 25112 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.174484 18806 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:39.975271 25199 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.184s
W20260709 12:24:40.220701 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.175333 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.123550 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1261) took 1099 ms. Trace:
I20260709 12:24:40.103996 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:40.224422 18809 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 12.
I20260709 12:24:40.198381 24972 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.113514 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.183221 18016 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:40.185425 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:40.186101 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:40.187352 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 85806.8us
I20260709 12:24:40.188642 18173 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:40.188944 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:40.190129 18167 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:24:40.190909 18097 raft_consensus.cc:614] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:40.063508 25088 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:40.190699 18169 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:40.130475 24883 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:40.194746 18095 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:40.137678 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.154073 25124 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:40.196979 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3798) took 1082 ms. Trace:
I20260709 12:24:40.202800 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:40.107266 25070 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.175310 25105 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.205801 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:40.159029 25134 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:40.190605 25211 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:40.219856 18098 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:40.102790 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:40.348703 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2902) took 1236 ms. Trace:
I20260709 12:24:40.364257 18810 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:40.204905 24781 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:24:40.223042 18807 rpcz_store.cc:276] 0709 12:24:39.023845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.024002 (+   157us) service_pool.cc:224] Handling call
0709 12:24:39.024430 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.024884 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.024918 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.025529 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.234019 (+208490us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:39.234091 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.234203 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.123162 (+888959us) raft_consensus.cc:1685] finished
0709 12:24:40.123202 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.123446 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":926,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":15063,"thread_start_us":173575,"threads_started":1,"wal-append.queue_time_us":295139}
W20260709 12:24:40.226269 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51317.6us
I20260709 12:24:40.375656 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:40.369908 18166 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:40.225629 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:40.365512 18170 rpcz_store.cc:276] 0709 12:24:39.112151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.135155 (+ 23004us) service_pool.cc:224] Handling call
0709 12:24:39.135478 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.135746 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:39.862072 (+726326us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.348120 (+486048us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.348314 (+   194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.348359 (+    45us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:40.348415 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.348487 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.348633 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62358,"prepare.queue_time_us":66858,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":61,"raft.queue_time_us":44414,"spinlock_wait_cycles":40983808,"thread_start_us":1121557,"threads_started":2}
W20260709 12:24:40.408329 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 270720us
I20260709 12:24:40.366220 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:40.369102 18015 rpcz_store.cc:276] 0709 12:24:39.114103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.136046 (+1021943us) service_pool.cc:224] Handling call
0709 12:24:40.196898 (+ 60852us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:40.439392 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.305989 25216 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:40.368182 24883 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:40.499934 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 294195us
W20260709 12:24:40.264564 25057 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.364s	user 0.017s	sys 0.000s
I20260709 12:24:40.640223 18168 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:40.368011 25105 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:40.341758 25106 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:40.225212 25112 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.576587 25092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.000s	sys 0.014s
W20260709 12:24:40.565768 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 498723us
W20260709 12:24:40.543917 18092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.014s	sys 0.025s
W20260709 12:24:40.674134 18092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.014s	sys 0.027s
I20260709 12:24:40.452204 25209 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.371030 24653 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.306511 25083 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.637072 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.198s (SENT) [suppressed 36 similar messages]
W20260709 12:24:40.708532 25191 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.548s	user 0.019s	sys 0.000s
W20260709 12:24:40.708537 25102 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.320s	user 0.015s	sys 0.008s
I20260709 12:24:40.636776 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:40.697749 25092 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:24:40.370725 25055 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.226931 25052 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.712550 25204 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:40.221652 24916 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.638187 25088 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.713923 25088 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:40.713658 24916 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.637768 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:40.715708 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 231324us
W20260709 12:24:40.655664 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2842) took 1451 ms (client timeout 1657 ms). Trace:
W20260709 12:24:40.716454 18016 rpcz_store.cc:269] 0709 12:24:39.204595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.449181 (+244586us) service_pool.cc:224] Handling call
0709 12:24:40.655390 (+1206209us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b540024de54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:40.655560 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1244519296}
I20260709 12:24:40.717234 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.265157 25190 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:40.640738 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:40.718020 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:40.718621 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77965.7us
I20260709 12:24:40.718771 18016 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:40.719472 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 956) took 1620 ms. Trace:
I20260709 12:24:40.720503 25054 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:40.721015 25054 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.450256 24972 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:40.722245 25106 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:40.722416 24776 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.722946 25106 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.725477 25106 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.667445 25129 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.683902 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3053) took 1279 ms. Trace:
W20260709 12:24:40.640090 25147 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.006s	sys 0.011s
I20260709 12:24:40.696676 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3788) took 1603 ms. Trace:
I20260709 12:24:40.728119 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:40.728916 18019 rpcz_store.cc:276] 0709 12:24:39.093450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.093613 (+   163us) service_pool.cc:224] Handling call
0709 12:24:39.094011 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.108428 (+ 14417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.108463 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.808739 (+700276us) spinlock_profiling.cc:243] Waited 7.56 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:39.809320 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 12:24:40.220033 (+410713us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:40.220073 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.220151 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.692344 (+472193us) raft_consensus.cc:1685] finished
0709 12:24:40.692367 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.696555 (+  4188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5329,"raft.queue_time_us":26490,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11229,"spinlock_wait_cycles":17395456,"thread_start_us":410359,"threads_started":1,"wal-append.queue_time_us":878696}
I20260709 12:24:40.731912 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:40.670936 24655 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.352s	user 0.018s	sys 0.009s
I20260709 12:24:40.640975 25203 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:40.697531 25145 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.737597 25145 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.738409 25139 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.010s
I20260709 12:24:40.738953 25139 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.740260 18165 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:24:40.741134 25145 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.715801 25191 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:40.653663 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 304345us
I20260709 12:24:40.741868 25191 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.698750 25072 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.742390 25191 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:40.742542 25072 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.744030 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:40.744680 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.712440 25055 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.747643 25055 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:40.648090 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.749203 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 959) took 1618 ms. Trace:
I20260709 12:24:40.751451 25190 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.752184 25190 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.753865 25191 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.754819 18018 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17. Re-sending same reply.
I20260709 12:24:40.721554 25054 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:40.756171 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2850) took 1528 ms (client timeout 1934 ms). Trace:
I20260709 12:24:40.756340 25191 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:40.756628 18018 rpcz_store.cc:269] 0709 12:24:39.228019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.934550 (+706531us) service_pool.cc:224] Handling call
0709 12:24:40.755978 (+821428us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:40.756104 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1880450944}
I20260709 12:24:40.749598 18015 rpcz_store.cc:276] 0709 12:24:39.131053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.647705 (+1516652us) service_pool.cc:224] Handling call
0709 12:24:40.749121 (+101416us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.759040 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.759687 18015 raft_consensus.cc:2376] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13. Re-sending same reply.
I20260709 12:24:40.760154 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 964) took 1132 ms. Trace:
I20260709 12:24:40.760444 18015 rpcz_store.cc:276] 0709 12:24:39.628220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.758724 (+1130504us) service_pool.cc:224] Handling call
0709 12:24:40.760096 (+  1372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.761386 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.732860 18019 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:40.664494 25165 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:40.661700 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:40.779573 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 138706us
I20260709 12:24:40.780699 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.781984 18023 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:40.639997 25128 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.736317 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3782) took 2835 ms. Trace:
I20260709 12:24:40.783238 18017 rpcz_store.cc:276] 0709 12:24:37.901209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.004278 (+1103069us) service_pool.cc:224] Handling call
0709 12:24:39.004659 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.015957 (+ 11298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.015996 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.033138 (+ 17142us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.223667 (+190529us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:39.286096 (+ 62429us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.297874 (+ 11778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.298107 (+   233us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.720466 (+1422359us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b240011ff78. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c13608fb0 00007f6c1627f488 00007f6c1627fbe8 00007f6c16257398 00007f6c1625276a 00007f6c16251bb6
0709 12:24:40.727590 (+  7124us) raft_consensus.cc:1685] finished
0709 12:24:40.736008 (+  8418us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.736209 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":190677,"prepare.queue_time_us":58264,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":63,"raft.queue_time_us":132022,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":31384,"spinlock_wait_cycles":13222272,"thread_start_us":131682,"threads_started":1,"wal-append.queue_time_us":289740}
I20260709 12:24:40.735919 25129 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:40.728852 25147 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.006s	sys 0.011s
I20260709 12:24:40.719811 18016 rpcz_store.cc:276] 0709 12:24:39.099583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.717686 (+1618103us) service_pool.cc:224] Handling call
0709 12:24:40.719396 (+  1710us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.762439 25197 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:40.727104 18091 rpcz_store.cc:276] 0709 12:24:39.404128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.655975 (+251847us) service_pool.cc:224] Handling call
0709 12:24:39.681604 (+ 25629us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.682108 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.682153 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.685452 (+  3299us) log.cc:844] Serialized 26 byte log entry
0709 12:24:40.221687 (+536235us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:40.221732 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.221804 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.683600 (+461796us) raft_consensus.cc:1685] finished
0709 12:24:40.683631 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.683815 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":95819,"raft.queue_time_us":56457,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":20991,"thread_start_us":478147,"threads_started":1,"wal-append.queue_time_us":978407}
I20260709 12:24:40.609625 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.766889 25190 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:40.767378 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:40.777631 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49631.6us
I20260709 12:24:40.774668 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2847) took 1547 ms. Trace:
I20260709 12:24:40.786866 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.745386 25072 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:40.838683 18019 rpcz_store.cc:276] 0709 12:24:39.226825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.731500 (+1504675us) service_pool.cc:224] Handling call
0709 12:24:40.774565 (+ 43065us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.841095 18022 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:40.841219 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.842128 18015 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:40.843072 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.800760 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:40.845494 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44917.4us
I20260709 12:24:40.840742 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.846464 18019 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:40.846601 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.851262 18808 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:40.788097 24697 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.852690 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3791) took 1757 ms. Trace:
I20260709 12:24:40.838881 18017 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:24:40.858991 18016 rpcz_store.cc:276] 0709 12:24:39.095233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.851342 (+1756109us) service_pool.cc:224] Handling call
0709 12:24:40.851781 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.852009 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.852034 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.852095 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:40.852145 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.852204 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.852612 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.860139 18017 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:24:40.854913 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.860877 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3793) took 1765 ms. Trace:
I20260709 12:24:40.861238 18017 rpcz_store.cc:276] 0709 12:24:39.095765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.859389 (+1763624us) service_pool.cc:224] Handling call
0709 12:24:40.859751 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.860568 (+   817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.860799 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:40.856899 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 247388us
I20260709 12:24:40.689743 24883 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:40.853175 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 195039us
I20260709 12:24:40.854063 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:40.855016 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:40.866425 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.787482 25165 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.824278 24776 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:40.842430 25092 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:40.867991 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:40.841547 25084 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.862691 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:40.868300 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:40.889424 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.869652 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:40.788491 25055 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:40.872575 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.852275 25051 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:40.794989 25203 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:40.906018 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:40.788666 25199 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:40.874698 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3797) took 1752 ms. Trace:
I20260709 12:24:40.874871 18015 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:40.924755 18017 rpcz_store.cc:276] 0709 12:24:39.113768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.862107 (+1748339us) service_pool.cc:224] Handling call
0709 12:24:40.862468 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.866132 (+  3664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.866162 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.866215 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:40.866250 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.866316 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.866453 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2052,"raft.queue_time_us":3204,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":7917}
W20260709 12:24:40.878425 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 180416us
I20260709 12:24:40.926064 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.926497 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3790) took 1831 ms. Trace:
I20260709 12:24:40.926919 18015 rpcz_store.cc:276] 0709 12:24:39.094977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.852764 (+1757787us) service_pool.cc:224] Handling call
0709 12:24:40.874596 (+ 21832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.926176 (+ 51580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.926432 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.928462 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:40.880900 25124 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:40.867499 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.935103 18095 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:40.848717 25106 raft_consensus.cc:2764] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:24:40.936197 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.890379 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:40.863583 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3792) took 1768 ms. Trace:
I20260709 12:24:40.938001 18019 rpcz_store.cc:276] 0709 12:24:39.095512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.855301 (+1759789us) service_pool.cc:224] Handling call
0709 12:24:40.862826 (+  7525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.863066 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.863106 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.863177 (+    71us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:40.863241 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.863323 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.863494 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.848502 25197 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.854390 25072 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:40.925733 24776 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:40.949705 25203 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.952111 18099 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:40.893658 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:40.953167 18812 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:40.770094 25102 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.955536 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:40.956025 25102 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:40.927745 25207 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.284s	user 0.000s	sys 0.002s
I20260709 12:24:40.858453 25190 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.777233 24655 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:40.806942 25128 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.875578 25165 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:40.909337 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:40.895953 25092 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.880769 18809 raft_consensus.cc:2410] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 12:24:40.929454 18015 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:40.874717 24892 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.940008 18019 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:40.983089 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3799) took 1841 ms. Trace:
I20260709 12:24:40.983546 18019 rpcz_store.cc:276] 0709 12:24:39.141324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.939174 (+1797850us) service_pool.cc:224] Handling call
0709 12:24:40.939574 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.982723 (+ 43149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.983020 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.940382 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.940856 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:40.892153 25191 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.889868 18807 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:40.941488 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:40.864403 24634 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 12:24:40.944546 18173 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:40.895022 25147 raft_consensus.cc:2764] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:40.999569 18095 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:40.892277 25070 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.022259 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3893) took 1915 ms. Trace:
I20260709 12:24:41.022787 18093 rpcz_store.cc:276] 0709 12:24:39.106387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.325328 (+218941us) service_pool.cc:224] Handling call
0709 12:24:39.325868 (+   540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.336690 (+ 10822us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.336772 (+    82us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.346692 (+  9920us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.865696 (+519004us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:39.865770 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.865939 (+   169us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:41.021954 (+1156015us) raft_consensus.cc:1685] finished
0709 12:24:41.021981 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.022175 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2875,"raft.queue_time_us":3153,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":8000,"thread_start_us":450214,"threads_started":1,"wal-append.queue_time_us":841693}
I20260709 12:24:41.024564 24883 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.024760 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:41.025382 25056 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.624s	user 0.024s	sys 0.000s
I20260709 12:24:41.025484 18093 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:40.932670 25224 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:40.890686 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:41.079353 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 188733us
W20260709 12:24:40.962177 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 325120us
I20260709 12:24:40.870893 24697 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:40.950023 25227 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:24:40.727382 24653 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:40.897089 25152 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:40.957808 25223 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:24:41.011829 25092 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:40.978875 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69670.3us
I20260709 12:24:40.945968 25139 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:40.943606 25094 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:40.956972 25106 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:41.003078 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.000376 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:41.128994 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 128787us
I20260709 12:24:41.006572 24655 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:40.984701 18094 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:41.128021 18095 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:41.016019 25197 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:40.888085 25184 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.046025 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:41.241988 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.025970 25056 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.081760 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:41.269873 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 188229us
W20260709 12:24:41.271391 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:41.084371 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:41.108862 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 236355us
I20260709 12:24:41.273568 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.275255 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.276328 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.276741 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.111032 25147 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:41.281916 18809 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:41.282235 25147 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:41.284446 18812 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:41.286499 25211 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:41.119043 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 481191us
I20260709 12:24:41.288285 25229 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:40.923494 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.291252 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:41.292136 18091 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.296149 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:41.298869 25055 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:41.238515 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.302006 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.302773 18095 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:41.126807 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:41.012389 24892 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:41.079962 25224 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.307539 25224 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.254087 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3895) took 1334 ms. Trace:
I20260709 12:24:41.277736 18810 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:24:41.278196 25199 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:41.310190 18092 rpcz_store.cc:276] 0709 12:24:39.920114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.921566 (+  1452us) service_pool.cc:224] Handling call
0709 12:24:39.922024 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.922298 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:39.972006 (+ 49708us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.871496 (+899490us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.871936 (+   440us) log.cc:844] Serialized 14 byte log entry
0709 12:24:41.253413 (+381477us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:41.253517 (+   104us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:41.253607 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.253728 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.253976 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4691,"prepare.queue_time_us":530465,"prepare.run_cpu_time_us":170,"prepare.run_wall_time_us":7220,"raft.queue_time_us":29081,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":34518,"thread_start_us":1275387,"threads_started":3,"wal-append.queue_time_us":712558}
I20260709 12:24:41.312764 18092 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:41.279247 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:41.313432 18815 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.315357 25231 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:41.121979 18093 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:41.279590 25072 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.281134 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.283766 25147 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.280493 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.330066 18092 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:40.976724 25190 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.322021 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:41.057180 25191 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:41.332459 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51466.1us
I20260709 12:24:41.338739 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.336197 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.333240 18807 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:41.241873 25051 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:41.343531 18096 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:41.236691 25106 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:41.346349 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48511.2us
I20260709 12:24:41.352883 25197 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:41.270694 24815 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:41.268572 25209 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:41.375015 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 248361us
W20260709 12:24:41.382203 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.362784 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45562 (request call id 27) took 1647 ms. Trace:
I20260709 12:24:41.364964 18015 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:24:41.384135 25054 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.628s	user 0.013s	sys 0.000s
I20260709 12:24:41.366056 25237 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:41.384757 25054 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.387638 25054 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.380646 24815 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:41.391932 25105 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.713s	user 0.012s	sys 0.004s
I20260709 12:24:41.392506 25105 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.358174 25051 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:41.394172 25051 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:41.394459 25134 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.023s	user 0.009s	sys 0.009s
I20260709 12:24:41.364415 25230 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:24:41.316817 25184 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.399124 25128 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.352851 25206 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:41.400236 25206 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.400743 25206 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.345999 25199 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.361702 25191 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:41.403532 24972 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.681s	user 0.013s	sys 0.000s
I20260709 12:24:41.404198 25112 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.404096 24972 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.376247 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:41.406248 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:41.407042 18099 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:41.408773 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2905) took 1783 ms (client timeout 1731 ms). Trace:
W20260709 12:24:41.409287 18172 rpcz_store.cc:269] 0709 12:24:39.625312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.625513 (+   201us) service_pool.cc:224] Handling call
0709 12:24:41.408486 (+1782973us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:41.408675 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2186348800}
I20260709 12:24:41.412595 18813 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 21. Ops: []
I20260709 12:24:41.369187 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:41.414597 18810 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.414652 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3803) took 1257 ms. Trace:
I20260709 12:24:41.361024 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1466) took 2190 ms. Trace:
I20260709 12:24:41.416060 18808 rpcz_store.cc:276] 0709 12:24:39.170840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.171059 (+   219us) service_pool.cc:224] Handling call
0709 12:24:39.176143 (+  5084us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.857724 (+1681581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.360718 (+502994us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54006153d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:41.360945 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3851617536}
I20260709 12:24:41.383111 18097 rpcz_store.cc:276] 0709 12:24:39.715743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.793615 (+ 77872us) service_pool.cc:224] Handling call
0709 12:24:41.362501 (+1568886us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54002959d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:41.362646 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627245440}
I20260709 12:24:41.431921 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:41.438462 18812 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.363211 24655 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:24:41.367692 25056 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.355644 25106 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:41.379668 25168 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:41.442960 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:41.457333 18168 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.443658 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:41.458657 18165 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:41.443912 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:41.445792 18169 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:41.446856 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:41.460640 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84540.2us
I20260709 12:24:41.460865 18170 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:41.448271 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:41.381623 24657 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:41.362784 24697 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.454128 18091 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:41.463515 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:41.468899 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:41.471781 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:41.474205 18164 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:41.471783 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:41.476414 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 137757us
W20260709 12:24:41.469131 18780 connection.cc:256] RPC call timeout handler was delayed by -1.7729262299999391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
I20260709 12:24:41.477686 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:41.478724 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3173190060001616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
I20260709 12:24:41.480363 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:41.482299 18097 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:41.487469 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.488781 18091 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:41.492997 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.380942 25209 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.496270 25184 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.497092 24655 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.464272 24657 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.498174 24657 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:41.497736 24655 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.498291 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.359308 25072 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.501040 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:41.523488 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.508623 18810 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:41.526157 18815 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:41.527760 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:41.528540 18091 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.467051 25152 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:41.526834 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:41.531817 18809 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:41.530181 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.532987 18064 leader_election.cc:400] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:24:41.533432 18064 leader_election.cc:403] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:41.529721 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:41.514979 18099 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:24:41.533783 18810 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 13.
I20260709 12:24:41.491256 25230 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.539991 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:41.491286 24892 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:41.551332 18810 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:41.505388 25211 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:41.543866 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.559516 18099 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:41.545111 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.542228 25105 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.546555 25129 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.547621 25209 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.548590 25244 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:41.548681 24910 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:41.565157 24653 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.566231 24972 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.566680 24697 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:41.550877 25242 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.567322 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.534456 25152 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.597283 25244 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:41.568001 25072 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.478475 25168 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.588346 18091 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:41.639693 25244 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.639591 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:41.640662 18808 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23. Re-sending same reply.
I20260709 12:24:41.539752 25054 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:41.641779 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.589075 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:41.644268 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.542726 24776 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:41.597559 18099 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:41.589118 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:41.600798 18098 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:41.602020 24655 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.550416 25237 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:24:41.500043 25235 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:24:41.588168 25128 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
W20260709 12:24:41.606614 25241 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:24:41.557554 25216 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.574100 25143 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:24:41.656085 25216 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.619581 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:41.622262 18812 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:24:41.622124 18780 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.813s (SENT)
I20260709 12:24:41.639634 25152 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:41.658396 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.658936 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:41.659487 25069 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:41.660070 18780 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.132s (SENT)
I20260709 12:24:41.660651 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.662144 24972 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:41.631699 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86703.5us
I20260709 12:24:41.637789 25242 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.666864 25190 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:41.642393 25054 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:41.668130 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.644923 18093 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:41.670547 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.671020 25199 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:41.671838 18095 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:41.671909 25168 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.673753 25072 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 19.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:41.675551 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:41.675846 25252 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:41.676388 25252 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:41.676738 18811 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: [19.16-19.16]
W20260709 12:24:41.677151 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 958) took 2563 ms (client timeout 1812 ms). Trace:
I20260709 12:24:41.674340 25216 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.678160 18016 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:41.679446 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.680192 18811 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:41.668519 25054 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.670749 25203 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:41.551680 25230 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.681906 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3900) took 1740 ms. Trace:
I20260709 12:24:41.682457 18096 rpcz_store.cc:276] 0709 12:24:39.941304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.984905 (+ 43601us) service_pool.cc:224] Handling call
0709 12:24:39.985243 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.352606 (+1367363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.681578 (+328972us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001afc54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:41.681806 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3123152768}
I20260709 12:24:41.683470 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:41.684240 25230 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.629488 24697 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:41.684954 25072 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:41.685511 25072 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.685954 25072 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:41.686512 25001 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:41.686055 24697 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.686941 25001 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:41.687690 24697 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.687978 25001 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.618899 24910 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.679446 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3794) took 2583 ms. Trace:
I20260709 12:24:41.703400 18016 rpcz_store.cc:276] 0709 12:24:39.096017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.860270 (+1764253us) service_pool.cc:224] Handling call
0709 12:24:41.677934 (+817664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.678613 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.679256 (+   643us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b54001cf550. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:41.679379 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1879344384}
I20260709 12:24:41.642494 18808 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:41.682116 18811 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:24:41.692284 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 957) took 2583 ms (client timeout 2198 ms). Trace:
W20260709 12:24:41.707437 18023 rpcz_store.cc:269] 0709 12:24:39.109319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.716895 (+1607576us) service_pool.cc:224] Handling call
0709 12:24:41.692086 (+975191us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b5400251554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:41.692205 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145039104}
I20260709 12:24:41.708940 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:41.709203 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 120235us
I20260709 12:24:41.693126 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:41.677599 18024 rpcz_store.cc:269] 0709 12:24:39.113454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.132647 (+ 19193us) service_pool.cc:224] Handling call
0709 12:24:41.676816 (+2544169us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54001cf554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:41.676994 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4222907648}
I20260709 12:24:41.668917 25242 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.680127 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.691787 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:41.712910 18098 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:41.705587 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:41.691726 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2922) took 1344 ms. Trace:
I20260709 12:24:41.714473 18173 rpcz_store.cc:276] 0709 12:24:40.346912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.373246 (+ 26334us) service_pool.cc:224] Handling call
0709 12:24:40.373578 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.998708 (+625130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.691420 (+692712us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54001b4254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:41.691654 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312200576}
W20260709 12:24:41.709715 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.709646 18023 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:41.717901 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.634158 25211 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:41.710194 18168 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:41.711119 18815 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:41.712100 24910 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.718844 18096 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
I20260709 12:24:41.677389 25252 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.735369 18808 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:41.735483 18812 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:41.720129 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.723508 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:41.729034 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:41.737488 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:41.737378 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.738011 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:41.738922 18093 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:41.733196 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:41.740252 18096 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:41.738726 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:41.595244 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:41.743026 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.743877 18097 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:41.711262 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:41.749670 18064 leader_election.cc:400] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:24:41.750286 18064 leader_election.cc:403] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:41.714164 24972 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.702346 25203 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:41.718077 24653 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:41.757851 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.741577 18024 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:41.758124 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:41.760224 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49061us
I20260709 12:24:41.769503 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:41.771428 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.725606 25199 raft_consensus.cc:2764] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:24:41.822301 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.823191 18090 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:41.415444 18019 rpcz_store.cc:276] 0709 12:24:40.156771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.017067 (+860296us) service_pool.cc:224] Handling call
0709 12:24:41.017484 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.017737 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:41.113723 (+ 95986us) log.cc:844] Serialized 14 byte log entry
0709 12:24:41.413411 (+299688us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:41.414145 (+   734us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:41.414226 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.414335 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.414554 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1827,"prepare.queue_time_us":95866,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":87,"raft.queue_time_us":402,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":12695,"thread_start_us":394146,"threads_started":2}
I20260709 12:24:41.824272 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:41.825115 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:41.764237 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 68822.3us
I20260709 12:24:41.826574 25236 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:41.827008 25236 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.828199 25236 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.825115 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:41.825626 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:41.834563 18095 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:41.746775 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.764698 25237 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:24:41.708498 25051 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.714432 25001 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.780053 25224 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.772047 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:41.832304 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3808) took 1651 ms. Trace:
I20260709 12:24:41.844130 18024 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.784255 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.787158 18023 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:41.788056 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:41.789917 25203 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:41.789218 24776 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:41.715821 24657 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.799847 25052 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:41.800916 25165 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.802s	user 0.012s	sys 0.002s
I20260709 12:24:41.746490 25056 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:41.772897 24653 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:41.762908 25211 raft_consensus.cc:2764] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:24:41.802815 24910 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.775202 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.767024 25128 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:41.875319 25083 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.807618 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:41.782069 25092 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:24:41.830158 25143 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:41.869776 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:24:41.839203 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92574.2us
I20260709 12:24:41.872972 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:41.849258 24892 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:41.876792 25152 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.881474 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:41.847409 25134 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:41.867662 25237 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.897737 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:41.869424 24653 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:41.915223 18815 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:41.882710 25143 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.850347 25096 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:24:41.925465 18808 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:41.920608 25139 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:41.885473 25216 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:24:41.923802 24892 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:41.923802 25143 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:41.932824 25139 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:41.950752 24910 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.882164 25165 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.959800 25139 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:41.971982 25224 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.934448 25203 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.941157 25166 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:41.976261 25165 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.026051 25166 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:42.001103 25219 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.158s	user 0.002s	sys 0.002s
W20260709 12:24:41.999675 25106 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.519s	user 0.014s	sys 0.000s
I20260709 12:24:42.004748 25113 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:42.005844 25206 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.604s	user 0.016s	sys 0.004s
I20260709 12:24:42.030244 25206 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:41.977805 25152 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.010130 25242 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:42.033970 25106 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.034699 25242 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:42.036090 25242 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.037452 25106 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.020803 25237 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:42.008307 25252 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:42.029361 25244 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.045181 25244 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.037163 25143 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.033560 25206 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.049361 25242 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:42.051046 25143 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.062223 25236 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.066180 25236 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:41.769485 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:41.847656 18092 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 12.
W20260709 12:24:42.114957 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:41.839152 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.115754 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 346161us
I20260709 12:24:42.118795 18016 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
W20260709 12:24:42.120009 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.844509 18015 rpcz_store.cc:276] 0709 12:24:40.180482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.117269 (+936787us) service_pool.cc:224] Handling call
0709 12:24:41.117529 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.383971 (+266442us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.832047 (+448076us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:41.832238 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":567771392}
I20260709 12:24:41.846218 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:42.124006 18023 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 13: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:41.846601 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:42.127125 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 228656us
I20260709 12:24:42.129094 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:42.134819 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:41.861860 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73845.2us
W20260709 12:24:42.138669 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 257089us
I20260709 12:24:41.832372 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:42.142266 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:42.144312 18098 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:41.884997 18099 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:42.152931 18167 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
W20260709 12:24:42.159533 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 951) took 3868 ms (client timeout 2132 ms). Trace:
W20260709 12:24:42.160364 18022 rpcz_store.cc:269] 0709 12:24:38.318680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.908133 (+589453us) service_pool.cc:224] Handling call
0709 12:24:42.158919 (+3250786us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:42.159242 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4442535680}
I20260709 12:24:42.163111 18017 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:42.164494 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3813) took 1275 ms. Trace:
I20260709 12:24:42.165203 18017 rpcz_store.cc:276] 0709 12:24:40.890895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.157568 (+1266673us) service_pool.cc:224] Handling call
0709 12:24:42.162465 (+  4897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.163985 (+  1520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.164339 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.171346 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3816) took 1279 ms. Trace:
I20260709 12:24:42.168859 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3814) took 1277 ms. Trace:
I20260709 12:24:42.203220 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:42.204429 18806 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:42.172339 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:42.180451 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:42.184042 18808 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:42.184322 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:42.210863 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1460) took 3922 ms. Trace:
I20260709 12:24:42.211598 18813 rpcz_store.cc:276] 0709 12:24:38.288025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.288922 (+   897us) service_pool.cc:224] Handling call
0709 12:24:40.145680 (+1856758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.413211 (+1267531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.894692 (+481481us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005f5854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:42.210205 (+315513us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54005f5850. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:42.210528 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7182196352}
I20260709 12:24:42.212419 18094 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:42.191064 24092 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:42.212982 24655 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.213157 18168 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:24:42.213600 24092 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:42.213898 24655 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.215361 24092 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.195412 24657 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.220182 24657 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:42.199550 24691 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
I20260709 12:24:42.203615 18022 rpcz_store.cc:276] 0709 12:24:40.891214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.166648 (+1275434us) service_pool.cc:224] Handling call
0709 12:24:42.167488 (+   840us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.168258 (+   770us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.168313 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:42.168415 (+   102us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:42.168476 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.168557 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.168758 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.172006 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:42.184696 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:42.186818 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:42.124449 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:42.196017 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:42.202801 18017 rpcz_store.cc:276] 0709 12:24:40.894037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.169743 (+1275706us) service_pool.cc:224] Handling call
0709 12:24:42.170364 (+   621us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.170843 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.170876 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:42.170945 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:42.170993 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.171088 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.171262 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.973583 25051 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:41.984709 25072 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.247634 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:42.242616 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3913) took 1279 ms. Trace:
I20260709 12:24:42.221185 24655 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.221000 24815 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:42.287329 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.290205 25268 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.212s	user 0.000s	sys 0.002s
I20260709 12:24:42.301291 24914 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.302389 25231 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.303282 25231 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:42.306548 25266 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:42.307132 24916 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.273478 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:42.311882 25269 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 12:24:42.315493 18015 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:42.316285 25084 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.316814 25084 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.317229 25084 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.067163 25139 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.277153 24958 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:24:42.258046 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:42.286250 24883 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:42.307344 25266 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:42.265105 24892 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.302700 24914 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.279939 18094 rpcz_store.cc:276] 0709 12:24:40.942307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.238053 (+295746us) service_pool.cc:224] Handling call
0709 12:24:42.179781 (+941728us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:24:42.221410 (+ 41629us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6633856}
I20260709 12:24:42.325307 18809 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:42.325739 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.317734 18015 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:42.322000 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:42.340979 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80849.3us
I20260709 12:24:42.321344 24910 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:24:42.342752 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:42.319199 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:42.324535 25139 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:42.332424 24883 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.325073 25072 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.321532 25199 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.366950 25266 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.369412 18015 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:24:42.370867 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 225359us
I20260709 12:24:42.324540 25230 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:42.324777 25143 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:42.320509 24624 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:24:42.392390 25112 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:42.393688 25273 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.126s	user 0.002s	sys 0.000s
I20260709 12:24:42.398226 25143 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:42.411998 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.412912 25206 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:42.364607 24883 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:42.415686 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:42.422292 25112 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:42.427711 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 289978us
I20260709 12:24:42.369414 25266 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:42.427621 18806 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:42.451180 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:42.436230 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.467944 18807 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:42.467204 25147 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:42.412992 25265 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.466390 25112 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.490782 25070 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.491734 25070 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.492652 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:42.494524 18020 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:42.498857 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:42.467319 25232 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.493537 18018 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
W20260709 12:24:42.511353 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.511938 25143 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:42.512856 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:42.513440 24914 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.514309 25265 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.516098 25166 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.519943 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:42.522454 25113 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:42.523734 18093 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.784s	user 0.010s	sys 0.024s
I20260709 12:24:42.524585 25203 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:42.524406 25232 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.528810 25147 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.529654 25070 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.547537 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:42.532858 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:42.533277 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:42.533896 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:42.508812 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.531018 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:42.533295 24697 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.844s	user 0.004s	sys 0.019s
W20260709 12:24:42.560042 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.560931 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.562834 25113 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.567481 25012 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:42.567796 25252 raft_consensus.cc:2764] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:42.568295 25012 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:42.570194 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:42.576948 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57251.3us
I20260709 12:24:42.577713 18171 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:42.547277 25230 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.476106 25199 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.582437 18093 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:24:42.581235 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:42.601794 25270 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.084s	user 0.002s	sys 0.000s
I20260709 12:24:42.601557 25186 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:24:42.604327 24914 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:42.605473 25147 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.610378 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3812) took 1721 ms. Trace:
I20260709 12:24:42.610881 18023 rpcz_store.cc:276] 0709 12:24:40.888777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.134071 (+1245294us) service_pool.cc:224] Handling call
0709 12:24:42.134850 (+   779us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.135843 (+   993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.135905 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.137224 (+  1319us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.253376 (+116152us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:42.253415 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.260920 (+  7505us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.516995 (+256075us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b1800460900. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:42.609943 (+ 92948us) raft_consensus.cc:1685] finished
0709 12:24:42.609971 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.610268 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139353,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":266,"spinlock_wait_cycles":104725888,"thread_start_us":115472,"threads_started":1,"wal-append.queue_time_us":144529}
I20260709 12:24:42.600762 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:42.614560 18811 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:42.594074 25168 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:42.615645 25168 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:42.616250 25168 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:42.589372 25166 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.621127 25166 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.575098 25203 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:42.623026 18166 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:42.622402 25072 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
W20260709 12:24:42.609520 25083 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.733s	user 0.016s	sys 0.009s
W20260709 12:24:42.632006 24972 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.865s	user 0.018s	sys 0.003s
I20260709 12:24:42.632709 24972 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:42.633054 25001 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.782s	user 0.016s	sys 0.009s
I20260709 12:24:42.634850 18815 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:42.636857 25096 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.638252 18021 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
W20260709 12:24:42.641348 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 965) took 3012 ms (client timeout 1813 ms). Trace:
W20260709 12:24:42.644776 18018 rpcz_store.cc:269] 0709 12:24:39.628636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.757665 (+1129029us) service_pool.cc:224] Handling call
0709 12:24:42.641019 (+1883354us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003bb5d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:42.641224 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3804750720}
W20260709 12:24:42.646553 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 133927us
W20260709 12:24:42.647825 25242 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.597s	user 0.018s	sys 0.003s
I20260709 12:24:42.648424 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:42.649405 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3796) took 3552 ms (client timeout 2119 ms). Trace:
W20260709 12:24:42.649966 18021 rpcz_store.cc:269] 0709 12:24:39.096553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.113084 (+1016531us) service_pool.cc:224] Handling call
0709 12:24:42.649133 (+2536049us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54001fc054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:42.649305 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5216150784}
I20260709 12:24:42.650143 24624 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:42.651888 25102 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:42.652356 25102 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:42.652504 25203 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.653864 25102 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.655370 25102 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:42.657277 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2851) took 3429 ms (client timeout 2034 ms). Trace:
W20260709 12:24:42.657838 18020 rpcz_store.cc:269] 0709 12:24:39.228324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.171135 (+942811us) service_pool.cc:224] Handling call
0709 12:24:42.656947 (+2485812us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:42.657178 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5339260800}
I20260709 12:24:42.626284 25147 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.662178 25184 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.663040 25184 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.665160 24914 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:42.668469 18024 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.823s	user 0.007s	sys 0.013s
I20260709 12:24:42.669160 18024 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:42.578580 25012 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.671343 25186 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.672420 25186 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.637845 25096 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:42.625255 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47191.8us
I20260709 12:24:42.700215 18017 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:42.694427 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 971) took 1075 ms. Trace:
W20260709 12:24:42.703795 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 218969us
I20260709 12:24:42.701493 18024 rpcz_store.cc:276] 0709 12:24:41.600794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.771704 (+170910us) service_pool.cc:224] Handling call
0709 12:24:42.669766 (+898062us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.695327 25012 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:42.719261 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70982.4us
W20260709 12:24:42.724236 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 290888us
I20260709 12:24:42.727195 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3822) took 1166 ms. Trace:
I20260709 12:24:42.698215 25184 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.727887 18017 rpcz_store.cc:276] 0709 12:24:41.560433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.255710 (+695277us) service_pool.cc:224] Handling call
0709 12:24:42.675758 (+420048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.708803 (+ 33045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.726851 (+ 18048us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b540010aed0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:42.727096 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":808159488}
I20260709 12:24:42.706422 25113 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:42.693137 25165 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.714038 25206 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:42.746694 25278 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:24:42.746596 25184 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:42.691797 25230 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.743384 25206 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.768985 25206 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:42.745222 25166 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.751634 25001 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:42.797982 18811 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:42.796751 25051 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.104s
W20260709 12:24:42.805066 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 279351us
I20260709 12:24:42.774266 24958 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.796878 25166 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.778237 25184 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:42.776417 24914 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.819463 25230 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.795872 25265 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:42.815809 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:42.833952 25084 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.516s	user 0.015s	sys 0.003s
W20260709 12:24:42.850694 18016 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.015s	sys 0.045s
W20260709 12:24:42.852694 18016 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.016s	sys 0.046s
I20260709 12:24:42.854231 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 969) took 1505 ms. Trace:
I20260709 12:24:42.828814 25001 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:42.844458 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:42.865243 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3810) took 2577 ms. Trace:
I20260709 12:24:42.803043 25165 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.814620 24958 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.843781 25057 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.843662 25244 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.855754 25203 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:42.854862 25230 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:42.877458 25231 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.573s	user 0.014s	sys 0.003s
I20260709 12:24:42.903219 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:42.858260 25084 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.879570 25057 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.857208 18016 rpcz_store.cc:276] 0709 12:24:41.348609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.188967 (+840358us) service_pool.cc:224] Handling call
0709 12:24:42.189750 (+   783us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:42.191580 (+  1830us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.191626 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:42.192876 (+  1250us) log.cc:844] Serialized 52 byte log entry
0709 12:24:42.270588 (+ 77712us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:42.853399 (+582811us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.853720 (+   321us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.853845 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.853908 (+    63us) raft_consensus.cc:1685] finished
0709 12:24:42.853924 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.854153 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":230844,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":95431,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":5387,"thread_start_us":649731,"threads_started":2,"wal-append.queue_time_us":71977}
I20260709 12:24:42.865921 18019 rpcz_store.cc:276] 0709 12:24:40.181054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.127553 (+1946499us) service_pool.cc:224] Handling call
0709 12:24:42.128483 (+   930us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.129223 (+   740us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:42.436240 (+307017us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.745486 (+309246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:42.745561 (+    75us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:42.755178 (+  9617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.755304 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.757935 (+  2631us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34920,"prepare.queue_time_us":371692,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":31672,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":2304,"thread_start_us":449525,"threads_started":2,"wal-append.queue_time_us":369394}
I20260709 12:24:42.882381 25083 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:42.884657 25199 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.911725 25051 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:42.857784 25143 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
W20260709 12:24:42.923146 25275 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:24:42.923355 18811 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 20 index: 18. (index mismatch)
I20260709 12:24:42.924310 25230 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.926827 24910 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:24:42.926998 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44947.8us
I20260709 12:24:42.928026 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:42.928330 25184 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.912531 25001 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.935336 25165 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:42.937467 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.911434 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87479.4us
I20260709 12:24:42.866407 25113 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:24:42.888885 25186 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:42.921856 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:42.896616 25232 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:42.922164 25279 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.144s	user 0.002s	sys 0.000s
I20260709 12:24:42.907358 25057 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.904354 25231 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:42.956552 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:42.957388 25072 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:42.959245 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:42.960230 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:42.961009 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:42.968433 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79007.8us
I20260709 12:24:42.977080 18809 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:42.979413 18167 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:42.980234 18807 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:42.985915 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:42.990540 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.991274 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.954941 25199 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:43.032527 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58297.7us
I20260709 12:24:42.950846 25203 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:42.943161 25244 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.041266 25232 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:43.049022 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:43.064622 25282 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:24:43.071134 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:42.988837 25051 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.071870 18807 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:24:43.073282 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.054359 25252 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:43.095520 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 174567us
I20260709 12:24:43.096208 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:43.060240 25232 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:43.097255 18809 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:43.070971 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.100504 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:43.101293 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 116262us
I20260709 12:24:43.056243 24624 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:24:43.115660 18813 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 14: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:43.128098 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:43.070134 25199 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.132536 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.151979 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.156980 18164 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:43.157601 25134 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:43.158399 25134 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:43.158509 25285 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.109s
I20260709 12:24:43.159783 25134 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.161062 25288 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.156998 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.161958 25288 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.164292 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:43.166154 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.167857 24958 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:43.168565 25287 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:24:43.171248 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1940447640004095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
I20260709 12:24:43.156742 18017 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:43.162477 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.162732 18020 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:43.181341 18021 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:43.173944 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62760.9us
I20260709 12:24:43.196785 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:43.176779 25166 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:43.171301 25288 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.204433 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:43.226171 18020 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 12:24:43.227764 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.139222 25252 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.188581 24910 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:43.235842 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 147457us
W20260709 12:24:43.246135 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49514.9us
I20260709 12:24:43.257985 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.258342 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.258695 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:43.259195 18018 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:43.259459 18020 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:43.260087 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:43.263088 25051 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.239195 25166 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:43.265503 24910 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:43.267036 24883 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.789s	user 0.010s	sys 0.006s
I20260709 12:24:43.267535 24883 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.270164 25299 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:43.270035 25293 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 12:24:43.270756 25299 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:43.271070 25244 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.271596 25293 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:43.272171 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67890.2us
I20260709 12:24:43.208637 24914 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:24:43.257476 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:43.279450 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:43.281358 25284 raft_consensus.cc:2764] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:24:43.282327 25199 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:43.284075 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89444.1us
I20260709 12:24:43.259372 25290 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.286520 25290 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.289324 25290 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.290911 24914 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:43.291493 24914 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:43.292677 24914 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.277043 25072 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:43.294298 25072 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:43.294358 24653 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.364s	user 0.010s	sys 0.015s
I20260709 12:24:43.294795 25072 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:43.295222 24653 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.299114 25230 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.272024 25299 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.301430 25299 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.303328 25252 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.304236 25139 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:43.304725 25139 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:43.306236 25147 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:43.306721 25147 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:43.307829 25147 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.309353 25057 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.310829 25203 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.314198 25304 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.315084 25304 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.315987 25231 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.320358 18017 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.272116 25293 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:43.352069 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 94775.6us
W20260709 12:24:43.299183 25112 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.745s	user 0.017s	sys 0.000s
I20260709 12:24:43.309643 25147 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:43.280220 25051 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.364886 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:43.317786 25304 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.369393 25293 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.371546 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:43.366453 25012 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.642s	user 0.008s	sys 0.008s
I20260709 12:24:43.362339 24958 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:43.349671 25285 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:43.363472 24883 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:43.361016 25199 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.350314 25184 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:24:43.412808 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:43.413448 25186 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:43.413928 25186 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:43.414175 18064 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.034s (SENT)
I20260709 12:24:43.414762 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:43.414983 25186 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.371968 24653 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.410280 25304 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:43.406210 25231 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.371224 25203 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.363426 25166 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:24:43.277530 25232 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.416651 25186 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:43.407357 25113 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:43.326845 25139 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.407544 25057 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:43.400302 24958 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.366420 25112 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:43.431259 25102 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.775s	user 0.019s	sys 0.000s
I20260709 12:24:43.421173 25184 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:43.446556 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.446513 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.428375 24841 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.399237 25051 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.439131 24914 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.468356 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.469141 18093 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:43.470043 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.470916 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:43.461490 25312 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.472399 18091 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:43.472937 25312 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.474238 25088 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:43.435823 25231 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.474790 25088 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.475219 25088 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:43.443864 25165 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.476786 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.462384 24883 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
I20260709 12:24:43.477725 18092 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:43.479408 25309 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:43.460310 18166 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:43.480195 25309 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.469689 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.482630 18096 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:43.452672 25304 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.485071 18806 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:43.484798 24841 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.460170 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:43.456804 25203 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.486263 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:43.487205 18809 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:43.469249 18090 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:43.482529 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.410212 25199 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:43.516573 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56545us
I20260709 12:24:43.513329 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:43.513773 25298 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:24:43.515359 25303 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:43.515925 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.450434 24958 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:43.534379 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.430542 25285 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.542027 18020 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:43.556033 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.549664 18095 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:43.557247 18098 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:43.476222 25165 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.555246 18164 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:43.529384 24653 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:43.570483 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 205670us
I20260709 12:24:43.574782 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:43.576417 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:43.567343 24653 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.579277 24653 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:43.582247 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.583153 18092 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:43.584666 25293 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.586917 25199 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.587837 25312 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.588248 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.588667 25203 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.590574 25304 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:43.591168 25168 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.974s	user 0.022s	sys 0.002s
I20260709 12:24:43.591053 25165 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.591085 25308 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:43.591696 24653 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.592319 24841 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:43.594251 18093 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:24:43.600876 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.601231 25084 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.592314 25305 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.606942 25096 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:43.562561 24892 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
W20260709 12:24:43.623099 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 222452us
I20260709 12:24:43.573652 25112 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:43.618577 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43621.6us
I20260709 12:24:43.632476 25184 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:43.623819 25311 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:43.694797 18097 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:43.695667 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.681874 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.680130 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:43.682880 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 104293us
W20260709 12:24:43.683622 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.691350 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.708184 18809 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:43.708642 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.715498 18094 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
W20260709 12:24:43.718221 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.622176 25290 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.725129 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.725955 18093 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.726991 18092 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:43.694818 25184 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.729061 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.729699 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:43.730389 18806 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.735311 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.735976 18091 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:43.730129 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:43.707312 25309 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.706985 25252 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.737056 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.744320 18098 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
W20260709 12:24:43.723384 25265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.004s	sys 0.019s
I20260709 12:24:43.746534 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.747782 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:43.748957 18024 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:43.749454 18020 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:43.746701 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.751351 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:43.675366 25168 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.757198 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.731238 25308 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.764531 25311 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.766976 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.770208 18018 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:43.734069 25184 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.732915 25305 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.776393 25308 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.779448 25308 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:43.756313 25252 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.782531 25252 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:43.790058 25290 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.792896 25290 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.796176 18807 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:43.800770 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:43.801090 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.801707 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.807507 24883 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:43.809628 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.812276 25325 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:43.814121 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.814484 18097 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:24:43.815483 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80681.5us
I20260709 12:24:43.816576 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:43.816661 18017 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:43.761111 25289 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.739648 25285 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:43.847661 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:43.797672 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:43.852977 25326 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.853983 25326 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.822501 25184 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:43.824050 25165 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.824489 25321 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.824050 18809 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:43.826858 25051 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.827047 25231 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.830550 25139 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:43.731909 25288 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.101s
I20260709 12:24:43.841665 18095 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:24:43.755033 25265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.004s	sys 0.020s
I20260709 12:24:43.863358 18811 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:43.863802 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:43.874914 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74246.7us
I20260709 12:24:43.875929 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:43.880656 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.881181 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:43.881507 18168 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:43.857712 25165 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:43.887918 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.889918 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.857462 25326 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.889400 25165 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.880203 18022 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:43.898121 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.902175 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.857769 25321 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.907562 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.910547 18164 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:43.903041 18097 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:43.903445 18098 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:43.904271 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:43.906766 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.914867 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:43.908293 18167 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:43.921684 18022 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:43.874037 25084 raft_consensus.cc:2764] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:43.874214 25285 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:43.873804 24653 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:43.926649 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:43.928532 18813 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:43.898452 25231 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:43.937860 18807 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:43.940629 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:43.944284 18778 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.692s (SENT)
I20260709 12:24:43.900367 25057 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:43.936496 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.945405 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:43.946277 18097 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:43.908362 25051 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:43.958662 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:43.954252 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:43.973599 18173 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:43.976352 25206 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.172s	user 0.012s	sys 0.015s
I20260709 12:24:43.978576 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.979377 18094 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
I20260709 12:24:43.980388 18094 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:24:43.981207 18094 raft_consensus.cc:606] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:43.982306 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 184776us
I20260709 12:24:43.982594 24624 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.983428 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:43.985941 25320 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:43.986085 25290 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:43.983240 24624 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:43.997773 18809 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:44.000931 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:43.989434 25186 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.007s
W20260709 12:24:44.008193 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.934381 25285 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.010083 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.015519 18093 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:44.017165 18024 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:44.018458 18017 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:44.019066 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.009701 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.021137 18806 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:44.023231 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:43.951439 25323 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:43.947379 25231 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:43.977144 25206 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.022985 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:44.054420 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.054673 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.055616 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.056077 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.056169 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.056329 18018 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:43.918320 25289 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:44.007885 24914 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.008s
I20260709 12:24:44.031271 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.068980 18024 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:24:44.069619 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.071152 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.071676 25289 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.057515 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.074445 18018 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:44.075832 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 131588us
I20260709 12:24:44.077165 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:44.078549 25084 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:44.079155 25084 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:44.080590 25084 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.072582 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.083573 18167 raft_consensus.cc:2368] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:44.033545 18098 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 13.
W20260709 12:24:44.034123 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79752.1us
I20260709 12:24:44.034983 18168 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 18. Ops: []
I20260709 12:24:44.009903 25230 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:44.031234 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.093070 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.036818 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.037024 18164 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:44.037744 25231 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.029356 25323 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:44.040591 24958 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.042632 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.046984 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.047833 25244 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:43.986099 25096 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.050838 18097 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 20 index: 18. (index mismatch)
I20260709 12:24:44.159869 25096 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:44.160809 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.052552 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:44.162303 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127458us
I20260709 12:24:44.060834 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.166536 18021 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:44.161543 18097 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:44.056468 25206 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:44.077502 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.807s (SENT)
W20260709 12:24:44.088761 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.030440 25308 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 12:24:44.019327 25285 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:44.095176 18020 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.954658 25326 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.085s
W20260709 12:24:44.104255 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.109014 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.110059 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.110169 24958 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:44.109764 25323 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.105994 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.110999 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.111737 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.115828 24624 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:44.119048 25184 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:44.027923 25320 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.094s
I20260709 12:24:44.109077 25168 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:44.109079 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.221143 18095 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:44.131321 25299 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.829s	user 0.012s	sys 0.000s
I20260709 12:24:44.136767 25321 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.066874 25311 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:44.094276 25305 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.097318 24841 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:24:44.158058 25266 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:44.168979 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55476.1us
I20260709 12:24:44.188211 18021 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:24:44.191905 18015 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:44.196508 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 119104us
I20260709 12:24:44.203812 18020 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:44.204427 18172 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:44.206540 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.231786 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.232605 18093 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.232878 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.208366 18169 raft_consensus.cc:2368] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:44.193748 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:44.197994 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:44.190939 25088 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.715s	user 0.006s	sys 0.018s
I20260709 12:24:44.210350 18091 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:44.215353 18165 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:44.217344 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.227005 18166 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:44.210119 25289 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:44.233079 18098 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:44.201516 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:44.216012 25293 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.593s	user 0.013s	sys 0.000s
I20260709 12:24:44.244647 18170 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:44.247373 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.248759 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.249346 18170 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:44.231133 25057 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:44.220670 25244 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:24:44.251173 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.252152 18164 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:44.253945 25088 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.256722 25088 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:44.238788 18167 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:44.258878 18167 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:44.260315 25326 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.260978 25326 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:44.263407 25326 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.264231 18168 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:44.266599 18019 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:44.267576 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.253651 18164 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:24:44.268770 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.273965 18171 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:44.277220 18015 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:44.277602 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.240438 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:44.284791 18097 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:44.269457 18091 raft_consensus.cc:2376] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13. Re-sending same reply.
I20260709 12:24:44.287640 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.288437 18091 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:44.289624 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.291685 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:44.292145 18164 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:44.293298 18814 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:44.294876 25051 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:44.295702 25051 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:44.266516 25096 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:44.266912 25323 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:44.301666 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:44.302795 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:44.303508 18806 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.241048 25299 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:44.305934 25299 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:44.306197 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:44.306973 25289 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:44.308251 25289 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:44.310989 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.305531 25088 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:44.312218 18090 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:24:44.313527 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.310937 18095 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:44.211457 25290 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
I20260709 12:24:44.316286 18090 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:44.306409 25299 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.293411 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.320709 18094 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:44.315753 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:44.276371 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 428725us
I20260709 12:24:44.330274 25290 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:44.330724 25290 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:44.331271 18169 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:44.331797 25290 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.332499 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:44.310894 25072 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.015s	user 0.019s	sys 0.005s
I20260709 12:24:44.329979 25266 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:44.333981 18095 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:24:44.334296 25072 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.335276 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.336241 18095 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 15: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:44.337379 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.338805 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.340271 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:44.342317 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.346670 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.357856 18093 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:44.361362 18098 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:44.315935 25332 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:24:44.333633 25290 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:44.402249 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86687.3us
I20260709 12:24:44.403709 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:44.404882 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.406617 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.407353 18091 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:44.414531 18091 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:24:44.424297 25285 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.425758 18807 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:44.428406 25072 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.429453 25072 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:44.430037 25072 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:44.435458 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2148162620001131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
I20260709 12:24:44.447568 25285 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:44.367157 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126820us
I20260709 12:24:44.454306 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.367743 18168 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:44.368723 18815 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:24:44.296980 25051 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.374224 18016 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14.
I20260709 12:24:44.374915 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.379302 18806 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:44.389292 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49116.5us
I20260709 12:24:44.370802 24691 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:44.392297 17916 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 284) took 22668 ms (client timeout 15000 ms). Trace:
I20260709 12:24:44.392946 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2938) took 1861 ms. Trace:
I20260709 12:24:44.393525 25134 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:44.456559 18169 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:44.468706 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:44.465728 18094 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:44.469956 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:44.471480 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:44.445192 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:44.471760 18173 rpcz_store.cc:276] 0709 12:24:42.531379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.534022 (+  2643us) service_pool.cc:224] Handling call
0709 12:24:42.534500 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:43.974374 (+1439874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.392570 (+418196us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540029dbd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:44.392841 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3067049344}
I20260709 12:24:44.478226 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.476411 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.503685 18171 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:24:44.482111 25325 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:44.506749 25328 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.441s	user 0.002s	sys 0.002s
I20260709 12:24:44.506430 25325 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.504086 25084 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:44.503455 25340 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:24:44.510439 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65371.7us
I20260709 12:24:44.509131 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:44.483043 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.482688 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:44.543506 18164 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:44.483296 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.517470 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:44.557242 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:44.477150 25337 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:24:44.516002 18092 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:44.516927 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:44.565085 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48238.6us
I20260709 12:24:44.566725 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:44.568022 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3936) took 1012 ms. Trace:
I20260709 12:24:44.567517 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.569296 18095 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:44.475468 25134 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:44.516927 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:44.522812 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:24:44.513586 25051 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:44.503695 25311 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:44.471259 25300 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:24:44.571477 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54634.3us
W20260709 12:24:44.582759 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.509s (SENT)
I20260709 12:24:44.510336 24691 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:44.572934 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:44.513890 25088 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:44.577172 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.510640 25293 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:24:44.596681 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:44.579481 18097 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:44.515314 25325 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:44.471423 17916 rpcz_store.cc:269] 0709 12:24:21.724218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.724665 (+   447us) service_pool.cc:224] Handling call
0709 12:24:23.261145 (+1536480us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16621090 00007f6c166bff3a 00007f6c1c475e9c 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:24:44.391907 (+21130762us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:40.688641 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:40.689511 (+   870us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:40.689610 (+    99us) write_op.cc:435] Acquired schema lock
0709 12:24:40.689634 (+    24us) tablet.cc:662] Decoding operations
0709 12:24:40.758480 (+ 68846us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:40.758610 (+   130us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:40.758643 (+    33us) tablet.cc:685] Acquiring locks for 22 operations
0709 12:24:40.761402 (+  2759us) tablet.cc:701] Row locks acquired
0709 12:24:40.761452 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:24:40.761695 (+   243us) write_op.cc:270] Start()
0709 12:24:40.791464 (+ 29769us) write_op.cc:276] Timestamp: P: 1783599880761622 usec, L: 0
0709 12:24:40.821745 (+ 30281us) op_driver.cc:348] REPLICATION: starting
0709 12:24:40.903171 (+ 81426us) log.cc:844] Serialized 8633 byte log entry
0709 12:24:41.306286 (+403115us) op_driver.cc:464] REPLICATION: finished
0709 12:24:41.525064 (+218778us) write_op.cc:301] APPLY: starting
0709 12:24:41.533279 (+  8215us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:42.016310 (+483031us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:42.016507 (+   197us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:42.075547 (+ 59040us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:42.325525 (+249978us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=44,mrs_lookups=0
0709 12:24:42.325641 (+   116us) write_op.cc:312] APPLY: finished
0709 12:24:42.344796 (+ 19155us) log.cc:844] Serialized 196 byte log entry
0709 12:24:42.554459 (+209663us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:42.609902 (+ 55443us) write_op.cc:454] Released schema lock
0709 12:24:42.610016 (+   114us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:42.610789 (+   773us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:42.610829 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2675923,"reported_tablets":50,"spinlock_wait_cycles":3360107392,"child_traces":[["op",{"apply.queue_time_us":170409,"apply.run_cpu_time_us":64558,"apply.run_wall_time_us":1107129,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96219,"num_ops":22,"prepare.queue_time_us":605835,"prepare.run_cpu_time_us":51374,"prepare.run_wall_time_us":422322,"raft.queue_time_us":28819,"raft.run_cpu_time_us":36114,"raft.run_wall_time_us":151153,"replication_time_us":484346,"thread_start_us":916607,"threads_started":5,"wal-append.queue_time_us":677930,"wal-append.run_cpu_time_us":25584,"wal-append.run_wall_time_us":2048427}]]}
I20260709 12:24:44.518738 25339 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:44.568779 18096 rpcz_store.cc:276] 0709 12:24:43.555965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.556162 (+   197us) service_pool.cc:224] Handling call
0709 12:24:43.831195 (+275033us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:43.831510 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:43.831537 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:43.832176 (+   639us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.030700 (+198524us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:44.030748 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.030841 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.112362 (+ 81521us) raft_consensus.cc:1685] finished
0709 12:24:44.112387 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.533205 (+420818us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400214150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:44.567946 (+ 34741us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":605535488,"thread_start_us":198038,"threads_started":1,"wal-append.queue_time_us":237072}
I20260709 12:24:44.581856 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:44.588368 18019 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:44.590939 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:44.530408 25147 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.159s	user 0.014s	sys 0.001s
W20260709 12:24:44.602754 18018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.023s	sys 0.014s
I20260709 12:24:44.603772 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:44.604038 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.565024 25072 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.513510 25084 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:44.569628 25001 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:44.608896 18017 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:44.578202 25134 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:44.608808 18018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.023s	sys 0.014s
I20260709 12:24:44.576390 25285 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:44.580801 25321 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:44.610445 18099 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:44.610975 18095 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:24:44.616112 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.580035 25139 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:44.623214 18098 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:44.619808 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 168339us
I20260709 12:24:44.620154 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.590579 25184 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:44.545594 25231 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.602716 25311 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:44.603463 25206 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:44.646060 18097 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:44.586645 25057 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:24:44.619089 24624 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:44.660957 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78262.3us
I20260709 12:24:44.661471 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.662088 25057 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:44.663918 25184 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:44.664325 25184 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:44.664783 25084 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:44.665887 18093 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:44.666067 18808 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:44.666386 18098 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:44.670051 25206 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.670624 25206 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:44.654367 24691 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.674577 25311 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:44.659585 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.661149 24624 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:44.677942 24624 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.645078 25001 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:44.665297 25184 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:44.682893 25001 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.631026 25232 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:44.687954 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:44.627308 25321 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:44.690930 18778 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.174s (SENT)
I20260709 12:24:44.617758 25285 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:44.628019 25339 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:44.626894 25139 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:44.637959 25301 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:44.710496 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.721032 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.723596 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:44.733024 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 90665.5us
I20260709 12:24:44.647164 25300 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
W20260709 12:24:44.742198 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.189s (SENT)
I20260709 12:24:44.742959 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:44.701860 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.702894 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.705212 25301 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.607513 24958 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 12:24:44.644245 24892 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:44.698297 25285 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:44.695442 25321 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:44.740641 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:44.704818 25139 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:44.784296 25072 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:44.802433 25301 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.606623 25284 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.082s
I20260709 12:24:44.748644 25147 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:44.750072 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.217s (SENT)
I20260709 12:24:44.759436 18090 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
W20260709 12:24:44.763690 25344 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:24:44.771507 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.753571 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.778255 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.835680 18020 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:44.779383 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.837190 18024 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:44.781772 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:44.790963 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.834671 18021 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:44.799036 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:44.802100 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3953) took 1204 ms. Trace:
I20260709 12:24:44.807576 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.700898 18170 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:24:44.847729 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66113.9us
I20260709 12:24:44.834493 18019 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:44.842509 18091 rpcz_store.cc:276] 0709 12:24:43.616980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.422428 (+805448us) service_pool.cc:224] Handling call
0709 12:24:44.422875 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.423131 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:44.439961 (+ 16830us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.801608 (+361647us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:44.801705 (+    97us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:44.801760 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.801859 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.802021 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2712,"prepare.queue_time_us":16268,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":2922,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":5011,"thread_start_us":301136,"threads_started":1}
W20260709 12:24:44.863600 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:44.904471 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 154444us
I20260709 12:24:44.909866 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.910923 18016 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:44.915818 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.916654 18020 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:24:44.822697 25339 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.847216 25096 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:44.927435 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.225s (SENT)
I20260709 12:24:44.928256 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:44.866096 25088 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:44.825795 25072 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:44.962494 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.967795 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.866482 25112 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:24:44.989257 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:44.903466 25232 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.068s
W20260709 12:24:45.028278 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 366399us
I20260709 12:24:44.964573 25051 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.025939 18016 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:45.031867 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 407624us
I20260709 12:24:44.972410 25088 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:45.053642 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 413965us
W20260709 12:24:45.057696 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1246408189999784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260709 12:24:45.058709 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0643408189998809s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20260709 12:24:45.059170 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0634408189998794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20260709 12:24:45.059602 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0175408189998052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
W20260709 12:24:45.060040 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0175408189998052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
I20260709 12:24:45.049621 24958 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:24:45.107132 18094 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.011s	sys 0.024s
W20260709 12:24:45.121776 18094 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.011s	sys 0.024s
I20260709 12:24:45.128216 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:45.129729 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68835.9us
W20260709 12:24:45.134383 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 227531us
I20260709 12:24:45.062958 25325 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:45.149881 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:45.152714 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:45.155054 18778 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:45.156675 18138 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.119s (SENT)
W20260709 12:24:45.138909 25347 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.396s	user 0.002s	sys 0.000s
W20260709 12:24:45.163932 25349 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:24:45.099543 25134 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.167752 18815 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:45.168943 25134 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:45.169235 18808 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:45.122512 24958 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:45.170953 24958 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.165534 25231 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:45.172264 25231 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:45.172534 24958 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:45.169622 25134 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.174732 25134 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.175202 25285 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.177435 25134 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:45.178890 25290 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:45.179327 25299 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.809s	user 0.015s	sys 0.000s
I20260709 12:24:45.190420 25088 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.191421 25299 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:45.194741 25351 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:24:45.195037 25299 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:45.177780 25285 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:45.195725 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.196252 25285 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:45.157406 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:45.119146 25147 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:45.207134 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.213984 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:45.223158 18171 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:45.108052 25232 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:24:45.053949 25051 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:45.108312 25084 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
W20260709 12:24:45.241181 18099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.018s	sys 0.015s
I20260709 12:24:45.242030 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:45.244238 18099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.019s	sys 0.015s
I20260709 12:24:45.244765 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.254637 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:45.258405 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3961) took 1001 ms. Trace:
W20260709 12:24:45.251101 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.023s	sys 0.033s
W20260709 12:24:45.259461 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.024s	sys 0.033s
W20260709 12:24:45.260046 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 636174us
I20260709 12:24:45.144583 25323 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:24:45.258927 18099 rpcz_store.cc:276] 0709 12:24:44.257239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.643591 (+386352us) service_pool.cc:224] Handling call
0709 12:24:44.685430 (+ 41839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.685694 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:44.686092 (+   398us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.248136 (+562044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:45.248227 (+    91us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:45.258068 (+  9841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.258162 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.258331 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":218,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":136,"thread_start_us":561451,"threads_started":1}
I20260709 12:24:45.239646 25096 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:45.300992 18023 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.023s	sys 0.029s
I20260709 12:24:45.223866 25325 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:45.308487 25139 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:45.318081 18018 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.017s	sys 0.048s
W20260709 12:24:45.318794 18018 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.018s	sys 0.048s
I20260709 12:24:45.327366 18165 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:45.238644 25147 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.329480 24691 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:24:45.340323 25266 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:45.301656 18023 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.023s	sys 0.029s
I20260709 12:24:45.342018 25266 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:45.344841 25300 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:45.345420 25300 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:45.346117 25319 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:45.346581 25353 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:24:45.346779 25319 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:45.348071 25319 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:45.350301 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1316) took 1052 ms. Trace:
I20260709 12:24:45.350979 25300 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:45.350845 18814 rpcz_store.cc:276] 0709 12:24:44.297704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.314967 (+ 17263us) service_pool.cc:224] Handling call
0709 12:24:44.363590 (+ 48623us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.376514 (+ 12924us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.376552 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.479249 (+102697us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.838277 (+359028us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:44.838327 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.838418 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:45.349936 (+511518us) raft_consensus.cc:1685] finished
0709 12:24:45.349967 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.350212 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17431,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"thread_start_us":296596,"threads_started":1,"wal-append.queue_time_us":517205}
I20260709 12:24:45.353160 25300 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:45.354372 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 204570us
I20260709 12:24:45.358282 18171 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:45.359428 18170 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: [17.12-19.13]
I20260709 12:24:45.361968 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:45.362453 25184 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:45.362969 25051 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:45.370941 25057 raft_consensus.cc:2764] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:24:45.371212 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:45.372062 18099 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:45.373659 25325 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:45.372076 18090 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
W20260709 12:24:45.379400 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.384035 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3826) took 1493 ms. Trace:
I20260709 12:24:45.384724 18018 rpcz_store.cc:276] 0709 12:24:43.826174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.075169 (+248995us) service_pool.cc:224] Handling call
0709 12:24:44.075635 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:44.076165 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.076196 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:44.076891 (+   695us) log.cc:844] Serialized 52 byte log entry
0709 12:24:44.611491 (+534600us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:45.319275 (+707784us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.319564 (+   289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.319662 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:45.319706 (+    44us) raft_consensus.cc:1685] finished
0709 12:24:45.319717 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.319857 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":106,"thread_start_us":1213114,"threads_started":2,"wal-append.queue_time_us":703121}
I20260709 12:24:45.415154 18164 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:45.425252 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:45.426671 18090 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:45.416596 25311 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.005s	sys 0.033s
W20260709 12:24:45.416817 25206 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.746s	user 0.016s	sys 0.000s
W20260709 12:24:45.417779 24624 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.736s	user 0.008s	sys 0.008s
I20260709 12:24:45.385691 24958 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:45.469316 24958 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.473232 24958 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:45.434721 25311 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.005s	sys 0.034s
W20260709 12:24:45.483345 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.507973 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.481375 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:45.504994 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 461943us
I20260709 12:24:45.514312 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:45.524448 18099 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:45.526424 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:45.544935 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:45.463975 24624 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:45.519307 25285 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.457182 25206 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:45.559531 18779 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.033s	sys 0.029s
I20260709 12:24:45.557511 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:45.564985 18779 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.033s	sys 0.030s
I20260709 12:24:45.560436 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:45.526377 25134 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:45.564180 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 436063us
I20260709 12:24:45.565671 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:45.552197 25362 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.545344 24958 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:45.576538 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.581177 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.582031 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:45.583410 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:45.560060 25051 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.589547 18022 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:45.602648 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:45.588306 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43484.6us
I20260709 12:24:45.612717 18021 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:45.584563 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:45.607692 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:45.618400 18020 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:45.378122 17913 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 11 to 12, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 12 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:45.623786 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:45.624830 18018 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:24:45.627745 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.575783 25363 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:45.579046 24958 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:45.640024 18781 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.007s	sys 0.024s
W20260709 12:24:45.640435 18781 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.007s	sys 0.025s
I20260709 12:24:45.568583 25285 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:45.658551 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 503585us
W20260709 12:24:45.675395 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.736754 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 605835us
I20260709 12:24:45.737625 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:45.639286 25363 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:45.751567 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104718us
I20260709 12:24:45.756291 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:45.709370 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 3.104s (SENT) [suppressed 21 similar messages]
I20260709 12:24:45.645318 24958 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:45.634107 25376 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:45.765882 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56577.1us
I20260709 12:24:45.613783 25051 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.775547 18021 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:45.669992 25370 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:45.656869 25139 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:45.589138 25362 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:45.671300 25368 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:45.790283 18064 leader_election.cc:400] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:45.715090 25147 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:45.763599 24892 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.739560 25339 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:45.765584 25363 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:45.812808 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 247197us
I20260709 12:24:45.816251 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:45.784425 25088 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:45.662356 25325 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.136s
W20260709 12:24:45.824887 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85828.8us
I20260709 12:24:45.827988 24892 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.831662 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:45.828733 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 156296us
I20260709 12:24:45.833163 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:45.833441 24892 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:45.784708 25375 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:45.789628 25232 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:45.804122 25370 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:45.842022 25366 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:24:45.842338 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:45.842965 25372 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
W20260709 12:24:45.844210 25378 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:24:45.845314 25072 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:45.857175 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.846251 25383 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:45.835227 25358 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.317s	user 0.001s	sys 0.001s
I20260709 12:24:45.853742 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:45.871832 18807 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:45.876406 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.867480 25321 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.046s	user 0.016s	sys 0.002s
I20260709 12:24:45.853742 25376 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:45.860623 25365 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 12:24:45.858809 25232 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.898481 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.900271 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.847195 24624 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:45.850775 25088 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:45.909730 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.910421 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.817552 25368 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.859056 25339 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:45.866626 25289 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:45.920030 18171 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:45.930256 18165 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:45.900271 25354 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.346s	user 0.000s	sys 0.001s
I20260709 12:24:45.944900 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:45.936216 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.222s (SENT)
I20260709 12:24:45.946941 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.957621 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.959095 18171 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:45.885427 25379 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:45.969024 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.971046 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.976986 18170 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:45.931399 25355 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.344s	user 0.001s	sys 0.000s
I20260709 12:24:45.960588 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.952521 25375 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:24:45.988808 18169 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:45.991742 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:46.001287 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 248557us
W20260709 12:24:46.000572 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 160251us
I20260709 12:24:45.912518 25386 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:46.026321 18172 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:45.928874 25232 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.007201 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.029017 18807 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:45.932333 25376 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:45.964177 25206 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.009332 25088 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.010200 25368 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:46.014664 25377 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 12:24:45.911855 25382 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.121s
I20260709 12:24:45.876282 17913 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 16, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 16 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:45.994848 25379 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.035961 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.054745 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.034240 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 989) took 1459 ms. Trace:
I20260709 12:24:46.040316 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.056933 18023 rpcz_store.cc:276] 0709 12:24:44.575191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.575343 (+   152us) service_pool.cc:224] Handling call
0709 12:24:44.754599 (+179256us) spinlock_profiling.cc:243] Waited 8.83 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:44.754921 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.755495 (+   574us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.755534 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.756296 (+   762us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.342189 (+585893us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:45.342237 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.342313 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.022883 (+680570us) raft_consensus.cc:1685] finished
0709 12:24:46.022907 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.034131 (+ 11224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":148,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":131,"spinlock_wait_cycles":20317952,"thread_start_us":585365,"threads_started":1,"wal-append.queue_time_us":770794}
I20260709 12:24:46.049248 25382 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:46.068823 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.042272 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:45.997424 25339 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.923975 25321 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.050443 25325 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.050037 24958 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.061206 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.061539 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.030671 25386 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:46.070557 25367 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.414s	user 0.001s	sys 0.001s
W20260709 12:24:46.074015 25301 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.228s	user 0.016s	sys 0.002s
W20260709 12:24:46.079049 25373 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.284s	user 0.000s	sys 0.002s
I20260709 12:24:46.006520 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.082309 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.079756 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.080036 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.054092 25379 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.071246 18172 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:46.040023 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:46.125172 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45271.8us
I20260709 12:24:46.128515 18018 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:24:46.102439 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.128410 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.129993 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:46.141564 18024 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:46.129658 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1130) took 1791 ms. Trace:
I20260709 12:24:46.141992 18096 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:46.139644 18171 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:24:46.141521 18015 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:24:46.142524 18097 rpcz_store.cc:276] 0709 12:24:44.338275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.652281 (+314006us) service_pool.cc:224] Handling call
0709 12:24:44.652628 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.691950 (+ 39322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.692007 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.800651 (+108644us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.394309 (+593658us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:45.394376 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.396176 (+  1800us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.129369 (+733193us) raft_consensus.cc:1685] finished
0709 12:24:46.129398 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.129556 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37069,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":139,"thread_start_us":475054,"threads_started":1,"wal-append.queue_time_us":816765}
I20260709 12:24:46.077503 25285 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.106027 24624 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.092478 25096 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:46.071683 25206 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:46.098567 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.159314 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.093226 25383 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:46.127883 25321 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:46.196949 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56474.5us
I20260709 12:24:46.189862 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:46.202601 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.221364 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77224.1us
I20260709 12:24:46.131354 24958 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.150167 25364 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:46.123430 25382 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:46.148546 25375 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.267961 18024 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:46.094260 25381 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.093395 25386 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:46.127223 25385 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
W20260709 12:24:46.259505 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 253082us
I20260709 12:24:46.222501 25362 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.259016 18091 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 12:24:46.044802 25370 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.134583 25051 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.132419 25376 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.248059 25244 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:46.079353 25339 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.200771 25096 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:46.278751 18091 pending_rounds.cc:85] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:24:46.253355 25285 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.123062 25325 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.264432 25382 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:46.188969 25363 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.137902 25301 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.270707 25380 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.221766 25206 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.303807 25285 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.315145 24958 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:46.316047 24958 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.316606 24958 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.318254 25285 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:46.318109 25382 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.319670 24958 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.320066 25382 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:46.320628 24958 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:46.274776 25375 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.324528 25375 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:46.326136 25383 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.328512 25206 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.307308 25051 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:46.329355 25051 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:46.330991 25421 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.330685 25051 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.331745 25421 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.332532 25051 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:46.290843 18091 pending_rounds.cc:107] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:24:46.207712 24624 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.336447 18091 raft_consensus.cc:2889] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:46.279274 25413 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:46.337529 25413 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.337754 25363 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.338231 25413 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:46.344069 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2966) took 1131 ms. Trace:
W20260709 12:24:46.355502 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 108257us
I20260709 12:24:46.282249 25381 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:46.362036 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84183.5us
I20260709 12:24:46.344633 18164 rpcz_store.cc:276] 0709 12:24:45.212593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.212780 (+   187us) service_pool.cc:224] Handling call
0709 12:24:45.245196 (+ 32416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.415905 (+170709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.343779 (+927874us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400383954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:46.343990 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":385555840}
I20260709 12:24:46.291076 25096 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.368078 18064 leader_election.cc:400] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 16. Current term is 17.
I20260709 12:24:46.364357 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:46.376475 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.387650 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.389262 25325 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.367856 25139 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.334378 25380 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.357085 25376 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.123658 17913 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 14 to 17, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 17 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:46.389547 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.416523 18024 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:46.402565 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.405974 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.417759 18806 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:46.367559 25392 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.405984 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.382417 25399 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:46.455420 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92443.2us
I20260709 12:24:46.419355 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:46.424292 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:46.420195 25376 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.427677 17913 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 11 to 14, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 14 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:46.390569 25139 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.444033 25301 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.445623 25421 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:46.447096 25394 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
W20260709 12:24:46.448161 25184 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.085s	user 0.013s	sys 0.004s
I20260709 12:24:46.448589 25414 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.449494 25407 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.449658 25424 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.450471 24624 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:46.450695 25397 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.370s	user 0.001s	sys 0.000s
I20260709 12:24:46.452905 25339 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:46.454811 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.462894 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.382063 25231 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.077s
W20260709 12:24:46.448192 25388 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 12:24:46.474072 25380 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.455696 25399 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.481282 18017 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
W20260709 12:24:46.485096 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60945.6us
I20260709 12:24:46.465768 25375 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:46.501106 18096 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
W20260709 12:24:46.531824 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 449602us
I20260709 12:24:46.534765 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.458912 25392 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.501893 25362 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:46.529873 25319 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:46.529218 25321 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.531595 25399 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:46.539600 25325 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:46.540818 25088 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.538125 25381 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.532936 25375 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.557674 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3971) took 1081 ms. Trace:
I20260709 12:24:46.560467 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3967) took 1116 ms. Trace:
I20260709 12:24:46.549439 25362 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.575871 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.484045 25414 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.553076 25383 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:46.563067 18098 rpcz_store.cc:276] 0709 12:24:45.476428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.477294 (+   866us) service_pool.cc:224] Handling call
0709 12:24:45.477748 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:45.480401 (+  2653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.480452 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:45.520216 (+ 39764us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.787158 (+266942us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:45.787522 (+   364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.787612 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.557416 (+769804us) raft_consensus.cc:1685] finished
0709 12:24:46.557441 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.557603 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11126,"raft.queue_time_us":2794,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":13841,"thread_start_us":217948,"threads_started":1,"wal-append.queue_time_us":469051}
I20260709 12:24:46.497530 25407 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:46.582150 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 305804us
W20260709 12:24:46.584646 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128248us
W20260709 12:24:46.586674 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.580822 25362 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.603657 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:46.604754 25422 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:46.605708 25422 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:46.501444 25424 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.606146 25422 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:46.606554 25422 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:46.608177 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3093) took 2808 ms. Trace:
I20260709 12:24:46.608282 25424 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.609195 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3964) took 1247 ms. Trace:
I20260709 12:24:46.609609 18099 rpcz_store.cc:276] 0709 12:24:45.361383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.361548 (+   165us) service_pool.cc:224] Handling call
0709 12:24:46.608970 (+1247422us) spinlock_profiling.cc:243] Waited 9.25 ms on lock 0x7b5400215054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:46.609104 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21280000}
I20260709 12:24:46.613418 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2967) took 1400 ms. Trace:
I20260709 12:24:46.613953 18167 rpcz_store.cc:276] 0709 12:24:45.212974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.213096 (+   122us) service_pool.cc:224] Handling call
0709 12:24:45.245785 (+ 32689us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:45.255937 (+ 10152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.255978 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:45.259192 (+  3214us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.622188 (+362996us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:45.622262 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.622380 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.612489 (+990109us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b24001bd808. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c13608fb0 00007f6c1627f488 00007f6c1627fbe8 00007f6c16257398 00007f6c1625276a 00007f6c16251bb6
0709 12:24:46.613071 (+   582us) raft_consensus.cc:1685] finished
0709 12:24:46.613092 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.613342 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15485,"raft.queue_time_us":10445,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":17335,"spinlock_wait_cycles":39668992,"thread_start_us":318480,"threads_started":1,"wal-append.queue_time_us":731450}
I20260709 12:24:46.563251 18090 rpcz_store.cc:276] 0709 12:24:45.444244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.444439 (+   195us) service_pool.cc:224] Handling call
0709 12:24:45.444981 (+   542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:45.479378 (+ 34397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.479432 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:45.520524 (+ 41092us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.794517 (+273993us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:45.794566 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.794640 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.560171 (+765531us) raft_consensus.cc:1685] finished
0709 12:24:46.560197 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.560385 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17998,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":12090,"thread_start_us":232902,"threads_started":1,"wal-append.queue_time_us":769395}
I20260709 12:24:46.617717 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:46.618871 25289 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:46.619020 18016 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:46.619387 25289 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:46.620420 25289 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.623340 17913 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 14, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 14 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:46.626448 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.570604 25375 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:46.630206 25369 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.979s	user 0.004s	sys 0.000s
I20260709 12:24:46.629464 25386 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.632794 25386 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:46.636701 25232 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.583s	user 0.010s	sys 0.000s
I20260709 12:24:46.556429 25321 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.640226 25139 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.641875 25414 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.586565 25383 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.648386 25383 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.564168 25088 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.653053 25088 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.653806 25392 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:46.654516 25299 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:46.655005 25299 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:46.655421 25374 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.084s	user 0.004s	sys 0.000s
I20260709 12:24:46.655933 25299 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.657348 25299 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:46.658908 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2871) took 1923 ms (client timeout 1904 ms). Trace:
W20260709 12:24:46.659415 18017 rpcz_store.cc:269] 0709 12:24:44.735051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.746599 (+ 11548us) service_pool.cc:224] Handling call
0709 12:24:46.658653 (+1912054us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:46.658807 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3762931456}
I20260709 12:24:46.608721 18094 rpcz_store.cc:276] 0709 12:24:43.799460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.526148 (+726688us) service_pool.cc:224] Handling call
0709 12:24:44.586512 (+ 60364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.586962 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.586995 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.599145 (+ 12150us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.126471 (+527326us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:45.126544 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.126736 (+   192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.606443 (+1479707us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b2400278378. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c13608fb0 00007f6c1627f488 00007f6c1627fbe8 00007f6c16257398 00007f6c1625276a 00007f6c16251bb6
0709 12:24:46.607819 (+  1376us) raft_consensus.cc:1685] finished
0709 12:24:46.607847 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.608097 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4945,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":8722,"spinlock_wait_cycles":155352832,"thread_start_us":526729,"threads_started":1,"wal-append.queue_time_us":880653}
W20260709 12:24:46.663722 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60164.2us
I20260709 12:24:46.668248 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.669076 18170 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:46.674418 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.675613 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:24:46.676122 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.677357 18173 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:46.677655 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.680575 25407 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.680749 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.681571 18022 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
W20260709 12:24:46.682991 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97628.1us
I20260709 12:24:46.684795 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:46.687254 25395 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.624s	user 0.001s	sys 0.001s
I20260709 12:24:46.700923 25376 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.706266 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.707190 18171 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:46.637317 25232 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.715505 18016 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:46.716689 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.724864 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.727336 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.731043 18806 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:46.728230 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.729753 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.732484 18814 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:46.730017 25364 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.694718 25289 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:46.734014 25088 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:46.736599 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:46.733405 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.738581 18807 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:46.739512 25376 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:46.733387 25364 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.740164 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:46.741143 25325 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:46.736642 25088 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:46.742879 25088 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:46.742617 25096 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.743561 25168 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:46.745217 25096 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:46.746109 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.740741 25321 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:46.747897 25379 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.003s
I20260709 12:24:46.747148 17913 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:46.748389 25379 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.696630 25407 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:46.746141 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.744004 25088 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.751632 25088 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:46.752362 25376 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:46.755550 18020 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:46.755617 25376 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:46.756263 25376 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.759213 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2884) took 1121 ms. Trace:
I20260709 12:24:46.759675 18018 rpcz_store.cc:276] 0709 12:24:45.637803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.647070 (+  9267us) service_pool.cc:224] Handling call
0709 12:24:45.750371 (+103301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:46.147580 (+397209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.758875 (+611295us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:46.759129 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856630144}
I20260709 12:24:46.741590 25325 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.761152 25232 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.761381 25325 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:46.750238 25389 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.907s	user 0.003s	sys 0.000s
I20260709 12:24:46.754599 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:46.748817 25396 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.728s	user 0.000s	sys 0.003s
W20260709 12:24:46.767000 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.767784 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.754338 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:46.768056 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.696376 25414 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
W20260709 12:24:46.774050 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.785413 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60636.9us
I20260709 12:24:46.786901 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.772459 25432 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.796072 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.794164 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:46.787083 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:46.799063 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 88593.8us
I20260709 12:24:46.718452 25051 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:24:46.803207 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.788594 25284 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:46.772541 25407 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:46.820101 18806 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:24:46.795449 25432 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.822907 25379 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:46.824333 25432 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:46.812155 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.825125 18170 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.787982 25424 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:46.826025 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.826339 25424 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.814720 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.828709 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.749248 17913 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:46.830241 25411 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:46.830786 25411 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.831225 25411 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.830982 17913 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:24:46.785954 25381 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:46.803526 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.834721 25359 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.835964 18165 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.835639 25359 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.807709 25386 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:46.793382 25231 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
W20260709 12:24:46.826961 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.840535 18170 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:24:46.824486 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.843837 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.844556 18172 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:46.850265 18164 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.825475 25423 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:46.841503 25420 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:46.857560 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.858690 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.859246 25057 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:46.859938 25381 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:46.805927 24691 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:46.817796 25407 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.870294 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:46.876330 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:46.852648 25311 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.878522 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.880381 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:46.880524 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.898598 18091 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.866791 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:46.884979 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:46.887167 18165 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:46.894171 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:46.878058 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:46.900080 18172 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:24:46.900058 18094 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.895716 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.896656 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.837538 25386 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:46.898119 18090 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:46.841074 25359 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.817898 25244 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:24:46.903637 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 2886) took 1106 ms. Trace:
I20260709 12:24:46.917972 18020 rpcz_store.cc:276] 0709 12:24:45.797393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.797649 (+   256us) service_pool.cc:224] Handling call
0709 12:24:46.903348 (+1105699us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b5400250b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:46.903546 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2144235392}
I20260709 12:24:46.906193 18096 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:46.873271 25057 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:46.908689 25375 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.909966 25427 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.910502 25416 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:46.911548 25408 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:46.921701 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43728.4us
I20260709 12:24:46.926100 24841 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:46.926378 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.928180 18090 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:46.928840 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3974) took 1430 ms. Trace:
I20260709 12:24:46.929088 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.893679 25407 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.927292 25431 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:46.917439 25414 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:46.917644 25380 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.892020 25423 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:46.955359 18166 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.904934 24691 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:46.963483 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.904291 25386 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.942245 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:46.936789 25381 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.883790 24958 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
W20260709 12:24:46.943710 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115094us
I20260709 12:24:46.940714 25424 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:46.954589 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.937475 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:46.940827 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.946707 18807 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:46.952121 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:46.966868 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:46.969163 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:46.944422 18164 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:46.945996 18095 rpcz_store.cc:276] 0709 12:24:45.498275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.498907 (+   632us) service_pool.cc:224] Handling call
0709 12:24:45.534376 (+ 35469us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:45.534862 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.534905 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:45.552256 (+ 17351us) log.cc:844] Serialized 52 byte log entry
0709 12:24:45.805699 (+253443us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:45.805746 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.805846 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.748282 (+942436us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b2400278378. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:24:46.914895 (+166613us) raft_consensus.cc:1685] finished
0709 12:24:46.914925 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.928756 (+ 13831us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2874,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":9491,"spinlock_wait_cycles":51610880,"thread_start_us":245622,"threads_started":1,"wal-append.queue_time_us":362236}
I20260709 12:24:46.948380 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:46.969985 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.977895 18811 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.915385 25311 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.952121 18806 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:46.962749 25414 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.982223 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:46.995275 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:46.999383 18811 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:47.000666 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:47.011977 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.013631 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.014667 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.014909 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.016042 25244 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:47.016518 25244 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.016930 25244 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.024487 18091 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:47.031142 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.032123 18098 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.996766 25293 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.984790 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:46.987978 25423 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.961975 24841 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:47.030148 25290 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:46.945643 25427 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:46.972638 18094 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:47.035125 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:46.998832 25414 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:47.025712 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.021116 25325 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:46.967471 25380 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.059105 18168 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:47.047662 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.051721 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.052019 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:46.992542 25435 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
W20260709 12:24:47.041888 25413 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.008s
I20260709 12:24:47.061029 18095 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:47.062534 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.030515 25441 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:47.081694 25325 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.095608 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.067708 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.070186 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.071256 25311 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.022264 24691 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.006538 25362 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.135703 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.136749 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 995) took 1035 ms. Trace:
I20260709 12:24:47.137106 18021 rpcz_store.cc:276] 0709 12:24:46.100911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.101064 (+   153us) service_pool.cc:224] Handling call
0709 12:24:47.135516 (+1034452us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:24:47.136667 (+  1151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3700096}
I20260709 12:24:47.073508 25380 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:47.117846 25422 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.511s	user 0.013s	sys 0.000s
I20260709 12:24:47.140514 25422 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.143368 25422 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.074000 18015 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: [21.15-21.15]
W20260709 12:24:47.076773 25399 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.006s
I20260709 12:24:47.144301 25399 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.078990 25375 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:47.146744 25399 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.076578 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.064985 25401 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.150250 25401 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.151966 25411 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.154259 18097 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:47.155022 18020 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:24:47.155540 18020 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 12
I20260709 12:24:47.155826 18020 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:24:46.939117 17913 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 15, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 15 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:47.058089 25408 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.078963 25289 raft_consensus.cc:2764] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:47.078845 24841 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.045280 25293 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.014369 25057 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.079002 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:47.083266 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:47.172538 18811 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:47.088284 18096 raft_consensus.cc:2368] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
I20260709 12:24:47.089681 18098 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:47.045159 25424 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.176923 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:47.178018 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99117.6us
I20260709 12:24:47.179279 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:47.042550 25379 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.119709 18815 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:47.117686 18017 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:47.052873 25051 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:47.093626 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.156078 18020 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:47.163643 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:47.203400 18090 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:47.204654 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.205237 18016 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:47.205595 18090 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:47.162609 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:47.203019 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:47.207381 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44908.1us
I20260709 12:24:47.208428 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.170140 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.095657 25441 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:47.175448 18138 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.599s (SENT)
I20260709 12:24:47.213626 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.183208 18015 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:47.137593 25362 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.184392 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.217041 18021 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:24:47.189100 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:47.194053 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.086381 25364 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.202283 18018 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:47.165205 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:47.209275 18098 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:47.209453 18806 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:47.208232 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:47.220515 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.230717 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:47.233258 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44322us
I20260709 12:24:47.234711 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.107688 25413 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.236833 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:47.238463 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73352.6us
W20260709 12:24:47.251495 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76095.6us
W20260709 12:24:47.253091 18138 leader_election.cc:336] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.683s (SENT)
I20260709 12:24:47.254740 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.255560 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.256539 18017 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:47.257264 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.236904 25375 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:47.206387 25325 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:47.208635 25290 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:47.266047 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.267634 18811 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:47.276656 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.277570 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:47.237975 25370 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:47.269681 18167 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:47.269673 18138 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.714s (SENT)
W20260709 12:24:47.268584 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 732943us
I20260709 12:24:47.239745 25057 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.291234 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.254308 25362 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.182047 25408 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.299376 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.178248 25376 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:47.286419 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.288125 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.278210 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:47.254125 25431 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:47.280233 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.283022 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:47.292780 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.308128 18020 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:47.308799 18016 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:47.297333 25452 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:47.206149 25424 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.309500 18090 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:24:47.310119 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62570.3us
I20260709 12:24:47.270622 25290 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.259176 25375 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.259169 25311 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:47.260458 25420 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:47.282336 18095 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:47.260382 25051 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.269315 25289 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:47.264621 25325 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.312592 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.271863 25427 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.306731 18019 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17. Re-sending same reply.
I20260709 12:24:47.297521 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.247548 25379 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.326974 25375 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:47.270609 25300 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:24:47.355355 18808 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:24:47.354354 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71458.3us
I20260709 12:24:47.359123 18023 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:24:47.362123 18094 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:47.390121 18094 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:47.313053 25431 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.322793 25290 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:47.325734 25411 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.357256 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.420912 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:47.325851 25401 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:47.395625 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:47.399104 25383 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.750s	user 0.010s	sys 0.008s
I20260709 12:24:47.399483 25293 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.364423 25392 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:47.436383 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.436681 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:47.408908 25399 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:47.369256 25420 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:47.473304 25420 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:47.385718 25386 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.410840 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.307799 25362 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.427004 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:47.433307 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.381414 25325 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.443051 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:47.450351 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:47.452175 18173 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:47.459084 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.462464 25399 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:47.463496 25447 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:24:47.470112 25289 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.458237 25057 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:47.496773 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60248.8us
W20260709 12:24:47.472110 25450 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:24:47.417390 25431 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:47.472524 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.483954 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.499854 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:47.513907 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2930) took 1598 ms (client timeout 1704 ms). Trace:
I20260709 12:24:47.528506 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.552286 18023 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.555660 18017 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:24:47.504377 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.497228 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.562667 18094 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:47.505789 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.534123 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:47.477697 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.564591 18806 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:24:47.486205 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:47.521082 25453 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:24:47.471333 25424 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.464744 17913 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 13, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 13 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:24:47.499073 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61570.9us
I20260709 12:24:47.531610 18172 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:47.589932 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.559258 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:47.506238 24624 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.591308 18096 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:47.566115 18098 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:47.535686 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.478513 25364 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:47.567792 18021 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:24:47.568462 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.611927 18022 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:47.481452 25407 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.517s	user 0.013s	sys 0.000s
I20260709 12:24:47.571648 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.571851 18024 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:47.574431 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.477013 25386 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.575675 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:47.631554 18015 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:47.444690 25456 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:47.576768 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.511204 25300 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.497109 25427 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:47.584316 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:47.649041 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.650417 18021 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:47.587499 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:47.552265 18168 rpcz_store.cc:269] 0709 12:24:45.916074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.916271 (+   197us) service_pool.cc:224] Handling call
0709 12:24:47.513645 (+1597374us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000433d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:47.513780 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2538386176}
I20260709 12:24:47.588651 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.588960 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.592482 18016 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
W20260709 12:24:47.483351 25284 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.635s	user 0.013s	sys 0.000s
I20260709 12:24:47.495361 25392 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:47.593242 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.479671 25408 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.593734 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:47.519776 25376 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.603885 25382 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:47.604719 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.566919 25057 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.617982 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:47.561928 25451 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:47.629266 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.138s (SENT)
I20260709 12:24:47.629544 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:47.630476 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.634860 18098 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:47.635932 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:24:47.626078 18018 raft_consensus.cc:2376] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 14. Re-sending same reply.
I20260709 12:24:47.444310 25383 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.587415 18811 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:47.653457 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:47.494141 25381 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.119s
W20260709 12:24:47.660751 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 76535.8us
I20260709 12:24:47.723420 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:47.561097 25416 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:24:47.490026 25455 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 12:24:47.730265 18019 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:47.729003 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:47.624909 25424 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:47.620714 25096 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:24:47.695493 18015 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:24:47.679751 18815 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:47.697717 18809 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:47.698625 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 401234us
I20260709 12:24:47.591475 25293 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.097s
W20260709 12:24:47.716188 25389 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:24:47.713296 18024 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:24:47.714150 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 357146us
I20260709 12:24:47.669777 25456 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:47.720801 18022 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:24:47.721182 18018 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:47.672564 25284 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:47.765733 18814 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:47.766047 25319 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:47.766772 25293 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:47.767328 25319 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:47.687135 25435 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:47.766994 17913 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 16. New cstate: current_term: 16 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:47.768023 25293 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:47.768141 25435 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:47.673950 18093 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:47.768539 25293 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:47.769400 25435 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.770869 25399 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:47.687397 25427 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:47.739612 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.696516 25382 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:47.772101 25427 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.772876 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:47.772578 25382 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.774286 25382 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.774752 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:47.776000 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 146765us
W20260709 12:24:47.752691 18138 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.776s (SENT)
I20260709 12:24:47.778119 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.778304 18809 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:47.778909 25381 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:47.609948 24892 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:24:47.779310 25381 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:47.779471 25168 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:47.780475 25424 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:47.780264 25381 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.781530 25424 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.704334 25370 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:47.789225 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:47.740187 24841 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:47.762195 18098 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:24:47.767469 18018 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:47.714672 25413 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.756546 18016 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:24:47.633496 25407 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.710412 18021 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:47.687665 25390 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:24:47.770093 18093 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:47.787333 18019 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:24:47.788987 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.767870 25319 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:47.669229 25300 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.787195 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:47.799753 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:47.787945 25244 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.771s	user 0.014s	sys 0.001s
I20260709 12:24:47.740841 25408 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:47.780258 18138 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.040s (SENT)
I20260709 12:24:47.829569 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.627609 24624 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.827119 18018 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:47.826357 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:47.825335 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.812086 18815 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:47.800437 25452 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:47.817005 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1315) took 1017 ms. Trace:
I20260709 12:24:47.781771 25381 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.816938 18094 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:47.836258 18099 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:24:47.786365 25457 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:47.807121 25409 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:47.812072 24958 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:47.812377 25411 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:47.842736 18814 rpcz_store.cc:276] 0709 12:24:46.799271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.799489 (+   218us) service_pool.cc:224] Handling call
0709 12:24:46.846985 (+ 47496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.766301 (+919316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.816596 (+ 50295us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54004b41d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:47.816900 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2112357248}
I20260709 12:24:47.792831 25311 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:24:47.769057 17913 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 15, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 15 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:47.846794 25379 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.869604 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:47.876396 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:47.881151 18170 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:47.769652 25293 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 20.18, Last appended by leader: 18, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.890692 25293 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:47.838543 25416 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:24:47.895833 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:47.812428 25376 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:24:47.896976 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:47.835428 25392 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.898013 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:47.898767 18167 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:47.878103 24841 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:47.829329 25408 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:47.846798 25096 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:47.884692 25362 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:47.905966 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 125729us
I20260709 12:24:47.721210 25451 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:47.857573 25457 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:47.907987 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.829545 25423 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:24:47.880963 25311 consensus_queue.cc:1243] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:47.915537 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.835196 25244 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.843189 25413 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 12:24:47.863396 25411 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:47.955564 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.878010 24892 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:47.957310 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:47.959758 18023 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:47.381934 25441 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:47.964104 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.965556 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.910043 25409 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:47.972364 25284 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:47.979903 18173 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:47.980195 25407 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.981096 25244 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:47.983211 25463 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:47.901194 25408 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:47.980986 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:47.983114 24624 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:47.978076 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:47.948056 25375 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.002s
I20260709 12:24:47.989470 18097 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:47.946611 25465 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.992110 18808 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:47.948732 25168 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:24:47.968984 25411 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.974179 25057 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:47.990502 25464 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:47.943480 25399 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:24:47.986924 24892 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.909330 25319 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:48.011651 25420 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.538s	user 0.007s	sys 0.007s
I20260709 12:24:47.910794 25456 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:48.021880 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 487923us
I20260709 12:24:47.978160 25432 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
W20260709 12:24:48.033546 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 385139us
W20260709 12:24:48.028805 25466 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:24:48.028623 25420 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:48.016209 25464 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:47.915921 25370 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.036386 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.005874 25390 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.022125 25408 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:48.020465 24892 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:48.044273 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:48.053963 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:48.043162 18138 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.690s (SENT)
I20260709 12:24:48.019021 25383 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:48.006942 25375 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:48.095795 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.094255 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.049798 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:48.079596 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:48.072897 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.137s (SENT)
I20260709 12:24:48.116079 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.118887 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:48.009357 25465 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:48.122325 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49510.6us
I20260709 12:24:48.123155 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:48.124420 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.133257 18815 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.13-14.13]
W20260709 12:24:48.138859 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89155.9us
I20260709 12:24:48.090807 25319 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:48.120157 25466 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:24:48.138007 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:48.088905 25413 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:48.112846 25422 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:48.132933 25414 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.043s	user 0.015s	sys 0.001s
I20260709 12:24:48.124771 25383 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:48.153998 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.086686 25057 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 12:24:48.148449 25244 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.181212 25244 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:48.161994 25300 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:48.223472 25299 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:24:48.225095 25461 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:24:48.226889 25323 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:48.227494 25323 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:48.233582 18099 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:48.238828 25451 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:48.228698 25323 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.241719 25323 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:48.243474 25459 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
I20260709 12:24:48.026082 25457 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.263959 25375 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:48.053046 17913 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:48.278577 25464 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.122921 25380 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
W20260709 12:24:48.290715 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 136934us
I20260709 12:24:48.178648 24624 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.127s
I20260709 12:24:48.304090 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:48.304512 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.369s (SENT)
W20260709 12:24:48.154456 25460 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.275s	user 0.002s	sys 0.000s
I20260709 12:24:48.318452 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:48.178905 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:48.155573 25390 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:48.237382 25401 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:48.324237 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:48.328193 25289 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.772s	user 0.006s	sys 0.009s
I20260709 12:24:48.331624 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3856) took 1160 ms. Trace:
I20260709 12:24:48.319331 25413 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:48.372473 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.380226 18022 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.009s	sys 0.006s
W20260709 12:24:48.407963 18022 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.010s	sys 0.007s
I20260709 12:24:48.409479 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:48.410462 18099 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:48.177556 25414 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:48.396643 18017 rpcz_store.cc:276] 0709 12:24:47.171104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.633242 (+462138us) service_pool.cc:224] Handling call
0709 12:24:47.633665 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.633959 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:48.049008 (+415049us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.331152 (+282144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.331227 (+    75us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:48.331295 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.331387 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.331546 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6938,"prepare.queue_time_us":414974,"prepare.run_cpu_time_us":343,"prepare.run_wall_time_us":9602,"thread_start_us":621385,"threads_started":2}
I20260709 12:24:48.400903 24624 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:48.353140 25470 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:24:48.402640 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.329882 25464 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:48.327698 25390 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.320086 25467 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:48.327955 25379 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:48.419852 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 438128us
W20260709 12:24:48.423588 25381 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.571s	user 0.009s	sys 0.004s
I20260709 12:24:48.334239 25401 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:48.356374 25451 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:48.462848 24691 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.034s	sys 0.029s
W20260709 12:24:48.429888 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.041s	sys 0.043s
I20260709 12:24:48.409677 25289 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.410163 25407 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:48.422087 25325 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.860s	user 0.015s	sys 0.000s
W20260709 12:24:48.555979 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.419003 25376 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:48.562201 18092 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:48.566784 18018 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:48.571516 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:48.428308 25168 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.574807 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:48.578106 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3855) took 1407 ms. Trace:
I20260709 12:24:48.424718 24624 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:48.580578 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.016s	sys 0.008s
W20260709 12:24:48.346424 25466 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
W20260709 12:24:48.660602 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.448627 25392 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.012s	sys 0.009s
I20260709 12:24:48.642376 18020 rpcz_store.cc:276] 0709 12:24:47.170854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.317025 (+146171us) service_pool.cc:224] Handling call
0709 12:24:47.357063 (+ 40038us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:47.357564 (+   501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.357601 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:47.364129 (+  6528us) log.cc:844] Serialized 52 byte log entry
0709 12:24:47.558267 (+194138us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:47.830195 (+271928us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.830655 (+   460us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.830857 (+   202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.830969 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.222218 (+391249us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b18003ff940. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:48.240770 (+ 18552us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b18003ff940. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:48.577760 (+336990us) raft_consensus.cc:1685] finished
0709 12:24:48.577791 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.578016 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":179558,"prepare.queue_time_us":852480,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":3343,"raft.queue_time_us":127626,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":36986,"spinlock_wait_cycles":40796928,"thread_start_us":412015,"threads_started":2,"wal-append.queue_time_us":418361}
W20260709 12:24:48.584712 18807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.006s	sys 0.017s
W20260709 12:24:48.587903 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.042s	sys 0.043s
I20260709 12:24:48.449476 25390 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:48.677651 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 852336us
I20260709 12:24:48.594570 25413 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:48.580838 25431 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.006s	user 0.016s	sys 0.000s
I20260709 12:24:48.641718 18023 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:48.670857 25382 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.896s	user 0.014s	sys 0.002s
I20260709 12:24:48.644160 25325 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:48.632969 18096 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:48.594748 24691 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s	user 0.034s	sys 0.029s
W20260709 12:24:48.658831 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.016s	sys 0.008s
W20260709 12:24:48.642429 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.010s	sys 0.017s
W20260709 12:24:48.602257 25051 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.223s	user 0.011s	sys 0.006s
I20260709 12:24:48.581590 25380 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:48.677098 18807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.007s	sys 0.018s
I20260709 12:24:48.449808 25467 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.678758 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.679678 18099 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
W20260709 12:24:48.680369 18015 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.001s	sys 0.018s
I20260709 12:24:48.595588 25407 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.584496 25455 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:48.604225 18812 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:24:48.650328 25427 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.006s	sys 0.011s
W20260709 12:24:48.673889 25290 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.126s	user 0.013s	sys 0.003s
I20260709 12:24:48.574110 25364 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:48.746402 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1080) took 1333 ms. Trace:
W20260709 12:24:48.817663 25456 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.011s	sys 0.018s
I20260709 12:24:48.596861 25301 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.144s
W20260709 12:24:48.791158 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.012s	sys 0.018s
W20260709 12:24:48.794348 18024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.000s	sys 0.024s
W20260709 12:24:48.869019 18024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.001s	sys 0.024s
I20260709 12:24:48.870190 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3887) took 1453 ms. Trace:
W20260709 12:24:48.713199 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.831s	user 0.047s	sys 0.033s
I20260709 12:24:48.593286 25451 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.690830 25413 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.802039 18090 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:48.870633 18024 rpcz_store.cc:276] 0709 12:24:47.416883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.750936 (+334053us) service_pool.cc:224] Handling call
0709 12:24:47.751307 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.751679 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:48.165928 (+414249us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.869643 (+703715us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.869739 (+    96us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:48.869794 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.869900 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.870100 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":399639,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"thread_start_us":1065026,"threads_started":2}
W20260709 12:24:48.812786 18015 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.002s	sys 0.019s
I20260709 12:24:48.787398 18023 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
W20260709 12:24:48.817977 18808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.014s	sys 0.011s
W20260709 12:24:48.883833 18808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.015s	sys 0.011s
W20260709 12:24:48.818539 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 139901us
I20260709 12:24:48.851537 18166 rpcz_store.cc:276] 0709 12:24:47.412886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.414068 (+  1182us) service_pool.cc:224] Handling call
0709 12:24:47.463087 (+ 49019us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.494805 (+ 31718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.494845 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.561068 (+ 66223us) log.cc:844] Serialized 26 byte log entry
0709 12:24:47.909004 (+347936us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:47.909058 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.909158 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.746055 (+836897us) raft_consensus.cc:1685] finished
0709 12:24:48.746088 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.746301 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31122,"raft.queue_time_us":31535,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":286417,"threads_started":1,"wal-append.queue_time_us":647823}
I20260709 12:24:48.357920 17913 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 17, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 17 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:24:48.817339 25467 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:48.741968 25465 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:48.755985 25460 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:24:48.658722 25168 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:48.674726 25392 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.012s	sys 0.010s
W20260709 12:24:48.863919 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3115) took 1924 ms (client timeout 1976 ms). Trace:
W20260709 12:24:48.871891 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s	user 0.047s	sys 0.037s
I20260709 12:24:48.918911 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4008) took 1377 ms. Trace:
I20260709 12:24:48.878415 25416 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:48.837900 25427 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.007s	sys 0.011s
I20260709 12:24:48.812870 25051 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.744429 25096 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.102s
W20260709 12:24:48.784145 25466 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:24:48.888760 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1083) took 1020 ms. Trace:
W20260709 12:24:48.748036 25293 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.857s	user 0.010s	sys 0.005s
I20260709 12:24:48.838107 25382 raft_consensus.cc:2764] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 12:24:48.868450 25456 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.011s	sys 0.019s
I20260709 12:24:48.850952 25455 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:48.959893 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.005s	sys 0.028s
I20260709 12:24:48.851104 25364 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:48.689354 25399 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:48.919495 18098 rpcz_store.cc:276] 0709 12:24:47.542066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.821646 (+279580us) service_pool.cc:224] Handling call
0709 12:24:47.844266 (+ 22620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.844512 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:47.881364 (+ 36852us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.918444 (+1037080us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.918512 (+    68us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:48.918569 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.918672 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.918835 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16395,"prepare.queue_time_us":36604,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":1036651,"threads_started":1,"wal-append.queue_time_us":1020323}
I20260709 12:24:48.889741 25413 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:48.865360 25290 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:48.886667 25375 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:48.918195 18092 rpcz_store.cc:269] 0709 12:24:46.939335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.948044 (+  8709us) service_pool.cc:224] Handling call
0709 12:24:48.855635 (+1907591us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400052654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:48.863810 (+  8175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3379200896}
I20260709 12:24:48.896096 25289 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
W20260709 12:24:48.925450 25323 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.683s	user 0.016s	sys 0.000s
W20260709 12:24:48.949182 25370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.019s	sys 0.013s
I20260709 12:24:48.897374 18099 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:48.751223 25431 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:48.913734 25465 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:48.967955 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.006s	sys 0.029s
W20260709 12:24:49.033895 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 855053us
I20260709 12:24:49.034603 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4009) took 1492 ms. Trace:
W20260709 12:24:48.974864 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.018s	sys 0.029s
W20260709 12:24:49.038096 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.018s	sys 0.029s
W20260709 12:24:49.038942 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 734894us
I20260709 12:24:49.039865 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.941308 18168 rpcz_store.cc:276] 0709 12:24:47.868590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.868759 (+   169us) service_pool.cc:224] Handling call
0709 12:24:47.894321 (+ 25562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.894591 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:47.894983 (+   392us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.888217 (+993234us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.888310 (+    93us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:48.888382 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.888482 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.888671 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":229,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":25595,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":5752,"spinlock_wait_cycles":11202560,"thread_start_us":973550,"threads_started":1,"wal-append.queue_time_us":1020094}
I20260709 12:24:49.042688 25380 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:48.959452 25057 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.021s	sys 0.016s
W20260709 12:24:49.043546 25057 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.021s	sys 0.016s
I20260709 12:24:48.985703 25399 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:49.044878 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.046311 25399 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.047025 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3129) took 1563 ms. Trace:
I20260709 12:24:49.047497 18096 rpcz_store.cc:276] 0709 12:24:47.483228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.613040 (+129812us) service_pool.cc:224] Handling call
0709 12:24:47.613469 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.748541 (+1135072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.046758 (+298217us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540011dd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:49.046960 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343711488}
I20260709 12:24:49.048964 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:49.050205 25057 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:49.046959 25399 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.053877 25399 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:49.055368 25490 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:24:49.056648 25476 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.452s	user 0.002s	sys 0.000s
I20260709 12:24:49.058549 25399 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.059587 25431 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:49.032423 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 452395us
I20260709 12:24:49.063627 25168 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:49.066167 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.066967 18168 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:49.070953 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.071774 18168 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:49.035233 18093 rpcz_store.cc:276] 0709 12:24:47.542508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.827488 (+284980us) service_pool.cc:224] Handling call
0709 12:24:47.827962 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.828475 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.828522 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.852338 (+ 23816us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.223744 (+371406us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b1800464440. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:48.713730 (+489986us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:48.713805 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.713955 (+   150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.020550 (+306595us) raft_consensus.cc:1685] finished
0709 12:24:49.020590 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.034483 (+ 13893us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2936,"raft.queue_time_us":52505,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":19008,"spinlock_wait_cycles":3437312,"thread_start_us":489626,"threads_started":1,"wal-append.queue_time_us":1114050}
I20260709 12:24:49.078492 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:49.010068 25364 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.076876 18165 raft_consensus.cc:2368] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20.
I20260709 12:24:49.023661 18095 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:48.982563 25424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.197s	user 0.022s	sys 0.030s
W20260709 12:24:49.018790 25492 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:24:49.047804 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:48.991020 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4003) took 1864 ms. Trace:
I20260709 12:24:49.086092 18090 rpcz_store.cc:276] 0709 12:24:47.127259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.594786 (+467527us) service_pool.cc:224] Handling call
0709 12:24:47.637668 (+ 42882us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.881347 (+1243679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.990658 (+109311us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400235d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:48.990929 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2677007744}
I20260709 12:24:49.088078 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:49.077683 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1520) took 1090 ms. Trace:
I20260709 12:24:49.090865 18807 rpcz_store.cc:276] 0709 12:24:47.987257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.987444 (+   187us) service_pool.cc:224] Handling call
0709 12:24:47.987998 (+   554us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.005925 (+ 17927us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.005978 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:48.006840 (+   862us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.792811 (+785971us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:48.792869 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.792952 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.077288 (+284336us) raft_consensus.cc:1685] finished
0709 12:24:49.077327 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.077563 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21835,"raft.queue_time_us":18597,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7329,"thread_start_us":759839,"threads_started":1,"wal-append.queue_time_us":820027}
W20260709 12:24:49.032604 25370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.020s	sys 0.013s
I20260709 12:24:49.009284 25289 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:49.096511 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:49.081952 25364 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:49.073800 25408 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.991s	user 0.009s	sys 0.002s
I20260709 12:24:49.008090 25290 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:49.076612 25319 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.921s	user 0.013s	sys 0.001s
W20260709 12:24:49.113238 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.013s	sys 0.032s
W20260709 12:24:49.114972 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.014s	sys 0.032s
I20260709 12:24:49.093482 25469 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:49.090802 25424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.305s	user 0.022s	sys 0.031s
W20260709 12:24:49.105863 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.111267 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.112335 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:49.115500 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 687490us
I20260709 12:24:49.135978 18171 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:49.098658 18092 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:49.127211 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:49.127821 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.008s	sys 0.033s
I20260709 12:24:49.076180 25455 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.100075 25057 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:49.138248 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.008s	sys 0.033s
I20260709 12:24:49.137169 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:49.108891 25427 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:24:49.133105 25422 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.016s	sys 0.024s
I20260709 12:24:49.100571 25289 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:49.150506 17987 leader_election.cc:400] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
I20260709 12:24:49.108341 25051 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.091885 25465 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.170056 17987 leader_election.cc:403] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:49.131414 25469 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.130441 25325 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.240653 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:49.206053 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.245916 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:49.247100 18807 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:24:49.247186 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 110890us
W20260709 12:24:49.170224 25422 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.016s	sys 0.026s
W20260709 12:24:49.251919 25483 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.270s	user 0.001s	sys 0.000s
W20260709 12:24:49.252535 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.264031 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.252961 25486 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:24:49.253526 25424 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.254232 25057 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.254917 18164 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:24:49.255141 25475 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.364s	user 0.000s	sys 0.001s
I20260709 12:24:49.255496 25427 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:49.257529 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:49.261878 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:49.180776 25411 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.162s	user 0.012s	sys 0.005s
W20260709 12:24:49.173056 25457 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.053s	sys 0.050s
I20260709 12:24:49.166643 25431 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:49.171495 25441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.021s	sys 0.034s
I20260709 12:24:49.183501 25407 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:49.283977 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:49.151209 25408 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.124s
W20260709 12:24:49.299378 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 506228us
I20260709 12:24:49.300906 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.249470 25469 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.301091 18171 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:49.253736 25420 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.190485 25432 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.308199 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.345679 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:49.346194 18170 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:49.346417 18807 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:49.283289 25487 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.359s	user 0.000s	sys 0.002s
I20260709 12:24:49.281183 25451 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:49.336167 25480 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.355s	user 0.000s	sys 0.001s
I20260709 12:24:49.269810 25380 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:24:49.190047 25051 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:49.305269 25441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.022s	sys 0.034s
I20260709 12:24:49.303359 25498 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:49.387071 18098 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:49.304203 25431 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:49.348191 18094 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.933s	user 0.006s	sys 0.019s
W20260709 12:24:49.391065 18094 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.007s	sys 0.020s
I20260709 12:24:49.301188 25502 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:49.270795 24841 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
W20260709 12:24:49.350096 18022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.035s	sys 0.076s
I20260709 12:24:49.276455 25504 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.065s
W20260709 12:24:49.301595 25457 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.054s	sys 0.051s
I20260709 12:24:49.191089 25289 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.298394 25411 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.460212 18097 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:49.454353 25435 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:49.461495 25411 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.294865 25057 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:49.347988 25381 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.012s	sys 0.013s
I20260709 12:24:49.461772 25435 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:49.462649 25381 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.012s	sys 0.013s
I20260709 12:24:49.463352 25381 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.463899 25381 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.464363 25057 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.465003 25435 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:49.465495 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.570s (SENT)
I20260709 12:24:49.365430 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1331) took 1500 ms. Trace:
I20260709 12:24:49.467185 25051 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:49.466930 18814 rpcz_store.cc:276] 0709 12:24:47.865535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.868249 (+  2714us) service_pool.cc:224] Handling call
0709 12:24:47.899195 (+ 30946us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.950136 (+ 50941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.950195 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.950956 (+   761us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.416210 (+465254us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:48.416265 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.416358 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.365130 (+948772us) raft_consensus.cc:1685] finished
0709 12:24:49.365161 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.365346 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20802,"raft.queue_time_us":55827,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":19494,"thread_start_us":415122,"threads_started":1,"wal-append.queue_time_us":998474}
I20260709 12:24:49.369385 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:49.468712 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:49.469317 18093 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:49.369616 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:49.470366 18096 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:49.370325 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:49.471343 25413 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.386400 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:49.404668 25465 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.405252 25431 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.472411 25457 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:49.472610 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.473105 25457 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:49.472437 18811 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:49.474148 25457 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.363822 25505 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:49.406631 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1333) took 1539 ms. Trace:
I20260709 12:24:49.391286 25380 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.475826 25505 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.476302 25505 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.476140 25380 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.478641 25380 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:49.354583 25432 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.414811 18090 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:49.418381 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:49.429987 25427 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.480940 25432 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:49.438391 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1518) took 1534 ms. Trace:
I20260709 12:24:49.475842 18806 rpcz_store.cc:276] 0709 12:24:47.867365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.867568 (+   203us) service_pool.cc:224] Handling call
0709 12:24:47.898384 (+ 30816us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.919641 (+ 21257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.919693 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.961304 (+ 41611us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.425194 (+463890us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:48.425252 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.425332 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.406140 (+980808us) raft_consensus.cc:1685] finished
0709 12:24:49.406319 (+   179us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.406527 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37274,"raft.queue_time_us":52512,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":13391,"thread_start_us":424052,"threads_started":1,"wal-append.queue_time_us":867549}
I20260709 12:24:49.482003 18809 rpcz_store.cc:276] 0709 12:24:47.903824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.904028 (+   204us) service_pool.cc:224] Handling call
0709 12:24:47.904463 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.961465 (+ 57002us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:47.995938 (+ 34473us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.241489 (+1245551us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.242260 (+   771us) log.cc:844] Serialized 26 byte log entry
0709 12:24:49.242464 (+   204us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:49.242514 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.242599 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.438065 (+195466us) raft_consensus.cc:1685] finished
0709 12:24:49.438095 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.438299 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50346,"prepare.queue_time_us":35504,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":324708,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":16566,"thread_start_us":1211445,"threads_started":2,"wal-append.queue_time_us":928914}
I20260709 12:24:49.364382 25451 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.484934 25451 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.489012 25451 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.278378 17913 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:49.387424 25498 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.443784 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:49.490064 25498 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.447124 25408 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:24:49.448614 25493 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:24:49.449273 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:49.492317 18099 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:49.493253 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1521) took 1463 ms. Trace:
I20260709 12:24:49.493707 18808 rpcz_store.cc:276] 0709 12:24:48.030425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.030587 (+   162us) service_pool.cc:224] Handling call
0709 12:24:48.031051 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.048557 (+ 17506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.048595 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:48.113752 (+ 65157us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.884563 (+770811us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:48.884636 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.884750 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.492983 (+608233us) raft_consensus.cc:1685] finished
0709 12:24:49.493011 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.493182 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11072,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":28754,"thread_start_us":726237,"threads_started":1,"wal-append.queue_time_us":1012604}
W20260709 12:24:49.450384 25503 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
I20260709 12:24:49.451997 25509 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:49.397182 18807 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:49.495988 25509 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.496694 17913 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:49.503788 25299 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:49.504359 25299 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:49.505424 25299 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.505702 24624 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.507156 24624 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:49.510352 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.511137 25414 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.513396 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 2989) took 1640 ms. Trace:
W20260709 12:24:49.454706 18022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.035s	sys 0.076s
I20260709 12:24:49.518990 18164 rpcz_store.cc:276] 0709 12:24:47.873179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.873344 (+   165us) service_pool.cc:224] Handling call
0709 12:24:49.513169 (+1639825us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400273dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:49.513318 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2836699392}
I20260709 12:24:49.520218 25362 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:49.520351 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1015) took 1735 ms. Trace:
W20260709 12:24:49.520678 25464 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.083s	user 0.009s	sys 0.006s
I20260709 12:24:49.520936 18164 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:49.456084 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:49.503726 25467 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.004s
I20260709 12:24:49.495280 24958 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:49.515508 24892 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.404s	user 0.017s	sys 0.000s
I20260709 12:24:49.521242 25455 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:49.471747 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113814us
I20260709 12:24:49.529381 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:49.530442 18024 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:49.384568 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.532074 18170 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:49.532212 18018 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:49.513384 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:49.533561 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.537128 18096 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:49.542038 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:49.541365 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.457464 25502 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:49.523489 18092 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:24:49.466463 25381 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.544129 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:49.565694 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:49.544481 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 205617us
I20260709 12:24:49.527137 24958 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:49.546607 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.549364 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.530803 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:49.539461 25380 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:49.544324 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 361515us
I20260709 12:24:49.527868 25299 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:49.520931 18022 rpcz_store.cc:276] 0709 12:24:47.785465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.785701 (+   236us) service_pool.cc:224] Handling call
0709 12:24:47.786122 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.786383 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:48.408607 (+622224us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.519820 (+1111213us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:49.519885 (+    65us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:49.519961 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.520080 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.520269 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17606,"prepare.queue_time_us":449463,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":1439,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":110,"thread_start_us":1714921,"threads_started":2}
I20260709 12:24:49.552394 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:49.572734 18096 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 21 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:49.540339 25383 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:49.584769 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.577562 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46942.5us
I20260709 12:24:49.587056 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:49.589892 18138 leader_election.cc:336] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.076s (SENT)
W20260709 12:24:49.601891 18138 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.123s (SENT)
I20260709 12:24:49.593117 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.541496 24892 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.591732 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.610164 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.598078 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:49.604826 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:49.544543 25455 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.622195 18096 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.608960 18019 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:49.592397 25301 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:49.621997 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:49.631233 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:49.628185 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.635696 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.590575 18023 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:49.539808 25467 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:49.643349 18138 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.210s (SENT)
I20260709 12:24:49.644292 25467 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.606346 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:49.646649 18138 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 3.104s (SENT)
I20260709 12:24:49.618460 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:49.597963 25413 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.646256 18016 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:24:49.651516 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.603291 25441 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.607993 24841 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.652101 18169 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:24:49.583184 25380 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:49.633669 25390 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.898s	user 0.015s	sys 0.006s
I20260709 12:24:49.659137 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:49.633359 25301 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.595093 25383 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.650403 18022 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
W20260709 12:24:49.665266 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93174us
I20260709 12:24:49.687032 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:49.645630 25432 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:49.667799 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1085) took 1797 ms (client timeout 2133 ms). Trace:
W20260709 12:24:49.693341 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.669746 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.674496 25392 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:49.678949 25362 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:49.681301 18096 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:49.704381 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:49.692478 18016 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:49.705466 18021 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:49.705188 18096 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:49.693284 18165 rpcz_store.cc:269] 0709 12:24:47.870534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.871200 (+   666us) service_pool.cc:224] Handling call
0709 12:24:49.667452 (+1796252us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:49.667715 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2713446272}
I20260709 12:24:49.711544 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1012) took 2100 ms. Trace:
I20260709 12:24:49.665947 25509 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.688858 25301 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.719578 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:49.722090 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.765s (SENT)
I20260709 12:24:49.726625 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:49.727624 25463 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:49.712486 18015 rpcz_store.cc:276] 0709 12:24:47.610903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.752689 (+141786us) service_pool.cc:224] Handling call
0709 12:24:47.803101 (+ 50412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.803364 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:48.882031 (+1078667us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.665041 (+783010us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.710858 (+ 45817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:49.710965 (+   107us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:49.711064 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.711201 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.711436 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46516,"prepare.queue_time_us":579836,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":38,"raft.queue_time_us":113093,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":1820013,"threads_started":4,"wal-append.queue_time_us":548033}
W20260709 12:24:49.733466 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 102359us
I20260709 12:24:49.698103 25392 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:49.737711 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.100s (SENT)
I20260709 12:24:49.706522 25362 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:49.744407 25515 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
W20260709 12:24:49.745563 25520 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 12:24:49.750977 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 95780.7us
I20260709 12:24:49.706579 25518 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.631059 24958 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.705154 25432 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:49.716593 25423 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:49.662456 25441 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.664304 24841 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.763620 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 3997) took 2714 ms. Trace:
I20260709 12:24:49.791363 18095 rpcz_store.cc:276] 0709 12:24:47.033851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.368637 (+334786us) service_pool.cc:224] Handling call
0709 12:24:47.369019 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.084753 (+1715734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.747746 (+662993us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b5400410ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:49.748013 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3804471680}
I20260709 12:24:49.764961 18016 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:49.689987 25390 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:49.779336 18813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1509) took 2179 ms (client timeout 2248 ms). Trace:
W20260709 12:24:49.794109 18813 rpcz_store.cc:269] 0709 12:24:47.587094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.587224 (+   130us) service_pool.cc:224] Handling call
0709 12:24:49.751527 (+2164303us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540059fe54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:49.766154 (+ 14627us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3348723456}
I20260709 12:24:49.732229 25414 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.771293 25424 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
W20260709 12:24:49.787230 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67777.2us
W20260709 12:24:49.812254 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.728693 25525 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:24:49.820408 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:49.773197 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:49.801204 25414 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.798718 25463 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.032s
W20260709 12:24:49.843586 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70497.6us
I20260709 12:24:49.843816 18168 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:49.694278 25502 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.849220 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:49.765784 25431 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:49.864233 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.836443 24958 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.778777 25505 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:49.878463 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:49.838711 25467 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:49.875383 25515 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:49.843366 25457 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.906262 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:49.912622 18096 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:49.907125 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1017) took 1341 ms. Trace:
I20260709 12:24:49.843264 25465 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.881489 25424 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:49.884900 25414 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.909077 18167 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:49.884109 25527 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.843914 25539 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:49.913741 18017 rpcz_store.cc:276] 0709 12:24:48.565364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.565970 (+   606us) service_pool.cc:224] Handling call
0709 12:24:49.039749 (+473779us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:49.039970 (+   221us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.040464 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:49.040497 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.041141 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:24:49.354454 (+313313us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:49.354510 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.354595 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.899094 (+544499us) raft_consensus.cc:1685] finished
0709 12:24:49.899126 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.907037 (+  7911us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2003,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"spinlock_wait_cycles":3501440,"thread_start_us":312701,"threads_started":1,"wal-append.queue_time_us":556160}
W20260709 12:24:49.900563 25364 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.770s	user 0.012s	sys 0.006s
I20260709 12:24:49.836663 25535 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:49.786362 25518 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.880448 25427 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.898629 25168 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.144s
I20260709 12:24:50.029742 25424 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:49.887094 25244 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:49.899719 25464 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.906083 25323 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.138s
I20260709 12:24:50.034816 25465 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.084236 25465 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:50.085736 25534 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
W20260709 12:24:50.099020 25529 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:24:50.101629 25390 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:49.910193 25096 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.144s
I20260709 12:24:49.840554 25392 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.908821 18018 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:50.078094 25464 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.034184 25457 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:49.909058 25432 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.068687 25424 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:50.123256 25489 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:50.072292 25300 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:24:50.063444 25427 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.173225 25300 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:50.178500 25464 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.179720 18813 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:50.179986 25457 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:50.058480 25401 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.181089 25401 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:50.181542 25505 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.186517 24841 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:50.152067 25435 consensus_queue.cc:1175] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" responder_term: 14 status { last_received { term: 12 index: 12 } last_committed_idx: 12 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:50.152638 25392 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:50.189913 25538 consensus_queue.cc:1175] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" responder_term: 21 status { last_received { term: 20 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:24:50.196830 25469 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.856s	user 0.009s	sys 0.004s
W20260709 12:24:50.169545 25536 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.325s	user 0.001s	sys 0.000s
I20260709 12:24:50.169873 25532 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:50.198424 24624 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.691s	user 0.013s	sys 0.002s
I20260709 12:24:50.161481 25508 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:50.168573 25519 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.298s	user 0.001s	sys 0.001s
I20260709 12:24:50.206594 25518 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.207748 25431 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:50.063936 25527 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.160487 25380 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.173731 25300 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:50.217394 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4013) took 2045 ms. Trace:
W20260709 12:24:50.234275 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 325896us
W20260709 12:24:50.248648 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.255930 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:50.286715 18097 rpcz_store.cc:276] 0709 12:24:48.172016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.172224 (+   208us) service_pool.cc:224] Handling call
0709 12:24:48.172586 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.460681 (+1288095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.217071 (+756390us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400214154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.217315 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2960630656}
I20260709 12:24:50.247318 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1334) took 1675 ms. Trace:
I20260709 12:24:50.261447 18098 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:50.264437 18095 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:50.267184 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.235195 25532 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:50.287570 18808 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
W20260709 12:24:50.296520 25299 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.704s	user 0.014s	sys 0.000s
I20260709 12:24:50.287981 18810 rpcz_store.cc:276] 0709 12:24:48.572247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.572783 (+   536us) service_pool.cc:224] Handling call
0709 12:24:48.596615 (+ 23832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.992176 (+395561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.992230 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.992299 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:48.992358 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.246890 (+1254532us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5400494154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.246972 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.247216 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":908925696}
W20260709 12:24:50.300833 18092 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.004s	sys 0.004s
W20260709 12:24:50.302126 18092 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.005s	sys 0.004s
W20260709 12:24:50.303877 18807 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.002s	sys 0.006s
W20260709 12:24:50.304919 18807 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.002s	sys 0.007s
I20260709 12:24:50.258575 25465 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.235319 25469 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.317621 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:50.235803 25301 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.284950 25521 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:50.298118 25532 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:50.352697 18137 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.000s	sys 0.015s
W20260709 12:24:50.357488 18137 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.000s	sys 0.016s
W20260709 12:24:50.359215 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.062s (SENT)
I20260709 12:24:50.346415 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:50.359815 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:50.363488 18016 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.014s	sys 0.011s
W20260709 12:24:50.339990 25498 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.849s	user 0.012s	sys 0.004s
W20260709 12:24:50.364576 18016 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.015s	sys 0.011s
I20260709 12:24:50.335762 25427 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.332640 25244 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:50.363516 25299 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.409803 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:50.414274 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:50.327913 25465 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.336479 25521 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.430994 25244 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:50.369557 25551 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.436856 18172 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:50.411070 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:50.460067 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:50.442342 25431 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:50.442698 25299 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:50.443063 25508 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.442368 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:50.443635 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.628s (SENT)
W20260709 12:24:50.454280 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.460974 18166 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:24:50.466614 25506 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.466866 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:50.484107 18164 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:50.467097 25527 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.471675 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2993) took 1398 ms. Trace:
I20260709 12:24:50.473678 18171 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:24:50.475203 25469 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.475859 25455 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:50.477098 25558 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:24:50.478940 25301 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.485005 18096 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:50.487658 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:50.487701 18168 rpcz_store.cc:276] 0709 12:24:49.073000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.075070 (+  2070us) service_pool.cc:224] Handling call
0709 12:24:49.075486 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.847469 (+771983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.471354 (+623885us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b54000433d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.471598 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1763815296}
I20260709 12:24:50.356844 25300 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.462495 25244 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:50.467131 25553 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:50.517591 25300 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:50.518536 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74945.1us
I20260709 12:24:50.490278 25506 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.520655 25301 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:50.520957 18165 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:50.522497 25506 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:50.487217 25431 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:50.520138 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.531895 18096 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:24:50.532733 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.486676 25508 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.426839 25498 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.541983 18164 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:50.539179 25508 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.548823 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:50.555195 18165 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:50.544221 25319 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:50.571593 25491 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.470s	user 0.000s	sys 0.003s
W20260709 12:24:50.574677 24958 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.672s	user 0.009s	sys 0.005s
I20260709 12:24:50.576851 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3148) took 1012 ms. Trace:
I20260709 12:24:50.579155 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3891) took 1048 ms. Trace:
I20260709 12:24:50.579843 18020 rpcz_store.cc:276] 0709 12:24:49.531113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.531516 (+   403us) service_pool.cc:224] Handling call
0709 12:24:49.548024 (+ 16508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.716801 (+168777us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:49.877968 (+161167us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.323835 (+445867us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:50.323932 (+    97us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:50.324020 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.578861 (+254841us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54003485d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.578919 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.579082 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5045,"prepare.queue_time_us":65305,"prepare.run_cpu_time_us":11042,"prepare.run_wall_time_us":91903,"raft.queue_time_us":11851,"raft.run_wall_time_us":5814,"spinlock_wait_cycles":407846144,"thread_start_us":591311,"threads_started":2,"wal-append.queue_time_us":464681}
I20260709 12:24:50.528743 25553 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.577443 18092 rpcz_store.cc:276] 0709 12:24:49.564087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.564332 (+   245us) service_pool.cc:224] Handling call
0709 12:24:49.595874 (+ 31542us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:49.621355 (+ 25481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.621390 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:49.643325 (+ 21935us) log.cc:844] Serialized 52 byte log entry
0709 12:24:49.662863 (+ 19538us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:50.302753 (+639890us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.303716 (+   963us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.575944 (+272228us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b18004178e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c1626dd70 00007f6c16261293 00007f6c16256cea 00007f6c1625723a 00007f6c166c1a2a 00007f6c166d10bf 00007f6c166d1038
0709 12:24:50.576311 (+   367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.576407 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:50.576546 (+   139us) raft_consensus.cc:1685] finished
0709 12:24:50.576564 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.576767 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":197401,"prepare.queue_time_us":223092,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":27719,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":25958,"spinlock_wait_cycles":31344128,"thread_start_us":624245,"threads_started":2,"wal-append.queue_time_us":131283}
I20260709 12:24:50.559351 25319 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.595968 25498 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.596382 25441 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.597218 25319 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:50.584831 25563 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:50.579087 25390 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:50.590559 25362 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.810s	user 0.012s	sys 0.005s
I20260709 12:24:50.601744 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:50.604609 25390 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:50.608496 18807 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:50.604986 25431 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.613407 25465 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:50.616006 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.617076 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3893) took 1085 ms. Trace:
I20260709 12:24:50.617770 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 2997) took 1512 ms. Trace:
I20260709 12:24:50.617516 18021 rpcz_store.cc:276] 0709 12:24:49.531814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.531954 (+   140us) service_pool.cc:224] Handling call
0709 12:24:49.551891 (+ 19937us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.705902 (+154011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.616802 (+910900us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.617002 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352276224}
I20260709 12:24:50.618291 18167 rpcz_store.cc:276] 0709 12:24:49.105621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.113773 (+  8152us) service_pool.cc:224] Handling call
0709 12:24:49.161013 (+ 47240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.041241 (+880228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.613809 (+572568us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.617704 (+  3895us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1719738624}
I20260709 12:24:50.620507 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3900) took 1086 ms. Trace:
I20260709 12:24:50.620962 18019 rpcz_store.cc:276] 0709 12:24:49.534486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.637116 (+102630us) service_pool.cc:224] Handling call
0709 12:24:49.637587 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.892707 (+255120us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.892746 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.914460 (+ 21714us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.318851 (+404391us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:50.371612 (+ 52761us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.373379 (+  1767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.373509 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:50.567906 (+194397us) spinlock_profiling.cc:243] Waited 7.03 ms on lock 0x7b1800254aa0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:24:50.611680 (+ 43774us) raft_consensus.cc:1685] finished
0709 12:24:50.620245 (+  8565us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.620430 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77896,"prepare.queue_time_us":52370,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":6309,"raft.queue_time_us":129949,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":10400,"spinlock_wait_cycles":104310400,"thread_start_us":637888,"threads_started":2,"wal-append.queue_time_us":498960}
W20260709 12:24:50.623612 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.627408 25465 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.630651 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:50.632052 18807 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:50.618683 25566 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:24:50.616757 25553 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.628332 25465 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.610200 25362 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.651022 25465 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.641518 25390 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.652915 25390 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:50.655395 25289 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:50.671181 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3901) took 1136 ms. Trace:
I20260709 12:24:50.671742 18023 rpcz_store.cc:276] 0709 12:24:49.534832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.641115 (+106283us) service_pool.cc:224] Handling call
0709 12:24:49.647364 (+  6249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.578156 (+930792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.578204 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:50.578249 (+    45us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:50.578293 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.670861 (+ 92568us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54001d1854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.670920 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.671094 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1152,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":5383,"spinlock_wait_cycles":2139385600}
I20260709 12:24:50.676286 25362 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.677739 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:50.664747 25424 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.688092 25424 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.695815 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.696794 18097 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:50.698287 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.699131 18093 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:50.708971 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3144) took 1336 ms. Trace:
I20260709 12:24:50.709532 18099 rpcz_store.cc:276] 0709 12:24:49.372464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.513682 (+141218us) service_pool.cc:224] Handling call
0709 12:24:49.514052 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.707460 (+1193408us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:50.707497 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:50.707547 (+    50us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:50.707590 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.708687 (+  1097us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.708743 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.708895 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2743538048}
I20260709 12:24:50.712137 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.689553 25532 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.691380 18095 pending_rounds.cc:85] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 11
I20260709 12:24:50.718616 18095 pending_rounds.cc:107] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20260709 12:24:50.719035 18095 raft_consensus.cc:2889] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:50.719967 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.720480 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.693528 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.721863 18090 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:50.690050 24841 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.713851 18099 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:24:50.714214 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.717202 18098 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:50.717622 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4012) took 3174 ms. Trace:
I20260709 12:24:50.722666 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.726173 18097 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:50.726533 18017 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:24:50.728178 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:50.726191 18094 rpcz_store.cc:276] 0709 12:24:47.543533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.846804 (+303271us) service_pool.cc:224] Handling call
0709 12:24:47.847168 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.882282 (+ 35114us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:48.414934 (+532652us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.597838 (+1182904us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.600883 (+  3045us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:49.600948 (+    65us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:49.600998 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.717305 (+1116307us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b540032b9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:50.717375 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.717548 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1952,"prepare.queue_time_us":515769,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":112,"raft.queue_time_us":15867,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":23593,"spinlock_wait_cycles":82054272,"thread_start_us":1674299,"threads_started":3,"wal-append.queue_time_us":1166906}
I20260709 12:24:50.728598 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:50.735267 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:50.743402 18020 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:50.741997 18815 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:24:50.744046 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.747712 18093 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:50.746683 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.751595 18021 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:50.738080 25539 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:50.756695 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3903) took 1221 ms (client timeout 1591 ms). Trace:
I20260709 12:24:50.738796 25474 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.764508 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.738457 25555 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:50.767290 18024 rpcz_store.cc:269] 0709 12:24:49.535392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.548940 (+ 13548us) service_pool.cc:224] Handling call
0709 12:24:50.756479 (+1207539us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54001d0454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:50.756609 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2257292672}
W20260709 12:24:50.759223 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.591s (SENT) [suppressed 30 similar messages]
I20260709 12:24:50.769603 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.753754 25463 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:50.738595 25057 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.776754 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.781383 18023 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:50.772352 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.742661 24841 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.801250 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.779808 25555 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.801481 25057 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.803860 18022 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:50.810966 25057 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.786324 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.792656 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.785684 18020 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:50.806407 18098 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:50.806067 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 120387us
I20260709 12:24:50.811244 25555 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.812127 25532 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.812148 18018 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:50.815462 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:50.815371 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.774845 25504 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.833992 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.821731 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:50.833290 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.834796 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.856681 18021 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:50.834311 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.860018 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.861066 18019 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:50.863267 25569 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:50.863353 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.863855 25504 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.774674 25539 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.824854 25300 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.866339 25300 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.867486 25539 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.868561 25471 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:50.869241 25471 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.869724 25471 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:50.871263 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.872635 18018 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:50.779698 25474 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:50.875684 25474 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.805630 25521 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.878545 24892 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.880689 25504 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:50.844352 25383 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:50.892170 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.818178 18064 connection.cc:256] RPC call timeout handler was delayed by -2.6534070059997248s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
I20260709 12:24:50.826650 25293 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.895342 25293 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:50.842499 25495 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.497s	user 0.006s	sys 0.000s
W20260709 12:24:50.898350 25554 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.419s	user 0.004s	sys 0.000s
I20260709 12:24:50.900324 25435 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:50.902659 25435 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:50.902988 25431 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:50.903626 25553 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:50.884397 25244 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:50.906752 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.864362 18021 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:50.780773 25463 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:50.910972 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.905936 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.894469 18020 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:50.912112 18019 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:50.912227 18017 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:50.891088 25300 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.914157 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.916126 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:24:50.908578 25311 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:50.917737 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.918700 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.919441 18024 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:50.919844 18099 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:24:50.925768 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.927585 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.928128 18092 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:50.929270 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.931396 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.933296 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.920434 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:50.881547 24841 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:50.940124 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.942739 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.944586 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.946393 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.947645 18018 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:50.950409 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.951370 18017 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:50.948499 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.917371 18090 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:50.916160 25463 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:50.955878 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:50.960594 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 196754us
I20260709 12:24:50.953900 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.959173 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.956722 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.962757 18018 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:50.962874 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.961581 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:50.963789 18090 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:50.963928 18024 raft_consensus.cc:2376] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19. Re-sending same reply.
I20260709 12:24:50.965333 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.965672 18093 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:50.966220 18024 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:50.966920 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.968007 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.970870 18023 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:50.971341 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:50.973800 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:50.972678 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.918388 25311 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:50.987910 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.983718 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:50.977144 18023 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:50.977272 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:50.979104 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:50.982477 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.985416 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.021284 18017 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:50.986714 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:50.991699 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:50.989477 18024 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:51.028663 18098 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:50.994199 18018 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:51.029100 18169 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:24:50.917899 25555 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:24:51.012135 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:51.034418 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.014503 18171 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [15.8-17.9]
I20260709 12:24:51.035262 18172 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:51.036016 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.015928 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.037384 18021 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:51.019835 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:51.038173 18093 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:50.973201 25432 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:51.018973 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.041651 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.042821 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.040309 18022 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:51.036968 18171 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:24:51.044880 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:51.045286 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.994156 18023 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:24:51.042855 18173 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:24:51.045203 18099 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:51.022547 25293 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.056584 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.059995 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:51.050603 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:51.032903 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:51.061035 18097 raft_consensus.cc:2376] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15. Re-sending same reply.
W20260709 12:24:51.047724 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 286159us
I20260709 12:24:51.051970 18168 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:51.003398 25299 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
W20260709 12:24:51.052627 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 3992) took 4094 ms (client timeout 2160 ms). Trace:
I20260709 12:24:51.063669 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:51.063939 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:51.063339 18091 rpcz_store.cc:269] 0709 12:24:46.947486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.025390 (+ 77904us) service_pool.cc:224] Handling call
0709 12:24:51.042101 (+4016711us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:51.042217 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5473353472}
I20260709 12:24:51.064602 18098 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:51.065244 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.065410 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.066524 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:51.066636 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83017.7us
I20260709 12:24:51.067601 18090 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 13. Current term is 14. Ops: [11.11-13.12]
I20260709 12:24:51.062203 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.070009 18167 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:24:51.071411 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.072348 18099 raft_consensus.cc:2376] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15. Re-sending same reply.
I20260709 12:24:51.074020 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.049927 25376 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:51.039006 25456 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:51.079950 18093 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:51.041568 25432 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.085743 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.092005 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.091322 25432 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:51.099825 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.079438 25376 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.110738 18171 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:24:51.100391 25575 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:24:51.115415 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.118932 18099 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:51.123960 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.126358 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:51.127012 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59678.4us
I20260709 12:24:51.084849 25539 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:24:51.061661 25311 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:51.151782 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.144286 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.145493 25576 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:24:51.141121 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:51.113909 25376 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.103909 25581 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:51.093219 25456 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.122260 25244 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.170322 25555 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:51.165797 25581 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.174515 25555 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.190419 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.193753 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:51.202310 18173 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:51.204028 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.196303 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:51.208214 18815 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:51.205780 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.209502 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.212061 18019 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:51.214874 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.213912 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:51.214854 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:51.216061 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:51.219576 18810 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:51.207789 25555 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:51.186501 25581 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:51.251276 25456 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.230513 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.253019 18090 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:24:51.258441 25471 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:51.224661 25577 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.175s	user 0.000s	sys 0.002s
I20260709 12:24:51.255273 18090 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:51.221532 18019 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:24:51.262811 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.263350 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.263610 18169 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:51.263962 18019 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:51.264062 18092 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:51.270682 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:51.307229 18167 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:51.310300 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:51.292479 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:51.311590 18092 raft_consensus.cc:2376] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21. Re-sending same reply.
I20260709 12:24:51.285516 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.236282 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:51.296643 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.314324 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.272110 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.312142 18167 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:51.317288 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.317240 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.312983 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:51.316164 18090 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:24:51.312067 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:51.314575 18169 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:51.325992 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1096) took 1677 ms. Trace:
W20260709 12:24:51.334828 17989 connection.cc:256] RPC call timeout handler was delayed by -1.0904603100002532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3138s for a response
I20260709 12:24:51.336897 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:51.337885 18098 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:51.349596 25376 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.351094 25392 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:51.351540 25392 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:51.352691 25376 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.352530 25392 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.358320 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.359014 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.361299 18092 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:24:51.362385 18092 raft_consensus.cc:606] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:51.362807 18092 raft_consensus.cc:614] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:51.327265 18170 rpcz_store.cc:276] 0709 12:24:49.648242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.648408 (+   166us) service_pool.cc:224] Handling call
0709 12:24:49.659202 (+ 10794us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.659519 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:51.148936 (+1489417us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b1800399640. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf
0709 12:24:51.154437 (+  5501us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.325472 (+171035us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:51.325570 (+    98us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:51.325640 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.325724 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.325909 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88978,"prepare.queue_time_us":140141,"prepare.run_cpu_time_us":222,"prepare.run_wall_time_us":6226,"raft.queue_time_us":4041,"spinlock_wait_cycles":26661248,"thread_start_us":245084,"threads_started":2}
I20260709 12:24:51.365406 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:51.327384 25431 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:51.340696 25364 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:51.343235 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:51.378600 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:51.380398 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:51.360926 25526 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:51.346895 25300 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:51.377612 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:51.357640 25392 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.368731 25244 raft_consensus.cc:2764] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 12:24:51.381431 25578 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.219s	user 0.002s	sys 0.000s
I20260709 12:24:51.372112 25431 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.405028 25431 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:51.404691 25581 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.384169 25364 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:51.408689 25364 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.409502 25244 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:51.410046 25244 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.410418 25244 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:51.393288 25526 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:51.412446 25526 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.386386 25465 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:51.398447 25471 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:51.380458 25521 raft_consensus.cc:2764] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:51.371189 25383 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:24:51.421975 25581 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.345528 25463 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:51.426086 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.437024 25383 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:51.437722 25383 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:51.438882 25383 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.404573 25555 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.440714 25383 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:51.442478 25588 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:24:51.444507 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3006) took 1004 ms. Trace:
I20260709 12:24:51.445099 18165 rpcz_store.cc:276] 0709 12:24:50.439979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.442002 (+  2023us) service_pool.cc:224] Handling call
0709 12:24:51.444250 (+1002248us) spinlock_profiling.cc:243] Waited 39.3 ms on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:51.444414 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90312064}
I20260709 12:24:51.383200 25299 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:51.450968 25590 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:24:51.451267 25299 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.454378 18813 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:51.453987 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61229.2us
I20260709 12:24:51.393884 25300 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:51.451699 25299 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.452497 25555 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:24:51.516728 25463 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.488356 25581 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.518493 25581 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:51.522940 25592 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:24:51.521653 25581 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:51.555037 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174702us
I20260709 12:24:51.558352 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:51.567420 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:51.591771 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73137.2us
I20260709 12:24:51.593696 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:51.596566 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138933us
I20260709 12:24:51.546753 25432 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.602351 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.571028 25521 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:51.611011 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.616501 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:51.620950 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.630676 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:51.633082 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.629608 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:24:51.609854 25463 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:51.570715 25300 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.608590 25364 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.657577 25599 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:24:51.637626 25399 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:51.607014 25319 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:24:51.637668 25576 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:51.618952 25602 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:51.646188 25456 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:51.666616 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.609346 25435 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:51.668875 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63536.6us
W20260709 12:24:51.655822 25600 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:24:51.670300 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:51.683765 18024 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:51.669309 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:51.685132 18018 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:51.671628 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:51.669438 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:51.686949 18017 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:51.643711 25585 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:51.688119 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:51.689495 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:51.694317 18022 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:51.618710 25584 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:51.693518 25435 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.676573 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:51.656935 25364 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.696363 18020 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:24:51.649744 25463 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:51.697427 25463 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.698218 25364 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.678174 18023 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:51.695607 25584 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.683336 25456 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.700271 25584 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:51.652365 25581 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:51.666189 25399 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:51.706502 25399 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.708510 25435 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.713268 25603 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:51.714097 25603 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:51.716267 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.718519 18024 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:24:51.644953 25293 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:24:51.715880 25602 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:51.748406 25456 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.748801 25293 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:51.754180 25613 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.793853 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.802985 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 170023us
I20260709 12:24:51.804967 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:51.807722 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:51.812251 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.797685 25502 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.818399 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91835.7us
W20260709 12:24:51.821132 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 137595us
I20260709 12:24:51.823630 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.797880 25293 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:51.821870 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3210181190002004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
I20260709 12:24:51.825976 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:51.828490 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2913181190001524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260709 12:24:51.828909 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2607181190001029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
W20260709 12:24:51.829208 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1806181189999734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:24:51.829394 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1752181189999646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
I20260709 12:24:51.810365 25456 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:51.829615 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1545181189999312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20260709 12:24:51.810636 25581 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:51.831434 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0915181190002841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
W20260709 12:24:51.833194 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0762181190002593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
I20260709 12:24:51.833990 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.826493 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:51.839792 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.841524 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.842568 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:51.803777 25576 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:51.830996 25617 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.846460 25617 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.851159 25617 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.852005 25617 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:51.852458 25617 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:51.852890 25617 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.825206 25502 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:51.854561 25604 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:24:51.856254 25603 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.856773 25502 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.839952 25456 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.848080 25616 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:51.844305 25614 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:51.859812 25456 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.861989 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:51.867452 25602 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.870188 25619 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:51.870779 25619 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.871186 25619 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:51.847700 25521 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:51.835471 25526 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.874409 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:51.874322 25526 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.874958 25618 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:51.876266 25618 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:51.876971 25618 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:51.877084 25526 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.869280 25299 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.804039 25613 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.848652 25432 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:51.849519 25431 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.842940 25244 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:51.889603 25293 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.914377 25311 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:51.915685 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.917074 25244 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:51.887177 25584 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.919217 25299 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:51.919775 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4026) took 1654 ms (client timeout 2167 ms). Trace:
W20260709 12:24:51.920243 18096 rpcz_store.cc:269] 0709 12:24:50.265548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.266747 (+  1199us) service_pool.cc:224] Handling call
0709 12:24:51.919557 (+1652810us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:51.919699 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447428864}
I20260709 12:24:51.921312 25584 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:51.923400 25431 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.925686 25293 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.927341 25399 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:51.914705 25392 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.523s	user 0.015s	sys 0.000s
I20260709 12:24:51.940088 25399 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:51.913084 25613 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:51.876418 25576 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 20.18, Last appended by leader: 18, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.878039 25618 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:51.950207 25463 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:51.963266 25556 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.483s	user 0.006s	sys 0.000s
W20260709 12:24:51.969149 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94838us
W20260709 12:24:51.971179 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 128688us
I20260709 12:24:51.971791 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:51.973052 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.974887 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:51.975879 18024 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:51.976205 18091 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:51.979732 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.992172 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:51.994833 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:51.993945 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:51.997855 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:51.998363 18024 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:52.001751 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.003916 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1029) took 2230 ms. Trace:
I20260709 12:24:52.002059 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.005843 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.004585 18016 rpcz_store.cc:276] 0709 12:24:49.768037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.799061 (+ 31024us) service_pool.cc:224] Handling call
0709 12:24:49.799522 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.819213 (+ 19691us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.819255 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.820728 (+  1473us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.427329 (+606601us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:50.427394 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.427495 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.962355 (+1534860us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:51.968772 (+  6417us) raft_consensus.cc:1685] finished
0709 12:24:51.968803 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.998631 (+ 29828us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9829760,"thread_start_us":605290,"threads_started":1,"wal-append.queue_time_us":658902}
I20260709 12:24:52.006595 18093 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:52.014472 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:52.018116 25584 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:52.019508 25524 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:52.020112 25524 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:51.994612 25244 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:24:52.028050 25244 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:52.028201 25582 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:52.009979 25603 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:52.021184 25524 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.029628 25524 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:52.028887 25582 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:51.996589 25463 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.031284 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:51.999971 25465 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:52.034852 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.047329 18168 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:52.047251 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.000451 25420 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:52.048574 18171 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:52.037930 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.048857 18168 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 13. (term mismatch)
I20260709 12:24:52.049223 18168 pending_rounds.cc:85] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 12
I20260709 12:24:52.049275 18091 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:52.049583 18168 pending_rounds.cc:107] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260709 12:24:52.035964 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.051867 18166 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:24:52.044284 18095 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:52.033631 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.048116 18172 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:52.056661 18168 raft_consensus.cc:2889] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:52.060446 18809 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:52.061640 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.062652 18165 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:52.066479 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.067899 18172 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:52.069033 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.028578 25244 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:52.055181 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.069792 18168 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:52.045221 25603 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:52.059091 25586 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:24:52.043452 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.097328 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.101012 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:52.110730 18138 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.670s (SENT)
W20260709 12:24:52.120960 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.122085 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.123688 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.124492 18098 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:52.092417 25581 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:52.102177 18095 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20. Re-sending same reply.
I20260709 12:24:52.103060 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:52.130295 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.132817 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:52.120764 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77388.6us
I20260709 12:24:52.135589 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:52.142453 25502 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:52.144863 25504 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:52.146934 25383 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:52.146781 18810 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:52.149435 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:52.160054 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:24:52.127535 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 90555.5us
I20260709 12:24:52.160902 18092 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:52.094774 25526 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:52.163818 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:52.183362 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:52.164783 18138 leader_election.cc:336] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.724s (SENT)
I20260709 12:24:52.126122 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:52.184255 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.184710 18809 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:52.175695 18168 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:52.184717 18093 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
W20260709 12:24:52.197113 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 188996us
I20260709 12:24:52.183512 18165 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:24:52.132051 25465 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:52.201182 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.205936 25635 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:52.207376 25300 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:52.212664 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.213374 18022 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20. Re-sending same reply.
I20260709 12:24:52.147295 25619 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:52.209092 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44404us
I20260709 12:24:52.214526 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.148428 25424 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:52.193746 25526 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.232151 18018 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:52.211452 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:52.215610 18138 leader_election.cc:336] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.771s (SENT)
I20260709 12:24:52.234529 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.230268 18021 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:52.235673 18811 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:52.215734 18017 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:52.149789 25383 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:52.187829 25456 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.192008 18091 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:24:52.226308 18092 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:24:52.196018 25598 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:24:52.209844 24841 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:52.238353 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:52.219698 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:52.239217 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:52.240000 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.246539 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47394.7us
I20260709 12:24:52.247738 18098 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:52.248538 18020 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:24:52.227811 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.181581 25623 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.250828 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.256794 18095 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 12:24:52.268347 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.259529 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.262251 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.250808 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:52.275872 18809 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:52.264407 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.255766 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.251991 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.271284 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.280892 18815 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:52.283238 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.284963 18168 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:52.285748 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.286134 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.286684 18808 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:52.286556 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.287289 18171 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:52.287321 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.288168 18168 raft_consensus.cc:2368] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
I20260709 12:24:52.288318 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.288908 18171 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:52.289538 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.290318 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.182065 25502 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:24:52.292251 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.296643 18165 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:24:52.297497 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.298310 18165 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:24:52.288928 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.228613 25424 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.294071 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.295331 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.300522 18171 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:52.300951 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.308817 18097 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:52.301795 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.103458 25582 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.249854 25602 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:52.322367 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.315819 18815 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:52.182193 25244 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.281731 25319 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.252820 25456 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.323447 18166 raft_consensus.cc:2376] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19. Re-sending same reply.
I20260709 12:24:52.342562 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.334510 18807 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:52.317329 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.343891 18091 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:52.336218 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.282135 25623 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:52.337383 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.340765 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.345580 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:52.346701 18093 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:52.341010 18172 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: [14.8-16.9]
W20260709 12:24:52.343883 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 132309us
I20260709 12:24:52.350610 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1343) took 1462 ms. Trace:
I20260709 12:24:52.308337 25424 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:52.351048 18813 rpcz_store.cc:276] 0709 12:24:50.888413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.888620 (+   207us) service_pool.cc:224] Handling call
0709 12:24:50.889058 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.455659 (+566601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.350349 (+894690us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54005ad554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:52.350530 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1299549440}
I20260709 12:24:52.353190 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:52.363924 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.374158 18815 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:24:52.374428 18168 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:52.366904 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.371028 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.382165 18097 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20260709 12:24:52.317724 25596 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:24:52.382601 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 78229.8us
I20260709 12:24:52.382956 18809 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:24:52.385345 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:52.386437 18093 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:24:52.385481 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:52.388695 18172 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:52.381100 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:52.393551 18168 raft_consensus.cc:2376] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18. Re-sending same reply.
I20260709 12:24:52.397699 18171 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:52.388628 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:52.350306 25456 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.390172 25637 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:52.376701 25598 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:24:52.419138 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.421305 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:52.422397 25634 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
W20260709 12:24:52.424955 25625 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:24:52.425096 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:52.376873 25427 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.434217 18091 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:52.452695 25639 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:52.438499 25632 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:24:52.442735 25619 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:52.351604 25319 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:52.445284 25311 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.530s	user 0.006s	sys 0.006s
I20260709 12:24:52.342244 25383 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.325202 25629 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.105s
I20260709 12:24:52.353758 25623 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.386893 25244 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:52.436908 18815 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:24:52.466274 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.489315 18091 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:52.384058 25617 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.500612 18168 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:52.505663 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.489056 25639 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:52.508992 18091 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:52.509146 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:52.490787 25635 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:52.517253 18173 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:52.523655 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.501962 25383 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:52.458660 25637 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.468501 25427 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.534523 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:52.531567 25576 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:52.514629 25624 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.499459 25602 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:52.538084 25581 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:52.527501 25619 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 12:24:52.550457 25624 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.547704 25526 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.552445 25463 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:52.563642 25624 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.560151 25635 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.566169 25526 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:52.584179 25526 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.586812 25526 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.587440 25623 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:52.588898 25524 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.559s	user 0.012s	sys 0.000s
I20260709 12:24:52.590155 25637 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.588954 25617 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:52.590415 25602 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.590113 25623 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.570458 25639 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.593479 25624 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:52.636003 17987 connection.cc:256] RPC call timeout handler was delayed by -1.0506979420001699s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
W20260709 12:24:52.468024 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 111933us
W20260709 12:24:52.641094 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.642254 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.646462 18024 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:52.647138 18091 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:24:52.641273 18094 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:52.639864 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.662369 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:52.663916 18020 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:52.665921 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 281202us
W20260709 12:24:52.668210 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.675002 25319 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:52.675886 25427 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.676941 25427 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.678869 25427 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.682595 25413 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:52.683145 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:52.571565 25502 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:52.684053 25413 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.685273 25502 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:52.687922 25502 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.692801 25427 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.698252 25617 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:52.699611 25617 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.703125 18022 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:52.706562 25624 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:52.697131 25502 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:52.708408 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.742093 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.744735 25637 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:52.762145 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:52.764478 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104856us
I20260709 12:24:52.767069 18021 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:52.766666 25623 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:52.768306 25623 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:52.769115 25651 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:52.772217 25617 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:52.777371 25413 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:52.773589 25617 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:52.788206 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2961) took 1712 ms (client timeout 2237 ms). Trace:
W20260709 12:24:52.800307 18173 rpcz_store.cc:269] 0709 12:24:51.033029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.212614 (+179585us) service_pool.cc:224] Handling call
0709 12:24:52.745270 (+1532656us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540007b554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:52.745576 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2938621696}
I20260709 12:24:52.808205 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:52.812455 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 498586us
I20260709 12:24:52.814219 25413 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:52.815066 25413 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:52.816856 25413 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.819523 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 2951) took 1824 ms. Trace:
W20260709 12:24:52.820423 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 396479us
I20260709 12:24:52.826880 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:52.823500 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113986us
I20260709 12:24:52.769227 25623 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:52.795801 25617 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:24:52.840401 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.787408 25637 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260709 12:24:52.854728 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:52.856120 18809 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:52.866262 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124697us
I20260709 12:24:52.867084 24841 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:52.870573 25431 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:24:52.884161 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:52.909188 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.909722 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:52.910812 18020 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:52.850895 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:52.923137 18809 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:52.921665 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:52.927507 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50559.5us
I20260709 12:24:52.824764 18164 rpcz_store.cc:276] 0709 12:24:50.995345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.995531 (+   186us) service_pool.cc:224] Handling call
0709 12:24:51.017183 (+ 21652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.022277 (+  5094us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.022331 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.961878 (+939547us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b1800086b60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:24:52.026533 (+ 64655us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b2c001268f8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:24:52.027199 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.306265 (+279066us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:52.329258 (+ 22993us) log.cc:844] Serialized 14 byte log entry
0709 12:24:52.818693 (+489435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.818889 (+   196us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.819001 (+   112us) raft_consensus.cc:1685] finished
0709 12:24:52.819025 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.819277 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":493095,"prepare.queue_time_us":22652,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"raft.queue_time_us":496726,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":9554,"spinlock_wait_cycles":7271552,"thread_start_us":278582,"threads_started":1,"wal-append.queue_time_us":422913}
W20260709 12:24:52.930879 25653 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.117s	user 0.005s	sys 0.000s
I20260709 12:24:52.931358 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:52.926455 24841 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.933352 24841 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:52.934381 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44159us
I20260709 12:24:52.936204 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:52.923804 25431 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:52.938350 25557 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:52.939353 25557 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:52.944024 24841 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:52.946372 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78881us
I20260709 12:24:52.946185 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:52.948827 25623 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:52.954758 25617 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:52.937498 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:52.978338 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:52.981089 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 171214us
W20260709 12:24:52.986946 25456 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.540s	user 0.021s	sys 0.004s
I20260709 12:24:52.948235 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:52.941061 25557 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:53.022289 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 84931.5us
I20260709 12:24:53.026535 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:53.034193 25383 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:53.035008 25383 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.035522 25383 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:53.038003 25424 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.015s
I20260709 12:24:53.038641 25424 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:53.043061 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77353.1us
I20260709 12:24:53.044947 18024 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:53.032191 18021 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:53.048646 18815 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:53.054975 25617 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:53.042241 25608 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:53.041751 25424 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.067651 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:53.068704 18809 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:53.053498 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.074242 25613 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:53.085624 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 150032us
I20260709 12:24:53.076577 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:53.091557 25244 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.012s
I20260709 12:24:53.089697 25648 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:53.081210 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.031931 25456 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:53.102380 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52485.2us
I20260709 12:24:53.092232 25648 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:53.104106 25608 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:53.117210 25431 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:53.118193 25431 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:53.119540 25431 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 20.18, Last appended by leader: 18, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.107318 25613 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:53.122349 18092 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:24:53.109728 25244 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.135488 18021 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
W20260709 12:24:53.141618 25652 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.335s	user 0.003s	sys 0.001s
I20260709 12:24:53.143294 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:53.145901 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88245.3us
W20260709 12:24:53.146677 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75232us
I20260709 12:24:53.148967 25613 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:24:53.145896 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115850us
I20260709 12:24:53.145888 25474 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.165869 25474 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.167824 25648 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:53.171077 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 143087us
I20260709 12:24:53.172372 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:53.175324 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:53.175544 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:53.177362 18807 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:53.177582 18024 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.179584 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:53.181972 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.182313 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:53.183353 18022 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
W20260709 12:24:53.186760 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71706.7us
W20260709 12:24:53.189463 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 145183us
I20260709 12:24:53.141741 25608 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.196760 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:53.199007 25471 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:53.199919 25471 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:53.201349 25390 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:24:53.212647 25603 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:24:53.214556 25526 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:53.215965 25526 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:53.216041 25602 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.216470 25526 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:53.216866 25602 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.218997 25390 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:53.223269 25390 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:53.224779 25526 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.225968 25645 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.226903 25645 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.227730 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.208099 25648 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:53.201968 25557 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:53.235225 25526 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:53.237004 25293 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.237300 25557 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:53.249059 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.240728 25456 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.201467 25471 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.259665 18097 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:24:53.248708 25557 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:53.262126 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.263067 24841 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:53.257473 25390 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.250140 25602 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:53.270265 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.278147 25521 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.278932 25390 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:53.285221 25582 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:53.294952 18097 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:53.300673 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.282617 25456 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:53.302632 25456 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:53.303217 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.290544 25521 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.300289 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.286374 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:53.297560 25645 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:53.342675 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62953.9us
I20260709 12:24:53.345963 25392 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:53.311479 25432 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:53.348399 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.348905 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.267707 25383 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.352897 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:53.366449 18094 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:24:53.344964 25456 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:53.375244 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3157) took 1113 ms. Trace:
I20260709 12:24:53.358721 25392 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:53.364729 25427 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:53.376169 18092 rpcz_store.cc:276] 0709 12:24:52.259064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.259220 (+   156us) service_pool.cc:224] Handling call
0709 12:24:53.365735 (+1106515us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400052154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:53.371972 (+  6237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884216704}
I20260709 12:24:53.386557 25399 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:24:53.387272 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.388232 18098 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:53.389070 25399 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:53.392019 25383 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.384191 25456 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:53.407163 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.405707 18024 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:53.402715 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:53.378940 25293 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:24:53.413105 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79299.4us
I20260709 12:24:53.413245 25392 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.414445 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:53.413801 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127647us
I20260709 12:24:53.403498 25399 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.430166 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:53.433531 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62196.2us
I20260709 12:24:53.442770 25432 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:53.450110 25464 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:53.450950 25464 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:53.458940 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.466163 25392 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.467911 25399 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:53.470911 25521 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:53.478148 25662 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.195s	user 0.001s	sys 0.004s
I20260709 12:24:53.452220 25464 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.413199 25293 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.486191 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:53.487914 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54634.2us
I20260709 12:24:53.488710 25293 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:53.488207 25427 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:53.492806 25427 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:53.494167 25383 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:53.494750 25383 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:53.495903 25383 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.428836 25602 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:53.500015 25539 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:53.501977 25456 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:53.502821 25456 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.503252 25244 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:53.505177 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90896.9us
I20260709 12:24:53.505470 25456 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.506457 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.456025 25557 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:53.508350 25664 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.124s	user 0.002s	sys 0.000s
I20260709 12:24:53.509940 25557 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:53.499432 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78212.2us
W20260709 12:24:53.486646 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 334189us
I20260709 12:24:53.533233 25244 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:53.534314 25244 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.542093 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:53.542994 18815 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:24:53.544210 25244 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:53.546128 17989 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.851s (SENT)
I20260709 12:24:53.550896 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:53.554128 25526 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:53.559855 25474 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:53.560420 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:53.568378 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62166.5us
I20260709 12:24:53.569945 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:53.568400 25244 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:53.575996 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:53.577587 18024 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:53.588272 18020 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 12:24:53.588851 18020 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 9
I20260709 12:24:53.589143 18020 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 12:24:53.589447 18020 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:53.589947 25557 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:53.565976 25608 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:24:53.594221 25602 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:53.595136 25602 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:53.596464 25602 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.598007 25376 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.598702 25376 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.601264 25376 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.602886 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.604122 25244 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:53.611034 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:53.612196 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.605700 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:53.634122 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 161363us
I20260709 12:24:53.662063 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.662415 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.663592 18090 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:24:53.664317 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:53.664798 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.665138 18096 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:53.665573 18094 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:53.666885 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.667773 18098 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:53.668989 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.646358 25244 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:53.670847 25244 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:53.672004 25244 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:53.678574 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.700157 18096 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 21 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:53.698588 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:53.702070 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.703853 18092 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:53.692849 25617 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.633s	user 0.011s	sys 0.008s
I20260709 12:24:53.701081 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:53.707932 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0317067360001602s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20260709 12:24:53.716959 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.709347 25464 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:53.745925 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:53.748447 18020 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:53.753505 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.747720 18090 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:24:53.755240 18090 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:24:53.756687 25413 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:53.756971 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:53.757771 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.759593 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:53.760548 18164 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:53.744607 25383 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.731536 25502 raft_consensus.cc:2764] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:53.755714 18090 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:53.754457 18094 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
W20260709 12:24:53.775208 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 81878.9us
I20260709 12:24:53.781040 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.782176 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:24:53.784457 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.780026 25464 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.794224 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.796578 25557 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:53.797180 25557 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:53.799368 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:53.801142 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.802372 25502 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.802907 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:53.803642 18169 raft_consensus.cc:2376] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18. Re-sending same reply.
I20260709 12:24:53.803315 25502 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.804559 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:53.805334 18017 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:53.793625 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:53.807178 25674 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:24:53.809523 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:53.816913 25427 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.821931 25675 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.822769 25427 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.822696 25675 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:53.833585 25675 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:53.798197 25464 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.836752 25502 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:53.742447 25613 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.841428 18138 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.536s (SENT)
I20260709 12:24:53.842147 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.850152 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.852766 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.853808 18170 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:53.856218 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:53.757531 25413 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.900334 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:53.851398 25675 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.920719 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.915158 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:53.840552 25613 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.905411 25608 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:53.930995 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89609.4us
I20260709 12:24:53.933189 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:53.908833 25392 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.937994 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:53.935231 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79126.8us
I20260709 12:24:53.930423 18169 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:53.947328 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:53.949337 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:53.949584 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:53.951012 18168 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:53.917520 25675 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.940101 25431 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.965451 18170 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:24:53.966476 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157099us
I20260709 12:24:53.975188 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:53.980656 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.991003 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:53.953436 25502 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.922534 25557 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.998207 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:53.999117 18097 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:53.943627 25392 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.003000 25687 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.993883 25431 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:53.996835 18173 raft_consensus.cc:2376] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18. Re-sending same reply.
I20260709 12:24:54.003943 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:54.005151 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.006132 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:53.997008 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52139.1us
I20260709 12:24:54.009858 18164 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:54.012614 18170 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:54.014372 18099 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:54.026958 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.028698 25672 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:54.029755 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.030207 25392 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:54.030581 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.031469 18097 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:54.033854 25557 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:54.034024 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.035748 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.036602 18166 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.038640 25413 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.022032 18169 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:24:54.041474 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.020954 18168 raft_consensus.cc:2376] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23. Re-sending same reply.
I20260709 12:24:54.040007 18167 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:54.051515 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.052778 18099 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:54.063030 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.064085 18095 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:54.064291 25427 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:54.065346 25427 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:54.066304 25613 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:54.067297 25675 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:54.067764 25683 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:54.030587 18165 raft_consensus.cc:2376] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18. Re-sending same reply.
W20260709 12:24:54.131852 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:54.126582 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 122689us
I20260709 12:24:54.079231 18170 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:54.021250 25582 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:54.099102 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.147396 18169 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:54.149535 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:54.152025 25682 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:24:54.152133 18167 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:24:54.153753 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.156049 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.156407 18167 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:54.148746 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:54.069093 25427 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.158582 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.159830 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:54.163715 18164 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260709 12:24:54.167151 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 207979us
I20260709 12:24:54.172041 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:54.178373 18094 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:54.180330 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.018975 25502 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:54.183662 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 192811us
I20260709 12:24:54.162396 25427 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:54.225247 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:54.163130 25582 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:24:54.236692 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82265.4us
I20260709 12:24:54.456797 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.170006 25681 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.240442 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.459192 18167 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:54.252211 25673 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.706s	user 0.007s	sys 0.000s
I20260709 12:24:54.223886 25413 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:54.050050 25687 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.470103 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.338027 25613 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:54.489235 25613 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:54.417534 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 179998us
I20260709 12:24:54.418546 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.418943 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.493902 25293 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:54.494444 25608 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:54.495687 25392 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:54.495749 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:54.419849 18092 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.709s	user 0.007s	sys 0.019s
I20260709 12:24:54.496328 25392 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:54.496603 18164 raft_consensus.cc:2376] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17. Re-sending same reply.
I20260709 12:24:54.497008 18092 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
W20260709 12:24:54.402501 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.497579 25392 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.420023 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.500263 18097 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:54.455683 25672 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:54.500993 25672 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:54.427692 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.503773 25672 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.504442 25696 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.505127 25696 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:54.505789 25392 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.507776 25696 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:54.427568 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 228595us
I20260709 12:24:54.428800 25645 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.510527 25645 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:54.513145 25645 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:54.429594 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:54.420641 25383 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.653s	user 0.013s	sys 0.013s
I20260709 12:24:54.431524 25431 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:54.514499 18171 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:24:54.516940 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:54.516422 17920 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:54.518157 18815 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 20 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:24:54.441799 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.444684 18091 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:54.521080 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:54.445634 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:54.447665 18022 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:54.448789 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:54.522467 18024 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.450054 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:54.450930 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:54.451663 25474 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:54.452423 18020 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:24:54.433203 18021 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.015s	sys 0.037s
I20260709 12:24:54.530804 18016 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:54.453667 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:54.531114 18021 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.015s	sys 0.038s
I20260709 12:24:54.182986 18165 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:54.442582 18166 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:54.460672 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.463232 25244 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.463946 25648 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:54.534219 25648 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.464840 25602 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.466073 25694 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:54.474741 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:54.477882 25582 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:54.538295 25602 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.539072 25431 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:54.539382 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64765us
I20260709 12:24:54.540617 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:54.540922 25602 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.463699 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.541671 25431 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:54.482882 18099 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:54.487254 25687 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.519886 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:54.426808 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.740s (SENT)
I20260709 12:24:54.542934 18015 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:54.542968 25648 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:54.543282 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116791us
I20260709 12:24:54.542285 25244 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.546216 25244 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.546518 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:54.524551 18169 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:54.527493 18167 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:24:54.521952 18018 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:24:54.532559 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:54.551237 18172 raft_consensus.cc:2368] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
I20260709 12:24:54.554587 18173 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:54.520444 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:54.557303 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63239.9us
I20260709 12:24:54.568077 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:54.580878 18023 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:54.559648 17920 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:54.556406 25681 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:54.587707 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45253.3us
I20260709 12:24:54.583282 25293 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:54.591058 18020 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:54.567342 25557 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.589222 18019 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:54.578253 25687 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.577117 25672 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:54.556167 25383 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:24:54.579252 25694 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:54.594100 25293 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.605286 25293 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:54.605703 25613 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:54.605789 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 142200us
I20260709 12:24:54.606472 25613 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:54.608158 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:54.608989 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:54.604796 25687 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.610917 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:54.611778 18015 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:54.612104 25687 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.612275 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.612974 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:54.614297 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.614652 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.615125 18095 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:54.619177 25502 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:54.620128 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.623346 18017 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:54.602065 25557 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.628204 25557 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:54.628309 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:24:54.630041 25672 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:54.634236 18023 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:54.601305 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:54.650367 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:54.651547 18095 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:54.634740 25427 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:54.653294 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45259us
I20260709 12:24:54.654431 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.655207 25427 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.643635 18016 raft_consensus.cc:2376] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20. Re-sending same reply.
I20260709 12:24:54.605988 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:54.589632 25474 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:54.666301 18015 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:54.678627 18024 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:54.643256 25383 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:54.685184 18016 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:54.721572 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127711us
W20260709 12:24:54.722275 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 307635us
I20260709 12:24:54.724625 25582 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.725523 25582 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:54.729626 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75303.2us
I20260709 12:24:54.733083 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:54.737663 25712 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:54.739254 25687 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:54.691469 25383 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:54.698880 25672 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:54.748698 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 109623us
I20260709 12:24:54.750093 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:54.753923 18093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.020s	sys 0.026s
W20260709 12:24:54.754348 18093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.020s	sys 0.026s
I20260709 12:24:54.741268 25413 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:54.743274 25712 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:54.746775 25383 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:54.735358 25582 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:54.743144 25675 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:54.749578 25427 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:54.767388 17920 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 6 to 12. New cstate: current_term: 12 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:54.792490 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 199662us
I20260709 12:24:54.794040 25427 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:54.802639 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.759251 25672 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:54.795503 25244 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.764644 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3177) took 1096 ms. Trace:
W20260709 12:24:54.815101 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 200535us
I20260709 12:24:54.816522 18778 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.816694 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:54.844376 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120698us
W20260709 12:24:54.853943 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 103998us
I20260709 12:24:54.843415 17920 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 7 to 13. New cstate: current_term: 13 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:54.842378 25602 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:54.797348 25675 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:54.891808 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:54.842028 18093 rpcz_store.cc:276] 0709 12:24:53.680482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.690362 (+  9880us) service_pool.cc:224] Handling call
0709 12:24:53.707665 (+ 17303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.714021 (+  6356us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:53.742809 (+ 28788us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.154006 (+411197us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b18001a2780. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:24:54.757230 (+603224us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.757330 (+   100us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:54.757407 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.757532 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.757754 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19844,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":110,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":119341696,"thread_start_us":600503,"threads_started":1,"wal-append.queue_time_us":1019995}
W20260709 12:24:54.892954 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74510.8us
I20260709 12:24:54.896407 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:54.842526 25709 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.156s
I20260709 12:24:54.876844 25645 raft_consensus.cc:2764] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:54.820225 25293 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:54.897540 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80990.2us
I20260709 12:24:54.842904 25244 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:54.811349 25427 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:55.032307 18091 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.511s	user 0.016s	sys 0.000s
W20260709 12:24:55.043361 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.040860 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.049638 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:54.897440 25602 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.052803 18091 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:55.032307 25413 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:55.062865 18169 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.065629 18169 raft_consensus.cc:614] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:55.041630 25715 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:55.048889 25474 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:55.090412 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 207423us
I20260709 12:24:54.978966 25712 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:55.097343 25702 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.220s	user 0.001s	sys 0.001s
I20260709 12:24:55.099885 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1065) took 1371 ms. Trace:
W20260709 12:24:55.101043 25704 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:24:55.104569 17913 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 296) took 26309 ms (client timeout 15000 ms). Trace:
I20260709 12:24:55.104516 25707 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:55.108100 25707 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:55.077795 25427 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:55.037145 25672 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:55.114508 25701 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:24:55.092871 25474 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:55.127504 25244 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.119997 25675 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.121784 25413 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:55.124028 25293 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:55.126390 25602 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.085458 25719 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.100375 18021 rpcz_store.cc:276] 0709 12:24:53.728296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.728465 (+   169us) service_pool.cc:224] Handling call
0709 12:24:53.742918 (+ 14453us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:54.531621 (+788703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.531657 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:54.550680 (+ 19023us) log.cc:844] Serialized 52 byte log entry
0709 12:24:54.747060 (+196380us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:54.749484 (+  2424us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.749795 (+   311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.749914 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.099554 (+349640us) raft_consensus.cc:1685] finished
0709 12:24:55.099585 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.099782 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11146,"prepare.queue_time_us":2186,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":123867,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":2725,"spinlock_wait_cycles":384000,"thread_start_us":947741,"threads_started":2,"wal-append.queue_time_us":216036}
I20260709 12:24:55.081619 25715 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:24:55.193218 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 437663us
W20260709 12:24:55.193946 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 297628us
W20260709 12:24:55.200294 25698 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.316s	user 0.001s	sys 0.000s
W20260709 12:24:55.198683 18138 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.623s (SENT)
I20260709 12:24:55.232898 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.095799 25645 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:55.142220 25707 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:24:55.236500 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.188102 25648 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.191377 25474 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.196519 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 383297us
I20260709 12:24:55.231626 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:55.234697 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 138144us
I20260709 12:24:55.209599 25413 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:55.241562 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.197356 25703 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.244344 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.244979 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.244875 25703 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.245198 18018 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:55.245855 18023 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:55.245955 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:55.248998 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50372.8us
W20260709 12:24:55.235152 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 337521us
I20260709 12:24:55.250814 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:55.105291 17913 rpcz_store.cc:269] 0709 12:24:28.795415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.852286 (+ 56871us) service_pool.cc:224] Handling call
0709 12:24:55.104292 (+26252006us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:50.475992 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:50.476832 (+   840us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:50.476888 (+    56us) write_op.cc:435] Acquired schema lock
0709 12:24:50.476914 (+    26us) tablet.cc:662] Decoding operations
0709 12:24:50.562833 (+ 85919us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:50.563006 (+   173us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:50.563041 (+    35us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:24:50.564546 (+  1505us) tablet.cc:701] Row locks acquired
0709 12:24:50.564600 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:24:50.564838 (+   238us) write_op.cc:270] Start()
0709 12:24:50.565037 (+   199us) write_op.cc:276] Timestamp: P: 1783599890564751 usec, L: 0
0709 12:24:50.565064 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:24:50.626819 (+ 61755us) log.cc:844] Serialized 6274 byte log entry
0709 12:24:51.091561 (+464742us) op_driver.cc:464] REPLICATION: finished
0709 12:24:51.399141 (+307580us) write_op.cc:301] APPLY: starting
0709 12:24:51.399270 (+   129us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:51.905212 (+505942us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:51.905370 (+   158us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:52.212485 (+307115us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:52.212847 (+   362us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 12:24:52.212897 (+    50us) write_op.cc:312] APPLY: finished
0709 12:24:52.213315 (+   418us) log.cc:844] Serialized 148 byte log entry
0709 12:24:52.312308 (+ 98993us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:52.355434 (+ 43126us) write_op.cc:454] Released schema lock
0709 12:24:52.378032 (+ 22598us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:52.426213 (+ 48181us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:52.426273 (+    60us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23182,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":302432,"apply.run_cpu_time_us":85355,"apply.run_wall_time_us":1008211,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71103,"num_ops":16,"prepare.queue_time_us":271600,"prepare.run_cpu_time_us":35436,"prepare.run_wall_time_us":288731,"raft.queue_time_us":184930,"raft.run_cpu_time_us":41924,"raft.run_wall_time_us":274124,"replication_time_us":526251,"spinlock_wait_cycles":1792,"thread_start_us":627109,"threads_started":5,"wal-append.queue_time_us":342517,"wal-append.run_cpu_time_us":25708,"wal-append.run_wall_time_us":2275191}]]}
I20260709 12:24:55.248317 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.254621 18016 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:55.255996 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:55.257947 25734 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.258800 25734 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:55.259491 25714 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:24:55.250591 18015 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:55.210979 25715 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:55.200762 25722 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:55.264339 18024 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:24:55.211649 25723 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.246471 25687 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:55.271453 25376 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:55.275137 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43660.5us
I20260709 12:24:55.306140 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:55.265529 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.275250 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.307415 18019 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:55.275395 25735 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:55.308957 25735 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:55.278769 25383 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.271842 25293 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.279688 25703 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.275391 18017 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 12:24:55.310559 18017 pending_rounds.cc:85] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 10
I20260709 12:24:55.310441 25293 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.283684 25613 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:55.285980 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.310750 25703 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:55.291620 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.311980 25703 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.312335 18093 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:55.312601 25703 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:55.246776 25474 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:55.304323 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 254831us
I20260709 12:24:55.315374 25293 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.310880 18017 pending_rounds.cc:107] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:55.318600 18017 raft_consensus.cc:2889] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:55.291077 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.320194 18091 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:55.321221 25383 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.322199 25705 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:55.323117 18095 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 18. Preceding OpId from leader: term: 22 index: 19. (index mismatch)
I20260709 12:24:55.322921 25705 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.324795 18094 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:55.326713 25721 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:55.328979 25727 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:55.266031 25712 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:55.292466 25392 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:55.331624 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.332595 18096 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:55.252329 25648 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:55.338290 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73083.8us
I20260709 12:24:55.337041 25413 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.211547 25719 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.342365 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:55.347446 18097 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:55.273624 25734 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:55.344600 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.337879 25645 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:55.309891 25735 consensus_queue.cc:1243] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 22.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:24:55.316439 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.353152 18022 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:24:55.343488 25687 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.360426 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.362471 25672 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.366257 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:55.375149 18092 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.366748 25645 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.350785 25376 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:55.382275 18017 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 8. (term mismatch)
I20260709 12:24:55.367973 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.368549 25736 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:55.383301 18017 pending_rounds.cc:85] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 7
I20260709 12:24:55.391430 18017 pending_rounds.cc:107] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:24:55.391782 18017 raft_consensus.cc:2889] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:55.392149 18017 pending_rounds.cc:107] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:24:55.392406 18017 raft_consensus.cc:2889] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:55.368738 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:55.370258 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64247.8us
I20260709 12:24:55.393229 18095 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:55.358486 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.367479 25648 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.389575 25502 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:55.393736 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:55.398983 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:55.395448 25502 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.404989 25502 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.407305 25712 raft_consensus.cc:2764] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:55.408700 25712 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:55.411746 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:55.417557 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:55.418792 25502 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:55.351958 25705 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.356328 25526 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:55.424070 18096 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
W20260709 12:24:55.424852 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108486us
I20260709 12:24:55.433706 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:55.422844 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51467.4us
I20260709 12:24:55.382539 25613 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:55.432536 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:55.382162 25383 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.438069 18022 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.440138 18092 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:24:55.442814 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82554.5us
I20260709 12:24:55.445855 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.447798 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.450068 18167 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:55.438823 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:55.443636 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 429028us
I20260709 12:24:55.423815 25602 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:55.454154 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 168292us
I20260709 12:24:55.452559 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:55.455174 18169 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:55.461540 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.462955 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.466396 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:55.467931 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.351514 25723 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.466111 18170 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:24:55.480011 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:55.442430 25735 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:55.355242 25722 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.476116 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.477399 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.474895 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:55.484972 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46247.5us
I20260709 12:24:55.494558 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:55.494983 25735 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:55.496011 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:55.496380 25427 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:55.487833 18809 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:55.452721 25502 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:55.489945 18815 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:55.492065 18173 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:55.490628 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:55.496217 25735 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.496882 25526 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:55.499780 18168 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:55.500469 25413 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.501255 18171 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:55.449795 25730 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.430039 25672 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.503875 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:55.507462 18169 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:55.554574 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:55.554924 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121308us
W20260709 12:24:55.527194 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.480471 25694 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:55.548987 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.557042 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50637.3us
I20260709 12:24:55.503780 25293 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:55.427107 25736 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.510604 25705 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:55.480799 25376 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.474404 25383 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.503360 25613 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:55.566151 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:55.585444 18092 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.578737 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:55.582520 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78830.5us
I20260709 12:24:55.546777 25427 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:55.587374 25427 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.589514 25741 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:55.561874 25293 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.594869 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.560885 25602 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:55.575978 25705 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.607684 18813 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:55.608249 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.719434 17989 leader_election.cc:400] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
I20260709 12:24:55.719970 17989 leader_election.cc:403] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:24:55.728717 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.598631 25293 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.731191 18815 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:24:55.738756 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141579us
I20260709 12:24:55.600142 25413 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:55.618119 25705 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.721698 25244 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.575264 25744 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.167s
I20260709 12:24:55.593053 25742 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:24:55.602728 25732 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:55.773343 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:55.796867 18096 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:24:55.800633 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 253009us
I20260709 12:24:55.801326 25695 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20.
W20260709 12:24:55.806344 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 219623us
I20260709 12:24:55.561461 25730 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.720875 25733 raft_consensus.cc:2764] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election decision vote started in defunct term 18: won
I20260709 12:24:55.611312 25602 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.807335 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:55.783047 18092 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:24:55.809546 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66396.3us
I20260709 12:24:55.809413 18815 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:55.810595 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.728708 25703 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.798506 25413 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:55.815814 25474 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.502s	user 0.013s	sys 0.000s
I20260709 12:24:55.834992 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.839179 18095 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:55.840648 25694 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.841374 25752 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:55.846937 25756 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:55.848239 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 288405us
I20260709 12:24:55.858852 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:55.850762 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.745554 25502 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.796229 25244 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.817414 25645 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.864697 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.870247 25672 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.862869 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.831516 25712 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:55.876331 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.879038 25602 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.880115 25756 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.863991 25752 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:55.862869 25694 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.885603 25431 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.888748 25431 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:55.890398 25694 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:55.892338 25431 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.897984 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:55.898393 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:55.899115 25756 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.900583 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:55.862071 25719 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.918560 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.917197 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:55.857733 25474 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:55.934739 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:55.946357 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45840.9us
I20260709 12:24:55.914541 25431 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:55.886827 25723 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:55.960389 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.974848 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:55.975670 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:55.977449 18022 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.002s
I20260709 12:24:55.978209 18022 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:55.984352 18019 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 12:24:55.931632 25719 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:55.934759 25742 raft_consensus.cc:2764] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 12:24:55.962095 25739 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.386s	user 0.002s	sys 0.000s
W20260709 12:24:55.963330 25747 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:24:55.927553 25722 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:55.931524 25687 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:56.004401 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 193899us
I20260709 12:24:56.014853 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:55.919206 25681 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 12:24:56.006682 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:56.009438 25749 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:55.952831 25431 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.973686 25383 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:56.012337 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67906.6us
I20260709 12:24:56.004974 25744 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.003372 25719 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.950284 25413 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.028491 25743 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:56.043433 25743 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.042609 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:56.038528 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:56.047537 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:56.038901 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.999s (SENT) [suppressed 33 similar messages]
I20260709 12:24:56.080932 18099 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:56.048435 25743 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:56.084748 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45917.1us
I20260709 12:24:56.048797 25703 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:56.089872 25293 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.092383 25293 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.094726 25474 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:56.090315 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75574.6us
I20260709 12:24:56.024969 25431 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:56.045507 25715 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.710s	user 0.011s	sys 0.003s
I20260709 12:24:56.041441 25721 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:56.029964 25742 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:56.046314 25719 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:56.048399 25754 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:56.082326 25613 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:56.144384 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46688.7us
I20260709 12:24:56.146271 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:56.148087 25707 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.148952 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:56.120553 25244 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:56.033794 25722 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.095s
I20260709 12:24:56.140245 25613 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:56.155251 25725 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.879s	user 0.009s	sys 0.006s
I20260709 12:24:56.155460 25613 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:56.156224 25755 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:56.157490 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3190) took 1518 ms. Trace:
W20260709 12:24:56.155473 25729 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.868s	user 0.006s	sys 0.018s
I20260709 12:24:56.135646 25754 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:56.164216 25754 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:56.166078 25700 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.166924 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.167344 25730 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.167870 18811 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:56.166831 25672 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.168447 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.126145 25721 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:56.149636 25707 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.192131 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:56.194343 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.044263 25752 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.168799 25672 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.245265 18172 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:56.157963 18090 rpcz_store.cc:276] 0709 12:24:54.639207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.658924 (+ 19717us) service_pool.cc:224] Handling call
0709 12:24:54.659466 (+   542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.660092 (+   626us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:54.798106 (+138014us) log.cc:844] Serialized 14 byte log entry
0709 12:24:55.233125 (+435019us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:55.274269 (+ 41144us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.274516 (+   247us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:55.274615 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.292962 (+ 18347us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:56.154381 (+861419us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:56.157118 (+  2737us) raft_consensus.cc:1685] finished
0709 12:24:56.157259 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.157417 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":182073,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":257,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":2280,"spinlock_wait_cycles":64880896,"thread_start_us":553995,"threads_started":2,"wal-append.queue_time_us":474944}
I20260709 12:24:56.248915 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:56.183493 25645 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260709 12:24:56.206694 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:56.210304 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61435.6us
I20260709 12:24:56.253042 18164 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:56.253521 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:56.210788 25759 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:24:56.211287 25711 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:56.212679 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:56.213182 25744 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:56.151477 25728 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 12:24:56.268257 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:56.133293 25742 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.223773 25721 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.19, Last appended by leader: 19, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.183699 25733 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:24:56.191179 25757 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:56.305807 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.250639 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.264658 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.269995 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:56.310708 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:56.271389 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.127231 25715 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.306453 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.312238 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:56.313928 18164 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:56.314009 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.317855 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.318528 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:56.319131 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:24:56.325309 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:56.312839 25705 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.004s
I20260709 12:24:56.330344 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.278074 25756 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:56.340651 18164 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:56.342097 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.343080 18164 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:56.308583 25722 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:56.350719 18138 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.627s (SENT)
I20260709 12:24:56.311286 25602 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.347672 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:56.362772 18170 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:56.345418 25756 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:56.352154 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:56.265149 25730 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20260709 12:24:56.305384 25744 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:56.361133 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 199072us
I20260709 12:24:56.321718 25749 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:56.360381 18813 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:56.305404 25648 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:24:56.307888 25707 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:24:56.380614 18164 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260709 12:24:56.372668 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.344166 25733 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:56.343990 25721 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:56.345010 25757 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:56.363067 25722 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:56.368708 25602 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.419718 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3964) took 1062 ms. Trace:
I20260709 12:24:56.405745 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.305022 25711 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.375528 25756 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:56.451277 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:56.450297 25608 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:24:56.457126 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60408.1us
I20260709 12:24:56.350057 25762 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:56.460855 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:56.523187 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62458.7us
I20260709 12:24:56.524583 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:56.461915 25427 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:56.462885 25700 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.404520 25749 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:56.463322 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.464241 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.464140 25431 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.400475 25733 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:56.472049 25726 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.190s	user 0.013s	sys 0.011s
W20260709 12:24:56.481516 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 130848us
I20260709 12:24:56.354671 25705 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.484126 18019 rpcz_store.cc:276] 0709 12:24:55.359000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.366008 (+  7008us) service_pool.cc:224] Handling call
0709 12:24:55.366426 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.994073 (+627647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.419184 (+425111us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54002ffb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:56.419595 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1406388096}
W20260709 12:24:56.512332 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54545.1us
I20260709 12:24:56.467147 25742 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:56.526952 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:56.510891 25608 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:56.531275 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:56.542898 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.453760 25744 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.491027 25722 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:56.553370 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:56.568439 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:56.532243 25413 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.528752 25762 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.543869 25733 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:56.594620 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 63930.2us
I20260709 12:24:56.529857 25502 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:56.595815 18164 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:56.568527 25695 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:56.561492 25602 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:56.609400 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:24:56.598372 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 328483us
I20260709 12:24:56.622077 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:56.538826 25392 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:24:56.628567 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:56.629544 25613 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.629946 25778 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:56.630722 25778 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:56.631780 25778 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.610227 25781 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:56.606878 25768 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:24:56.635032 25672 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:56.599766 25775 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:24:56.629307 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.642448 25722 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.643963 25427 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:56.644465 25427 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:56.644835 25427 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:56.646024 25722 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.649596 25754 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.650358 18169 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:56.652822 25754 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:56.655303 25705 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.660782 25736 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:56.661913 25711 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:56.662437 25771 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
W20260709 12:24:56.667495 25779 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:24:56.602082 25431 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.634181 25749 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:56.676067 25743 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.592s	user 0.013s	sys 0.001s
I20260709 12:24:56.556926 25767 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:24:56.678260 25757 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.688963 25711 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.689985 25711 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.692221 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.693485 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:56.694036 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 140788us
I20260709 12:24:56.676810 25743 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:56.607858 25762 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:56.700306 18169 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.19-21.19]
I20260709 12:24:56.715478 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:56.716506 18810 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:24:56.718137 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.718336 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.718551 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.718844 18022 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:56.718997 18020 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:56.723181 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.716419 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.717653 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.725802 18024 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:24:56.726502 18021 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:24:56.726863 18021 pending_rounds.cc:85] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 12
I20260709 12:24:56.727128 18021 pending_rounds.cc:107] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:24:56.727376 18021 raft_consensus.cc:2889] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:56.727136 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:56.719149 18165 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:56.728402 18017 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:56.728605 18016 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:56.730093 18019 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:56.732946 18020 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:56.735249 18170 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:56.736441 18168 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:56.740366 25761 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.740908 25474 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:56.741180 25761 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.694449 25744 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:56.745044 25761 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:56.735224 25695 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:56.748051 18164 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:24:56.765797 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.781371 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:56.681411 25602 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20260709 12:24:56.784826 18165 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:56.783983 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 242460us
I20260709 12:24:56.800683 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:56.801542 18094 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:56.802074 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.802994 18019 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:56.804364 25711 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:56.732066 25735 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 12:24:56.806223 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.809235 25695 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.810585 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:56.810417 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:56.812072 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:56.815526 18099 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:56.816824 25695 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:56.778461 25722 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:56.789702 25770 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:24:56.750723 25613 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:56.791558 25719 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.640s	user 0.014s	sys 0.000s
I20260709 12:24:56.940984 18099 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:56.959645 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:56.973811 18022 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:56.981220 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.982605 18022 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:56.943094 25431 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:56.996593 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 280110us
I20260709 12:24:56.943668 25789 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.001693 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:57.017386 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.018263 18164 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:24:57.018467 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.019433 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.023175 18170 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:57.024508 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.025647 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.025967 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.028585 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1129) took 1561 ms. Trace:
W20260709 12:24:57.031328 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 250097us
I20260709 12:24:57.028000 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.036412 18022 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 12:24:57.036897 18022 pending_rounds.cc:85] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 12
I20260709 12:24:56.970376 25755 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:57.043334 18091 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:24:57.044317 18091 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:57.044754 18091 raft_consensus.cc:614] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:57.046187 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:57.029708 18166 rpcz_store.cc:276] 0709 12:24:55.466946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.556531 (+ 89585us) service_pool.cc:224] Handling call
0709 12:24:55.574272 (+ 17741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.574546 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:55.593496 (+ 18950us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.663236 (+1069740us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b18003f15a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:24:57.027425 (+364189us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.027624 (+   199us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:57.027773 (+   149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.028027 (+   254us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.028383 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":170759,"prepare.queue_time_us":18651,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":57,"raft.queue_time_us":157718,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":22408,"spinlock_wait_cycles":6750720,"thread_start_us":208900,"threads_started":2,"wal-append.queue_time_us":1390608}
I20260709 12:24:56.979451 25711 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:57.037150 18022 pending_rounds.cc:107] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 12:24:57.050426 18022 raft_consensus.cc:2889] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:57.048207 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.944343 25722 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.052369 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.991766 25754 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:56.977020 25756 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:56.988857 25645 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:57.056185 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:57.037514 25789 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.072623 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:56.970968 25719 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.071394 25687 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:57.035642 25675 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:57.076972 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:56.978157 25613 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.073457 25754 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.101468 18164 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
W20260709 12:24:57.076256 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 135445us
I20260709 12:24:57.091789 18024 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:24:57.094547 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:57.095916 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:57.058171 25784 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:24:57.115397 18165 raft_consensus.cc:2368] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19.
I20260709 12:24:57.098501 25788 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.113126 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:57.115818 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.020490 25431 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.099617 25755 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.123580 25431 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:57.123921 25755 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:57.046841 25648 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:24:57.124639 25755 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:57.066478 25602 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:57.126128 25602 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:57.126636 25602 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:57.127204 25756 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:57.079206 25645 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.079682 25687 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:57.130340 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.130767 18024 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:57.131541 25687 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.132584 25794 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:57.132859 25687 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:57.134671 18165 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:24:57.133205 25794 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.087582 25715 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:57.099866 25736 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:57.167016 18016 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:57.168468 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.115765 25778 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:57.173413 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.174470 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:57.130405 25645 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:57.178215 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.181181 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.186805 18016 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:24:57.198923 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.117475 25700 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:24:57.168259 25794 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:57.251379 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:57.254925 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 204243us
I20260709 12:24:57.181035 25766 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:57.262637 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.182156 25743 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:57.257135 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:57.274168 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:57.280136 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:57.281912 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:57.298292 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.345883 18023 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:57.148780 25788 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.255280 25795 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:24:57.363292 25733 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.736s	user 0.015s	sys 0.001s
I20260709 12:24:57.428803 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:57.346536 25803 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:57.435109 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.436328 18016 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:57.430198 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:57.441860 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:57.440117 18810 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
W20260709 12:24:57.495409 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53764.1us
W20260709 12:24:57.519547 25797 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:24:57.521116 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:57.526595 25792 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.446s	user 0.000s	sys 0.001s
I20260709 12:24:57.530262 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.531442 18099 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:57.532837 25722 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:57.534332 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:57.431232 25786 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.450s	user 0.000s	sys 0.002s
I20260709 12:24:57.540905 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.542408 18098 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:57.513718 25613 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:57.372443 25787 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:57.545050 25787 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.547688 25787 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:57.548368 25787 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:57.499260 25795 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.550253 18024 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:57.448112 25803 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:57.552743 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.409417 25741 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:57.554179 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.555554 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:57.556334 25788 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:57.494804 25734 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.557715 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3055) took 1285 ms. Trace:
I20260709 12:24:57.559581 25719 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:57.493862 25733 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.564839 18096 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:57.566397 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:57.494029 25794 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.567444 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.568917 18099 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
W20260709 12:24:57.582298 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.584411 18810 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:24:57.586238 18016 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:57.588055 18171 rpcz_store.cc:276] 0709 12:24:56.272269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.326778 (+ 54509us) service_pool.cc:224] Handling call
0709 12:24:56.349056 (+ 22278us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:56.353872 (+  4816us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.353924 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:56.365182 (+ 11258us) log.cc:844] Serialized 52 byte log entry
0709 12:24:56.608615 (+243433us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:56.627542 (+ 18927us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.707712 (+ 80170us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.746725 (+ 39013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.746816 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:57.557420 (+810604us) raft_consensus.cc:1685] finished
0709 12:24:57.557446 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.557638 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":257888,"prepare.queue_time_us":50338,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":114,"raft.queue_time_us":114315,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":46744,"spinlock_wait_cycles":8461696,"thread_start_us":228841,"threads_started":1,"wal-append.queue_time_us":352925}
I20260709 12:24:57.597352 18098 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:24:57.601231 18098 pending_rounds.cc:85] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 8
I20260709 12:24:57.601577 18098 pending_rounds.cc:107] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:24:57.601963 18098 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:57.598922 18099 pending_rounds.cc:85] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 10
I20260709 12:24:57.605063 18099 pending_rounds.cc:107] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:57.605504 18099 raft_consensus.cc:2889] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:57.593745 25413 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.990s	user 0.016s	sys 0.012s
I20260709 12:24:57.621048 25413 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:57.622344 25413 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.622587 18022 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:57.622706 25795 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:57.622924 25413 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:57.624372 25427 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.979s	user 0.009s	sys 0.005s
I20260709 12:24:57.624969 25427 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.627305 18016 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:57.613677 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.631044 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.632571 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.635555 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.638945 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.583428 25803 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:57.616222 25799 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
W20260709 12:24:57.649315 18173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.043s	sys 0.031s
W20260709 12:24:57.650365 18173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.044s	sys 0.031s
I20260709 12:24:57.657985 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:24:57.660281 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.017s	sys 0.022s
W20260709 12:24:57.661402 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.017s	sys 0.022s
W20260709 12:24:57.664398 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 555597us
I20260709 12:24:57.666376 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:57.600939 25741 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:57.612625 25754 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:57.653126 25762 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.916s	user 0.021s	sys 0.007s
I20260709 12:24:57.646188 25413 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:57.604192 25794 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.688392 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:57.595427 25734 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.687062 18017 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:57.711334 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:57.714632 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 217174us
I20260709 12:24:57.714957 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:57.685070 25602 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.015s
I20260709 12:24:57.716064 18171 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:24:57.684144 25700 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:57.728945 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.008s	sys 0.012s
I20260709 12:24:57.685256 25766 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:57.735060 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:57.745965 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.010s	sys 0.012s
I20260709 12:24:57.755826 18096 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:57.757309 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:57.757941 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:57.747475 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:57.748770 18170 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:24:57.770031 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.771332 18099 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:57.772770 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.687040 25762 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.741020 25645 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:57.717057 25754 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:57.780851 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.780755 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.742321 25700 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.783942 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.785919 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:57.783519 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.787014 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
W20260709 12:24:57.779933 25687 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.646s	user 0.011s	sys 0.017s
I20260709 12:24:57.817620 18015 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:24:57.817883 18169 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:57.826769 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.804955 25700 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:57.828249 18095 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:24:57.801028 25754 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:57.742419 25602 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:57.840319 25817 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:57.840397 25809 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.198s	user 0.000s	sys 0.002s
I20260709 12:24:57.845041 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3977) took 1113 ms. Trace:
I20260709 12:24:57.845508 18020 rpcz_store.cc:276] 0709 12:24:56.737612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.737809 (+   197us) service_pool.cc:224] Handling call
0709 12:24:56.738404 (+   595us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.738912 (+   508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.738951 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:56.739470 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.396007 (+656537us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:57.396055 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.396140 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:57.844744 (+448604us) raft_consensus.cc:1685] finished
0709 12:24:57.844771 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.844958 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":203739,"spinlock_wait_cycles":29025664,"thread_start_us":455862,"threads_started":1,"wal-append.queue_time_us":844885}
I20260709 12:24:57.840855 25813 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.847941 25813 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.850944 25703 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:57.853142 25703 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:57.854789 25719 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:57.855347 25719 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.855746 25719 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:57.858538 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3047) took 1656 ms. Trace:
I20260709 12:24:57.859151 18168 rpcz_store.cc:276] 0709 12:24:56.246146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.247549 (+  1403us) service_pool.cc:224] Handling call
0709 12:24:57.858189 (+1610640us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54000320d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:57.858448 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":977572224}
I20260709 12:24:57.861315 18168 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:24:57.826740 25810 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:24:57.867630 18171 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:24:57.853605 25703 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:57.846462 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 2999) took 2400 ms (client timeout 1962 ms). Trace:
I20260709 12:24:57.817010 25766 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:57.874330 18172 rpcz_store.cc:269] 0709 12:24:55.446266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.447430 (+  1164us) service_pool.cc:224] Handling call
0709 12:24:57.846160 (+2398730us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:57.846321 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1831490816}
I20260709 12:24:57.876685 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.880927 18168 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:57.827514 25687 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:57.812538 25413 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:24:57.885363 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:57.767458 25741 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.890398 18172 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:57.873710 25711 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.815s	user 0.019s	sys 0.002s
I20260709 12:24:57.895916 18099 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:24:57.842000 25754 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:57.884975 25811 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:24:57.900725 18095 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:57.903867 18099 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:57.907650 25813 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:57.888696 25757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.007s	sys 0.027s
I20260709 12:24:57.918246 18172 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:24:57.918169 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:24:57.922852 25815 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:24:57.839823 25795 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:24:57.882628 25703 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:57.917459 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:57.944856 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:57.950580 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.956550 18095 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:57.927109 25711 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:57.904498 25687 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:24:57.962049 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134351us
I20260709 12:24:57.962297 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:24:57.950407 25757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.007s	sys 0.028s
I20260709 12:24:57.966205 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:57.973115 18095 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:57.885454 25766 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:57.987403 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:24:57.971457 25781 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:58.002815 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 244984us
I20260709 12:24:58.003386 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:58.009187 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 261829us
I20260709 12:24:58.009459 18024 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:57.908939 25608 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.034586 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.037237 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:58.037545 18016 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:57.998104 25794 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.040338 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:58.003157 25781 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.999120 25819 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:58.008229 25557 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:58.051702 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:58.075631 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1128) took 2583 ms. Trace:
I20260709 12:24:58.014467 25734 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.073405 25819 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:58.078089 18173 rpcz_store.cc:276] 0709 12:24:55.466657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.546964 (+ 80307us) service_pool.cc:224] Handling call
0709 12:24:56.790108 (+1243144us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:24:56.790390 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.944594 (+154204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.944637 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:56.955739 (+ 11102us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.650983 (+695244us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:57.651065 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.651236 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.031905 (+380669us) raft_consensus.cc:1685] finished
0709 12:24:58.031933 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.050013 (+ 18080us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":185467,"raft.queue_time_us":153754,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":14516,"spinlock_wait_cycles":29752704,"thread_start_us":637188,"threads_started":1,"wal-append.queue_time_us":782559}
I20260709 12:24:58.041709 25820 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:58.048249 25795 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:58.049477 25762 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.074911 25781 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:58.077174 25794 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:58.014597 25376 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:24:58.098584 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:58.082388 25557 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.154243 25557 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:58.158602 25431 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:58.160161 25821 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:24:58.124593 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1137) took 1091 ms. Trace:
I20260709 12:24:58.160847 18170 rpcz_store.cc:276] 0709 12:24:57.033579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.830956 (+797377us) service_pool.cc:224] Handling call
0709 12:24:57.848231 (+ 17275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.848707 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.848740 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.849324 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.954198 (+104874us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:57.954244 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.954322 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.124247 (+169925us) raft_consensus.cc:1685] finished
0709 12:24:58.124278 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.124500 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":875,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":32640,"thread_start_us":104337,"threads_started":1,"wal-append.queue_time_us":271917}
I20260709 12:24:58.164261 18170 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:58.126468 25827 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:24:58.127566 25819 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.131067 25602 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.099462 25795 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.087457 25687 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.098563 25734 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:24:58.142549 25721 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.685s	user 0.013s	sys 0.006s
I20260709 12:24:58.159585 25431 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.117259 18016 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
W20260709 12:24:58.176127 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77618us
I20260709 12:24:58.184266 25829 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:58.185774 25766 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:58.193940 25785 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.591s	user 0.005s	sys 0.000s
I20260709 12:24:58.192931 25719 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.208513 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:58.210472 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 164522us
I20260709 12:24:58.213156 18809 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 22 index: 19. (term mismatch)
I20260709 12:24:58.219894 18809 pending_rounds.cc:85] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 18
I20260709 12:24:58.220206 18809 pending_rounds.cc:107] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 21.19
I20260709 12:24:58.220467 18809 raft_consensus.cc:2889] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:58.199900 25826 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
I20260709 12:24:58.222064 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:58.223058 18095 raft_consensus.cc:2376] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19. Re-sending same reply.
I20260709 12:24:58.211042 25794 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:58.217718 25819 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:58.255772 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.272964 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.225236 25842 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:58.211197 25830 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
W20260709 12:24:58.282568 25836 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:24:58.283948 25719 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.288275 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3067) took 1148 ms. Trace:
I20260709 12:24:58.290432 25427 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.292189 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1081) took 1212 ms. Trace:
I20260709 12:24:58.292933 25431 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.294518 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.298375 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:58.279129 25769 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.788s	user 0.006s	sys 0.000s
I20260709 12:24:58.299122 25795 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.299471 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:24:58.294617 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3012) took 1270 ms. Trace:
W20260709 12:24:58.302533 25803 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.004s
I20260709 12:24:58.303174 25803 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.305732 25803 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.306358 25413 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:58.306766 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1368) took 2567 ms. Trace:
I20260709 12:24:58.307253 18813 rpcz_store.cc:276] 0709 12:24:55.739899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.773384 (+ 33485us) service_pool.cc:224] Handling call
0709 12:24:55.773765 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.380441 (+606676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.306441 (+1926000us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54007823d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:58.306652 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333300480}
I20260709 12:24:58.308305 25413 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:58.308969 25413 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.300047 25848 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:58.269279 25754 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.314219 18094 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:58.288837 18172 rpcz_store.cc:276] 0709 12:24:57.140244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.951817 (+811573us) service_pool.cc:224] Handling call
0709 12:24:57.952366 (+   549us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:57.959070 (+  6704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.959127 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:57.960051 (+   924us) log.cc:844] Serialized 52 byte log entry
0709 12:24:58.083400 (+123349us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:58.102338 (+ 18938us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.102757 (+   419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.102881 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.288040 (+185159us) raft_consensus.cc:1685] finished
0709 12:24:58.288063 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.288195 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5978,"prepare.queue_time_us":20419,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":77,"raft.queue_time_us":10696,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":26668,"thread_start_us":80458,"threads_started":1,"wal-append.queue_time_us":238258}
I20260709 12:24:58.301234 18167 rpcz_store.cc:276] 0709 12:24:57.024202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.075281 (+ 51079us) service_pool.cc:224] Handling call
0709 12:24:57.114934 (+ 39653us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.592407 (+477473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.592442 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.606010 (+ 13568us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.762489 (+156479us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:57.767345 (+  4856us) log.cc:844] Serialized 14 byte log entry
0709 12:24:57.767702 (+   357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.767829 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.294361 (+526532us) raft_consensus.cc:1685] finished
0709 12:24:58.294390 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.294542 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21793,"prepare.queue_time_us":10515,"raft.queue_time_us":59990,"raft.run_cpu_time_us":609,"raft.run_wall_time_us":44157,"spinlock_wait_cycles":12479360,"thread_start_us":579126,"threads_started":2,"wal-append.queue_time_us":315561}
I20260709 12:24:58.306656 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.292667 18022 rpcz_store.cc:276] 0709 12:24:57.080425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.080704 (+   279us) service_pool.cc:224] Handling call
0709 12:24:57.081130 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.623152 (+542022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.291828 (+668676us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54001cf554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:58.292107 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1244106112}
W20260709 12:24:58.280444 25801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.150s	user 0.011s	sys 0.014s
I20260709 12:24:58.345507 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.350122 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:58.353904 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 133373us
I20260709 12:24:58.308332 25781 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.356760 25842 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:58.357125 25781 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:58.344980 25801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.214s	user 0.011s	sys 0.014s
I20260709 12:24:58.362794 18168 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:58.364926 25413 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.362789 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:58.275727 25700 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.374825 25700 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.375850 25829 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:58.374877 18020 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:24:58.382361 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:58.369736 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.371624 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:58.384913 25829 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:58.385695 18097 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:24:58.415797 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:58.426271 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.427284 18097 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:24:58.429208 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:24:58.435458 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 120800us
I20260709 12:24:58.435217 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:58.447700 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.448940 18024 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:58.434084 25700 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:58.452003 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:58.454475 18138 connection.cc:256] RPC call timeout handler was delayed by -1.7646853330002159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
I20260709 12:24:58.455355 18808 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:58.456199 18815 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:58.456964 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.456221 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.458490 18020 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.438585 25801 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:58.474347 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.533504 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0613580680001178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:24:58.466308 25754 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.535301 25766 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.535151 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.536062 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:58.454804 25700 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:58.545120 25392 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.548178 25780 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:58.435101 25829 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.459873 25837 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:58.551673 25798 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.141s	user 0.001s	sys 0.011s
I20260709 12:24:58.553799 25787 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:58.572703 18024 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:24:58.582399 25700 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.460057 25737 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.460299 25645 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:58.595364 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:58.604842 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.608500 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:24:58.605212 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 136846us
I20260709 12:24:58.575947 18020 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:58.609844 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.611058 18024 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:24:58.601375 25703 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.005s
I20260709 12:24:58.557528 25801 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.597462 25837 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:58.612936 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:58.615717 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.593307 25293 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 12:24:58.620810 18024 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.584024 25766 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.624413 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:58.589854 25392 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.597340 25557 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.638666 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.616163 25741 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:58.604805 25787 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.619458 25837 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:58.618988 25695 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:58.677348 18173 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:24:58.672559 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4118) took 1871 ms. Trace:
I20260709 12:24:58.678172 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.615586 25645 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.678676 18090 rpcz_store.cc:276] 0709 12:24:56.800663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.802512 (+  1849us) service_pool.cc:224] Handling call
0709 12:24:56.802856 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.823100 (+ 20244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.823154 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:56.823957 (+   803us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.038347 (+214390us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:57.038419 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.038551 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.395403 (+1356852us) spinlock_profiling.cc:243] Waited 91 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:58.668215 (+272812us) raft_consensus.cc:1685] finished
0709 12:24:58.668266 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.672420 (+  4154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19670,"raft.queue_time_us":21659,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"spinlock_wait_cycles":209411456,"thread_start_us":213655,"threads_started":1,"wal-append.queue_time_us":353448}
I20260709 12:24:58.625991 25794 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:58.679142 18023 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:58.697888 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:58.668725 25741 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.616597 25703 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:24:58.634562 25850 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260709 12:24:58.675597 18778 connection.cc:256] RPC call timeout handler was delayed by -1.6461193929999354s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27599999999999997s for a response
I20260709 12:24:58.649152 18024 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:58.623634 25767 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:58.636556 25801 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.613591 25737 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.715807 25846 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:58.628679 25730 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
W20260709 12:24:58.719187 25856 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:24:58.680246 25852 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.728029 18024 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:58.735194 18024 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:58.736660 18022 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:58.738677 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.724126 25721 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:58.722806 25796 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.361s	user 0.004s	sys 0.000s
I20260709 12:24:58.710305 25741 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:58.720892 25557 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.717700 25737 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.719363 25824 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.748154 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.766891 18020 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:24:58.748492 18778 connection.cc:256] RPC call timeout handler was delayed by -1.2416042169998036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
I20260709 12:24:58.767731 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.768862 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:24:58.697495 25695 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.741343 25842 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:24:58.680754 25376 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.742520 25852 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.779942 25376 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.782403 25842 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.712992 25794 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.749365 25721 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:58.755065 25829 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:58.823699 25856 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 12:24:58.848140 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150746us
I20260709 12:24:58.816869 25842 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.770888 25855 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:24:58.855751 25376 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.788568 25824 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.863180 25855 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:24:58.865170 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 240849us
I20260709 12:24:58.845939 25849 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.871006 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:24:58.875939 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 112004us
W20260709 12:24:58.874405 25861 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:24:58.886750 25837 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.888137 18092 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:58.888172 18095 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:58.891004 25794 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.891858 25794 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.892462 25794 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:58.894573 25859 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
W20260709 12:24:58.895536 25814 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.112s	user 0.001s	sys 0.001s
I20260709 12:24:58.900034 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3015) took 1874 ms. Trace:
I20260709 12:24:58.900717 18171 rpcz_store.cc:276] 0709 12:24:57.033406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.732818 (+699412us) service_pool.cc:224] Handling call
0709 12:24:57.866338 (+133520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.877804 (+ 11466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.899695 (+1021891us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54000320d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:58.899926 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240222080}
I20260709 12:24:58.906728 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3014) took 1881 ms. Trace:
I20260709 12:24:58.907549 18165 rpcz_store.cc:276] 0709 12:24:57.024913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.135802 (+110889us) service_pool.cc:224] Handling call
0709 12:24:57.136214 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.234283 (+ 98069us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.234317 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.275869 (+ 41552us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.603358 (+327489us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:57.603404 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.603477 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.894763 (+1291286us) spinlock_profiling.cc:243] Waited 6.14 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:58.906131 (+ 11368us) raft_consensus.cc:1685] finished
0709 12:24:58.906170 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.906604 (+   434us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33017,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":9266,"spinlock_wait_cycles":14111616,"thread_start_us":422468,"threads_started":2,"wal-append.queue_time_us":440425}
W20260709 12:24:58.915818 25862 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.086s	user 0.001s	sys 0.001s
W20260709 12:24:58.918035 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3009) took 1894 ms (client timeout 2315 ms). Trace:
W20260709 12:24:58.918603 18169 rpcz_store.cc:269] 0709 12:24:57.023167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.027589 (+  4422us) service_pool.cc:224] Handling call
0709 12:24:58.917748 (+1890159us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:58.917953 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1786447744}
I20260709 12:24:58.920007 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1201) took 1103 ms. Trace:
I20260709 12:24:58.920737 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3070) took 1779 ms. Trace:
I20260709 12:24:58.921331 18173 rpcz_store.cc:276] 0709 12:24:57.141438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.091835 (+950397us) service_pool.cc:224] Handling call
0709 12:24:58.106779 (+ 14944us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.678054 (+571275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.920345 (+242291us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b5400030a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:58.920648 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1303604096}
I20260709 12:24:58.925895 25801 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:58.926494 25801 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:58.926757 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1373) took 2783 ms. Trace:
I20260709 12:24:58.929529 25793 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:58.930282 25793 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:58.931579 25645 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.932540 25707 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:58.932773 25703 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.932953 25787 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.933106 25707 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:58.933619 25707 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:58.933655 25787 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:58.934367 25787 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.936645 25787 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:58.931452 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60685us
I20260709 12:24:58.940054 25766 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.827816 25829 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.917761 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3068) took 1777 ms. Trace:
I20260709 12:24:58.942116 25392 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:58.942288 25819 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.693s	user 0.008s	sys 0.008s
I20260709 12:24:58.870401 25721 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.874878 25842 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:58.915861 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1085) took 1788 ms. Trace:
W20260709 12:24:58.950924 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.953291 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:58.927289 18809 rpcz_store.cc:276] 0709 12:24:56.143086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.152935 (+  9849us) service_pool.cc:224] Handling call
0709 12:24:56.153313 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.226352 (+2073039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.926360 (+700008us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:58.926660 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4713786112}
I20260709 12:24:58.920621 18099 rpcz_store.cc:276] 0709 12:24:57.816722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.937969 (+121247us) service_pool.cc:224] Handling call
0709 12:24:57.938454 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.938867 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.938900 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.953717 (+ 14817us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.556571 (+602854us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b1800295840. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:24:58.607738 (+ 51167us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:58.607790 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.608296 (+   506us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.914834 (+306538us) spinlock_profiling.cc:243] Waited 6.05 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:58.919628 (+  4794us) raft_consensus.cc:1685] finished
0709 12:24:58.919677 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.919911 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14918,"raft.queue_time_us":44188,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":5954,"spinlock_wait_cycles":18595840,"thread_start_us":50913,"threads_started":1,"wal-append.queue_time_us":772571}
I20260709 12:24:58.951815 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.948047 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:58.948320 18168 rpcz_store.cc:276] 0709 12:24:57.140778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.960729 (+819951us) service_pool.cc:224] Handling call
0709 12:24:57.972488 (+ 11759us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.363522 (+391034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.917352 (+553830us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b540029e0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:58.917661 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":893070080}
I20260709 12:24:58.951143 25842 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:58.963317 25846 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:24:58.964146 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.964051 25846 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.964830 18092 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:24:58.964531 25842 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.931367 25793 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.948140 25849 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:58.968909 25849 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:24:58.906876 25858 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:24:58.943993 25789 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.947045 25829 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:58.970853 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.971786 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:24:58.974103 25766 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:58.974645 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:58.975548 18097 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 14.
I20260709 12:24:58.978041 25842 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:58.952127 18015 rpcz_store.cc:276] 0709 12:24:57.127345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.129981 (+  2636us) service_pool.cc:224] Handling call
0709 12:24:58.915451 (+1785470us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b54001d2254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:58.915734 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1577749760}
I20260709 12:24:58.983439 25695 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.993927 25733 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:58.927817 25801 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:58.999343 25801 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.003804 25794 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:59.008095 25813 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:59.011018 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57831.3us
W20260709 12:24:59.011932 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.013180 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:59.013545 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.015566 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:58.957265 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.018814 25766 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:59.018450 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61316.3us
I20260709 12:24:59.021848 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.023984 25846 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:59.024205 25837 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:58.977612 25824 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:59.019400 25687 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:59.027530 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1192) took 1256 ms. Trace:
I20260709 12:24:59.028345 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:59.029309 18016 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:59.029484 18093 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4113) took 2853 ms (client timeout 2154 ms). Trace:
W20260709 12:24:59.030246 18093 rpcz_store.cc:269] 0709 12:24:56.175947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.212379 (+ 36432us) service_pool.cc:224] Handling call
0709 12:24:59.029044 (+2816665us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:59.029337 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3952547200}
I20260709 12:24:59.028149 18094 rpcz_store.cc:276] 0709 12:24:57.770802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.801464 (+ 30662us) service_pool.cc:224] Handling call
0709 12:24:57.822527 (+ 21063us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.314804 (+492277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.027120 (+712316us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:59.027432 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130171776}
W20260709 12:24:59.040855 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 3981) took 2238 ms (client timeout 2218 ms). Trace:
I20260709 12:24:59.041014 18019 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:58.977617 25721 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:59.041790 18017 rpcz_store.cc:269] 0709 12:24:56.802079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.805777 (+  3698us) service_pool.cc:224] Handling call
0709 12:24:59.040379 (+2234602us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:24:59.040684 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2017893120}
I20260709 12:24:59.042309 25766 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:59.055256 25766 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.057641 18021 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:24:59.061149 25852 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:59.062023 25723 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:24:59.067808 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46538.7us
I20260709 12:24:59.069322 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.064853 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.072369 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:59.073138 18093 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:59.073370 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 286553us
I20260709 12:24:59.074275 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:59.068675 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:59.080690 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:59.082517 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:59.084995 18015 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:59.089462 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:59.043479 25791 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 2.005s	user 0.004s	sys 0.002s
I20260709 12:24:59.091477 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.103583 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:59.105202 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:59.120862 18016 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:24:59.205960 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99896.2us
I20260709 12:24:59.081730 25846 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:59.125640 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.088447 25715 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:59.212400 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86933.1us
I20260709 12:24:59.086031 25852 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:59.210016 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104911us
I20260709 12:24:59.214380 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.210116 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.216606 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:59.218575 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 137979us
I20260709 12:24:59.220232 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.221837 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:59.212054 25703 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:59.206267 25824 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:59.240412 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:59.089413 25700 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.243474 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.216390 25724 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:59.245805 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.215312 25852 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:59.222952 25413 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:59.257988 18095 raft_consensus.cc:2368] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:59.234411 25474 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:59.259416 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.233062 25789 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:59.266434 25557 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:59.271816 25557 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.272390 25557 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.262147 25376 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:59.274565 25376 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:24:59.276173 25376 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.270495 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:59.278411 18810 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:59.238847 25849 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:59.241914 25502 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:59.298832 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:59.299860 18095 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:24:59.215845 25794 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:24:59.295390 25789 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:59.312645 25789 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.314141 18097 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:59.291076 25687 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.252480 25824 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:59.251973 25846 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:59.318727 25703 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:59.315450 25675 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:59.316952 25865 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:59.297719 25502 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.261219 25852 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:59.334339 25431 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:59.244606 25787 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:24:59.261468 25413 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:59.293112 25474 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:24:59.330523 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:24:59.405942 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.047s	sys 0.090s
I20260709 12:24:59.374501 25502 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:59.375967 25700 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:59.338289 25703 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.322283 25742 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:24:59.440294 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 109879us
I20260709 12:24:59.404448 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:59.409545 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.039s	sys 0.079s
I20260709 12:24:59.441278 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:24:59.441663 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.040s	sys 0.079s
I20260709 12:24:59.331214 25824 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:59.367415 25813 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:24:59.412662 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.047s	sys 0.090s
W20260709 12:24:59.451262 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.033s	sys 0.058s
I20260709 12:24:59.331007 25687 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.418406 25474 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:59.449896 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:59.462875 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.464039 18024 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:24:59.336122 25866 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:59.440536 25703 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.455332 25846 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:59.331139 25867 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.087s
I20260709 12:24:59.455478 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.412081 25413 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:59.470599 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.459467 25293 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.451814 25813 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:59.439245 25557 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:24:59.508615 18091 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.009s	sys 0.013s
I20260709 12:24:59.481341 25787 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:24:59.509922 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 295662us
W20260709 12:24:59.453073 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.034s	sys 0.058s
I20260709 12:24:59.452666 25742 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.496119 25846 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.511081 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:24:59.509346 18091 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.009s	sys 0.013s
I20260709 12:24:59.502158 25819 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:59.529520 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.532367 25742 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:59.538405 25741 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.772s	user 0.013s	sys 0.003s
I20260709 12:24:59.509938 25787 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:24:59.541308 18093 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:24:59.541057 25703 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.541993 18810 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:24:59.542434 25707 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.608s	user 0.014s	sys 0.000s
I20260709 12:24:59.558380 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:59.560858 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.577752 18020 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:24:59.485075 25866 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.578868 18806 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:59.522378 25846 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.577666 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:59.579658 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:59.562155 18815 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:24:59.560112 25721 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:59.574788 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50087.5us
I20260709 12:24:59.585742 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:59.587620 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:24:59.589457 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.594030 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:59.596194 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:59.596974 18021 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.010s	sys 0.010s
W20260709 12:24:59.597376 18021 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.010s	sys 0.010s
I20260709 12:24:59.600512 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:24:59.602288 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.611608 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.614933 18810 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:59.598387 18017 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.006s	sys 0.012s
W20260709 12:24:59.617563 18017 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.006s	sys 0.012s
W20260709 12:24:59.592386 25801 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.592s	user 0.013s	sys 0.004s
W20260709 12:24:59.595765 25750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.008s	sys 0.029s
I20260709 12:24:59.632674 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.622632 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:59.634724 18810 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:59.639533 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.642544 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.643388 18809 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:24:59.578058 25819 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.652420 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:24:59.630340 25750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.008s	sys 0.030s
I20260709 12:24:59.614490 25877 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.582679 25707 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.658859 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.589414 25474 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.669550 25687 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:59.670960 25474 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:59.673326 18023 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:59.577873 25741 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.588722 25787 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.683035 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.686756 18164 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:59.688835 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:59.683929 18808 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:24:59.692884 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:24:59.693557 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.711655 18164 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:24:59.716508 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:24:59.717974 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.718968 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.720587 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.719519 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.674495 25877 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.735831 25819 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:59.658761 25883 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:24:59.736830 25877 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:59.739410 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:59.684650 25750 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.694828 25766 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.750716 25741 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:59.754383 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:24:59.759581 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:59.762945 25781 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:59.763687 25781 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:59.765565 25703 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:24:59.766691 25866 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:24:59.768846 25886 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:59.769521 25886 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.772133 25886 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:59.774562 25876 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:24:59.776130 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3227) took 1419 ms. Trace:
I20260709 12:24:59.776546 18098 rpcz_store.cc:276] 0709 12:24:58.356388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.883011 (+526623us) service_pool.cc:224] Handling call
0709 12:24:58.883489 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.883974 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.884005 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.884598 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.442173 (+557575us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:59.442219 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.442298 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.775878 (+333580us) raft_consensus.cc:1685] finished
0709 12:24:59.775904 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.776058 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5016,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":138,"thread_start_us":557098,"threads_started":1,"wal-append.queue_time_us":697571}
I20260709 12:24:59.778318 25766 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.781111 25766 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:59.785082 25872 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
I20260709 12:24:59.786698 25801 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.787413 25801 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.715209 25849 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.081s
I20260709 12:24:59.791401 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.792237 18815 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:24:59.792479 25875 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:59.796191 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:24:59.800794 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:24:59.777371 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:24:59.803128 25837 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:59.803772 25837 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:24:59.765055 25781 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.804927 25837 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:24:59.810963 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:24:59.837877 18810 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:24:59.847038 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.848079 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.737762 25474 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.850817 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4129) took 2367 ms. Trace:
I20260709 12:24:59.853058 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:24:59.855789 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.864568 18170 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:59.864703 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:24:59.880116 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:24:59.880980 18173 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.881032 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:59.865572 18091 rpcz_store.cc:276] 0709 12:24:57.442006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.442140 (+   134us) service_pool.cc:224] Handling call
0709 12:24:57.442521 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.442777 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:57.888145 (+445368us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.950668 (+1062523us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b18004478e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287
0709 12:24:59.557189 (+606521us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.809239 (+252050us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:59.809302 (+    63us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:59.809367 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.809454 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.809614 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1213619,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":119,"raft.queue_time_us":52631,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":48475,"spinlock_wait_cycles":59627904,"thread_start_us":1324212,"threads_started":5,"wal-append.queue_time_us":188122,"wal-append.run_cpu_time_us":1637,"wal-append.run_wall_time_us":1124324}
I20260709 12:24:59.798139 25707 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:24:59.892603 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3226) took 1537 ms. Trace:
I20260709 12:24:59.843946 25848 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:59.896003 18090 rpcz_store.cc:276] 0709 12:24:58.355495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.731584 (+376089us) service_pool.cc:224] Handling call
0709 12:24:58.732017 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.732458 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.732492 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.743640 (+ 11148us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.854350 (+110710us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:58.854420 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.854531 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.784322 (+929791us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:24:59.892123 (+107801us) raft_consensus.cc:1685] finished
0709 12:24:59.892163 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.892500 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":600,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":6851,"spinlock_wait_cycles":254920064,"thread_start_us":82788,"threads_started":1,"wal-append.queue_time_us":605137}
I20260709 12:24:59.692471 25675 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:24:59.939392 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:59.902360 25721 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:59.902750 25723 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:59.966996 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:24:59.902408 25875 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:59.906769 25801 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:24:59.910885 25842 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.932s	user 0.012s	sys 0.004s
I20260709 12:24:59.911128 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1096) took 1158 ms. Trace:
I20260709 12:24:59.912858 25852 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.886215 25819 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:59.940776 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:24:59.799041 25750 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.900074 25413 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.887223 25474 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.866879 25557 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:24:59.947100 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.943163 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:24:59.972491 18020 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:24:59.972050 18022 rpcz_store.cc:276] 0709 12:24:58.753089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.753249 (+   160us) service_pool.cc:224] Handling call
0709 12:24:58.753647 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.836070 (+ 82423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.836111 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.836161 (+    50us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:58.836216 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.910813 (+1074597us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b540010aed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:24:59.910884 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.911048 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188943232}
I20260709 12:24:59.973532 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:24:59.977617 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 229427us
I20260709 12:24:59.982713 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:24:59.983451 18809 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:59.939543 25877 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.991890 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:24:59.993074 18023 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:59.993571 18136 leader_election.cc:400] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:24:59.888922 25824 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:24:59.994511 18136 leader_election.cc:403] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:24:59.970081 18017 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.995273 18173 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:24:59.996160 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:00.000386 18022 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:59.971449 25721 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:00.012312 18018 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
W20260709 12:25:00.003947 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 123074us
I20260709 12:24:59.983160 25819 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.949666 25700 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:25:00.016225 18024 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:59.981355 25293 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:59.943352 25502 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.022750 18020 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:00.024333 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 3.115s (SENT)
W20260709 12:25:00.023874 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80841.1us
I20260709 12:25:00.026553 18168 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:00.009359 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.030196 25890 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.004s
I20260709 12:25:00.031358 25474 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:00.032287 25687 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:00.032794 25687 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:24:59.997478 25877 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.021271 25675 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.038300 25413 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:00.038477 25893 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:25:00.015940 25875 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:00.039546 25700 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:00.040369 25700 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.040256 25875 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.018349 25557 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.041531 25846 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.043648 25750 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:00.044554 25721 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 21.19, Last appended by leader: 19, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.046077 25721 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.046695 25824 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:00.021003 25819 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:00.050537 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:00.050351 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.050951 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:00.051638 18021 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:00.052917 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 230675us
I20260709 12:25:00.054554 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.033200 25687 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.058250 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:00.064366 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.048956 25293 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:00.068979 18022 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:00.072933 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.074930 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.088222 18023 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
W20260709 12:25:00.080711 25789 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.017s	sys 0.006s
W20260709 12:25:00.114655 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 132072us
I20260709 12:25:00.126941 25897 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:00.127925 25873 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.608s	user 0.001s	sys 0.001s
I20260709 12:25:00.128178 25897 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.122017 18023 pending_rounds.cc:85] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 13
I20260709 12:25:00.134272 25608 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.134858 18023 pending_rounds.cc:107] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:25:00.135602 18023 raft_consensus.cc:2889] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:00.119974 25789 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.018s	sys 0.006s
I20260709 12:25:00.141316 18093 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
W20260709 12:25:00.104732 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 258435us
I20260709 12:25:00.142570 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:00.150417 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 75613.8us
I20260709 12:25:00.061488 25890 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:00.151957 18064 leader_election.cc:400] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19.
W20260709 12:25:00.152622 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79915.1us
I20260709 12:25:00.161391 18064 leader_election.cc:403] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:25:00.079031 25787 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.166426 25431 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.086083 25846 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:00.155730 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:00.165261 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.182667 18022 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.167450 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:00.184098 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:00.184612 18171 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:00.185196 18021 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:00.083611 25750 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.169234 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.173676 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1394) took 1500 ms. Trace:
I20260709 12:25:00.158234 18015 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:00.072539 25824 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:25:00.228502 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.228502 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.160233 25695 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.258783 18811 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:00.187083 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.182780 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:00.262007 18172 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:00.267514 18022 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:00.274652 18167 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:00.078811 25899 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.178s
I20260709 12:25:00.187011 25852 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:00.379520 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.380244 18094 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.275767 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.255156 18813 rpcz_store.cc:276] 0709 12:24:58.663743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.663982 (+   239us) service_pool.cc:224] Handling call
0709 12:24:58.664458 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.726718 (+ 62260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.726782 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.744669 (+ 17887us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.093198 (+348529us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:59.093247 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.127143 (+1033896us) spinlock_profiling.cc:243] Waited 68.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:00.163865 (+ 36722us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b5400494154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:00.163930 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:00.163985 (+    55us) raft_consensus.cc:1685] finished
0709 12:25:00.164000 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:00.164172 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3228,"raft.queue_time_us":14696,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":6916,"spinlock_wait_cycles":291842176,"thread_start_us":304834,"threads_started":1,"wal-append.queue_time_us":700539}
I20260709 12:25:00.381474 18170 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:25:00.315007 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.383733 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:00.276190 25901 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:25:00.384617 18172 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:00.282308 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:00.108314 25903 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.391469 18095 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:25:00.216748 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.392678 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.169075 25700 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19.
I20260709 12:25:00.172324 25789 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:00.330785 25911 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:00.342208 25904 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:25:00.354666 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.374660 18017 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:00.374840 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.428192 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.376883 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.377485 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.378602 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.273878 25824 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.232620 25898 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:00.381472 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 197471us
I20260709 12:25:00.394726 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.382963 25750 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.382899 25908 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:00.401665 25721 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:25:00.450735 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.416997 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1193) took 2645 ms. Trace:
I20260709 12:25:00.452433 18096 rpcz_store.cc:276] 0709 12:24:57.771170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.808264 (+ 37094us) service_pool.cc:224] Handling call
0709 12:24:57.891021 (+ 82757us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.207766 (+1316745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.207805 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:59.207860 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:59.207895 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.416597 (+1208702us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400214b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:00.416679 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:00.416900 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3027500928}
I20260709 12:25:00.419903 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.392040 18095 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:00.176168 25376 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.207s
I20260709 12:25:00.454293 18165 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:00.434168 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.455406 18092 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:00.455518 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:00.436249 18093 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:00.456977 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:00.457469 25906 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:25:00.443763 25824 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.323328 25837 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.109s
I20260709 12:25:00.463343 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:00.466852 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:00.438264 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:00.383970 25707 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.441596 18172 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:00.172066 25896 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.240s
I20260709 12:25:00.423177 25789 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:00.443375 18099 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:00.439858 25502 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.453504 18166 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:00.467754 18813 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:00.463877 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:00.439217 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.464596 25557 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20.
I20260709 12:25:00.442229 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.425899 25911 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.470089 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.489560 18167 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:25:00.471540 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:00.474833 18090 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:25:00.471000 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.467636 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:25:00.490018 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47928.5us
I20260709 12:25:00.490860 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.491369 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:00.491611 18165 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:00.492802 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.492862 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.491665 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:00.494544 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.497746 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.499653 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.501313 18165 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:00.477386 25789 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:00.502450 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.503508 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:00.503821 25910 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:25:00.465646 25750 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.487747 25877 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.475492 25707 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.508508 18165 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:25:00.465381 25898 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.501631 25837 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:00.509428 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.478621 25742 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.512226 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.526938 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.493930 25911 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.485450 25903 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:00.501443 25912 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:00.524812 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:00.465956 25908 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.526325 18167 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:00.528254 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.529378 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.529428 18172 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:00.516750 25793 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.513806 18094 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:00.540971 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.541257 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:00.531373 25877 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.523645 25837 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:00.543787 25842 raft_consensus.cc:2764] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:25:00.533164 25903 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.562217 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.562446 25903 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.548012 25608 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:00.564007 25608 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:00.564397 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.544317 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.547235 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:00.564030 25687 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.567631 18809 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.569519 25687 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:00.542258 25896 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.565114 25608 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.572598 18167 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:00.574913 18167 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:25:00.578635 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.553375 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.554320 25837 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.583870 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.584230 25908 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:00.531692 25721 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:25:00.527415 25888 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:25:00.587105 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.554804 25899 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.555303 25675 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.555404 25413 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:00.594022 25413 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:00.595049 25413 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.596110 25413 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.557210 25707 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:00.588083 18096 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:00.589711 18167 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:25:00.599912 25675 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.603231 25750 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.603915 25675 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.550048 25824 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:00.605870 25819 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.545s	user 0.015s	sys 0.001s
I20260709 12:25:00.531446 25742 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.606577 25819 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.609189 25898 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.612555 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:00.613070 18815 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:00.613400 18806 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:00.589509 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:00.614953 25898 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:00.615358 25898 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.615670 25898 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.612541 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.618626 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:00.591615 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.541993 25917 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:00.623505 18813 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:25:00.624456 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.590394 25877 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:00.636180 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:00.609581 25819 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.641462 18809 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
W20260709 12:25:00.644188 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135140us
I20260709 12:25:00.647096 18806 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:25:00.655670 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.661127 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.623667 25813 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.701064 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1082) took 3597 ms. Trace:
I20260709 12:25:00.701700 18019 rpcz_store.cc:276] 0709 12:24:57.093707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.093917 (+   210us) service_pool.cc:224] Handling call
0709 12:24:59.040658 (+1946741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.042047 (+  1389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.690581 (+1648534us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b5400268250. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:00.690779 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4475884288}
I20260709 12:25:00.685721 25896 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.681478 25899 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:00.686463 25824 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:00.718866 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.720068 18094 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:00.720988 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74734.8us
I20260709 12:25:00.708649 25813 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.711809 25899 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:00.747349 18022 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:00.708313 25912 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:00.752164 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:00.753162 18022 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:00.716086 25376 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 12:25:00.771337 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:00.772136 18023 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.772795 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.775857 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.754436 25899 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:00.781420 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.783905 18094 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:00.785682 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.792100 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:00.793955 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90218.5us
I20260709 12:25:00.794696 18093 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.793764 18022 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:00.795967 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:00.791659 18099 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:25:00.797369 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:00.792192 18092 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:25:00.796677 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:00.766695 25824 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:00.817592 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.818038 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.819001 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.820284 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:00.823333 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:00.824689 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:00.840930 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.843819 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.844003 25413 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.845623 25413 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:00.847500 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.848819 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.849148 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.849385 18099 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:25:00.849884 18092 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:00.850244 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.850862 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:00.850986 18022 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:00.851591 25787 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.852079 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.852790 18092 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:00.840790 25293 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.854805 25293 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.855309 25293 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.853802 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:00.857193 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.858964 18093 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:00.859462 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.860484 25787 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:00.861322 25787 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:25:00.862995 25926 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:25:00.862582 25912 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 21.19, Last appended by leader: 19, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:00.864800 25930 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:25:00.792995 25842 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
W20260709 12:25:00.870826 25915 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
W20260709 12:25:00.872015 25920 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
W20260709 12:25:00.874032 25921 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:25:00.815192 25824 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.881134 25824 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:00.828012 25819 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.790359 25928 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.105s
I20260709 12:25:00.842880 25608 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.842633 25687 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.852094 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:00.913585 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54227.9us
I20260709 12:25:00.922035 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:00.928730 25912 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.929795 25912 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.933422 25903 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.935020 18165 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:00.935390 25912 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:00.937865 25898 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.938764 25743 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:00.939340 25743 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:00.940604 25898 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:00.940553 25743 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.942718 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 3999) took 1496 ms. Trace:
I20260709 12:25:00.943281 18018 rpcz_store.cc:276] 0709 12:24:59.446356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.449369 (+  3013us) service_pool.cc:224] Handling call
0709 12:24:59.471302 (+ 21933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.017895 (+546593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.942349 (+924454us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54001d1854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:00.942622 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220274048}
I20260709 12:25:00.946491 25877 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.947196 25877 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.948127 18019 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: [16.11-18.12]
I20260709 12:25:00.949515 25898 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:00.950170 25898 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:00.951519 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.953416 18023 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:00.955128 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.956115 18165 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:00.955628 25911 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:00.900444 25936 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.082s
W20260709 12:25:00.974529 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.958992 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4136) took 1876 ms. Trace:
I20260709 12:25:00.979045 18098 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:01.027590 18097 rpcz_store.cc:276] 0709 12:24:59.082435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.082721 (+   286us) service_pool.cc:224] Handling call
0709 12:24:59.083148 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.325803 (+242655us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.958718 (+1632915us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:00.958924 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":519720704}
I20260709 12:25:01.028522 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4154) took 1304 ms. Trace:
I20260709 12:25:00.957096 18096 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:01.029245 18098 rpcz_store.cc:276] 0709 12:24:59.724469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.778487 (+ 54018us) service_pool.cc:224] Handling call
0709 12:25:00.978326 (+1199839us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:25:00.978709 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.028006 (+ 49297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.028446 (+   440us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59499776}
I20260709 12:25:00.999408 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:00.972992 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1144) took 1283 ms. Trace:
W20260709 12:25:01.034596 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112685us
I20260709 12:25:01.034564 18168 rpcz_store.cc:276] 0709 12:24:59.698777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.708550 (+  9773us) service_pool.cc:224] Handling call
0709 12:24:59.738456 (+ 29906us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.049219 (+310763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.972635 (+923416us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54003ba1d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:00.972886 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":661990272}
I20260709 12:25:01.035785 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.036942 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:01.043612 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44390.5us
W20260709 12:25:01.046278 17989 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.515s (SENT) [suppressed 21 similar messages]
I20260709 12:25:01.045439 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.055656 18099 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
W20260709 12:25:00.965317 25902 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.932s	user 0.003s	sys 0.001s
I20260709 12:25:01.069146 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:00.984865 25700 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:01.091535 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4006) took 1642 ms. Trace:
I20260709 12:25:01.032428 25781 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.956667 25898 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:01.032989 25687 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:00.970351 25392 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.952260 25837 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.095700 18024 rpcz_store.cc:276] 0709 12:24:59.448636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.475816 (+ 27180us) service_pool.cc:224] Handling call
0709 12:24:59.476349 (+   533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.020194 (+543845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.091184 (+1070990us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54001d1354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:01.091449 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241032064}
I20260709 12:25:01.041389 25803 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.032651 25794 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:01.051795 25743 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:01.068228 25865 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:01.105358 25687 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.960021 25899 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.119000 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.115511 25911 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.119822 25898 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.965659 25789 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.095266 25700 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:01.129781 25687 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:01.152009 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.155479 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.156217 25842 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:01.156837 25842 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:01.156976 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.115038 25392 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.160156 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.161151 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:01.162520 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.163997 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.164407 18095 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.165235 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.123926 18091 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:01.107038 25608 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:01.124555 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 300064us
W20260709 12:25:01.125578 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89861.4us
I20260709 12:25:01.170265 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:01.126552 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.130005 18167 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:01.131714 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 324483us
I20260709 12:25:01.150315 25700 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.120651 25794 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.143837 25789 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.117434 25803 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.131305 25557 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:25:01.168008 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.153656 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.165580 18095 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:01.170151 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.123304 25865 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.142326 18170 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:25:01.174278 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:01.179191 18094 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:25:01.180812 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.161609 25392 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.191869 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.193125 18172 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:25:01.188961 18811 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:01.195997 18172 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:01.196485 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.202339 18172 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:01.180269 25741 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:01.178817 25376 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.203186 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.195410 25865 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:01.227275 25608 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:01.200093 25413 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:01.237350 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.237629 18095 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:25:01.195689 25789 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:25:01.199713 25813 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:01.290319 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9372227919998295s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2315s for a response
W20260709 12:25:01.291195 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7257227919999423s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.255s for a response
I20260709 12:25:01.270784 18095 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
W20260709 12:25:01.291617 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5226259410001148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260709 12:25:01.291883 18095 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:25:01.292474 18095 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:01.292650 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 246385us
I20260709 12:25:01.205233 25700 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:01.292042 18064 connection.cc:256] RPC call timeout handler was delayed by -1.429925940999965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
W20260709 12:25:01.288949 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51731.7us
W20260709 12:25:01.294744 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2928227920001518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30310000000000004s for a response
I20260709 12:25:01.297209 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.266920 18167 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:01.298380 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.299196 18167 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:01.299422 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:01.300242 18095 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:01.300285 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.302480 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.303309 18097 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:01.303850 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.304171 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.304886 18097 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:01.293275 25898 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:25:01.306002 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:01.306182 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:01.307027 18099 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:01.308190 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.311607 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.313493 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:01.312207 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.316373 18095 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:01.322206 25931 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.322898 25837 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:01.350018 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1136) took 4293 ms (client timeout 2716 ms). Trace:
W20260709 12:25:01.350621 18166 rpcz_store.cc:269] 0709 12:24:57.033248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.068687 (+ 35439us) service_pool.cc:224] Handling call
0709 12:25:01.326569 (+4257882us) spinlock_profiling.cc:243] Waited 3.33 s on lock 0x7b54000438d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:01.326790 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7666192128}
W20260709 12:25:01.356365 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 176503us
I20260709 12:25:01.267031 25819 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:01.207597 25793 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:01.244510 25877 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:01.382546 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71194.5us
I20260709 12:25:01.383586 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.385493 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.344204 25928 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:01.391736 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 208039us
I20260709 12:25:01.323063 25931 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:01.395572 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 215467us
I20260709 12:25:01.203085 25842 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.254235 25376 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:01.386523 25896 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.669s	user 0.011s	sys 0.003s
I20260709 12:25:01.380730 25793 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.411108 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.415084 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:01.379380 25947 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:01.423023 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.427457 18024 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:01.424685 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:01.428772 18022 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:01.417119 25942 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:01.394797 25837 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:01.392423 25819 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:01.391860 25940 raft_consensus.cc:2764] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260709 12:25:01.437228 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.444358 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:01.438728 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:01.422127 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.424647 25793 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:01.425946 25898 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:01.429593 25941 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:01.418037 25896 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:01.418288 25700 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:01.437662 25842 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:01.392827 25687 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.431514 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:01.456827 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.453797 25946 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:01.453967 25941 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:01.435680 25877 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:01.454644 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:01.475203 18018 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:01.475322 25948 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:01.479656 25961 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:01.480334 25961 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.480379 25866 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:01.480894 25961 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:01.481158 25866 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:01.466629 25939 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:25:01.456853 25899 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.482141 25877 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:01.440440 25942 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.435122 25947 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.483431 25877 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.485066 25794 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:01.494560 25962 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:25:01.404472 25928 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.466068 18022 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:01.466665 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.472470 25898 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.473902 25941 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:01.518962 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.529009 18099 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:25:01.529701 18781 connection.cc:256] RPC call timeout handler was delayed by -1.1576642029999675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260709 12:25:01.527465 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.533136 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.539378 18813 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:01.539727 25965 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:01.468572 25803 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:01.544584 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78073.5us
I20260709 12:25:01.546447 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.548974 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:01.552441 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1143) took 1864 ms (client timeout 2254 ms). Trace:
I20260709 12:25:01.552634 18173 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20. Ops: []
W20260709 12:25:01.552990 18164 rpcz_store.cc:269] 0709 12:24:59.688408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.688543 (+   135us) service_pool.cc:224] Handling call
0709 12:25:01.552035 (+1863492us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54001b5654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:01.552366 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49912704}
W20260709 12:25:01.555472 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 151134us
I20260709 12:25:01.558092 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:01.560122 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 483512us
I20260709 12:25:01.499354 25431 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:25:01.563644 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.567408 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.569097 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:01.550856 25917 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:01.556974 25813 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:01.573143 25953 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:01.585311 18171 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:01.582777 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:01.558087 25946 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:01.540827 25965 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.565532 25948 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:01.604363 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.604929 18168 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:01.609908 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.579154 25917 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:01.588865 25803 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:01.589699 25813 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:01.571995 25741 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
W20260709 12:25:01.616731 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 96227us
W20260709 12:25:01.634598 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67252.7us
I20260709 12:25:01.637326 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:01.641038 18099 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:25:01.558398 25940 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:25:01.642930 18095 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:25:01.643221 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.604363 25392 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:01.646878 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:01.572496 25819 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.607869 25846 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.651142 18022 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:01.648434 18095 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.655325 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.659452 18095 raft_consensus.cc:614] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:01.660916 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.559924 25866 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.615849 25946 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:25:01.683434 18021 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:01.668272 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.685060 18018 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:01.682260 18778 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:01.668572 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.674695 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:01.639930 25813 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:01.644300 25945 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:01.644073 25963 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:01.680853 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 135028us
I20260709 12:25:01.686838 18778 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.690567 18099 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:25:01.697168 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.716980 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:01.717761 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:01.722952 18171 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:01.723323 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.680380 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:01.639415 25948 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:01.724053 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:01.719945 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.645711 25803 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:01.677222 25955 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
W20260709 12:25:01.759763 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79500.8us
I20260709 12:25:01.764564 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:01.765517 18171 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: [18.9-18.9]
W20260709 12:25:01.769984 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 83211.4us
I20260709 12:25:01.650025 25707 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.093s
I20260709 12:25:01.721601 25945 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:01.776322 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 134535us
I20260709 12:25:01.719110 25392 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:01.769726 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:01.778793 18169 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:01.737581 25877 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:01.735921 25855 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:25:01.736802 25687 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:01.818154 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3028) took 1066 ms. Trace:
I20260709 12:25:01.788121 18171 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:01.819016 18169 rpcz_store.cc:276] 0709 12:25:00.751611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.668277 (+916666us) service_pool.cc:224] Handling call
0709 12:25:01.818027 (+149750us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:01.665607 25961 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.802222 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:01.810971 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:01.678287 25966 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.123s
W20260709 12:25:01.823063 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3091) took 1497 ms (client timeout 1515 ms). Trace:
I20260709 12:25:01.824558 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4160) took 1448 ms. Trace:
I20260709 12:25:01.825382 18094 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:25:01.767604 25781 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:01.849542 18170 rpcz_store.cc:269] 0709 12:25:00.325672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.417198 (+ 91526us) service_pool.cc:224] Handling call
0709 12:25:01.822791 (+1405593us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b54003016d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:01.822973 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1639844224}
I20260709 12:25:01.823772 25962 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.678747 25846 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:01.821664 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4173) took 1207 ms (client timeout 1502 ms). Trace:
W20260709 12:25:01.837185 25959 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:25:01.837558 18020 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:25:01.719975 25953 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 12:25:01.839105 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033336
I20260709 12:25:01.839326 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:01.839696 25957 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:25:01.786070 25973 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:01.840199 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:01.841894 18021 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:25:01.849905 25431 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:02.081097 18021 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:02.082644 18021 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:01.783615 25945 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:01.761898 25931 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.777261 25413 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:01.847491 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:01.848106 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.091034 18171 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:01.720438 25970 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.145s
W20260709 12:25:01.830238 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94633.3us
I20260709 12:25:01.912597 25976 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.906673 25897 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.769225 25922 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.089s
I20260709 12:25:01.737609 25963 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:01.777757 25917 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:01.858582 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.030818 18165 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 22 index: 20. (index mismatch)
I20260709 12:25:02.027665 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3248) took 1241 ms. Trace:
I20260709 12:25:02.006346 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d440327a63754b889f7edd91e1373942"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:01.834192 25741 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:02.110913 25741 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.111495 25741 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:02.114781 25981 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.115602 25981 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.011833 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4012) took 2057 ms. Trace:
W20260709 12:25:02.031800 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.118016 25979 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.941457 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.118809 25979 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:02.119993 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 178708us
I20260709 12:25:02.121085 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.003021 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3024) took 1253 ms. Trace:
I20260709 12:25:02.122001 18173 rpcz_store.cc:276] 0709 12:25:00.749298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.196335 (+447037us) service_pool.cc:224] Handling call
0709 12:25:01.552335 (+356000us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.553313 (+   978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.002668 (+449355us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54001b5650. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:02.002913 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":817827712}
I20260709 12:25:02.123773 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.126071 25931 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:01.912256 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3116) took 1120 ms. Trace:
W20260709 12:25:01.943311 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.126996 18168 rpcz_store.cc:276] 0709 12:25:00.791943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.068778 (+276835us) service_pool.cc:224] Handling call
0709 12:25:01.911940 (+843162us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54003011d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:01.912145 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1183291776}
I20260709 12:25:01.955904 18172 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:25:01.860648 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.129180 18811 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:01.917164 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.969050 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.130192 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1155) took 1360 ms. Trace:
I20260709 12:25:02.130949 18015 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.130846 18167 rpcz_store.cc:276] 0709 12:25:00.769608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.300083 (+530475us) service_pool.cc:224] Handling call
0709 12:25:01.300485 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.300893 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.300929 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.301501 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.463713 (+162212us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:01.463795 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.463908 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.129793 (+665885us) raft_consensus.cc:1685] finished
0709 12:25:02.129872 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.130102 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1144,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":107,"thread_start_us":161709,"threads_started":1,"wal-append.queue_time_us":293793}
I20260709 12:25:02.132314 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.133612 18015 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:02.042143 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:02.138757 25608 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:25:02.040727 18096 rpcz_store.cc:269] 0709 12:25:00.614536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.618395 (+  3859us) service_pool.cc:224] Handling call
0709 12:25:01.821291 (+1202896us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54001afc54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:01.821566 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":762890496}
I20260709 12:25:01.999783 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:01.834981 25376 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:01.835543 25877 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:02.159204 25741 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.159359 25877 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.149077 25973 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.162272 25973 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.165237 25973 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.166533 25813 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.140587 25976 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.168727 25867 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:02.169230 25867 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:02.169376 25813 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.170449 25867 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.171177 25976 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.171753 25867 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:02.099838 25413 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.175050 25413 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.840252 25928 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:01.840575 25871 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:01.983050 25946 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:01.899943 25793 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.178048 25946 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:01.904940 25965 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:02.025242 25923 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.285s	user 0.001s	sys 0.003s
W20260709 12:25:01.903244 25865 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.579s	user 0.014s	sys 0.000s
W20260709 12:25:01.899554 25964 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:25:02.068368 18020 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:02.069527 18018 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
W20260709 12:25:01.974583 25934 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.108s	user 0.001s	sys 0.004s
I20260709 12:25:01.977582 25829 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:02.022063 25925 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:02.121657 25979 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:01.970104 25912 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:01.982585 25787 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:01.951009 25942 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:01.973384 25392 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:01.907328 25293 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:01.778402 25967 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:01.967949 25977 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:01.857717 25781 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.235875 25781 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:02.098416 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.240940 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:02.241652 18097 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:01.977701 25961 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.099987 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.245625 18022 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
W20260709 12:25:02.080765 25837 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.627s	user 0.014s	sys 0.001s
I20260709 12:25:02.104283 18091 rpcz_store.cc:276] 0709 12:25:00.786723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.786908 (+   185us) service_pool.cc:224] Handling call
0709 12:25:00.787325 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.170010 (+382685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.027243 (+857233us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:02.027556 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":773277952}
I20260709 12:25:02.248564 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:02.198868 25829 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:02.118070 18016 rpcz_store.cc:276] 0709 12:24:59.957173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.973138 (+ 15965us) service_pool.cc:224] Handling call
0709 12:25:02.011419 (+2038281us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b54001fc054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:02.011713 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1127153536}
I20260709 12:25:02.135475 25931 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.137921 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:02.138168 25723 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:02.138609 25700 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.139199 25801 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:02.125175 25963 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.148029 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.145076 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.152220 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:01.840338 25803 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.268743 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1156) took 1498 ms. Trace:
I20260709 12:25:02.269385 18166 rpcz_store.cc:276] 0709 12:25:00.769931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.351750 (+581819us) service_pool.cc:224] Handling call
0709 12:25:01.384429 (+ 32679us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.384894 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.384923 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.385522 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.579039 (+193517us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:01.579113 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.579245 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.268430 (+689185us) raft_consensus.cc:1685] finished
0709 12:25:02.268458 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.268657 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":355,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"thread_start_us":192985,"threads_started":1,"wal-append.queue_time_us":283745}
I20260709 12:25:02.155198 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3256) took 1306 ms. Trace:
I20260709 12:25:01.867750 25962 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.152246 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.182796 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.088325 25431 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:01.993948 25707 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:01.851596 25948 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.191244 25928 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:02.197590 25941 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.660s	user 0.010s	sys 0.005s
I20260709 12:25:02.225394 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:02.279888 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:02.225612 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.179302 25946 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.285651 18808 raft_consensus.cc:2368] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17.
I20260709 12:25:02.264583 25979 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:02.239486 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1112) took 1776 ms. Trace:
W20260709 12:25:02.246249 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.249210 18091 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:02.268548 25924 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.378s	user 0.026s	sys 0.017s
I20260709 12:25:02.201145 25925 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:02.215318 25912 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:02.260198 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122429us
I20260709 12:25:02.301143 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:02.302018 18808 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:02.198650 25865 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.266917 18165 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:02.246913 25837 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:02.272886 18092 rpcz_store.cc:276] 0709 12:25:00.848580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.858751 (+ 10171us) service_pool.cc:224] Handling call
0709 12:25:00.859158 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.111917 (+252759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.111960 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.130739 (+ 18779us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.399704 (+268965us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:01.399802 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.400076 (+   274us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.154923 (+754847us) raft_consensus.cc:1685] finished
0709 12:25:02.154955 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.155119 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76816,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":62,"thread_start_us":439313,"threads_started":2,"wal-append.queue_time_us":281282}
I20260709 12:25:02.283991 25700 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.262177 25931 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.277000 18015 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:02.281528 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.284538 18806 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:02.317167 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:02.229473 25787 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.190088 25871 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:02.293854 25918 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.288057 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.289803 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.291034 18017 rpcz_store.cc:276] 0709 12:25:00.462854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.463946 (+  1092us) service_pool.cc:224] Handling call
0709 12:25:00.464978 (+  1032us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.473281 (+  8303us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:00.502184 (+ 28903us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.691703 (+189519us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.703875 (+ 12172us) log.cc:844] Serialized 26 byte log entry
0709 12:25:00.707961 (+  4086us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:00.708027 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.708148 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.024354 (+1316206us) spinlock_profiling.cc:243] Waited 1.63 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:02.239037 (+214683us) raft_consensus.cc:1685] finished
0709 12:25:02.239192 (+   155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.239400 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28010,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":6490,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":23735,"spinlock_wait_cycles":3746176,"thread_start_us":163014,"threads_started":1,"wal-append.queue_time_us":192693}
I20260709 12:25:02.330678 18016 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:01.849653 18090 rpcz_store.cc:276] 0709 12:25:00.375929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.388553 (+ 12624us) service_pool.cc:224] Handling call
0709 12:25:00.389107 (+   554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.490096 (+100989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.824137 (+1334041us) spinlock_profiling.cc:243] Waited 73.3 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:01.824459 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":168628096}
I20260709 12:25:02.332302 18090 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:02.267835 25967 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:02.300025 18137 leader_election.cc:400] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17.
I20260709 12:25:02.302670 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:02.279896 25431 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:02.306424 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.279245 25707 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:02.310261 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.219488 25933 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:25:02.293854 25973 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:25:02.201524 25848 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.163s	user 0.013s	sys 0.030s
I20260709 12:25:02.326923 18092 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:02.324314 18091 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:02.302651 25819 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.026s
W20260709 12:25:02.294795 25924 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.410s	user 0.027s	sys 0.017s
I20260709 12:25:02.335486 18164 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: [16.11-18.12]
I20260709 12:25:02.299401 25793 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:02.291893 25941 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.337654 18137 leader_election.cc:403] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 12:25:02.374743 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 74842.7us
W20260709 12:25:02.375362 25984 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:25:02.338752 18167 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:02.290596 25928 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.347512 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:02.358421 25848 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.322s	user 0.014s	sys 0.031s
I20260709 12:25:02.361483 18090 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:02.321317 25700 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.359773 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.399330 18091 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:02.367183 18808 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:02.399940 18093 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:02.356560 25787 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:02.360777 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:02.386301 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.410145 18173 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:02.387982 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.389660 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.330130 25976 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:02.384748 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.392616 18097 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:02.427683 25976 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:02.327208 25946 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:02.428364 25976 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:02.348901 25973 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.396068 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.431213 25973 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:02.401990 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.336107 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:02.433648 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97685.5us
I20260709 12:25:02.309100 25829 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.436096 25829 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:02.361598 25925 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.438725 25925 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:02.405853 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1207) took 1104 ms. Trace:
I20260709 12:25:02.440379 18094 rpcz_store.cc:276] 0709 12:25:01.301333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.310112 (+  8779us) service_pool.cc:224] Handling call
0709 12:25:01.824941 (+514829us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.856759 (+ 31818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.405579 (+548820us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54001afc50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:02.405747 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1182707072}
W20260709 12:25:02.441606 25983 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:25:02.445052 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.445498 25863 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:02.446081 25863 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.349397 25967 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.446489 25863 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:02.448379 25967 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:02.349860 25842 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.449652 25842 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.450383 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:02.451506 25741 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:02.452209 25842 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.453413 18815 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 22 index: 20. (index mismatch)
I20260709 12:25:02.348138 25912 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.460983 25912 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:02.461411 25743 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:02.462037 25743 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:02.462811 25962 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.463063 25743 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.464318 25743 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:02.465044 25700 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.452615 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:02.468300 18096 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:25:02.468832 18096 pending_rounds.cc:85] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 12
I20260709 12:25:02.469116 18096 pending_rounds.cc:107] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:25:02.469405 18096 raft_consensus.cc:2889] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:02.406140 25871 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.415256 18169 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:25:02.415737 25866 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:02.415580 25781 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.337060 25918 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.393898 25608 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:02.423137 25931 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.454224 25742 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:02.454473 25928 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:02.454775 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.456117 25865 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:02.515270 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 154597us
I20260709 12:25:02.412755 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 2954) took 1674 ms. Trace:
I20260709 12:25:02.515983 18019 rpcz_store.cc:276] 0709 12:25:00.738777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.738951 (+   174us) service_pool.cc:224] Handling call
0709 12:25:00.739431 (+   480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:00.948698 (+209267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.412314 (+1463616us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b540024de54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:02.412659 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478035328}
I20260709 12:25:02.413013 25945 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:02.471222 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.472002 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.472635 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.521996 18171 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:02.473181 18172 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:02.413983 25707 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:02.477625 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.507225 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.507434 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:02.488831 25942 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:02.475976 25750 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:02.514639 18170 raft_consensus.cc:606] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.535048 18170 raft_consensus.cc:614] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:02.456276 25721 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.522661 25877 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.548758 18098 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:25:02.552147 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.528613 25960 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.533886 25819 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:02.529781 25942 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:02.523880 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:02.574193 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97671.1us
I20260709 12:25:02.577288 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.523097 18173 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:25:02.480319 25897 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.582471 25897 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.585285 25897 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.536010 18806 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:25:02.519528 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.585469 25557 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:02.586442 18090 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:02.587495 25557 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.512888 18018 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:02.526271 25961 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
W20260709 12:25:02.580197 17910 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.200s	user 0.005s	sys 0.005s
I20260709 12:25:02.588557 25742 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.589152 25941 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.589805 25819 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:02.590287 25945 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.594000 25707 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:02.618727 18811 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:25:02.621272 18171 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:02.594668 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3032) took 1790 ms. Trace:
I20260709 12:25:02.623122 18164 rpcz_store.cc:276] 0709 12:25:00.803888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.560130 (+756242us) service_pool.cc:224] Handling call
0709 12:25:01.560637 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:02.364550 (+803913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.594314 (+229764us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b540002add4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:02.594585 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1781175552}
I20260709 12:25:02.595985 25942 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:02.558563 25829 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.596987 25967 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:02.599467 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 296843us
I20260709 12:25:02.600752 25924 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:25:02.602996 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.603502 25985 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:02.609663 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.675248 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.676187 18021 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
W20260709 12:25:02.571807 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.667032 18018 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:25:02.668294 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.672720 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.673298 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.680925 18019 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.681160 18024 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:02.543558 25750 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.680459 18021 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:02.684890 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:02.685611 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.536294 25392 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:02.690621 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:02.693394 18024 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.692092 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:02.702713 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.703115 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.703338 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.703915 18171 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:02.705174 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:02.705605 18021 raft_consensus.cc:2368] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:02.705477 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.705214 18164 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:02.711423 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:02.711570 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 205542us
I20260709 12:25:02.706363 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.712620 18169 raft_consensus.cc:2368] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:02.708633 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:02.713953 18016 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.568948 25877 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.607286 25960 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.722991 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:02.724642 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:02.725487 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.698750 25933 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.683283 25941 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:25:02.699134 25961 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.636340 25557 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.731884 18015 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:02.698835 25392 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.698542 25947 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:02.732972 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:02.620085 25742 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.733916 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.706071 25967 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:02.707484 25932 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:02.739476 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:02.669847 25829 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.699636 25922 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.046s
W20260709 12:25:02.751123 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.737093 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:02.751754 18020 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.757253 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:02.758222 18022 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:02.760197 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:02.706794 25888 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
W20260709 12:25:02.752241 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 232698us
I20260709 12:25:02.775509 25924 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:02.778446 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:02.778085 25948 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:02.731134 25928 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.763516 25911 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:02.738029 25846 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:02.738778 25392 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:02.785540 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:02.784446 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:02.735950 25940 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.789865 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.750168 25848 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:02.687808 25985 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.798924 18022 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:02.742566 25932 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:02.780367 25924 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:02.800935 18021 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.749791 25781 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:02.750834 25918 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:02.764590 25931 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:02.730195 25961 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:02.805207 25996 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:25:02.781590 25941 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.765221 25925 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.803619 18811 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:02.780071 25948 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:02.739112 25700 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 12:25:02.805485 25974 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:02.823030 25924 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.837173 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:02.755517 25967 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.733541 25819 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:02.781950 25979 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
W20260709 12:25:02.957460 25431 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.598s	user 0.006s	sys 0.004s
I20260709 12:25:02.785173 25945 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:25:02.807487 25848 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:02.846163 25903 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:02.800040 25721 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:25:02.794804 25928 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.888628 18024 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:02.822517 25932 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:25:02.878005 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60765.4us
I20260709 12:25:02.970167 25750 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:02.912612 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 107654us
I20260709 12:25:03.004873 18064 leader_election.cc:400] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:03.005313 18064 leader_election.cc:403] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:25:02.990625 25981 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:03.010722 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.011101 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.816073 26003 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:02.818149 25911 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:25:02.921873 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.012621 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.946175 25924 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:03.016868 18807 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:02.948108 25829 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:03.026467 25863 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.579s	user 0.012s	sys 0.003s
I20260709 12:25:03.027220 25863 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:02.950939 25877 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.908078 25925 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:02.949954 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.958884 25812 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.963717 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:03.030645 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67133.6us
W20260709 12:25:02.984833 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.031543 18094 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 23 because replica has last-logged OpId of term: 22 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 18.
I20260709 12:25:02.992148 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4190) took 1011 ms. Trace:
I20260709 12:25:03.032720 18092 rpcz_store.cc:276] 0709 12:25:01.981144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.372582 (+391438us) service_pool.cc:224] Handling call
0709 12:25:02.372873 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.520767 (+147894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.520815 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.520863 (+    48us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:02.520913 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.991814 (+470901us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400291b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:02.991875 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.992070 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339600640}
I20260709 12:25:03.034293 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:02.924391 25974 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:02.994307 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:03.036852 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:02.892606 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 155595us
I20260709 12:25:03.039058 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:03.041641 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:02.988692 26000 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.290s	user 0.000s	sys 0.002s
I20260709 12:25:03.000749 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:02.823346 25940 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.014979 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.015270 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.018976 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:03.018989 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:03.010741 25557 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:03.019985 26005 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:25:03.016523 25945 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.065560 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:03.023466 26006 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 12:25:03.022015 26008 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:25:03.070304 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69662.6us
I20260709 12:25:03.072690 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.023213 25824 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:03.023589 25967 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.020955 25960 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:02.999374 25848 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:02.952386 25742 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:03.044070 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.068953 25557 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:03.020929 26002 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:25:03.071044 18815 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:03.073735 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.039263 26003 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:03.050338 25700 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:03.075958 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:03.050400 25974 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.061573 25897 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:03.060670 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:03.089047 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.066826 25977 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:03.087574 25866 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.003s
I20260709 12:25:03.091090 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.076820 25824 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:03.113703 18092 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:03.111301 25942 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.114836 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.011914 25431 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.116391 25961 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.076599 25877 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:25:03.118630 25431 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:03.119166 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.111574 25743 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.647s	user 0.014s	sys 0.004s
I20260709 12:25:03.121973 25998 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:03.123234 26012 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:03.126008 25608 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.622s	user 0.013s	sys 0.001s
W20260709 12:25:03.102210 25973 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.670s	user 0.012s	sys 0.004s
I20260709 12:25:03.126758 25608 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17.
I20260709 12:25:03.126951 25973 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.127729 18167 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:25:03.094264 18811 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:03.063541 25750 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:03.006541 25948 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.116427 18092 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:03.082042 25413 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:03.096906 26003 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:03.132484 18173 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:03.136653 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:03.140790 18166 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:03.103297 25700 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:03.131907 25413 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.132042 25897 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:03.295619 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 235127us
I20260709 12:25:03.067723 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:03.298610 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 225373us
I20260709 12:25:03.157580 25413 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:03.096393 25557 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:03.379386 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 342897us
I20260709 12:25:03.147508 25700 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.144183 25743 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:03.391196 25743 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.391976 25743 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:03.056644 25812 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.395494 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4179) took 1478 ms. Trace:
I20260709 12:25:03.396188 18093 rpcz_store.cc:276] 0709 12:25:01.916962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.919501 (+  2539us) service_pool.cc:224] Handling call
0709 12:25:01.949786 (+ 30285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.400589 (+450803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.394644 (+994055us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54002972d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:03.395163 (+   519us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034473728}
I20260709 12:25:03.135162 25890 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.398979 25890 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.399717 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4193) took 1417 ms. Trace:
I20260709 12:25:03.210386 25888 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:03.402451 25888 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:03.404659 25888 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:03.217386 25912 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.755s	user 0.025s	sys 0.008s
I20260709 12:25:03.228099 25925 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:03.212648 25928 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.406944 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:03.226733 25931 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:03.408677 25931 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:03.411791 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 344013us
I20260709 12:25:03.233143 25932 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.222168 25940 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:03.383149 25392 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.563s	user 0.009s	sys 0.012s
I20260709 12:25:03.381598 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:03.236366 25941 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:03.235240 25942 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.238376 25945 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:03.228706 25963 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:03.211668 25973 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:03.232417 25976 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.803s	user 0.008s	sys 0.007s
I20260709 12:25:03.230212 25978 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:03.146700 25985 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:03.234009 25986 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.868s	user 0.005s	sys 0.001s
W20260709 12:25:03.158723 26001 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.152s	user 0.003s	sys 0.000s
I20260709 12:25:03.229224 26003 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:03.237192 26007 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.315s	user 0.002s	sys 0.000s
W20260709 12:25:03.219223 26009 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.296s	user 0.000s	sys 0.006s
W20260709 12:25:03.158625 26018 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:25:03.207000 26021 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
W20260709 12:25:03.362650 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.363741 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.366530 25707 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.749s	user 0.019s	sys 0.000s
I20260709 12:25:03.363524 25824 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.332720 25897 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.415896 25974 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.460000 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4178) took 1543 ms. Trace:
I20260709 12:25:03.417474 18099 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:25:03.460603 18098 rpcz_store.cc:276] 0709 12:25:01.916531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.916732 (+   201us) service_pool.cc:224] Handling call
0709 12:25:03.459388 (+1542656us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:03.459658 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961816960}
I20260709 12:25:03.462786 25974 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.464991 25812 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.418613 25392 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.419216 25941 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:03.471969 25941 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.420739 25945 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:03.421276 25963 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:03.476780 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 86) took 1509 ms. Trace:
I20260709 12:25:03.422886 25976 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.479839 18809 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:03.423394 25978 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:03.429080 25962 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.432854 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4192) took 1450 ms. Trace:
I20260709 12:25:03.483840 25978 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:03.431313 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.443878 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.488960 25940 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.484022 18091 rpcz_store.cc:276] 0709 12:25:01.981801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.403679 (+421878us) service_pool.cc:224] Handling call
0709 12:25:02.404128 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:02.601618 (+197490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.601656 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:02.602349 (+   693us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.735374 (+133025us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:02.735422 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.753054 (+ 17632us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:03.236449 (+483395us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:03.432006 (+195557us) raft_consensus.cc:1685] finished
0709 12:25:03.432074 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.432573 (+   499us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":196760,"raft.queue_time_us":198774,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":108,"spinlock_wait_cycles":97565952,"thread_start_us":132527,"threads_started":1,"wal-append.queue_time_us":221769}
I20260709 12:25:03.417866 25925 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:03.489935 25940 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.490340 25925 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.445725 18812 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: [22.20-22.20]
I20260709 12:25:03.495313 25940 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.437395 25898 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:03.496819 25787 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:03.497290 25981 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.498394 25981 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.499573 18165 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:03.487645 25945 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.502081 26021 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:03.446727 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:03.504474 26021 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:03.504709 18814 raft_consensus.cc:2368] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:03.505033 26021 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:03.506631 26021 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 22.20, Last appended by leader: 20, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.506009 25945 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.425583 25932 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.447299 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.476914 25963 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.514480 25963 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.519394 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4185) took 1600 ms. Trace:
I20260709 12:25:03.439486 25675 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:03.477407 18170 rpcz_store.cc:276] 0709 12:25:01.967384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.006138 (+ 38754us) service_pool.cc:224] Handling call
0709 12:25:03.476136 (+1469998us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:03.476419 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":943092480}
I20260709 12:25:03.520682 25675 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.444867 26003 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:03.519439 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.523877 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.524066 25981 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.483325 25962 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.527352 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.530158 18090 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:03.530261 18098 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:25:03.502614 25863 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:03.531802 25917 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:03.400195 18095 rpcz_store.cc:276] 0709 12:25:01.982134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.417072 (+434938us) service_pool.cc:224] Handling call
0709 12:25:02.417624 (+   552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:02.605792 (+188168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.605854 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:02.606452 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.716841 (+110389us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:02.716912 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.398705 (+681793us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:03.398806 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:03.398927 (+   121us) raft_consensus.cc:1685] finished
0709 12:25:03.398944 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.399426 (+   482us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":152285,"raft.queue_time_us":152861,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"spinlock_wait_cycles":81254272,"thread_start_us":109966,"threads_started":1,"wal-append.queue_time_us":237434}
I20260709 12:25:03.460053 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:03.528347 25824 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:03.439261 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58151.9us
I20260709 12:25:03.531994 25557 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.469424 25392 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.520073 18096 rpcz_store.cc:276] 0709 12:25:01.918970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.334042 (+415072us) service_pool.cc:224] Handling call
0709 12:25:02.349189 (+ 15147us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.469949 (+120760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.518660 (+1048711us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400410ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:03.519102 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272827776}
I20260709 12:25:03.497435 25787 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:03.504849 25813 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.473591 25941 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:03.481614 25976 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:03.558475 25925 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.550434 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:03.559617 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.528566 25962 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.524107 25675 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.504506 25793 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.578370 25793 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.582962 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1408) took 1135 ms. Trace:
I20260709 12:25:03.583742 18809 rpcz_store.cc:276] 0709 12:25:02.447946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.457464 (+  9518us) service_pool.cc:224] Handling call
0709 12:25:02.457897 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.480441 (+1022544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.582452 (+102011us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54005429d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:03.582821 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2348994304}
I20260709 12:25:03.591298 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1592) took 1227 ms. Trace:
I20260709 12:25:03.592103 18806 rpcz_store.cc:276] 0709 12:25:02.363353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.363939 (+   586us) service_pool.cc:224] Handling call
0709 12:25:03.590503 (+1226564us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b540059fe54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:03.590907 (+   404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232045824}
I20260709 12:25:03.593318 25948 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:03.594396 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.594645 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:03.597383 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:03.573598 25813 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.616032 25813 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.591440 25787 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.599826 18022 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:03.617321 18094 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:25:03.612284 18024 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:03.613045 18016 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:03.611116 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.617365 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 204049us
I20260709 12:25:03.615121 18017 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:03.618294 25787 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:03.605162 18170 raft_consensus.cc:606] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:03.650444 25819 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:03.651695 25819 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:03.660265 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:25:03.665479 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62631.8us
I20260709 12:25:03.676509 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:25:03.679476 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.687690 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:03.682935 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:03.696686 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.690979 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.722679 25940 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:03.723413 25940 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:03.723899 25940 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:03.725111 25998 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.726054 25998 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:03.725992 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.726732 25743 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.728814 25998 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.731158 25743 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:03.731853 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3125) took 1353 ms (client timeout 1515 ms). Trace:
W20260709 12:25:03.732491 18167 rpcz_store.cc:269] 0709 12:25:02.381458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.394213 (+ 12755us) service_pool.cc:224] Handling call
0709 12:25:03.731337 (+1337124us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54003016d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:03.731586 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1567708032}
W20260709 12:25:03.740151 25750 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.007s
I20260709 12:25:03.740887 25750 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:03.743723 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:03.754292 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:03.755971 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.757302 18090 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:03.747964 25813 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:25:03.769402 18017 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:03.779716 18063 leader_election.cc:400] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:03.784210 18063 leader_election.cc:403] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
W20260709 12:25:03.784950 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72674.8us
I20260709 12:25:03.781384 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.789322 18806 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:03.666306 25819 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:03.790539 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 133124us
I20260709 12:25:03.802350 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:03.807253 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:03.810487 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.806188 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:03.806747 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:03.816763 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 103935us
I20260709 12:25:03.790326 25793 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.818825 25918 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:03.818771 25922 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.819886 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:03.795666 25932 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.821100 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.821401 18093 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:03.827589 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:03.807934 25945 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:03.841866 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45060.2us
I20260709 12:25:03.848603 18806 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:03.851696 18024 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:03.850340 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.855644 18170 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:25:03.863765 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 242681us
I20260709 12:25:03.830453 25890 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:03.870368 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.853286 25962 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:03.871318 18167 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:25:03.869613 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.881073 25932 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.867045 25922 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:03.894042 25961 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.895038 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.898764 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:03.906420 18095 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:03.911543 18097 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:03.906903 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:03.914016 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 144854us
I20260709 12:25:03.915545 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:03.918301 26022 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:03.927266 18016 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:03.752974 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:03.939901 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:03.942570 25812 raft_consensus.cc:993] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:25:03.943225 25812 raft_consensus.cc:1081] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Signalling peer cab4a2f5d3ca48a2a803ee8ec3906e1d to start an election
I20260709 12:25:03.947829 26021 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:03.950706 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:03.951893 18019 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
W20260709 12:25:03.953260 25924 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.902s	user 0.008s	sys 0.019s
I20260709 12:25:03.953788 25924 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:03.956327 25924 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:03.962153 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:03.964768 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:03.970583 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68017.4us
I20260709 12:25:03.997658 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:03.989413 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:03.998963 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 94281.4us
I20260709 12:25:03.976999 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4033) took 1295 ms. Trace:
I20260709 12:25:03.969784 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:04.010156 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:04.003928 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 261346us
I20260709 12:25:04.018158 18015 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:25:04.018951 25945 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:04.018349 25890 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:04.020162 25890 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:04.020545 25945 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:04.021073 25945 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:04.012198 25812 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:04.021473 25890 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.022254 18095 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:04.022904 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:04.024009 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.025537 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.014365 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:03.970944 25888 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:04.032876 18092 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:25:04.012419 18810 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:03.972231 25978 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:04.045503 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.905316 25700 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.047982 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:04.048943 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.071710 18095 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:04.049302 25866 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:04.050693 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.051100 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.051744 25924 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:04.069672 25922 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:04.085592 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.086306 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.058483 25945 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:04.060523 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 181) took 64110 ms (client timeout 15000 ms). Trace:
W20260709 12:25:04.063529 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.904s (SENT)
W20260709 12:25:04.060725 25917 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.528s	user 0.013s	sys 0.002s
I20260709 12:25:04.003168 18018 rpcz_store.cc:276] 0709 12:25:02.681327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.681459 (+   132us) service_pool.cc:224] Handling call
0709 12:25:02.681952 (+   493us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:02.764871 (+ 82919us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.764923 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:02.791973 (+ 27050us) log.cc:844] Serialized 52 byte log entry
0709 12:25:03.074833 (+282860us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:03.453319 (+378486us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.454838 (+  1519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.976285 (+521447us) spinlock_profiling.cc:243] Waited 61.1 ms on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:03.976346 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:03.976429 (+    83us) raft_consensus.cc:1685] finished
0709 12:25:03.976543 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.976767 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":243753,"prepare.queue_time_us":297378,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":3355,"raft.queue_time_us":60419,"raft.run_cpu_time_us":473,"raft.run_wall_time_us":18864,"spinlock_wait_cycles":140433664,"thread_start_us":320455,"threads_started":2,"wal-append.queue_time_us":351308}
W20260709 12:25:04.063417 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 93833.5us
W20260709 12:25:04.064558 26026 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
W20260709 12:25:04.065711 25848 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.830s	user 0.004s	sys 0.019s
W20260709 12:25:04.066972 26029 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:25:04.093129 18094 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:25:04.067435 25392 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.066851 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.940s	user 0.051s	sys 0.021s Metrics: {"bytes_written":4604,"delete_count":0,"lbm_write_time_us":990,"lbm_writes_lt_1ms":5,"reinsert_count":0,"thread_start_us":106729,"threads_started":1,"update_count":169,"wal-append.queue_time_us":138575,"wal-append.run_cpu_time_us":966,"wal-append.run_wall_time_us":1042924}
I20260709 12:25:04.070465 18016 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:25:04.093720 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:04.092643 25974 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:04.050765 25960 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:04.108040 25931 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.698s	user 0.019s	sys 0.002s
W20260709 12:25:04.081544 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.108700 25931 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:04.109405 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.088995 18017 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:04.093187 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:04.078649 18099 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:04.103019 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:04.105401 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.100630 18064 leader_election.cc:400] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:04.104362 25974 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:04.110236 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.110005 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:04.101572 25941 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.539s	user 0.014s	sys 0.003s
I20260709 12:25:04.122227 18810 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:04.119747 18019 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:04.090454 25963 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.121968 26013 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.143567 18097 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:04.125988 18021 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:04.126338 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.258071
W20260709 12:25:04.126168 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 208365us
I20260709 12:25:04.126974 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.160036 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.129267 25974 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.164413 25974 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.130368 25945 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.165982 25932 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:04.166227 25413 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.784s	user 0.015s	sys 0.004s
I20260709 12:25:04.166833 25413 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.138238 25392 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.170449 18806 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:04.175252 25700 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:04.092379 25750 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.178506 25750 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:04.139868 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76511us
W20260709 12:25:04.139914 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93347.3us
I20260709 12:25:04.146970 18095 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 19.
I20260709 12:25:04.035653 25961 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.153307 25376 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:04.154186 18091 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:04.099840 25940 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.162268 18017 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:04.105849 25848 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:04.215852 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.216645 18093 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:04.115425 26022 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.117184 25787 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:04.221388 18023 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:04.217778 25392 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.277215 18099 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:04.223523 25413 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.310194 25392 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:04.311097 25392 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.295639 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:04.294454 26029 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:25:04.323086 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:04.323849 25750 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:04.324677 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.159457 26013 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.340939 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.219807 25963 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.324170 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.296752 25700 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:04.345629 18020 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:04.346290 18810 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:25:04.087492 17920 rpcz_store.cc:269] 0709 12:23:59.949685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.949979 (+   294us) service_pool.cc:224] Handling call
0709 12:25:04.060154 (+64110175us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:55.474313 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:55.835963 (+361650us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:55.856836 (+ 20873us) write_op.cc:435] Acquired schema lock
0709 12:24:55.859703 (+  2867us) tablet.cc:662] Decoding operations
0709 12:24:55.927369 (+ 67666us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:55.952316 (+ 24947us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:55.974384 (+ 22068us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:24:56.041778 (+ 67394us) tablet.cc:701] Row locks acquired
0709 12:24:56.041863 (+    85us) write_op.cc:260] PREPARE: finished
0709 12:24:56.042105 (+   242us) write_op.cc:270] Start()
0709 12:24:56.042302 (+   197us) write_op.cc:276] Timestamp: P: 1783599896042017 usec, L: 0
0709 12:24:56.042347 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:24:56.043313 (+   966us) log.cc:844] Serialized 1674 byte log entry
0709 12:24:57.096410 (+1053097us) op_driver.cc:464] REPLICATION: finished
0709 12:24:57.563047 (+466637us) write_op.cc:301] APPLY: starting
0709 12:24:57.563164 (+   117us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:57.618437 (+ 55273us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:57.618635 (+   198us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:57.862008 (+243373us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:57.862390 (+   382us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=8,mrs_lookups=0
0709 12:24:57.862455 (+    65us) write_op.cc:312] APPLY: finished
0709 12:24:57.876578 (+ 14123us) log.cc:844] Serialized 49 byte log entry
0709 12:24:58.042576 (+165998us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:58.070788 (+ 28212us) write_op.cc:454] Released schema lock
0709 12:24:58.084526 (+ 13738us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:58.199538 (+115012us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:58.203085 (+  3547us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":220426,"reported_tablets":31,"spinlock_wait_cycles":201216,"child_traces":[["op",{"apply.queue_time_us":466238,"apply.run_cpu_time_us":62441,"apply.run_wall_time_us":540472,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20871,"num_ops":4,"prepare.queue_time_us":164825,"prepare.run_cpu_time_us":50389,"prepare.run_wall_time_us":912307,"raft.queue_time_us":376792,"raft.run_cpu_time_us":37970,"raft.run_wall_time_us":236117,"replication_time_us":1017405,"spinlock_wait_cycles":30848,"thread_start_us":824771,"threads_started":5,"wal-append.queue_time_us":513530,"wal-append.run_cpu_time_us":39890,"wal-append.run_wall_time_us":2669700}]]}
I20260709 12:25:04.298076 25962 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:04.344885 25750 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.300379 25787 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:04.312196 25940 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:04.350432 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 166930us
I20260709 12:25:04.329371 25932 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:04.372752 18098 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:25:04.373412 18098 pending_rounds.cc:85] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 10
W20260709 12:25:04.361675 25948 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.707s	user 0.018s	sys 0.000s
I20260709 12:25:04.356976 25376 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:04.373716 18098 pending_rounds.cc:107] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:25:04.392510 18098 raft_consensus.cc:2889] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:04.392565 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87186.8us
W20260709 12:25:04.378575 25824 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.828s	user 0.014s	sys 0.006s
I20260709 12:25:04.370105 25787 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:04.397847 18096 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:25:04.401958 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 236218us
I20260709 12:25:04.408293 25392 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.415582 25963 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.389420 25700 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.388065 25376 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:04.416781 25931 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.365465 26022 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.387735 25750 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.408341 25812 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.434314 26030 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
W20260709 12:25:04.438481 25918 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.602s	user 0.013s	sys 0.003s
W20260709 12:25:04.437038 25897 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.016s	sys 0.026s
I20260709 12:25:04.433892 25376 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.418062 25932 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.456547 25980 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:04.455953 17914 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 8 to 16. New cstate: current_term: 16 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:04.457175 25897 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.016s	sys 0.026s
W20260709 12:25:04.467621 26033 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:25:04.475011 25961 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.477254 25413 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.482245 25700 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:04.525352 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.450608 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:04.482313 25392 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.486979 26022 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.526512 18808 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:25:04.489929 25750 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.404449 25962 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.435636 25890 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:04.516198 25812 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.541719 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:04.525174 25931 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.499745 25918 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
W20260709 12:25:04.529975 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.532881 25948 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.520884 18024 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:04.533429 25700 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.529887 25848 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:04.562640 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.566769 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:04.566757 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.577548 18806 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:04.577263 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:04.582636 17914 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 8 to 16, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 16 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:04.583891 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.584914 18018 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:25:04.586536 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 154016us
I20260709 12:25:04.589057 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.529683 25413 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.559156 25750 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.563410 25392 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.579461 25948 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.608124 25940 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:04.623001 25940 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:04.624396 25940 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.629979 18016 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:49542
I20260709 12:25:04.631018 25963 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:04.631815 25963 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.633966 25750 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:04.636018 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.601521 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:04.649747 18019 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:04.582430 25812 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:04.650885 26029 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:25:04.663926 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 222223us
I20260709 12:25:04.586604 25931 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.571192 25890 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.664937 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:04.575779 25700 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:04.679035 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 162976us
I20260709 12:25:04.666849 25948 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:04.683421 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.684650 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.665565 25918 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:04.694969 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.677639 25945 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.703645 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 185038us
W20260709 12:25:04.705770 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150169us
I20260709 12:25:04.709336 26036 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:04.691766 25962 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:04.710189 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70795.3us
I20260709 12:25:04.710202 26039 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:04.712311 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:04.712419 25918 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.715823 26047 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:04.710625 25392 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.716565 26047 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.717047 26047 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:04.718003 25866 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:04.719869 25932 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:04.720142 25974 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.008s
I20260709 12:25:04.714007 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.721751 18021 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:04.725911 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1596) took 1015 ms. Trace:
I20260709 12:25:04.726573 25945 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.726441 18815 rpcz_store.cc:276] 0709 12:25:03.646733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.669774 (+ 23041us) service_pool.cc:224] Handling call
0709 12:25:03.685249 (+ 15475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.708079 (+ 22830us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:03.934880 (+226801us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.661147 (+726267us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.661370 (+   223us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:04.661446 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.661610 (+   164us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.661770 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":396133,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":3278,"raft.queue_time_us":25786,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"spinlock_wait_cycles":27202432,"thread_start_us":917601,"threads_started":3,"wal-append.queue_time_us":715703}
I20260709 12:25:04.729024 25945 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.729138 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.703150 25812 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:04.730906 26013 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.730842 25918 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:04.704522 17914 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:04.732821 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.734704 25413 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:04.735575 25978 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.665s	user 0.017s	sys 0.000s
I20260709 12:25:04.746382 18806 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:04.737447 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:04.737202 25376 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.755558 25700 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:04.759799 26044 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 12:25:04.759728 26040 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.152s	user 0.005s	sys 0.000s
I20260709 12:25:04.748059 26048 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:04.769229 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:04.753152 25918 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:04.750059 25963 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.776883 17914 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:04.779208 25888 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:04.780037 25918 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.782722 18016 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:04.791955 18016 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:04.790340 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.794348 25961 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:04.795128 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.796849 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.796955 17914 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 13 to 14, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 14 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:04.798413 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54333.4us
I20260709 12:25:04.798779 25750 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:04.799166 17914 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 11 to 22, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 22 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:04.797667 18019 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:04.806916 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:04.807670 18017 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:04.812337 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.771464 25978 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.819891 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:04.822028 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1129) took 2030 ms. Trace:
I20260709 12:25:04.822619 18020 rpcz_store.cc:276] 0709 12:25:02.791367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.791584 (+   217us) service_pool.cc:224] Handling call
0709 12:25:02.794764 (+  3180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.379369 (+1584605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.819879 (+440510us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b5400268754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:04.821931 (+  2052us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3518971776}
I20260709 12:25:04.793408 25918 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:04.831887 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:04.837568 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.842746 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:04.845899 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 95358.8us
I20260709 12:25:04.858002 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.861639 25819 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:04.864236 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.867550 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:04.868999 18806 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:25:04.868417 18015 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:04.873441 18021 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:04.876277 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:04.877185 18024 raft_consensus.cc:2368] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:04.862221 25931 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:04.862361 26013 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:04.846549 25978 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.889021 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:04.890509 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:04.891010 18017 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:25:04.891798 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.883912 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:04.904697 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:04.908403 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 107970us
I20260709 12:25:04.910518 18096 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:25:04.896080 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.875010 25413 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:04.855468 25963 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:25:04.918035 18016 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:04.922710 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 210568us
W20260709 12:25:04.925168 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49967.1us
I20260709 12:25:04.926349 18019 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:04.926980 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.927871 18098 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:04.930888 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:04.931448 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.933780 25855 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:04.934502 25855 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:04.934515 18099 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:04.935623 25855 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.936193 25700 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.936873 25700 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:04.942863 26045 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.316s	user 0.000s	sys 0.003s
W20260709 12:25:04.946450 26041 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.320s	user 0.000s	sys 0.003s
I20260709 12:25:04.950040 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.950975 18095 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:04.952011 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:04.952759 18091 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:04.954221 25750 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:04.956739 25925 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:04.954278 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:04.958767 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.959426 18021 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:04.957176 25925 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:04.962038 25392 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:04.961676 25925 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.966380 25925 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:04.968111 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:04.970722 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.975474 26042 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.361s	user 0.001s	sys 0.003s
I20260709 12:25:04.975692 25978 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:04.976547 26052 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.977275 26052 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:04.978354 26035 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.311s	user 0.004s	sys 0.000s
I20260709 12:25:04.922309 26013 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:04.980787 26052 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.980588 26013 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:04.981560 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72146.1us
W20260709 12:25:04.983608 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.985105 25945 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:04.985984 25945 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.882968 25819 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.987619 25931 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:04.988571 25931 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.989290 25974 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:04.990127 25974 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:04.991752 18064 leader_election.cc:400] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:04.988739 25819 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.992865 18020 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:04.993667 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:04.995733 25932 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:04.996194 25890 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:04.996415 25932 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:04.932166 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:05.007057 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:05.009162 18090 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:25:05.007123 18091 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:25:05.008137 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:05.010668 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.011503 18090 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 23 because replica has last-logged OpId of term: 22 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 19.
I20260709 12:25:05.012384 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.008983 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.009043 18092 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:05.007550 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:05.013552 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.019325 18016 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:05.030434 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 166796us
I20260709 12:25:05.032356 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.033432 26062 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:05.034152 26062 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:04.991742 26048 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:05.035406 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:05.035894 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.036017 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:05.039898 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:05.020217 25787 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.590s	user 0.005s	sys 0.011s
I20260709 12:25:04.985163 25376 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:04.801321 17914 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 12 to 18, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 18 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:04.988654 25945 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.046888 18098 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 23 because replica has last-logged OpId of term: 22 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 19.
I20260709 12:25:05.056475 18021 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:05.017231 25978 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.050926 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:05.051924 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.053001 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:05.059526 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.060670 18094 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:05.061409 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:05.064028 18091 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:05.041481 25948 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.068393 18092 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:05.072971 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.074204 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:05.074317 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.075089 18016 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:05.077320 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.077793 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.042429 26048 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:05.085777 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.086833 18018 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:05.082717 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.088551 18099 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:05.094022 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.086185 25392 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:05.098910 25392 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.099406 25392 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:05.060814 26037 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:05.102312 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:05.128232 18094 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.067716 25855 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:05.068357 26043 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.103304 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.081607 25897 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260709 12:25:05.104875 26062 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.106040 25376 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:05.106184 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.107831 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.108466 18809 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:05.108637 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.109200 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:05.149200 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 130773us
I20260709 12:25:05.149200 18091 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:25:05.110138 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:05.110814 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.110692 26048 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.150955 18092 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:05.112155 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.113444 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.156144 18016 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:05.061854 25974 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:05.114559 25932 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:05.115703 26057 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:25:05.117094 25961 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.117872 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.122231 25940 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:05.122820 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.123179 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.124097 18167 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:25:05.122270 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:05.124804 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.114449 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.102312 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:05.130226 18093 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:05.124131 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.138260 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.056317 25787 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.147019 18015 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:05.154345 18021 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:05.111690 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:05.162277 25918 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.146953 26043 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.168309 25750 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:05.168835 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:05.154933 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.172664 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.155961 18097 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:05.174940 18092 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:05.082444 25948 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.069169 25978 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.144230 25812 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:05.232357 25812 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:05.233623 25812 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.234931 25700 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.177429 26048 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:05.176999 26047 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.148780 25855 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:05.184201 18020 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:05.133186 26037 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:05.204805 25940 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.237267 18169 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:05.253300 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:05.255031 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.255338 25813 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.256224 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.237502 25819 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:05.257601 25819 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.258592 25897 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.259331 25897 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.263159 25948 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.204433 18095 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:05.225183 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:05.266042 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.267822 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.268798 18167 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:05.272256 26043 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:05.274704 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.194837 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106590us
W20260709 12:25:05.227737 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72879.5us
I20260709 12:25:05.230993 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.231352 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:05.286249 18168 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.224510 18172 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:05.262619 25813 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.239718 18090 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:05.258080 25819 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:05.237636 18017 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:25:05.286305 18166 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:25:05.297506 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 128752us
W20260709 12:25:05.301946 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 252812us
I20260709 12:25:05.213652 18094 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:05.329322 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.334492 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:05.336273 18167 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:05.337167 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.333423 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.335311 18813 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:05.246639 26037 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:05.362185 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.360720 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.354980 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:05.361096 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:05.245128 26047 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.241237 25931 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:05.247167 25940 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.343044 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:05.360633 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:05.371382 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:05.373060 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:05.373190 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:05.251699 25787 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:05.364907 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.371238 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.241019 25918 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:05.213091 26052 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:05.212960 25961 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.202943 25974 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.150s
I20260709 12:25:05.405794 18169 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:05.413961 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.405014 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:05.416396 18806 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:05.377051 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:05.371564 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:05.392367 18022 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:05.409479 18023 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:05.410211 25700 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:05.410818 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46010.9us
I20260709 12:25:05.411093 25932 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:05.439949 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.423045 18172 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:05.333423 25925 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.379858 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:05.411507 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50990.7us
W20260709 12:25:05.411314 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123941us
I20260709 12:25:05.371802 26043 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:05.325265 25413 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
W20260709 12:25:05.446628 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66885.5us
I20260709 12:25:05.446532 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:05.446760 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 359718us
I20260709 12:25:05.418900 18167 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:05.455775 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.423905 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:05.462751 18170 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 21 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:25:05.446271 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.463176 18021 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:05.467423 26039 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:05.468390 18165 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:05.469508 26039 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.474341 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:05.470402 26039 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:05.501202 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56480.9us
I20260709 12:25:05.246127 25855 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.511507 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.377305 25948 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:05.350616 25866 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.100s
I20260709 12:25:05.423822 25700 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:05.441717 25932 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:05.458791 26043 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.523376 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:05.540917 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65439.7us
I20260709 12:25:05.532951 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:05.547778 18172 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:05.376909 25962 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.134s
I20260709 12:25:05.548187 18172 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:05.558319 18021 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:05.575429 18806 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:05.516345 25918 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:05.524256 25980 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.577503 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:05.536779 25700 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:05.532328 25897 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.239187 17914 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 7 to 10, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 10 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:05.475975 25392 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.536615 25787 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:05.541599 25932 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:05.584241 18166 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:05.525339 25948 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:05.580674 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:05.587009 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126410us
I20260709 12:25:05.556684 26043 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:05.616966 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 243949us
I20260709 12:25:05.618943 25813 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.615522 18172 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.630343 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.629256 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:05.631892 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51335.1us
W20260709 12:25:05.630919 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 119486us
I20260709 12:25:05.572355 26071 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:05.621757 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44397.5us
I20260709 12:25:05.620891 18169 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:05.581350 25918 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.631886 18172 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:05.623466 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:05.580116 26052 raft_consensus.cc:2764] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 12:25:05.629362 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 519273us
W20260709 12:25:05.662343 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80484.1us
I20260709 12:25:05.563346 26074 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.091s
I20260709 12:25:05.614735 25431 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:05.638127 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.633955 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:05.678112 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.625699 25974 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.678226 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.644992 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.682245 18173 raft_consensus.cc:2368] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22.
I20260709 12:25:05.628160 25948 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:05.619488 26073 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:05.624564 25750 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:05.623888 26063 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:05.725203 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 96041.8us
W20260709 12:25:05.726135 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48002.9us
W20260709 12:25:05.725414 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80533.5us
I20260709 12:25:05.632587 26064 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:05.640971 25824 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:05.728016 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:05.631656 25897 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:05.677006 25813 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:05.734263 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100384us
I20260709 12:25:05.636866 26071 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.644750 25918 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:05.741775 18809 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:25:05.742054 26076 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:05.743261 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:05.744155 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 235536us
W20260709 12:25:05.775399 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55921.7us
W20260709 12:25:05.745214 26065 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
I20260709 12:25:05.745146 26081 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:05.732306 25962 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:05.745975 26064 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:05.746167 25890 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:05.729972 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.766371 18023 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:05.620651 25980 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:05.799924 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70110.2us
I20260709 12:25:05.764964 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:05.726662 26073 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.728793 26063 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.808058 26073 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:05.808189 26063 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:05.808429 26064 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:05.752810 25675 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:05.811851 25675 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.812369 25675 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:05.734769 25978 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.783128 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:05.818548 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47131.4us
I20260709 12:25:05.783838 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:05.819540 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:05.821030 26085 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:05.635980 26078 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.144s
I20260709 12:25:05.800248 26081 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:05.768332 25813 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.823571 25813 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.823177 26081 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.797550 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:05.772559 26071 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.825932 26084 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:25:05.776960 26076 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.730894 25974 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:05.817631 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.781970 25376 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:05.799502 25890 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:05.832041 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67204.8us
I20260709 12:25:05.835646 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.796285 25962 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:05.797681 26077 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.838820 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:05.838891 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:05.846962 26066 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
I20260709 12:25:05.821687 26085 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:05.807078 25961 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:05.848064 26080 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:25:05.834299 26076 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:06.124984 26086 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.125926 25978 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:06.126276 26039 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:06.126442 25978 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.126914 26039 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:05.859513 26067 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260709 12:25:05.806313 25824 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:06.127883 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 330413us
I20260709 12:25:06.128309 25824 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.128754 25824 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.774744 25897 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.130055 25961 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:06.130625 25961 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:06.132007 25897 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.131688 25961 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:06.132733 26072 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.497s	user 0.001s	sys 0.000s
I20260709 12:25:06.140933 26063 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:06.115819 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 277056us
I20260709 12:25:06.116672 26070 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.114360 26048 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.867s	user 0.006s	sys 0.025s
W20260709 12:25:06.123203 26083 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.381s	user 0.006s	sys 0.015s
I20260709 12:25:06.118055 26074 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:06.126942 25978 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.133852 25890 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.134404 26013 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.135231 25974 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:05.834576 25812 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:05.850133 25413 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:25:06.137089 26052 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.140161 25431 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:06.142356 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1186) took 1023 ms. Trace:
I20260709 12:25:06.143491 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1148) took 1026 ms. Trace:
I20260709 12:25:06.220207 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.220598 18018 rpcz_store.cc:276] 0709 12:25:05.116774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.145541 (+ 28767us) service_pool.cc:224] Handling call
0709 12:25:05.147383 (+  1842us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:05.190422 (+ 43039us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:05.190483 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:05.208689 (+ 18206us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.418575 (+209886us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:05.418772 (+   197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.418860 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.132012 (+713152us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:06.143135 (+ 11123us) raft_consensus.cc:1685] finished
0709 12:25:06.143161 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.143418 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":282,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120,"spinlock_wait_cycles":27640192,"thread_start_us":178212,"threads_started":1,"wal-append.queue_time_us":284631}
I20260709 12:25:06.143694 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1185) took 1024 ms. Trace:
I20260709 12:25:06.146323 25888 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:25:06.148173 26037 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.707s	user 0.004s	sys 0.024s
I20260709 12:25:06.148273 25948 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.129478 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:06.150640 26085 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:05.831578 25945 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:25:06.118693 25940 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.009s
I20260709 12:25:06.248556 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.119663 18172 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:06.162427 26053 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:06.162497 26070 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.170482 25392 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.226022 25888 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:06.172725 25824 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.242512 25675 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.265654 25945 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.266269 25945 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.266912 25675 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:06.219589 18173 rpcz_store.cc:276] 0709 12:25:05.119247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.121730 (+  2483us) service_pool.cc:224] Handling call
0709 12:25:06.142057 (+1020327us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:06.142274 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993814912}
I20260709 12:25:06.116385 25962 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:06.221907 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:06.231608 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 426743us
W20260709 12:25:06.187506 25819 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.817s	user 0.013s	sys 0.016s
W20260709 12:25:06.234405 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 105006us
I20260709 12:25:06.308705 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:06.232537 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 396968us
I20260709 12:25:06.223567 18164 rpcz_store.cc:276] 0709 12:25:05.118895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.119387 (+   492us) service_pool.cc:224] Handling call
0709 12:25:05.119830 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:05.120338 (+   508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.120371 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:05.121026 (+   655us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.323925 (+202899us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:05.323981 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.324073 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.143341 (+819268us) raft_consensus.cc:1685] finished
0709 12:25:06.143367 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.143494 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19848,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":2706,"thread_start_us":202386,"threads_started":1,"wal-append.queue_time_us":410810}
W20260709 12:25:06.227227 25932 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.000s
W20260709 12:25:06.251271 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.253700 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:06.318605 18809 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:06.218896 25931 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.258850 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 427349us
I20260709 12:25:06.250236 25890 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.186058 25813 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.118835 25376 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:06.258280 18018 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:06.266566 25824 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:06.182900 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 399135us
I20260709 12:25:06.268296 25980 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.268586 25945 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.259739 25940 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.323329 25932 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.360972 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:06.361562 26070 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.274825 18015 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:06.331151 25931 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.362735 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:06.362975 25940 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.233209 25974 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:06.364009 25974 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.365113 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.223060 18020 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:06.246052 26084 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:06.306996 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.306964 18022 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:06.229856 26052 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.227708 26013 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.330629 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:06.384317 18094 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.331597 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:06.307934 25700 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.681s	user 0.013s	sys 0.000s
I20260709 12:25:06.383818 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:06.347486 25948 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.237772 26037 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.390790 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:06.391667 18810 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.393028 26037 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.393267 26052 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.394536 26063 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.395536 26013 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:06.352319 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.322471 25819 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:06.356089 26043 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.715s	user 0.011s	sys 0.000s
I20260709 12:25:06.358471 25888 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.260978 26053 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.364288 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.406682 26053 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:06.407202 18097 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:06.364411 25931 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.224419 25918 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.407989 25888 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:06.410328 25918 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.366819 18017 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:06.410825 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:06.368667 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:06.366849 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:06.412467 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45732.8us
W20260709 12:25:06.372678 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64084.3us
I20260709 12:25:06.380505 25974 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.326454 25787 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:06.380872 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.371457 26084 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:06.367352 26048 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:06.427349 25431 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:06.391093 25932 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.351795 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:06.352319 26071 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 12:25:06.322132 25963 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:06.385730 26073 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.577s	user 0.011s	sys 0.002s
I20260709 12:25:06.419008 18810 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:06.429529 26084 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.381740 25376 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:06.431103 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62541.3us
W20260709 12:25:06.435438 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 83747us
I20260709 12:25:06.453416 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:06.438227 18815 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:06.451726 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:06.454783 25813 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.396754 25700 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.465468 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:06.473425 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.438705 25431 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.442483 25897 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:06.483034 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:06.486886 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70543.2us
I20260709 12:25:06.487257 18810 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:25:06.489049 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:06.426410 26043 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.491382 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:06.427735 25978 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.510439 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:06.433610 25787 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.522770 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.504235 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:06.460090 25813 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:06.506100 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.450503 26091 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.541421 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:06.499696 18094 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:06.525799 18806 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:25:06.485394 25431 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.479491 26048 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.451432 26063 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:25:06.548964 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.451988 26071 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.523926 18018 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:06.545531 18093 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:06.548233 18095 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:06.456028 26073 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.459434 25963 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.537930 18019 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:06.522049 26075 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:06.537390 25932 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.558332 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:06.559420 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49104us
I20260709 12:25:06.578637 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:06.561457 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:06.554665 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:06.554368 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:06.580084 18097 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.532892 25812 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.580183 18018 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.503821 25675 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:25:06.544807 26053 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:25:06.591595 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102657us
I20260709 12:25:06.593006 18016 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:06.595569 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:06.574673 26092 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:25:06.600564 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77821.3us
I20260709 12:25:06.605160 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:06.571990 26075 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.612470 18019 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
W20260709 12:25:06.618088 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73924.1us
I20260709 12:25:06.581672 25392 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:06.599494 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.534s (SENT) [suppressed 12 similar messages]
I20260709 12:25:06.553690 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:06.620708 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:06.637681 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:06.638653 18017 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.573967 26094 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:25:06.642086 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:06.559930 26048 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.597221 26089 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:06.585943 26053 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:06.643860 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51373.1us
I20260709 12:25:06.571787 25962 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.607733 26070 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:06.657151 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:06.626420 26075 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.633190 25813 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.623827 25945 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:06.598105 25413 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:25:06.641983 26085 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:06.600330 25897 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:25:06.586625 26076 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:06.670495 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116811us
I20260709 12:25:06.586308 25932 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.688998 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:06.689769 18098 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 25
W20260709 12:25:06.690621 26088 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.349s	user 0.003s	sys 0.000s
I20260709 12:25:06.673524 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:06.673954 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.674949 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.712894 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:06.713907 18810 raft_consensus.cc:2368] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17.
I20260709 12:25:06.671247 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:06.707491 18173 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:06.670437 26063 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.713073 18019 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:06.680064 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:06.684697 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:06.693392 26097 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 12:25:06.695201 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.665506 26070 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.697953 26085 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.700286 18098 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:06.700623 26071 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:06.701442 26099 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:25:06.693568 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:06.664215 26086 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:25:06.672470 18780 leader_election.cc:400] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22.
I20260709 12:25:06.706559 18809 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:25:06.714386 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72423.1us
I20260709 12:25:06.734350 18780 leader_election.cc:403] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:25:06.655422 26053 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:06.735272 26102 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:06.735998 26102 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.736474 26102 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:06.737171 25932 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.670380 26062 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:06.738140 26062 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.740650 26062 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.735052 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:06.700208 25962 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.747093 25700 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.746905 25962 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:06.686818 25392 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:06.664672 25819 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.756583 26071 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22.
I20260709 12:25:06.756984 25962 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:06.758313 25962 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.762068 25963 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:06.731040 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:06.755553 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84456us
I20260709 12:25:06.767398 26070 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.732591 26037 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.768388 18023 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:06.737948 26063 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.659998 26090 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 12:25:06.720808 18017 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:06.723176 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:06.735023 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135571us
I20260709 12:25:06.785254 18097 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:06.786286 18813 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:06.786422 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 124987us
I20260709 12:25:06.787146 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:06.789505 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54538.3us
I20260709 12:25:06.733145 25961 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:06.789440 18022 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:25:06.798854 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114260us
I20260709 12:25:06.799926 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:06.749281 26048 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.804234 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:06.768894 25978 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.781198 26089 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:06.808068 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:06.813165 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:06.815635 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:06.788203 25812 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:06.817272 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.793181 25392 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:06.803856 26043 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.835536 25392 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.838680 25945 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:06.809016 25700 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:25:06.812350 26091 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:25:06.847860 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:06.849798 25866 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:06.873844 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:06.864037 25431 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.854869 25787 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.809633 26037 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:06.880450 26037 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.880941 26037 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.861397 17988 leader_election.cc:400] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17.
I20260709 12:25:06.863903 25750 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:06.882030 17988 leader_election.cc:403] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:25:06.882155 25750 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:06.883172 25750 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:06.884222 26063 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.884601 25750 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.824223 26073 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.884379 26084 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.863834 26085 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:06.824873 26116 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:06.812964 25932 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:06.815997 25974 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:06.841065 26053 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.897061 25812 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:06.899372 26053 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.900456 25932 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.900928 25932 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:06.894544 26086 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:06.902544 26086 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.905064 26086 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:06.906991 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:06.908182 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46964.2us
I20260709 12:25:06.875466 18018 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:06.825991 18064 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:06.826555 26095 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:06.879920 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3145) took 1703 ms. Trace:
I20260709 12:25:06.886209 18815 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:06.867030 25945 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:06.876854 25675 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:06.889971 26076 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:06.889878 26084 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:06.891407 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:06.894567 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.936264 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.865125 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.910068 18064 leader_election.cc:403] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:25:06.878533 25866 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:06.912842 25890 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.005s
I20260709 12:25:06.947482 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.953505 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:06.911386 18165 rpcz_store.cc:276] 0709 12:25:05.175384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.469355 (+293971us) service_pool.cc:224] Handling call
0709 12:25:05.486770 (+ 17415us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:05.619350 (+132580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.619385 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:05.688467 (+ 69082us) log.cc:844] Serialized 52 byte log entry
0709 12:25:06.188635 (+500168us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:06.624095 (+435460us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:06.826965 (+202870us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b1800346360. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf
0709 12:25:06.827171 (+   206us) log.cc:844] Serialized 14 byte log entry
0709 12:25:06.844765 (+ 17594us) log.cc:844] Serialized 14 byte log entry
0709 12:25:06.844987 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.845081 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.878450 (+ 33369us) raft_consensus.cc:1685] finished
0709 12:25:06.878467 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.878593 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42911,"prepare.queue_time_us":245693,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":104,"raft.queue_time_us":53959,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":8091,"spinlock_wait_cycles":254791296,"thread_start_us":585824,"threads_started":2,"wal-append.queue_time_us":599299}
I20260709 12:25:06.948513 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.917423 26120 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17.
W20260709 12:25:06.957090 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65855.1us
I20260709 12:25:06.911178 26095 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.962452 26095 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.966456 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.968086 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.954974 26116 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.970563 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.971031 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.974192 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.975414 26102 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.976521 18099 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:06.978206 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:06.979178 18094 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:06.979456 26084 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.980150 26084 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:06.954377 25812 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:06.921773 26085 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:25:06.984436 25888 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:06.985132 25888 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.985612 25888 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.967991 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:06.980840 18094 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:07.005925 18096 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:06.987671 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:07.008399 18092 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 19.
I20260709 12:25:07.007193 18090 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
W20260709 12:25:07.011989 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 186330us
I20260709 12:25:07.023571 18099 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:07.012887 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.014474 25431 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:07.014688 26075 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:07.016098 26119 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
W20260709 12:25:07.016607 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 312940us
I20260709 12:25:07.016677 26116 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:07.017235 26120 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:07.018821 26102 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:07.018932 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.019415 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:06.951786 26086 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.034888 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 233930us
I20260709 12:25:07.050005 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.054097 18092 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:07.038642 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:07.021557 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:07.041162 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:07.042022 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:07.056619 18810 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:07.047950 18097 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:07.035609 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:07.050104 18165 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:07.057511 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:07.004774 26063 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.058209 18813 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:25:07.059401 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.064041 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:07.068617 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.040325 26043 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
W20260709 12:25:07.087260 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.069525 18099 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:07.075047 18171 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:07.064785 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:07.032065 25974 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.079979 18024 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:07.079813 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.088645 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:07.091974 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.094841 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.096000 18097 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:07.100217 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:07.100517 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:07.068838 25700 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:07.079200 26094 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:07.079859 25978 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:07.113021 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.116242 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.125434 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.087534 26120 consensus_queue.cc:1243] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:25:07.164479 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.165304 18172 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:07.165431 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.162021 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.166841 18165 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.115150 25978 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.167459 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.177539 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.178117 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.178866 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.179833 18168 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.182636 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.080888 26063 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.181475 18167 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:07.183609 18166 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:07.080291 26086 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:07.160200 26097 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:07.200531 18171 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:07.100576 25974 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:07.117671 26123 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:07.231223 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 143214us
I20260709 12:25:07.236675 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:07.242476 25932 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:07.246124 26123 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:07.247702 26123 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.175925 25978 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:07.250691 18812 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:07.250986 26086 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:07.160225 26121 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:07.252122 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.257335 18090 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:25:07.245457 25932 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:07.251505 25888 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.266487 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.275684 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: [22.20-22.20]
I20260709 12:25:07.267871 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.278775 18169 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:07.269479 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.282258 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.283383 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:07.289836 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.290467 18166 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:07.290621 18171 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.292905 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:07.293776 26127 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:25:07.294911 26075 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:07.296360 26117 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.244s	user 0.000s	sys 0.000s
I20260709 12:25:07.299710 26089 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:07.303133 25974 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:07.304569 26116 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:07.311450 25888 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:07.258203 26121 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:07.316934 26131 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:07.316850 25945 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.271095 26084 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:07.319492 26063 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:07.323029 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.325064 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:07.327268 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 241566us
W20260709 12:25:07.328549 26128 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.087s	user 0.000s	sys 0.002s
I20260709 12:25:07.335505 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.337200 18092 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:07.337374 18165 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:25:07.340283 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 275586us
I20260709 12:25:07.387694 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:07.351487 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.353978 18168 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:07.389539 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.339171 26091 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:07.394282 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.395360 18165 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:07.351290 26089 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:07.287770 26094 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:07.385546 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.350533 25932 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.316039 26037 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:07.396715 18170 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:07.403035 18169 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:25:07.405360 18167 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:07.420055 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.359386 26116 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:07.422967 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:07.374796 26131 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.428181 18020 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:07.377727 26121 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:07.423864 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:07.436336 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:07.438414 26116 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.444983 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:07.446126 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.435892 26131 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.454571 18172 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:07.456184 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.459038 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.480569 18173 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:07.484454 18015 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.481549 18024 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:07.486791 18020 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:07.489189 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.411967 26075 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:07.491398 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67634.5us
I20260709 12:25:07.503286 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.398329 25962 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.090s
I20260709 12:25:07.506620 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:07.514520 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.443953 26134 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:07.406093 26091 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.512486 18172 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260709 12:25:07.484226 26095 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.002s
I20260709 12:25:07.511272 26094 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:07.502660 26120 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:07.569141 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.444795 25824 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260709 12:25:07.496055 25978 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.573053 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:07.414319 25932 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.534201 26037 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:07.549927 25945 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:07.589146 18020 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:07.586256 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:07.616722 18811 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:25:07.559674 26116 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:07.635810 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:07.572371 25750 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.687s	user 0.016s	sys 0.000s
I20260709 12:25:07.637125 18172 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:25:07.633040 17910 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.528s	user 0.000s	sys 0.000s
I20260709 12:25:07.585179 26134 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:07.511752 26075 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:07.641528 26124 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:25:07.569340 26095 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:07.662393 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.665158 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.672739 18098 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:07.679852 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.656739 26123 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:07.686913 18164 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:07.688817 18171 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
W20260709 12:25:07.695588 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 209960us
I20260709 12:25:07.689201 26074 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:07.747479 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.752918 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:07.800314 18092 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:07.796432 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:07.802171 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 241229us
I20260709 12:25:07.803606 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.812749 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:07.813952 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.814947 18171 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:07.822324 18022 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:07.760115 25974 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.826443 18091 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:07.703636 26123 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.834901 18093 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:25:07.787823 26074 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:07.799664 25700 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.791797 26069 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:07.838775 18093 pending_rounds.cc:85] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
W20260709 12:25:07.842284 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 335763us
I20260709 12:25:07.862336 18093 pending_rounds.cc:107] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:25:07.842514 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.846662 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.862663 18093 raft_consensus.cc:2889] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:07.864924 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:07.865873 18094 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:07.757050 26086 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 12:25:07.867449 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:07.861941 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 289018us
I20260709 12:25:07.872689 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:07.877255 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.875327 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:07.873788 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69590.9us
I20260709 12:25:07.879164 18021 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:07.882812 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:07.883360 18097 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:07.882932 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.883785 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.885649 18096 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.888571 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:07.889645 18094 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:07.890345 18172 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:07.891485 18094 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:07.892808 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.893376 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:07.893599 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.898998 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:07.841252 25888 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:07.862262 26069 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.920574 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.923791 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:07.907841 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.912427 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:07.859620 26120 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:07.916580 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:07.892035 25819 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:07.933763 18091 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:07.934521 18094 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2217b6329873485ead60419dc5acbff0"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:25:07.936555 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:07.939881 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:07.926534 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:07.945562 18094 raft_consensus.cc:606] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:07.946835 18094 raft_consensus.cc:614] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:07.947602 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:07.949023 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:07.941572 18015 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:07.960577 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.963102 18171 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:07.928784 26069 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:07.914475 25932 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:07.935685 26048 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:07.749226 26134 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.841682 25974 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:07.897527 25750 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:25:07.898481 25700 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:07.999231 18021 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:25:08.039024 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.039611 18097 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:08.004367 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.053032 18024 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:07.943871 25819 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:08.014581 26147 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 12:25:08.011193 26074 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.012226 26086 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:08.080549 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:08.081640 18021 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:08.082836 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:08.014564 26144 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.265s	user 0.000s	sys 0.000s
I20260709 12:25:08.098832 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.099952 18168 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:08.101171 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:07.932037 25888 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.103242 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:08.103700 25888 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.015324 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:08.023716 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.104980 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.038120 26161 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:07.957156 25978 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:08.107952 18093 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:08.109400 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 236781us
I20260709 12:25:08.032421 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.113586 26063 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:08.035802 25866 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:08.038223 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:07.969542 26047 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:08.000765 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:08.126853 18811 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:08.054704 18094 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:08.010613 26091 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:07.935415 26131 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.109313 26132 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:08.031088 18815 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 15. Current term is 16. Ops: []
W20260709 12:25:08.075755 26121 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.594s	user 0.010s	sys 0.004s
I20260709 12:25:08.091991 26086 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:08.092087 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.038897 26156 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:08.093986 26163 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:08.094542 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:08.095623 26145 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:25:08.043067 26053 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:08.107143 18168 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.113642 25932 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:08.064872 25819 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:08.119467 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:08.121657 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:08.125119 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.126406 26047 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:08.129129 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:08.073983 26095 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.131808 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.133545 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.172025 18165 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:08.089154 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.140008 18016 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:08.030288 18170 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.620s	user 0.016s	sys 0.000s
I20260709 12:25:08.159637 18021 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:08.161546 18093 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:25:08.178102 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59196.3us
I20260709 12:25:08.177947 18170 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:08.180171 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:08.118484 26089 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.693s	user 0.015s	sys 0.000s
I20260709 12:25:08.201553 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.203271 18171 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:08.210992 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:08.212245 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 307109us
I20260709 12:25:08.213150 18021 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:25:08.213860 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.214906 18170 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:08.216504 18096 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:08.218473 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:08.218783 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.175096 26160 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:25:08.169378 26047 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.228276 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:08.234476 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.123041 26120 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.168955 26053 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:08.237195 26172 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:25:08.175707 26091 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:08.261796 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.263530 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:08.265091 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.249256 18016 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:25:08.140141 26121 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.254750 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.269375 18165 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:08.211465 25824 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:25:08.234721 26145 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:25:08.224069 26123 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:08.247316 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
W20260709 12:25:08.287577 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61891.6us
W20260709 12:25:08.287957 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 158962us
I20260709 12:25:08.283211 18171 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:08.287969 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.277120 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.289987 18170 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:08.290373 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.291496 18171 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:08.292735 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.294296 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.295707 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.296290 18165 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:08.288537 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:08.168608 26163 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.308095 18023 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:08.293211 25750 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.009s
I20260709 12:25:08.322501 25750 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:08.322980 25750 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:08.324028 25750 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.325446 25750 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:08.314407 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.328439 18165 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:08.357565 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.359186 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.360311 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.361342 18165 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:08.337965 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49527.6us
I20260709 12:25:08.221136 26161 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.086s
I20260709 12:25:08.369622 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.341935 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:08.370283 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3160) took 1243 ms. Trace:
I20260709 12:25:08.167182 26156 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.271912 26168 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:08.273617 26091 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:08.351289 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 248109us
I20260709 12:25:08.352270 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.311954 26123 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.313534 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.216215 26089 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.370882 18171 raft_consensus.cc:606] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:08.402474 18171 raft_consensus.cc:614] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:08.361258 18167 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:08.370865 18173 rpcz_store.cc:276] 0709 12:25:07.134422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.177791 (+ 43369us) service_pool.cc:224] Handling call
0709 12:25:08.369998 (+1192207us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54003e2bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:08.370197 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628366208}
I20260709 12:25:08.330166 26053 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.398939 25700 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.302704 26086 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.340809 25824 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:25:08.422438 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:08.423717 26181 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:08.402451 26169 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:25:08.400076 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:08.399531 26159 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:08.430338 18170 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:08.430425 26159 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:08.431000 26159 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:08.432184 18170 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:08.418452 26163 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.444424 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:08.443068 18170 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:25:08.454285 18811 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:08.368558 26172 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:08.383693 26158 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:08.467753 18165 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:08.460647 26089 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.475437 18166 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:08.368270 26048 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.107s
W20260709 12:25:08.437685 26116 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.795s	user 0.016s	sys 0.000s
I20260709 12:25:08.498327 18171 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:08.454859 26154 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:08.494501 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:08.488245 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43913.9us
I20260709 12:25:08.468453 26074 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.407454 26123 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:08.499951 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 232548us
I20260709 12:25:08.496222 25700 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:08.448975 26174 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.121s
I20260709 12:25:08.506258 26158 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:08.520341 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.520308 18019 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:25:08.526913 18019 pending_rounds.cc:85] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 13
I20260709 12:25:08.527228 18019 pending_rounds.cc:107] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:25:08.527550 18019 raft_consensus.cc:2889] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:08.489261 26069 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.532133 26156 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.533095 26123 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:08.534773 26158 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.488947 26172 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.516682 25700 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.502044 26047 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.533771 26123 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:08.516906 26074 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.511835 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:08.554102 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 284338us
W20260709 12:25:08.555016 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.551074 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.569418 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:08.570676 18091 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:08.575286 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:08.520380 25974 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:08.576252 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:08.510608 26154 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:08.583071 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 239641us
I20260709 12:25:08.565145 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:08.562731 25700 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:08.520220 26183 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
W20260709 12:25:08.589308 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77614.6us
I20260709 12:25:08.584376 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.562731 26123 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:08.610400 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:08.617965 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:08.619024 18096 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:08.614738 18024 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:08.584553 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:08.621172 26192 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.621418 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:08.622036 26192 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.626874 18171 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:08.627197 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:08.628114 18096 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:08.631331 18099 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:08.591738 25824 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.593403 26048 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:25:08.641103 26043 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:08.635010 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50595.3us
I20260709 12:25:08.635069 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:08.641883 26043 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:08.642755 18024 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.644625 26043 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.645159 18778 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:08.649987 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.650827 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:08.653136 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.608280 25974 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:08.655858 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1207) took 1077 ms. Trace:
I20260709 12:25:08.659252 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:08.607554 26121 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:08.605895 26185 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:08.665736 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:08.660714 18173 rpcz_store.cc:276] 0709 12:25:07.578246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.411033 (+832787us) service_pool.cc:224] Handling call
0709 12:25:08.466404 (+ 55371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.466690 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:08.483806 (+ 17116us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.655170 (+171364us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.655324 (+   154us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:08.655439 (+   115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.655552 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.655750 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16561,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":26277,"thread_start_us":144543,"threads_started":1}
I20260709 12:25:08.668015 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:08.675103 18815 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:08.686395 18024 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:08.687225 18811 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:25:08.687705 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.689085 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:08.651100 26168 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:08.694105 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:08.695303 18812 raft_consensus.cc:2368] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:08.658692 26053 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:08.664674 26134 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.029s
W20260709 12:25:08.703177 26180 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:25:08.702862 26123 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:08.703817 26177 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:25:08.703699 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:08.704198 26183 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:08.685693 26091 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:08.706743 26172 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.708725 18024 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:08.717355 26154 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.761308 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3174) took 1233 ms. Trace:
W20260709 12:25:08.704900 26182 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:25:08.728986 26090 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.804988 18166 rpcz_store.cc:276] 0709 12:25:07.527760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.539894 (+ 12134us) service_pool.cc:224] Handling call
0709 12:25:08.313324 (+773430us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:25:08.760954 (+447630us) spinlock_profiling.cc:243] Waited 75 ms on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:08.761200 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175762176}
I20260709 12:25:08.698395 26168 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:08.763744 25819 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.589s	user 0.013s	sys 0.002s
I20260709 12:25:08.682966 26178 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.129s
W20260709 12:25:08.829892 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.813798 26123 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:08.869668 26147 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:25:08.828455 26168 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:08.828534 26043 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:08.687939 26048 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.113s
I20260709 12:25:08.942652 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1182) took 1382 ms. Trace:
I20260709 12:25:08.943241 18809 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:08.847232 25819 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:08.944080 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 265736us
I20260709 12:25:08.828311 26053 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
I20260709 12:25:08.830341 26090 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:08.943372 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 257054us
I20260709 12:25:08.933780 26121 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:08.762857 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 4.586s	user 0.110s	sys 0.102s Metrics: {"cfile_cache_miss":8,"cfile_cache_miss_bytes":26410,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":489620,"dirs.run_cpu_time_us":916,"dirs.run_wall_time_us":5101,"lbm_read_time_us":49825,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":21,"lbm_write_time_us":313753,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":2,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":6,"peak_mem_usage":270284,"reinsert_count":0,"thread_start_us":414590,"threads_started":2,"update_count":200}
I20260709 12:25:08.952090 18018 rpcz_store.cc:276] 0709 12:25:07.560744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.560986 (+   242us) service_pool.cc:224] Handling call
0709 12:25:07.561503 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.561861 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:07.583651 (+ 21790us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.681546 (+1097895us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b18001059a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:25:08.681900 (+   354us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.942041 (+260141us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.942122 (+    81us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:08.942222 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.942336 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.942552 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100909,"prepare.queue_time_us":962179,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":138,"raft.queue_time_us":252578,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":28726,"spinlock_wait_cycles":8013184,"thread_start_us":434081,"threads_started":2}
I20260709 12:25:08.950002 26091 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:08.950567 26043 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:08.953457 26172 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:08.972173 18022 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:08.984565 26190 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.987430 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260709 12:25:08.999922 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
I20260709 12:25:08.993243 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3028) took 1035 ms. Trace:
I20260709 12:25:09.002490 18018 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:25:08.956238 26069 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:08.970232 26047 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.003255 18015 rpcz_store.cc:276] 0709 12:25:07.958295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.958562 (+   267us) service_pool.cc:224] Handling call
0709 12:25:07.959105 (+   543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.959460 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:08.718140 (+758680us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.982403 (+264263us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.992608 (+ 10205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.992711 (+   103us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:08.992798 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.992913 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.993145 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":733001,"prepare.queue_time_us":804419,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":10263,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":3023,"thread_start_us":261406,"threads_started":1}
I20260709 12:25:08.997114 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:09.049760 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 384082us
I20260709 12:25:09.051003 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.052768 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:08.995368 26048 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:25:09.057119 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:09.057943 18018 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.059752 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:09.056515 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60256.7us
I20260709 12:25:09.075923 25819 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.008038 26178 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:09.089785 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.064795 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.014190 26074 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:09.096625 26159 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.665s	user 0.017s	sys 0.000s
I20260709 12:25:08.987512 26091 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.098495 18018 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:08.980859 26121 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:09.107744 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56848.9us
I20260709 12:25:08.998970 26190 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:08.997112 26172 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.093463 26178 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:09.092007 26185 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:09.157384 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92679.3us
I20260709 12:25:09.164551 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:09.096236 25819 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:09.120957 26048 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:25:09.176698 26208 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:25:09.178821 26091 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.179910 26090 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.118125 26187 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.135174 26209 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:09.134985 26159 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.103859 26043 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.180941 26053 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:09.177608 26172 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.203986 18170 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:25:09.205719 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3054) took 2105 ms (client timeout 1738 ms). Trace:
W20260709 12:25:09.206230 18164 rpcz_store.cc:269] 0709 12:25:07.096840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.103734 (+  6894us) service_pool.cc:224] Handling call
0709 12:25:09.201414 (+2097680us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54001b5154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:09.201609 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1223442688}
I20260709 12:25:09.183559 26215 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.208799 26209 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:09.210350 26043 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:09.221793 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4225) took 2266 ms. Trace:
I20260709 12:25:09.222326 18098 rpcz_store.cc:276] 0709 12:25:06.949678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.971671 (+ 21993us) service_pool.cc:224] Handling call
0709 12:25:06.972056 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.673387 (+701331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.215996 (+1542609us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540035d9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:09.216272 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1610818944}
I20260709 12:25:09.211567 26159 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.212559 26121 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.188052 25974 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:09.249923 26037 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:09.193986 26187 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.239302 26185 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.205624 26178 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.215449 26215 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:09.207728 26053 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:09.269553 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105079us
I20260709 12:25:09.255144 26037 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:09.271171 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:09.269027 25700 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.638s	user 0.010s	sys 0.007s
I20260709 12:25:09.261714 26215 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.221215 25819 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:09.275444 26204 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.194s	user 0.001s	sys 0.001s
I20260709 12:25:09.284710 18064 leader_election.cc:400] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:09.285745 18064 leader_election.cc:403] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:25:09.282539 26037 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.296442 26075 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.296872 26037 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.288677 26053 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:09.299695 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1211) took 1686 ms. Trace:
I20260709 12:25:09.300221 18168 rpcz_store.cc:276] 0709 12:25:07.613336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.463556 (+850220us) service_pool.cc:224] Handling call
0709 12:25:08.761189 (+297633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.761498 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.761544 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.761611 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:08.761660 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.761744 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.299431 (+537687us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54002742d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:09.299608 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":683098496}
I20260709 12:25:09.294564 25819 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:09.304901 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 252228us
I20260709 12:25:09.305907 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:09.307296 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.274852 26221 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:09.308279 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 249070us
I20260709 12:25:09.319597 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.321592 26091 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:09.309036 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.322809 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.307843 25750 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:09.323887 25750 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.327569 26187 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.311532 26194 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:09.312135 25866 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:09.313017 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.313899 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:09.316702 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.322993 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.321517 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.332093 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.310643 26209 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:09.284674 26178 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:09.352974 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:09.305058 26095 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20.
I20260709 12:25:09.338260 25962 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:09.338481 26209 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:09.354645 18167 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.361266 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.365088 25962 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:09.277556 25700 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.372238 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:09.325855 26134 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:25:09.377233 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71488us
I20260709 12:25:09.367499 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:09.381740 18168 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:09.382817 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.382174 26199 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.388062 26227 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:09.387955 25962 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.388599 26199 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.365897 26205 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:09.332242 26190 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.350402 26224 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:09.391294 26199 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:09.380396 25866 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:25:09.371709 26209 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:09.384634 25787 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:25:09.382946 26134 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:09.400571 26214 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:09.401160 26214 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:09.401505 26214 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:09.403333 18168 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:09.406064 26091 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:09.408597 26165 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.394s	user 0.004s	sys 0.000s
I20260709 12:25:09.387220 18137 leader_election.cc:400] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:09.410172 18137 leader_election.cc:403] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:09.384187 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:09.408255 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:09.355130 25819 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:09.430323 25819 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.430780 26215 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.435667 26131 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.435859 26215 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:09.442112 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 55043.6us
I20260709 12:25:09.422248 26137 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:09.442761 26153 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.432s	user 0.004s	sys 0.000s
I20260709 12:25:09.443210 26137 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:09.422729 26205 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:09.446077 26205 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:09.446224 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3171) took 2021 ms. Trace:
I20260709 12:25:09.447386 26205 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.449102 26192 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.444427 26137 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.449472 26205 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:09.457242 26196 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.506s	user 0.001s	sys 0.000s
I20260709 12:25:09.462059 26159 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:09.462812 26159 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.465399 26159 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:09.424880 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.363289 25974 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.428261 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1175) took 2006 ms. Trace:
I20260709 12:25:09.428743 26168 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.431262 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3320) took 1543 ms. Trace:
I20260709 12:25:09.492585 18093 rpcz_store.cc:276] 0709 12:25:07.887502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.891502 (+  4000us) service_pool.cc:224] Handling call
0709 12:25:09.431039 (+1539537us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54002968d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:09.431188 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":418692992}
I20260709 12:25:09.433060 25819 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.417691 26210 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:09.442303 26075 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.469780 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4091) took 1354 ms. Trace:
I20260709 12:25:09.505532 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:09.470239 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:09.446806 18169 rpcz_store.cc:276] 0709 12:25:07.424719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.424892 (+   173us) service_pool.cc:224] Handling call
0709 12:25:07.425293 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.425787 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:07.425844 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.458291 (+ 32447us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.802125 (+343834us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:07.802177 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.802255 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.441779 (+1639524us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:09.445739 (+  3960us) raft_consensus.cc:1685] finished
0709 12:25:09.445769 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.446131 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2760,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":25519872,"thread_start_us":230964,"threads_started":1,"wal-append.queue_time_us":424980}
I20260709 12:25:09.471216 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:09.510257 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123085us
I20260709 12:25:09.507972 18023 rpcz_store.cc:276] 0709 12:25:08.115176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.216614 (+101438us) service_pool.cc:224] Handling call
0709 12:25:08.223074 (+  6460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.318203 (+ 95129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.469518 (+1151315us) spinlock_profiling.cc:243] Waited 80.9 ms on lock 0x7b54000ff854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:09.469705 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186074880}
I20260709 12:25:09.513834 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:09.514606 18018 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:25:09.472245 26132 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.424625 25700 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:09.475356 26200 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.524s	user 0.005s	sys 0.000s
I20260709 12:25:09.475780 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:09.527042 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:09.527604 18092 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:25:09.477465 26183 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.527935 18093 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:09.473801 18091 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:25:09.526346 18812 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:09.480763 18099 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:09.481184 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.482359 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:09.530543 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.526261 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:09.531415 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:09.484202 18097 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:09.415767 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.489108 18019 rpcz_store.cc:276] 0709 12:25:07.421792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.422094 (+   302us) service_pool.cc:224] Handling call
0709 12:25:07.443535 (+ 21441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.528258 (+1084723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.427939 (+899681us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001d0954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:09.428182 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2475300224}
W20260709 12:25:09.432694 25824 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.795s	user 0.014s	sys 0.000s
I20260709 12:25:09.497877 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:09.533013 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:09.536892 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:09.537796 18092 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.506937 18021 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 12:25:09.418967 25750 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.517952 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:09.544713 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 129093us
I20260709 12:25:09.522351 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:09.545133 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.545878 18171 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:09.521344 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:09.545921 18172 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:09.546932 18018 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:09.547025 18023 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:09.548677 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:09.550531 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:09.551453 18091 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:25:09.545799 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.525085 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:09.521498 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.478181 26174 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:09.559733 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.560791 26209 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.562106 26190 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.562769 26190 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.563184 26214 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.564180 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.566469 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:09.567204 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.562109 26043 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:09.567991 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.568747 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.569111 26190 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.569344 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.570140 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.570297 26210 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:09.571070 26210 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:09.571420 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:09.572348 18016 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:09.573882 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:09.574993 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.575205 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.575942 18806 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:09.530189 25866 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:25:09.577198 26173 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.577905 26176 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:09.577975 26173 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.578704 26176 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:09.579861 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.579782 26176 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.580009 26134 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.581554 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:09.581791 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.571440 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3021) took 1713 ms. Trace:
I20260709 12:25:09.511912 26075 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:09.582973 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.583343 26075 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.575058 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:09.584084 18810 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:09.585016 26170 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:09.518985 26159 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:09.585675 26170 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:09.586313 26169 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:09.586249 26159 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.587070 26169 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:09.588449 26170 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.588166 26169 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.588923 26159 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.589545 18808 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
W20260709 12:25:09.594264 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48614.4us
I20260709 12:25:09.596014 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:09.596856 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:09.597015 18091 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:09.607785 18092 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:09.509982 26192 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:09.621867 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.651269 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:09.582911 18022 rpcz_store.cc:276] 0709 12:25:07.858296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.858534 (+   238us) service_pool.cc:224] Handling call
0709 12:25:09.571192 (+1712658us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400108bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:09.571343 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2494634880}
I20260709 12:25:09.565629 26214 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.594726 25700 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:09.675740 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:09.996145 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1245) took 2201 ms. Trace:
W20260709 12:25:09.999688 26189 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.131s	user 0.007s	sys 0.003s
W20260709 12:25:10.000053 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.597754 26134 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:10.003276 25888 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:10.004555 26218 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:10.006007 26084 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.006596 26084 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.007661 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3024) took 2109 ms. Trace:
I20260709 12:25:10.008100 18020 rpcz_store.cc:276] 0709 12:25:07.898213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.900458 (+  2245us) service_pool.cc:224] Handling call
0709 12:25:07.900958 (+   500us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.204635 (+303677us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.204691 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.227602 (+ 22911us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.583261 (+355659us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:08.583478 (+   217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.584039 (+   561us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.998853 (+1414814us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:10.007232 (+  8379us) raft_consensus.cc:1685] finished
0709 12:25:10.007260 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.007556 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":137258,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":21773568,"thread_start_us":583577,"threads_started":2,"wal-append.queue_time_us":427974}
I20260709 12:25:10.005199 26218 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:09.996639 18090 rpcz_store.cc:276] 0709 12:25:07.795128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.795276 (+   148us) service_pool.cc:224] Handling call
0709 12:25:07.812174 (+ 16898us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.069746 (+257572us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.069787 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.102691 (+ 32904us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.344862 (+242171us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:08.344938 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.995677 (+1650739us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b540035edd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:09.995738 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.995793 (+    55us) raft_consensus.cc:1685] finished
0709 12:25:09.995809 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.996073 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":399,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":23580,"spinlock_wait_cycles":591141120,"thread_start_us":241678,"threads_started":1,"wal-append.queue_time_us":407297}
I20260709 12:25:10.011716 26218 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.689594 26168 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:09.733496 26197 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.635s	user 0.000s	sys 0.003s
I20260709 12:25:10.009598 26084 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:09.590730 26183 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.015451 26154 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.016089 26154 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:10.017441 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 425137us
I20260709 12:25:10.018013 26159 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:10.018579 26159 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.019615 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3087) took 1993 ms. Trace:
I20260709 12:25:10.020052 18170 rpcz_store.cc:276] 0709 12:25:08.026254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.506333 (+480079us) service_pool.cc:224] Handling call
0709 12:25:09.201613 (+695280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.204729 (+  3116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.019393 (+814664us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54001b5150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:10.019549 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1597852416}
I20260709 12:25:10.021072 26159 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:10.024296 26089 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.024902 26089 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.026014 25787 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:10.026713 25787 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.029268 25787 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:10.029549 26226 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.667s	user 0.000s	sys 0.005s
I20260709 12:25:10.034045 26089 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:10.036147 26154 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:09.852568 26158 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s	user 0.006s	sys 0.008s
W20260709 12:25:09.855530 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.018s	sys 0.025s
W20260709 12:25:10.039433 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.018s	sys 0.025s
W20260709 12:25:10.039394 26158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.081s	user 0.006s	sys 0.008s
W20260709 12:25:10.040539 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 708486us
I20260709 12:25:10.045244 25962 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:09.990151 26037 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.693s	user 0.016s	sys 0.008s
I20260709 12:25:09.982813 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:09.988786 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:09.990041 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 393283us
I20260709 12:25:09.526412 26091 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:25:09.990876 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 421592us
W20260709 12:25:10.076538 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87926.4us
I20260709 12:25:10.077770 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:09.991014 26173 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:09.991452 26227 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.991912 26121 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:09.591857 25824 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:09.530441 26132 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:10.050760 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59217.2us
I20260709 12:25:10.066629 25700 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:10.070237 18064 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:09.577853 25866 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:10.086448 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:10.083005 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:10.096994 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.103713 26132 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:10.099555 26215 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.105173 26183 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:10.105089 26215 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.085247 26227 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.106266 26227 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:10.107685 26215 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:10.054442 26047 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.101588 26043 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 12:25:10.108527 26047 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.051359 25675 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:10.110076 25675 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.109951 25700 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.085175 26073 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:10.111514 26047 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:10.085177 26090 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.112352 26158 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
W20260709 12:25:10.112887 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.114024 26158 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:10.114570 26158 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:10.115053 26158 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:10.115151 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:10.116369 26159 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:10.116644 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.117565 26073 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:10.112461 25700 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:10.118355 26194 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:10.119573 18096 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:25:10.119448 26194 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:25:10.120671 26199 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:10.121122 26199 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.122165 26199 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:10.123350 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.124251 25750 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:10.124727 25750 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.126278 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:10.126013 25750 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.129679 26173 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:10.131294 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:10.132306 26172 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.132483 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.133394 26172 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:10.133852 26172 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:10.134250 26172 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:25:10.134681 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.134992 25392 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:10.135753 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 52937.1us
I20260709 12:25:10.135612 25392 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.137009 26132 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.137292 25932 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:10.138002 25932 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:10.137063 26091 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.139309 26091 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.139851 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:10.107967 26190 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 12:25:10.143435 18806 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:10.118372 26073 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:10.145049 26073 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:10.147895 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51037us
I20260709 12:25:10.133538 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.150772 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.154587 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:10.099932 25866 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.159202 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.160037 18813 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:10.162823 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.123683 26199 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:10.165980 18099 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:25:10.165069 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:10.171288 18090 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:10.174775 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.176549 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:25:10.179993 18166 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:25:10.156234 26121 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.159389 26190 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:10.182708 18099 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:10.210556 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.183427 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:10.147586 26037 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.083s
I20260709 12:25:10.163746 26090 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:10.228670 18810 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:10.210371 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.137586 26132 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.235101 18093 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:25:10.225162 18091 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:10.166744 26170 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:10.228012 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.229777 18090 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:10.231534 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.232366 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:10.224792 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:10.239323 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.240489 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:10.189159 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.182822 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.184517 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.261473 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.279542 18096 raft_consensus.cc:606] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.280102 18096 raft_consensus.cc:614] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:10.258863 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.281973 18167 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:10.286944 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.304011 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:10.301594 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88138.1us
I20260709 12:25:10.139258 25932 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 22.20, Last appended by leader: 20, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.303436 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.308641 18099 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:10.309420 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.211490 26089 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.289047 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.223006 26190 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.305522 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:10.241346 26205 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.787s	user 0.017s	sys 0.006s
I20260709 12:25:10.333644 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.334730 18167 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:10.182524 26070 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.063s
W20260709 12:25:10.272750 26151 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.462s	user 0.009s	sys 0.019s
I20260709 12:25:10.288708 26170 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:10.309432 26228 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:10.317077 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 238981us
I20260709 12:25:10.314451 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.355401 18093 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:10.229030 26090 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:10.322449 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.367400 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.331032 26190 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:10.324805 18090 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:10.328099 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.320731 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.337002 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.341101 18020 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:10.379477 18024 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:25:10.310108 26075 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.726s	user 0.010s	sys 0.012s
I20260709 12:25:10.229909 25897 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.089s
I20260709 12:25:10.353637 18094 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 12:25:10.371291 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48930.8us
I20260709 12:25:10.374248 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:10.342249 26178 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.986s	user 0.017s	sys 0.008s
I20260709 12:25:10.344340 26043 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:25:10.353768 26151 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.562s	user 0.010s	sys 0.020s
W20260709 12:25:10.388399 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60413us
I20260709 12:25:10.409888 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.354553 26228 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.277643 25866 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.386557 25974 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.426369 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:10.431018 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59845.7us
I20260709 12:25:10.428748 18808 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:10.432912 18173 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:10.437446 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.370110 26090 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:10.440374 18064 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.593s (SENT)
I20260709 12:25:10.322587 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.311272 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:10.341574 26214 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:25:10.430547 18094 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:10.459266 18170 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:25:10.468139 18094 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:10.467700 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:10.459928 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 277478us
I20260709 12:25:10.463764 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.332787 26089 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.472683 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.475469 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.476892 18170 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:10.424671 26174 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:25:10.481146 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54883.1us
I20260709 12:25:10.490026 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.490876 18020 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:10.331956 26121 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:10.466051 26229 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.296s	user 0.000s	sys 0.002s
I20260709 12:25:10.455586 26215 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.458487 26228 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:10.456204 26043 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:10.490404 18094 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.507536 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.421023 25700 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.068s
I20260709 12:25:10.508776 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:10.517174 18094 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:10.460530 26205 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:10.482012 25866 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:10.519865 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.417507 26075 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.437150 26168 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:25:10.453954 26181 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
W20260709 12:25:10.518880 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.519609 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.520629 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.538190 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:10.547992 18172 raft_consensus.cc:2368] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:10.545367 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.545607 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:10.554795 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.556172 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.559441 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:10.508178 26246 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.561395 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:10.562805 26242 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 12:25:10.518678 26205 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:10.566411 26231 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20260709 12:25:10.570235 26181 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:10.548087 25819 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:10.571254 26245 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:25:10.600703 18091 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:10.571780 26132 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:10.547297 26235 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:25:10.572979 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1218) took 1206 ms. Trace:
I20260709 12:25:10.609903 18164 rpcz_store.cc:276] 0709 12:25:09.367033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.367209 (+   176us) service_pool.cc:224] Handling call
0709 12:25:09.367739 (+   530us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.368295 (+   556us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.368378 (+    83us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.382360 (+ 13982us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.497881 (+115521us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:09.497932 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.498012 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.572560 (+1074548us) raft_consensus.cc:1685] finished
0709 12:25:10.572588 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.572895 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":716,"raft.queue_time_us":3692,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":8175,"thread_start_us":103488,"threads_started":1,"wal-append.queue_time_us":674040}
W20260709 12:25:10.575937 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.576342 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.614455 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.615204 18168 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 16.
I20260709 12:25:10.576581 18021 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:10.519814 26248 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:10.579845 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.519016 26123 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:25:10.571318 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:10.616911 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:10.618924 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.643253 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:25:10.619889 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 144541us
I20260709 12:25:10.596659 26205 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:10.638868 18172 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:25:10.596081 26181 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.510357 26170 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.653481 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.657406 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.578280 26174 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:25:10.658525 18164 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:10.658560 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:10.582908 26255 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.659309 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:10.654023 18064 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.679s (SENT)
I20260709 12:25:10.606652 25819 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.664736 18168 raft_consensus.cc:2376] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20. Re-sending same reply.
W20260709 12:25:10.662001 26243 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:25:10.664019 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4224) took 3714 ms. Trace:
W20260709 12:25:10.671350 18064 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.724s (SENT)
I20260709 12:25:10.672107 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.668318 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.679441 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:10.679903 18064 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.736s (SENT)
I20260709 12:25:10.680227 18815 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:10.669207 18095 rpcz_store.cc:276] 0709 12:25:06.949336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.949513 (+   177us) service_pool.cc:224] Handling call
0709 12:25:06.962921 (+ 13408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.007463 (+ 44542us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:07.308067 (+300604us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b1800075700. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:25:07.378000 (+ 69933us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.568583 (+190583us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.986761 (+1418178us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b1800075700. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:25:09.439997 (+453236us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:09.440609 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.876206 (+435597us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:09.926619 (+ 50413us) log.cc:844] Serialized 14 byte log entry
0709 12:25:09.927267 (+   648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.927439 (+   172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.663688 (+736249us) raft_consensus.cc:1685] finished
0709 12:25:10.663714 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.663932 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92646,"prepare.queue_time_us":645965,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":107,"raft.queue_time_us":58543,"raft.run_cpu_time_us":579,"raft.run_wall_time_us":63982,"spinlock_wait_cycles":16420992,"thread_start_us":692963,"threads_started":4,"wal-append.queue_time_us":1027753,"wal-append.run_cpu_time_us":3087,"wal-append.run_wall_time_us":1420797}
I20260709 12:25:10.680418 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:10.682638 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.692952 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.694104 18164 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
W20260709 12:25:10.696368 18064 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.747s (SENT)
I20260709 12:25:10.697607 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:10.698987 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.665086 26154 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:10.665912 26159 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:10.733711 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 162487us
I20260709 12:25:10.677603 26221 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:10.682873 26168 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:10.689355 26132 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:10.784776 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 300712us
I20260709 12:25:10.775240 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.778332 18091 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:10.786583 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:10.693662 26257 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.821519 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1437) took 1297 ms. Trace:
W20260709 12:25:10.825095 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 181933us
I20260709 12:25:10.826253 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:10.819192 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:10.771821 26159 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:10.757331 26154 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:10.664259 26043 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.830034 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.822084 18811 rpcz_store.cc:276] 0709 12:25:09.524048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.524244 (+   196us) service_pool.cc:224] Handling call
0709 12:25:09.575466 (+ 51222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.575880 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.575913 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.576496 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.852165 (+275669us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:09.852246 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.852382 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.790289 (+937907us) raft_consensus.cc:1685] finished
0709 12:25:10.821249 (+ 30960us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.821443 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2990,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":275089,"threads_started":1,"wal-append.queue_time_us":646811}
W20260709 12:25:10.841782 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 145534us
W20260709 12:25:10.845628 18064 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.798s (SENT)
W20260709 12:25:10.848723 18064 leader_election.cc:336] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.865s (SENT)
I20260709 12:25:10.849360 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:10.852674 18064 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.925s (SENT)
W20260709 12:25:10.853848 18064 leader_election.cc:336] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.004s (SENT)
I20260709 12:25:10.854956 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.823513 26094 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:10.860055 26185 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.860984 26185 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.865064 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:10.838219 26159 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:10.869604 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.842275 25974 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:10.873442 18164 raft_consensus.cc:2376] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20. Re-sending same reply.
W20260709 12:25:10.876741 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50617us
I20260709 12:25:10.842813 26246 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.876842 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:10.838639 26154 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:10.816345 26168 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:10.883811 18064 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.028s (SENT)
I20260709 12:25:10.884409 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.784283 26181 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:10.874936 26249 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20260709 12:25:10.855036 25897 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.885386 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:10.875576 26236 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.328s	user 0.002s	sys 0.000s
I20260709 12:25:10.820844 26257 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.875182 26260 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:25:10.903605 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 133274us
I20260709 12:25:10.886528 26246 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.918969 26246 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:10.920692 26073 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.775s	user 0.007s	sys 0.007s
I20260709 12:25:10.906872 25897 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.923406 18164 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:10.929036 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:10.890933 26215 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:10.889770 26256 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:10.891569 26255 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
W20260709 12:25:10.940625 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.915894 25974 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:10.938901 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 152423us
I20260709 12:25:10.900925 26266 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.068s
I20260709 12:25:10.962672 26266 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:10.963217 26266 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:10.964270 26266 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 22.19, Last appended by leader: 19, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:10.932384 26181 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:10.975049 18064 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.102s (SENT)
I20260709 12:25:10.975595 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:10.974515 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:10.978377 18064 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.110s (SENT)
I20260709 12:25:10.978960 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:10.981292 26070 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:10.939594 25700 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
W20260709 12:25:10.988224 18064 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.136s (SENT)
I20260709 12:25:10.988812 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:10.995878 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.937623 26073 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:10.965783 26266 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:10.942827 25932 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:25:10.956449 26256 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.007594 25392 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.008628 26256 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.009112 25897 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:11.009763 26252 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260709 12:25:11.010087 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3186) took 1497 ms. Trace:
I20260709 12:25:10.967518 26257 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:11.021737 18064 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.180s (SENT)
W20260709 12:25:11.022608 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.022770 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:11.010509 18169 rpcz_store.cc:276] 0709 12:25:09.512660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.516551 (+  3891us) service_pool.cc:224] Handling call
0709 12:25:09.517058 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:09.520016 (+  2958us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.520072 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:09.531049 (+ 10977us) log.cc:844] Serialized 52 byte log entry
0709 12:25:09.907905 (+376856us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:10.016679 (+108774us) log.cc:844] Serialized 14 byte log entry
0709 12:25:10.128134 (+111455us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.128231 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.009727 (+881496us) raft_consensus.cc:1685] finished
0709 12:25:11.009763 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.010008 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113435,"prepare.queue_time_us":90999,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":123332,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":16054,"thread_start_us":233417,"threads_started":1,"wal-append.queue_time_us":583310}
I20260709 12:25:10.970044 26278 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:10.982861 26248 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:10.993198 26261 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.303s	user 0.001s	sys 0.001s
I20260709 12:25:10.985526 25819 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:11.040015 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:11.044118 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:11.045538 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 102664us
I20260709 12:25:10.993507 26260 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:11.049944 26254 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:11.044885 18097 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:11.085668 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:11.107381 18815 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:11.079665 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:11.028270 25392 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.082544 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:11.085548 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63901.7us
W20260709 12:25:11.111940 18064 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.240s (SENT)
I20260709 12:25:11.029501 26151 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:11.117475 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.120287 26172 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.986s	user 0.006s	sys 0.009s
W20260709 12:25:11.120874 26190 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.707s	user 0.012s	sys 0.000s
I20260709 12:25:11.121699 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:11.117437 26248 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:11.127095 18094 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:11.122380 26073 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:11.124441 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:11.086905 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:11.124703 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.086452 25750 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:11.078491 26258 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
W20260709 12:25:11.070849 26265 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:25:11.074468 26274 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:11.074123 26132 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:11.029870 26256 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.100s
I20260709 12:25:11.132274 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:11.158983 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72209us
I20260709 12:25:11.236195 18024 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:25:11.238497 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.067658 26273 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:11.129215 26248 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.129745 26190 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:11.148358 26228 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.621s	user 0.013s	sys 0.000s
W20260709 12:25:11.156343 25866 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.611s	user 0.000s	sys 0.011s
I20260709 12:25:11.070376 26168 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:11.242000 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 117641us
I20260709 12:25:11.111847 18019 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:11.339643 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:11.157120 26260 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:11.354996 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:11.336510 18017 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:11.316465 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.358410 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:11.365494 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.123687 25392 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:11.431069 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76231.5us
I20260709 12:25:11.186931 26073 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:11.335481 25819 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.315248 26273 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:11.455852 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97594.6us
I20260709 12:25:11.455060 18024 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:25:11.124588 26173 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.994s	user 0.007s	sys 0.009s
I20260709 12:25:11.434283 26132 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.348271 26228 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.457294 26073 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:11.491957 18024 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:25:11.477679 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1198) took 1162 ms. Trace:
I20260709 12:25:11.352078 25812 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:11.498142 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.464155 26227 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.357s	user 0.012s	sys 0.006s
I20260709 12:25:11.492983 18022 rpcz_store.cc:276] 0709 12:25:10.315270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.318269 (+  2999us) service_pool.cc:224] Handling call
0709 12:25:10.345650 (+ 27381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.506997 (+161347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.507037 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.507601 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.778103 (+270502us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:10.778161 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.778273 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.477364 (+699091us) raft_consensus.cc:1685] finished
0709 12:25:11.477392 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.477594 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3452,"raft.queue_time_us":75987,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":28857,"thread_start_us":353502,"threads_started":2,"wal-append.queue_time_us":343629}
I20260709 12:25:11.433768 26043 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.454357 25866 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.483182 25819 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.558933 18093 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 21. Ops: []
I20260709 12:25:11.454859 26274 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:11.552204 26227 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:11.542137 26090 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.058s	user 0.011s	sys 0.005s
I20260709 12:25:11.544191 25392 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.542362 25812 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.564389 25866 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.544024 18166 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:11.470634 26199 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.295s	user 0.017s	sys 0.000s
W20260709 12:25:11.568971 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 229521us
I20260709 12:25:11.575702 18170 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:25:11.488830 26158 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.373s	user 0.016s	sys 0.003s
I20260709 12:25:11.576902 18168 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:25:11.486439 26273 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:11.079321 26271 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.392s
W20260709 12:25:11.578732 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.581708 25812 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.468581 26284 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:11.143503 26172 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.506891 26073 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:11.588393 18098 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:11.560781 26228 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:11.572176 26205 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.910s	user 0.015s	sys 0.001s
I20260709 12:25:11.543659 26248 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:11.554702 26190 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.552042 26279 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:11.516882 26173 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:11.573777 26090 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.508589 26132 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.623564 18022 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:11.592142 26199 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.575562 26043 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.593370 26284 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:11.587713 26121 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.027s	sys 0.068s
W20260709 12:25:11.629369 26154 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.749s	user 0.013s	sys 0.000s
I20260709 12:25:11.639528 26090 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.640242 18023 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:11.625486 26279 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:11.643662 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.588744 26158 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.613857 26205 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:11.637230 26121 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.027s	sys 0.069s
W20260709 12:25:11.685227 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 162672us
I20260709 12:25:11.634573 26284 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:11.707051 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.723049 18780 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.695s (SENT) [suppressed 32 similar messages]
I20260709 12:25:11.645586 25819 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.634292 26199 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.677193 26279 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:11.761922 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172979us
I20260709 12:25:11.666270 26154 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.766263 25392 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.780687 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:11.759918 26248 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:11.763394 26290 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.760381 25812 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:11.788008 26209 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:11.788717 26210 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:11.801007 26290 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.801390 25812 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.792284 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:11.794239 26281 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:11.797511 26181 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.824s	user 0.008s	sys 0.003s
I20260709 12:25:11.775733 25392 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.772696 25750 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:11.819245 25750 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.798964 26248 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.799770 26273 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.835630 26290 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:11.835675 26273 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.835803 25866 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.842067 25750 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:11.800406 26227 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:11.801584 26210 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.802384 26172 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:11.790692 26251 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.208s	user 0.006s	sys 0.005s
I20260709 12:25:11.805243 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:11.856667 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51565.9us
I20260709 12:25:11.860945 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:11.764683 26121 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:11.818248 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1204) took 1244 ms. Trace:
I20260709 12:25:11.870899 18019 rpcz_store.cc:276] 0709 12:25:10.573745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.490231 (+916486us) service_pool.cc:224] Handling call
0709 12:25:11.570956 (+ 80725us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.571418 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.571460 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.579572 (+  8112us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.763966 (+184394us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:11.764022 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.764100 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.817927 (+ 53827us) raft_consensus.cc:1685] finished
0709 12:25:11.817955 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.818157 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2012,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":4035,"thread_start_us":152787,"threads_started":1,"wal-append.queue_time_us":214215}
I20260709 12:25:11.873209 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:11.848592 18169 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
W20260709 12:25:11.874019 26232 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.643s	user 0.011s	sys 0.011s
I20260709 12:25:11.876255 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:11.877476 26264 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.986s	user 0.008s	sys 0.005s
I20260709 12:25:11.858722 26227 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:11.866220 26215 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:25:11.855403 26286 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.882416 26286 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.884035 25974 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:11.884534 25974 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:11.861742 18094 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:11.831719 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:11.887645 18172 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
W20260709 12:25:11.872076 26266 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.838s	user 0.014s	sys 0.000s
I20260709 12:25:11.876402 26275 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:11.835630 25392 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.885438 25974 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.865310 26248 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:11.890820 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.899920 25974 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:11.900843 25675 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.902225 25675 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:11.883113 26283 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:25:11.909557 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3198) took 1926 ms. Trace:
I20260709 12:25:11.909780 26132 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.910159 18165 rpcz_store.cc:276] 0709 12:25:09.982749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.982929 (+   180us) service_pool.cc:224] Handling call
0709 12:25:09.983331 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.100213 (+116882us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.100271 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.100964 (+   693us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.388071 (+287107us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:10.388147 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.395667 (+  7520us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.789941 (+1394274us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:11.909158 (+119217us) raft_consensus.cc:1685] finished
0709 12:25:11.909186 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.909470 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116234,"raft.queue_time_us":136589,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":14511488,"thread_start_us":205246,"threads_started":1,"wal-append.queue_time_us":350642}
I20260709 12:25:11.892201 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3126) took 1005 ms. Trace:
I20260709 12:25:11.912019 18172 rpcz_store.cc:276] 0709 12:25:10.887088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.831274 (+944186us) service_pool.cc:224] Handling call
0709 12:25:11.892123 (+ 60849us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:11.918200 18094 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:11.918361 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:11.923192 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 200203us
I20260709 12:25:11.926903 26199 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.927691 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:11.928512 18165 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:11.926743 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:11.930544 18018 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:11.929711 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:11.933588 18165 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:11.920626 26121 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:11.934923 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.937677 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:11.910969 26275 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:11.956429 25812 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:11.961764 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:11.962725 18165 raft_consensus.cc:2376] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24. Re-sending same reply.
I20260709 12:25:11.963892 26158 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:11.964885 26205 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:11.945792 26159 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.062s	user 0.013s	sys 0.003s
I20260709 12:25:11.965905 26159 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:11.967392 18024 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:11.968777 26159 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:11.970618 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3049) took 1318 ms. Trace:
I20260709 12:25:11.972419 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:11.920679 26073 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:11.971324 18024 rpcz_store.cc:276] 0709 12:25:10.652153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.509864 (+857711us) service_pool.cc:224] Handling call
0709 12:25:11.526925 (+ 17061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.967907 (+440982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.969648 (+  1741us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400109ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:11.970507 (+   859us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1012252416}
I20260709 12:25:11.981427 18018 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:11.963888 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:11.977413 18169 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:11.970440 26289 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.982417 26275 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:11.988590 26247 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.473s	user 0.037s	sys 0.081s
I20260709 12:25:11.991355 26279 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.005951 26154 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:12.006000 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.006994 26075 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:12.014573 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.033859 26275 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:12.003852 26121 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.036283 26121 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.037030 26121 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.016204 26174 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.039883 26174 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.044116 26075 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:12.049948 26174 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.018797 26073 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.045038 26075 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.024772 18165 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:12.024299 25675 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.029266 26289 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.050474 26210 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:12.023211 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98242.5us
I20260709 12:25:12.016155 18019 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
W20260709 12:25:12.057255 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.051201 26297 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.062086 18018 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:25:12.065589 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.055550 26174 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.083633 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.085206 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1464) took 1908 ms. Trace:
I20260709 12:25:12.085832 18810 rpcz_store.cc:276] 0709 12:25:10.177051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.177210 (+   159us) service_pool.cc:224] Handling call
0709 12:25:10.177621 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.229222 (+ 51601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.084896 (+1855674us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.085132 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":116809600}
I20260709 12:25:12.069502 26267 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.060689 25675 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.056529 26301 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:25:12.101243 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45562 (request call id 33) took 1939 ms. Trace:
I20260709 12:25:12.102020 18096 rpcz_store.cc:276] 0709 12:25:10.161774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.176343 (+ 14569us) service_pool.cc:224] Handling call
0709 12:25:12.100934 (+1924591us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.101094 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228437888}
I20260709 12:25:12.105372 25675 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.098246 26267 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.108744 26267 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:12.110556 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.116019 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.122999 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.123971 18165 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:12.124660 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1230) took 1142 ms. Trace:
I20260709 12:25:12.125092 18165 rpcz_store.cc:276] 0709 12:25:10.983111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.122585 (+1139474us) service_pool.cc:224] Handling call
0709 12:25:12.124568 (+  1983us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.126441 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.127203 18165 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.068666 26297 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:12.131911 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72958.9us
I20260709 12:25:12.084666 26174 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.112752 26134 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:12.131569 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0020854109998254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
I20260709 12:25:12.154755 18165 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:12.157347 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.156118 26279 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.159660 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.163029 18165 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:12.167344 18815 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:12.168838 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.168900 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.170890 18165 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.122306 26284 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.144192 26298 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.181075 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:12.190090 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:12.144165 26134 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.195248 26267 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:12.204196 26267 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:12.205498 26267 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:12.207947 26241 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.736s	user 0.000s	sys 0.010s
W20260709 12:25:12.208600 26305 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:25:12.209034 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3107) took 1700 ms (client timeout 2004 ms). Trace:
W20260709 12:25:12.209477 18170 rpcz_store.cc:269] 0709 12:25:10.508658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.559159 (+ 50501us) service_pool.cc:224] Handling call
0709 12:25:12.208813 (+1649654us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b540007bf54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.208957 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1814424960}
I20260709 12:25:12.218338 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.219798 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.222314 26304 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:12.223057 26304 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.227131 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.228766 18170 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:12.232620 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.196622 26190 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.239375 18165 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:12.242063 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.246608 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:12.251974 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.252693 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.253511 18170 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:12.254199 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3213) took 1670 ms. Trace:
I20260709 12:25:12.254343 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.254628 18170 rpcz_store.cc:276] 0709 12:25:10.583940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.252357 (+1668417us) service_pool.cc:224] Handling call
0709 12:25:12.254120 (+  1763us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.255388 18165 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:12.258615 26297 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.259294 26134 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.259552 26158 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:12.260385 26158 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:12.264791 26174 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.266127 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3125) took 1396 ms. Trace:
I20260709 12:25:12.266038 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.266732 18165 rpcz_store.cc:276] 0709 12:25:10.870068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.254016 (+1383948us) service_pool.cc:224] Handling call
0709 12:25:12.266025 (+ 12009us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.268016 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.267210 18170 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:12.268818 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:12.269340 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3133) took 1266 ms. Trace:
I20260709 12:25:12.270100 26075 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.269937 18170 rpcz_store.cc:276] 0709 12:25:11.002818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.265605 (+1262787us) service_pool.cc:224] Handling call
0709 12:25:12.269194 (+  3589us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.269428 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1227) took 1343 ms. Trace:
I20260709 12:25:12.276583 18165 rpcz_store.cc:276] 0709 12:25:10.925735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.267654 (+1341919us) service_pool.cc:224] Handling call
0709 12:25:12.269358 (+  1704us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.277328 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.278203 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.194211 26298 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.280000 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.280637 18165 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:12.248311 26073 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.284449 26073 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.202297 26121 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.306394 26121 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.307461 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1191) took 2062 ms. Trace:
I20260709 12:25:12.307312 26121 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.308251 18017 rpcz_store.cc:276] 0709 12:25:10.245159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.256087 (+ 10928us) service_pool.cc:224] Handling call
0709 12:25:10.256623 (+   536us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.357268 (+1100645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.307032 (+949764us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.307360 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2482844032}
I20260709 12:25:12.314461 26134 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.315315 26134 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.317210 26209 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.317935 26209 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.319105 26134 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.320226 26209 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.321436 18094 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:12.322770 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:12.323089 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3206) took 1949 ms (client timeout 1982 ms). Trace:
W20260709 12:25:12.323715 18166 rpcz_store.cc:269] 0709 12:25:10.373501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.373754 (+   253us) service_pool.cc:224] Handling call
0709 12:25:12.322828 (+1949074us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001b4254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.323000 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2614574976}
I20260709 12:25:12.325011 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.327752 18090 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:25:12.327991 26240 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.904s	user 0.004s	sys 0.013s
I20260709 12:25:12.328348 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.329272 18095 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.329434 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3356) took 2006 ms. Trace:
I20260709 12:25:12.359761 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1201) took 2011 ms. Trace:
I20260709 12:25:12.329452 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1192) took 2072 ms. Trace:
I20260709 12:25:12.361132 18021 rpcz_store.cc:276] 0709 12:25:10.257453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.257636 (+   183us) service_pool.cc:224] Handling call
0709 12:25:10.258051 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.637588 (+379537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.329191 (+1691603us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54001d0454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.329382 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731848576}
I20260709 12:25:12.362324 26185 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.355705 26298 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:12.300320 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3190) took 2779 ms (client timeout 1694 ms). Trace:
W20260709 12:25:12.367380 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1461) took 2220 ms (client timeout 1883 ms). Trace:
W20260709 12:25:12.367569 18173 rpcz_store.cc:269] 0709 12:25:09.521069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.521223 (+   154us) service_pool.cc:224] Handling call
0709 12:25:12.299961 (+2778738us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b54000505d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.300191 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2015654272}
W20260709 12:25:12.367909 18808 rpcz_store.cc:269] 0709 12:25:10.146658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.146919 (+   261us) service_pool.cc:224] Handling call
0709 12:25:12.367108 (+2220189us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.367248 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":636448000}
I20260709 12:25:12.369163 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.369870 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4106) took 2860 ms. Trace:
I20260709 12:25:12.302538 26091 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:12.371490 26091 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:12.370378 18015 rpcz_store.cc:276] 0709 12:25:09.510004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.513844 (+  3840us) service_pool.cc:224] Handling call
0709 12:25:09.514169 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.591201 (+ 77032us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:09.663242 (+ 72041us) log.cc:844] Serialized 14 byte log entry
0709 12:25:10.176886 (+513644us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.176958 (+    72us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:10.177016 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.369494 (+2192478us) spinlock_profiling.cc:243] Waited 76.7 ms on lock 0x7b5400347bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.369582 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.369775 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3629,"prepare.queue_time_us":402593,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":39118,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":15874,"spinlock_wait_cycles":176478592,"thread_start_us":177612,"threads_started":1,"wal-append.queue_time_us":294824,"wal-append.run_cpu_time_us":2753,"wal-append.run_wall_time_us":1298406}
I20260709 12:25:12.319173 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3355) took 2008 ms. Trace:
I20260709 12:25:12.373152 18093 rpcz_store.cc:276] 0709 12:25:10.310952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.482672 (+171720us) service_pool.cc:224] Handling call
0709 12:25:10.509303 (+ 26631us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.566103 (+1056800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.318767 (+752664us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54002cae54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.319070 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2413280512}
I20260709 12:25:12.323772 18096 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:12.330845 18097 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.372632 26091 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.225878 26304 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.331524 18092 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:25:12.375936 26123 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.376590 26123 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.379598 26257 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.380486 26257 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.382674 18173 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:12.384719 26284 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.334582 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.336900 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:12.339927 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.340205 26073 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.391433 26123 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.261711 26158 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.360028 18098 rpcz_store.cc:276] 0709 12:25:10.323524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.469224 (+145700us) service_pool.cc:224] Handling call
0709 12:25:10.469645 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.594892 (+1125247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.329154 (+734262us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000523d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.329364 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2572108544}
I20260709 12:25:12.360342 18023 rpcz_store.cc:276] 0709 12:25:10.348900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.349072 (+   172us) service_pool.cc:224] Handling call
0709 12:25:10.354513 (+  5441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.648340 (+1293827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.359338 (+710998us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:12.359669 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2956243712}
I20260709 12:25:12.325382 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.398530 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1220) took 1483 ms. Trace:
I20260709 12:25:12.398933 26304 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.352666 26286 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.399027 18166 rpcz_store.cc:276] 0709 12:25:10.914846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.325004 (+1410158us) service_pool.cc:224] Handling call
0709 12:25:12.398449 (+ 73445us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.387696 26121 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.356994 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.416276 18172 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:12.414582 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.375355 26091 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:12.333052 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.418228 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.418882 18095 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:12.391765 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.392309 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.399570 26304 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.424316 18097 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:25:12.424672 18064 leader_election.cc:336] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.720s (SENT)
W20260709 12:25:12.426435 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.404394 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.441960 18807 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:12.355216 26250 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.527s	user 0.000s	sys 0.010s
I20260709 12:25:12.298125 26089 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.321604 26303 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.278234 18170 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:25:12.366442 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3195) took 2833 ms (client timeout 2705 ms). Trace:
I20260709 12:25:12.420408 18094 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:12.420598 18092 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:25:12.375134 18165 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:12.452564 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.452744 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:12.450917 18171 rpcz_store.cc:269] 0709 12:25:09.533497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.568431 (+ 34934us) service_pool.cc:224] Handling call
0709 12:25:12.366111 (+2797680us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54003b9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.366321 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2105304960}
I20260709 12:25:12.454622 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.415297 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.456058 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.429426 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.459993 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.431684 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.460915 18096 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:12.461201 18814 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 24 index: 21. (index mismatch)
I20260709 12:25:12.403263 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.461917 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.435081 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.291968 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:12.462810 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:12.463042 18090 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:12.463419 18173 raft_consensus.cc:606] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.419776 18098 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:25:12.444214 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.441790 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.426545 26221 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:12.435209 25974 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.535s	user 0.017s	sys 0.000s
I20260709 12:25:12.449746 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.456228 18093 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:12.428857 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.458137 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.460098 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:12.467170 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 175364us
I20260709 12:25:12.436167 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.464617 18808 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:12.444744 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.478778 18171 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:12.479027 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:12.465215 18173 raft_consensus.cc:614] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:12.465309 18170 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:12.467167 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.467708 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.469812 18172 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:12.471505 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.472067 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.453989 26137 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.479300 18812 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
W20260709 12:25:12.483989 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.491675 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.505336 18098 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:25:12.509743 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.503657 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:12.511814 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:12.513141 18092 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
W20260709 12:25:12.509800 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53846.3us
I20260709 12:25:12.515237 26267 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.516662 26257 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.517110 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.518222 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.519224 18095 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:12.515302 26194 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:12.523407 18173 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:12.468113 26089 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:12.524694 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:12.512308 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:12.533370 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.526878 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3210) took 2010 ms. Trace:
I20260709 12:25:12.534112 18173 rpcz_store.cc:276] 0709 12:25:10.516179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.514600 (+1998421us) service_pool.cc:224] Handling call
0709 12:25:12.514976 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.526485 (+ 11509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.526802 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.535014 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.467993 25812 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.535713 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:25:12.525571 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55379us
I20260709 12:25:12.535537 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.469067 26221 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.531126 26089 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:12.536430 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.551644 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.554687 18023 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:12.558768 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.559499 18023 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:25:12.458906 26303 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.527163 25819 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:12.562063 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:12.560004 18173 raft_consensus.cc:606] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:25:12.522928 26260 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:12.530256 26209 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:12.508724 26123 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.564602 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:12.564893 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3214) took 1924 ms. Trace:
I20260709 12:25:12.562493 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:12.579037 18015 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:12.578919 18171 rpcz_store.cc:276] 0709 12:25:10.640729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.534971 (+1894242us) service_pool.cc:224] Handling call
0709 12:25:12.564271 (+ 29300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.564506 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.564555 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.564604 (+    49us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:12.564649 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.564711 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.564832 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.569518 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:12.567548 26260 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.571355 18807 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:12.549980 26134 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:12.572151 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.587630 26257 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:12.588181 26257 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.588657 26257 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:12.590745 26227 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:12.592235 25675 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:25:12.593461 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3212) took 2033 ms (client timeout 2419 ms). Trace:
W20260709 12:25:12.594072 18168 rpcz_store.cc:269] 0709 12:25:10.560204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.667891 (+107687us) service_pool.cc:224] Handling call
0709 12:25:12.593148 (+1925257us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.593368 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2076988288}
I20260709 12:25:12.595868 18168 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:12.596729 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1219) took 1682 ms. Trace:
I20260709 12:25:12.594801 18171 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:12.570899 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:12.567642 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:12.604212 18021 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:12.551658 26089 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:12.585678 26260 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:12.557313 26073 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:12.532150 26298 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:25:12.626132 26073 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.626555 26298 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:12.578189 26185 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:12.627244 26298 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:12.627568 26185 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:12.628022 26185 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:12.628372 26298 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.614434 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3127) took 1702 ms. Trace:
I20260709 12:25:12.629706 18171 rpcz_store.cc:276] 0709 12:25:10.911838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.580250 (+1668412us) service_pool.cc:224] Handling call
0709 12:25:12.594576 (+ 14326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.613808 (+ 19232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.614335 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.631459 26303 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.613997 26267 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:12.633361 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.634112 26260 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:12.638674 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1223) took 1722 ms (client timeout 1695 ms). Trace:
I20260709 12:25:12.597172 18168 rpcz_store.cc:276] 0709 12:25:10.914420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.595208 (+1680788us) service_pool.cc:224] Handling call
0709 12:25:12.595612 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.596375 (+   763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.596653 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:12.639254 18169 rpcz_store.cc:269] 0709 12:25:10.916089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.875944 (+959855us) service_pool.cc:224] Handling call
0709 12:25:12.638381 (+762437us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b5400050ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.638554 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92176640}
W20260709 12:25:12.643338 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72528.3us
W20260709 12:25:12.644948 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.550384 25974 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:12.573513 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.647003 18095 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 15.
I20260709 12:25:12.626688 26073 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:12.631539 18171 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:12.651506 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1221) took 1736 ms. Trace:
I20260709 12:25:12.531553 26174 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:25:12.567103 26297 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:12.537874 26037 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:12.651921 18171 rpcz_store.cc:276] 0709 12:25:10.915409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.630735 (+1715326us) service_pool.cc:224] Handling call
0709 12:25:12.631151 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.651120 (+ 19969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.651438 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.617570 18813 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:12.643807 18099 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:25:12.656651 26301 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:12.662366 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3124) took 1819 ms. Trace:
I20260709 12:25:12.675794 18164 rpcz_store.cc:276] 0709 12:25:10.842636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.876494 (+ 33858us) service_pool.cc:224] Handling call
0709 12:25:12.651535 (+1775041us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:12.662243 (+ 10708us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98727296}
I20260709 12:25:12.677845 18164 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:12.523311 26137 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.548410 26289 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.669090 18166 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:12.641535 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1224) took 1725 ms. Trace:
I20260709 12:25:12.579030 25819 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:12.674736 18168 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:25:12.689196 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1222) took 1773 ms. Trace:
I20260709 12:25:12.650353 26084 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:12.678805 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1226) took 1753 ms. Trace:
I20260709 12:25:12.562748 25812 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.682283 18169 rpcz_store.cc:276] 0709 12:25:10.924399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.640252 (+1715853us) service_pool.cc:224] Handling call
0709 12:25:12.640729 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.641024 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.641070 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.641137 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:12.641189 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.641271 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.641452 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.692533 18164 rpcz_store.cc:276] 0709 12:25:10.925358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.677015 (+1751657us) service_pool.cc:224] Handling call
0709 12:25:12.677549 (+   534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.678401 (+   852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.678720 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.689196 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1225) took 1764 ms. Trace:
I20260709 12:25:12.694027 18166 rpcz_store.cc:276] 0709 12:25:10.924834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.660585 (+1735751us) service_pool.cc:224] Handling call
0709 12:25:12.668810 (+  8225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.688878 (+ 20068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.689105 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.605007 26246 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.094s
I20260709 12:25:12.653846 25974 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.673465 26298 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.704577 18808 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:12.695609 18166 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:12.689729 18168 rpcz_store.cc:276] 0709 12:25:10.915753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.639569 (+1723816us) service_pool.cc:224] Handling call
0709 12:25:12.674332 (+ 34763us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.688803 (+ 14471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.689105 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.693607 18169 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.696481 18164 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:25:12.711293 18098 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 13.
I20260709 12:25:12.716303 18164 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 13
I20260709 12:25:12.618245 26123 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.718995 18164 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:25:12.729619 18164 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:12.730643 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1229) took 1772 ms. Trace:
I20260709 12:25:12.731029 18164 rpcz_store.cc:276] 0709 12:25:10.971441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.693546 (+1722105us) service_pool.cc:224] Handling call
0709 12:25:12.693904 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.730305 (+ 36401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.730571 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.716667 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3130) took 1765 ms. Trace:
I20260709 12:25:12.732187 18169 rpcz_store.cc:276] 0709 12:25:10.951401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.693004 (+1741603us) service_pool.cc:224] Handling call
0709 12:25:12.693395 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.716277 (+ 22882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.716585 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.717506 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3131) took 1741 ms. Trace:
I20260709 12:25:12.734208 18166 rpcz_store.cc:276] 0709 12:25:10.976539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.694897 (+1718358us) service_pool.cc:224] Handling call
0709 12:25:12.695240 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.715259 (+ 20019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.717413 (+  2154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.737668 18166 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:25:12.738692 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3132) took 1736 ms. Trace:
I20260709 12:25:12.740664 18169 rpcz_store.cc:276] 0709 12:25:11.002262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.733365 (+1731103us) service_pool.cc:224] Handling call
0709 12:25:12.735558 (+  2193us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.735884 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.735933 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.735999 (+    66us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:12.736051 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.736136 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.738585 (+  2449us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.743464 18169 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:25:12.744693 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3136) took 1740 ms. Trace:
I20260709 12:25:12.745425 18169 rpcz_store.cc:276] 0709 12:25:11.003819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.742383 (+1738564us) service_pool.cc:224] Handling call
0709 12:25:12.742887 (+   504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.744087 (+  1200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.744562 (+   475us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.747710 18169 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 11. (term mismatch)
I20260709 12:25:12.748250 18169 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 10
I20260709 12:25:12.748616 18169 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 22.11
I20260709 12:25:12.749287 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3134) took 1737 ms. Trace:
I20260709 12:25:12.749955 18166 rpcz_store.cc:276] 0709 12:25:11.003179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.736703 (+1733524us) service_pool.cc:224] Handling call
0709 12:25:12.737305 (+   602us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.740011 (+  2706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.740345 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.752977 18166 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:12.749014 18169 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:12.761428 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3137) took 1757 ms. Trace:
I20260709 12:25:12.762245 18169 rpcz_store.cc:276] 0709 12:25:11.004139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.746659 (+1742520us) service_pool.cc:224] Handling call
0709 12:25:12.747154 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.761029 (+ 13875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.761327 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.764597 18169 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:12.766201 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3140) took 1153 ms. Trace:
I20260709 12:25:12.766681 18169 rpcz_store.cc:276] 0709 12:25:11.622404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.763698 (+1141294us) service_pool.cc:224] Handling call
0709 12:25:12.764251 (+   553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.765806 (+  1555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.766129 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.768533 18169 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:25:12.769397 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3142) took 1144 ms. Trace:
I20260709 12:25:12.769897 18169 rpcz_store.cc:276] 0709 12:25:11.625339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.767720 (+1142381us) service_pool.cc:224] Handling call
0709 12:25:12.768210 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.769051 (+   841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.769331 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:12.778225 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.110s	sys 0.139s
W20260709 12:25:12.778793 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.111s	sys 0.139s
W20260709 12:25:12.779263 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 644715us
I20260709 12:25:12.780854 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.782163 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1231) took 1657 ms. Trace:
I20260709 12:25:12.782902 18166 rpcz_store.cc:276] 0709 12:25:11.126028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.751192 (+1625164us) service_pool.cc:224] Handling call
0709 12:25:12.752401 (+  1209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.759707 (+  7306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.782092 (+ 22385us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.709916 26190 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:12.695601 26047 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:12.710260 25974 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.791498 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:12.793239 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:12.800427 18097 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:12.792899 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.794107 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.802462 18096 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:12.794315 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:12.805035 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:12.795070 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:12.797637 18811 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.696267 26300 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.088s
I20260709 12:25:12.808576 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.811889 18095 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:12.822072 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3223) took 1054 ms. Trace:
I20260709 12:25:12.822561 18168 rpcz_store.cc:276] 0709 12:25:11.767621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.796143 (+1028522us) service_pool.cc:224] Handling call
0709 12:25:12.821385 (+ 25242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.821613 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.821640 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.821687 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:12.821731 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.821787 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.822002 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.812835 26246 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:25:12.824846 26246 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:12.825295 26246 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:12.825693 26309 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:25:12.826221 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.827013 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.827862 18096 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:12.826287 26246 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.814219 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:12.830977 18092 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:12.834139 26089 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.834941 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:25:12.846474 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.846848 26123 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.848826 25812 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:12.849658 25819 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.797420 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3224) took 1025 ms. Trace:
I20260709 12:25:12.728482 26304 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.816730 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.818280 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:12.864053 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.702369 26183 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:12.858408 18166 rpcz_store.cc:276] 0709 12:25:11.771666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.796143 (+1024477us) service_pool.cc:224] Handling call
0709 12:25:12.796566 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.796867 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.796911 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.796984 (+    73us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:12.797041 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.797133 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.797321 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:12.860679 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:12.800390 26047 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:12.867918 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.818874 26267 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:12.858037 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45946.4us
I20260709 12:25:12.875401 18807 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:25:12.878311 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 337356us
I20260709 12:25:12.867913 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.880247 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:12.882156 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 321484us
I20260709 12:25:12.883985 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.884785 18172 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:12.886232 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:12.886929 18172 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:12.862871 26089 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:12.890442 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:12.891580 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.892843 18812 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:12.893424 18166 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
W20260709 12:25:12.894418 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 337300us
I20260709 12:25:12.906862 18093 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:12.896242 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:12.869792 26183 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:12.907795 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3205) took 2519 ms. Trace:
I20260709 12:25:12.909165 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:12.901683 18097 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:12.923234 18172 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:12.854439 26275 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:25:12.933770 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:12.943133 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:12.936173 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 353915us
I20260709 12:25:12.945024 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:12.937394 18167 rpcz_store.cc:276] 0709 12:25:10.367739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.449463 (+ 81724us) service_pool.cc:224] Handling call
0709 12:25:10.449913 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.450197 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:10.497340 (+ 47143us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.214439 (+1717099us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b180041a940. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:25:12.886384 (+671945us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.886747 (+   363us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.886820 (+    73us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:12.886893 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.887053 (+   160us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.887247 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":665,"prepare.queue_time_us":37045,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":4695,"raft.run_cpu_time_us":403,"raft.run_wall_time_us":54741,"spinlock_wait_cycles":4199296,"thread_start_us":670372,"threads_started":2,"wal-append.queue_time_us":2303490}
W20260709 12:25:12.947793 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.948983 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:12.952143 18015 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:12.886229 25787 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:12.937417 26089 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:12.954360 26089 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.954849 26089 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:12.955367 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:12.958534 26094 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:12.958195 26047 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:12.959942 26047 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:12.967162 26310 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:25:12.968509 26174 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:12.969151 26174 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.969664 26174 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.939255 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:12.909996 26260 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:12.979103 26260 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:12.892351 26205 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:12.990703 26205 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:12.907256 26246 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:12.975682 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:13.027431 18015 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:13.029161 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.030426 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:13.031921 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:13.021137 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:13.032866 26306 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:13.033654 26306 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:13.034132 26306 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.937527 26221 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:13.040271 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 384720us
I20260709 12:25:13.042550 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:13.044034 18022 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:13.038484 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:13.045660 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:13.041855 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:12.981443 26260 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:25:12.937721 26209 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.102s
I20260709 12:25:13.074359 18022 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:13.041067 26308 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:13.083221 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:13.086059 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:13.099591 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:13.101004 18167 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:25:13.102577 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:13.105158 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.106778 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:13.107612 18097 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:13.081079 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:13.110667 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:13.111886 18093 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:13.113296 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:13.114238 18090 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:13.115666 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:13.116662 18097 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:13.056249 26276 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:13.123934 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:13.125454 18167 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:13.127887 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:13.130590 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.131940 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:13.138309 18090 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:13.140157 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:13.075361 26084 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:13.071973 25812 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:13.050532 26304 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:25:13.051072 26176 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:25:13.100664 26308 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.063094 26303 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
I20260709 12:25:13.083521 26315 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:13.082100 26043 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:13.171300 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:13.175361 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:13.198992 18170 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:25:13.077350 26301 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:25:13.143236 26137 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:13.145056 26084 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:13.159418 25812 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:13.202397 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 259145us
I20260709 12:25:13.064702 26183 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:13.187783 26308 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:13.162257 26320 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:25:13.200011 26315 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:13.264945 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 198128us
W20260709 12:25:13.262414 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 181402us
W20260709 12:25:13.278362 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 339173us
I20260709 12:25:13.202916 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:13.317664 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114831us
I20260709 12:25:13.318738 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:13.205291 18170 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 24 index: 21. (index mismatch)
I20260709 12:25:13.206911 26123 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:13.291462 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 395374us
I20260709 12:25:13.279556 26190 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:13.292228 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 359802us
I20260709 12:25:13.304054 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3226) took 1246 ms. Trace:
I20260709 12:25:13.330219 18168 rpcz_store.cc:276] 0709 12:25:12.058097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.823604 (+765507us) service_pool.cc:224] Handling call
0709 12:25:12.824016 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.824271 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:13.051355 (+227084us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.303577 (+252222us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:13.303653 (+    76us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:13.303722 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.303811 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.303975 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":344705,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":2496,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":153,"thread_start_us":467115,"threads_started":2}
I20260709 12:25:13.294688 25812 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:13.306624 26304 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:13.307852 26324 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:13.309259 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111202us
I20260709 12:25:13.223126 26094 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:13.304953 26043 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:13.323712 26123 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:13.492930 26123 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:13.322759 26183 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:13.356449 26190 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:13.500699 26190 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:13.322389 26315 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:13.356963 25819 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.145s
I20260709 12:25:13.503252 25819 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:13.503738 25819 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:13.504760 25819 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:13.506649 26321 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.304s	user 0.000s	sys 0.000s
I20260709 12:25:13.507326 26320 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:13.303844 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:13.507804 26320 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:13.333504 26037 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:13.509143 26037 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.509563 26037 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:13.284107 26084 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:13.324319 26257 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.735s	user 0.018s	sys 0.000s
W20260709 12:25:13.482992 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.314164 26314 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.375s	user 0.000s	sys 0.002s
I20260709 12:25:13.506827 26123 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:13.334513 18170 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 10. (term mismatch)
W20260709 12:25:13.513065 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 176576us
I20260709 12:25:13.538496 18170 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 9
I20260709 12:25:13.539229 18170 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 20.10
I20260709 12:25:13.539710 18170 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:13.329432 26091 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.897s	user 0.009s	sys 0.016s
I20260709 12:25:13.337414 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:13.546707 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.557067 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.550812 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 247121us
I20260709 12:25:13.558480 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:13.560145 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 222838us
I20260709 12:25:13.552863 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:13.470544 26303 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:13.577059 18165 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:13.535156 26318 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:13.576855 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 261275us
I20260709 12:25:13.550518 26043 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:13.577603 26275 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:13.575960 25819 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:13.609115 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50758.3us
W20260709 12:25:13.642289 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0562915020000219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27730000000000005s for a response
W20260709 12:25:13.618230 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 299584us
I20260709 12:25:13.636596 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1468) took 1236 ms. Trace:
I20260709 12:25:13.644239 18810 rpcz_store.cc:276] 0709 12:25:12.399990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.400623 (+   633us) service_pool.cc:224] Handling call
0709 12:25:12.401065 (+   442us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.401542 (+   477us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:12.401579 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.402317 (+   738us) log.cc:844] Serialized 26 byte log entry
0709 12:25:12.755580 (+353263us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:12.755634 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.755716 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.636198 (+880482us) raft_consensus.cc:1685] finished
0709 12:25:13.636243 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.636484 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2260,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":352724,"threads_started":1,"wal-append.queue_time_us":414972}
W20260709 12:25:13.645236 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.645517 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:13.545795 26257 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:13.586817 26318 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.689735 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:13.585337 26303 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:13.600064 26205 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:13.703382 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:13.707026 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:13.584745 26176 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 12:25:13.652076 26260 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:13.728091 26303 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.730031 26303 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:13.635134 26043 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.693034 26318 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:13.686832 26275 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:13.738965 26047 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.778s	user 0.014s	sys 0.003s
I20260709 12:25:13.709048 26221 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:25:13.751873 26275 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.691779 26209 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:25:13.716946 26320 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.706087 26284 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:13.757460 26073 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.087s	user 0.014s	sys 0.001s
I20260709 12:25:13.716866 26205 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:25:13.757504 26185 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.129s	user 0.014s	sys 0.000s
W20260709 12:25:13.769742 26325 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:25:13.733505 26043 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:13.717410 26190 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.079s
W20260709 12:25:13.828501 26298 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.101s	user 0.014s	sys 0.003s
I20260709 12:25:13.818068 26205 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:13.829288 26047 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:13.769711 26328 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:13.806259 26330 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:13.823823 26338 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:25:13.824019 26185 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:13.791761 26089 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.806s	user 0.022s	sys 0.020s
I20260709 12:25:13.686086 26094 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.769654 26332 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:13.853084 26047 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.806237 26331 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:13.843652 26328 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.844192 26205 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:13.850461 26330 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.932711 26330 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:13.884083 26185 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:13.891088 26257 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:13.867707 26089 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:13.898770 26336 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:25:13.873525 26331 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:13.872128 26123 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:13.943367 26246 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.914s	user 0.014s	sys 0.006s
I20260709 12:25:13.906126 26328 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:13.950356 26331 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:13.917754 26306 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.883s	user 0.010s	sys 0.006s
W20260709 12:25:13.952415 26174 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.982s	user 0.012s	sys 0.020s
I20260709 12:25:13.966274 26174 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.951944 26306 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.822062 26284 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:13.972793 26306 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:13.978404 26315 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.962641 26318 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:13.931226 26308 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.602s	user 0.011s	sys 0.001s
I20260709 12:25:13.981089 26332 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.005431 26328 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.052611 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1211) took 1101 ms. Trace:
I20260709 12:25:14.060345 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3230) took 1965 ms. Trace:
I20260709 12:25:14.064767 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3239) took 1922 ms. Trace:
W20260709 12:25:14.075855 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.077560 18808 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:14.078459 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.004s	sys 0.007s
I20260709 12:25:14.078270 26134 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:13.958590 26084 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:13.821321 26073 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.090354 26308 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.102764 26315 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.111299 26350 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.113543 26328 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.113636 26174 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.136858 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.004s	sys 0.008s
W20260709 12:25:14.138281 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.138790 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52977.2us
W20260709 12:25:14.152364 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.155687 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.159617 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 835450us
I20260709 12:25:14.166541 26354 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:14.168249 26354 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:14.168752 26073 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.168771 26354 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:14.205754 26354 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:14.208002 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.209085 26354 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:13.990096 26324 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:14.195720 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 491090us
I20260709 12:25:14.212405 26324 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 22.19, Last appended by leader: 19, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.214494 26352 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.214346 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.215749 18810 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:14.215493 26352 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.169793 18023 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:14.218178 26174 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.119127 18166 rpcz_store.cc:276] 0709 12:25:12.142048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.945326 (+803278us) service_pool.cc:224] Handling call
0709 12:25:13.024744 (+ 79418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.025016 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:13.281062 (+256046us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.063480 (+782418us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.063662 (+   182us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:14.063883 (+   221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.064093 (+   210us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.064475 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":163065,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":2380,"raft.queue_time_us":43556,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":161,"thread_start_us":1017772,"threads_started":3,"wal-append.queue_time_us":449907}
I20260709 12:25:14.219125 26174 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.220659 26315 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:14.223534 26315 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:14.173108 26084 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:14.177982 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.008s	sys 0.001s
I20260709 12:25:14.178311 26134 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:14.179862 26355 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:25:14.180003 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:14.244082 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.008s	sys 0.002s
I20260709 12:25:14.244309 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:14.118957 18171 rpcz_store.cc:276] 0709 12:25:12.094901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.843075 (+748174us) service_pool.cc:224] Handling call
0709 12:25:12.843619 (+   544us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:12.844125 (+   506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.844160 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:12.845029 (+   869us) log.cc:844] Serialized 52 byte log entry
0709 12:25:13.164443 (+319414us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:13.164495 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.164577 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.059332 (+894755us) raft_consensus.cc:1685] finished
0709 12:25:14.059404 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.059947 (+   543us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1972,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":146,"thread_start_us":318789,"threads_started":1,"wal-append.queue_time_us":478174}
I20260709 12:25:14.191597 26350 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.194306 26047 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:14.248263 26037 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.738s	user 0.005s	sys 0.014s
I20260709 12:25:14.249006 26350 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.248880 26037 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.195652 26308 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.118432 18017 rpcz_store.cc:276] 0709 12:25:12.950621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.954335 (+  3714us) service_pool.cc:224] Handling call
0709 12:25:12.954682 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.955051 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.955091 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.955757 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.205216 (+249459us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:13.205571 (+   355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.206097 (+   526us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.051282 (+845185us) raft_consensus.cc:1685] finished
0709 12:25:14.051410 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.052062 (+   652us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11867,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":248283,"threads_started":1,"wal-append.queue_time_us":696921}
W20260709 12:25:14.156755 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 817280us
I20260709 12:25:14.199833 25675 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.249756 26134 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.203271 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.224462 26246 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.225730 26123 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.227023 26257 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.228807 26094 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.229915 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.230816 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.172358 26318 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.231534 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:14.245841 18165 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3221) took 2560 ms (client timeout 3342 ms). Trace:
I20260709 12:25:14.258029 26037 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:14.263161 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83514.3us
I20260709 12:25:14.255283 26174 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.255033 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:14.256374 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:14.255029 18171 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:14.300047 26260 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:25:14.259179 25974 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.378s	user 0.018s	sys 0.007s
I20260709 12:25:14.257098 25675 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.250844 26089 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.251262 18172 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:14.254144 26303 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.013s
I20260709 12:25:14.250349 26352 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.278604 26084 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:12.871012 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:14.326366 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 117034us
W20260709 12:25:14.325011 18165 rpcz_store.cc:269] 0709 12:25:11.685688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.452271 (+766583us) service_pool.cc:224] Handling call
0709 12:25:14.245151 (+1792880us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:14.245528 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2583638400}
I20260709 12:25:14.338955 26084 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.339567 26084 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.339828 26297 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:14.344132 25787 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.344609 26246 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.345016 25787 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.348377 26246 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.348755 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.010s	sys 0.012s
W20260709 12:25:14.349347 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.011s	sys 0.012s
I20260709 12:25:14.350108 26094 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.353343 26094 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.354996 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.023s	sys 0.043s
W20260709 12:25:14.355545 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.023s	sys 0.043s
W20260709 12:25:14.356003 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 771912us
I20260709 12:25:14.358445 18166 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 16.
I20260709 12:25:14.360183 25675 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.360942 26351 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:25:14.361213 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49010us
I20260709 12:25:14.361508 26351 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.362272 26257 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.363308 26351 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.329407 26260 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:14.366303 18170 raft_consensus.cc:2376] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23. Re-sending same reply.
I20260709 12:25:14.367995 26260 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:14.370105 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 113994us
I20260709 12:25:14.370942 26123 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.374745 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:14.377136 26134 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:14.377903 26134 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.378438 26134 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.380962 18813 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: [22.20-24.21]
I20260709 12:25:14.378176 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.361339 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:14.383155 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43779.5us
I20260709 12:25:14.374919 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:14.388711 18166 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
W20260709 12:25:14.387215 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.007s	sys 0.011s
I20260709 12:25:14.343461 26297 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:14.385178 26276 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:14.399344 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.007s	sys 0.012s
W20260709 12:25:14.360183 26326 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.050s	user 0.007s	sys 0.014s
I20260709 12:25:14.342018 25974 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.382839 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.365589 26257 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.409715 26227 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.410231 26316 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.410531 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:14.411055 26316 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.411624 18097 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:14.406340 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:14.387115 25819 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.734s	user 0.007s	sys 0.011s
I20260709 12:25:14.416831 18811 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.416599 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.400977 26276 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:14.403719 26351 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:14.396709 25787 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.381783 26320 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.422974 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:14.425747 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.427819 18168 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:14.428464 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:14.429263 18093 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.429546 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.431669 18812 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 13.
W20260709 12:25:14.433141 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44167.7us
W20260709 12:25:14.438297 26322 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.135s	user 0.024s	sys 0.010s
W20260709 12:25:14.441972 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.432713 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 879898us
I20260709 12:25:14.423890 26316 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.430991 26257 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:14.457983 18171 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:14.443404 26260 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.463227 18090 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:14.453584 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3220) took 2808 ms. Trace:
I20260709 12:25:14.463786 25819 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:14.466766 25819 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.442608 26227 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.467897 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1208) took 1518 ms. Trace:
I20260709 12:25:14.468354 18023 rpcz_store.cc:276] 0709 12:25:12.949392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.951080 (+  1688us) service_pool.cc:224] Handling call
0709 12:25:12.951455 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.217809 (+1266354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.467514 (+249705us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:14.467818 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2800509824}
I20260709 12:25:14.472079 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:14.473088 18099 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:14.474360 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:14.474560 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:14.475472 18094 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.475663 18098 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:14.476215 26284 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.477782 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:14.479022 18096 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:14.469856 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:14.461539 26037 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.488669 26037 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.490504 26297 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.492057 26297 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.463958 18169 rpcz_store.cc:276] 0709 12:25:11.644736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.770952 (+1126216us) service_pool.cc:224] Handling call
0709 12:25:12.771434 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.771774 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.771815 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.772534 (+   719us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.043883 (+271349us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:13.043927 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.044004 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.414156 (+1370152us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:14.440627 (+ 26471us) raft_consensus.cc:1685] finished
0709 12:25:14.440658 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.453359 (+ 12701us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8364928,"thread_start_us":222658,"threads_started":1,"wal-append.queue_time_us":380936}
I20260709 12:25:14.461334 26123 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.486855 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:14.503721 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1453) took 1558 ms. Trace:
I20260709 12:25:14.504312 18808 rpcz_store.cc:276] 0709 12:25:12.945738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.945947 (+   209us) service_pool.cc:224] Handling call
0709 12:25:12.957185 (+ 11238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.137722 (+1180537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.503312 (+365590us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000e2ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:14.503596 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2574483712}
W20260709 12:25:14.506436 26205 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.606s	user 0.014s	sys 0.010s
I20260709 12:25:14.507108 26205 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.512682 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:14.498827 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:14.464251 26246 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.520551 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:14.515694 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:14.527801 18097 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:14.530306 26332 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:14.531383 26332 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.532006 26037 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.535823 25819 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.525456 26330 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.542382 18099 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:14.543749 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:14.555804 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56255.4us
I20260709 12:25:14.545009 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:14.546881 18098 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:14.483377 26257 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.554215 18166 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:25:14.564900 26094 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:14.565807 26094 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.566265 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:14.566711 26360 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:14.567451 26360 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.568686 26094 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.569295 26094 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.570775 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:14.571857 25974 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.525612 26318 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.578073 18095 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 15.
I20260709 12:25:14.577878 26318 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.552568 26330 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.585952 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.586233 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138431us
I20260709 12:25:14.586789 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3228) took 2517 ms. Trace:
I20260709 12:25:14.587297 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:14.589562 26303 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:14.587306 18173 rpcz_store.cc:276] 0709 12:25:12.069695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.833619 (+763924us) service_pool.cc:224] Handling call
0709 12:25:12.850420 (+ 16801us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:12.851080 (+   660us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.851131 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:12.852610 (+  1479us) log.cc:844] Serialized 52 byte log entry
0709 12:25:13.069960 (+217350us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:13.577507 (+507547us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.581209 (+  3702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.581312 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.359057 (+777745us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:14.586183 (+227126us) raft_consensus.cc:1685] finished
0709 12:25:14.586341 (+   158us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.586572 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33707,"prepare.queue_time_us":132093,"prepare.run_cpu_time_us":9935,"prepare.run_wall_time_us":79500,"raft.queue_time_us":5731,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":9957,"spinlock_wait_cycles":9333120,"thread_start_us":624555,"threads_started":2,"wal-append.queue_time_us":356189}
I20260709 12:25:14.591336 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.592331 18814 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:25:14.593097 18813 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:14.590124 26303 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:14.593864 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70267.3us
I20260709 12:25:14.594095 26303 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:14.602571 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.591626 26338 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:25:14.603883 18810 raft_consensus.cc:2410] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:14.604171 26338 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:14.604728 26338 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:25:14.608466 26134 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:14.603914 26362 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:25:14.615658 26332 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.617473 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46822.2us
I20260709 12:25:14.621134 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:14.622061 18097 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:14.625046 26315 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:14.628131 25974 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:25:14.628743 25974 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:14.629150 25974 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:25:14.623086 26246 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.631735 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:14.633028 18023 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:14.634483 26137 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:14.635268 26137 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.636343 26257 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.623272 26094 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.646960 26324 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.647805 26324 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.651229 26324 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.655463 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:14.655881 26246 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.668901 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:14.616803 26227 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.672051 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 211758us
I20260709 12:25:14.676175 26075 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:14.691059 18094 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
W20260709 12:25:14.541807 25812 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.008s	user 0.014s	sys 0.009s
I20260709 12:25:14.688867 18807 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:14.679932 26134 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.691421 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66717.7us
I20260709 12:25:14.697938 25812 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.699697 26360 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:14.704596 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43863.5us
I20260709 12:25:14.716856 26318 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.718206 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 258527us
I20260709 12:25:14.718437 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:14.719245 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:14.720012 26190 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:14.705731 26315 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:14.690711 26075 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.708241 25812 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.708870 26316 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:14.730569 26075 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.731065 26316 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.709421 18814 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:14.711170 18167 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:14.693703 26330 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:14.712574 26159 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.736377 26094 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.713547 26332 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:14.738188 26332 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.740993 26332 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.714440 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 215922us
I20260709 12:25:14.742586 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:14.745461 26328 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.670459 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:14.750530 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 133398us
I20260709 12:25:14.751473 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:14.725343 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.722164 26315 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:14.730159 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:14.752519 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:14.756765 26134 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.758262 25819 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:14.759228 25819 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.760053 26315 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.21, Last appended by leader: 21, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.757366 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:14.762395 25812 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:14.762966 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43913.7us
I20260709 12:25:14.765197 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:14.763020 25812 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:14.766183 26358 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:14.766479 25812 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.766721 26358 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:14.767253 26358 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:14.768880 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3145) took 1681 ms (client timeout 1910 ms). Trace:
I20260709 12:25:14.760818 25819 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:14.773727 17914 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 182) took 52689 ms (client timeout 15000 ms). Trace:
W20260709 12:25:14.769378 18167 rpcz_store.cc:269] 0709 12:25:13.087912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.139749 (+ 51837us) service_pool.cc:224] Handling call
0709 12:25:14.768620 (+1628871us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54001b4c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:14.768809 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3609329024}
W20260709 12:25:14.770711 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.771095 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 910f8c8d787e4abbb092413c1f90ebf3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 22 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:14.771812 26205 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.765494 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:14.778309 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60246.2us
I20260709 12:25:14.779398 26284 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:14.780069 26315 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:14.790453 26246 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:14.790159 26330 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:14.791092 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.796831 26246 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.810601 26246 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.810473 26205 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:14.812271 26257 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:14.812781 26257 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:14.791699 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:14.798242 18807 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:14.798962 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.814116 26257 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.815402 18810 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:14.799921 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.816087 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:14.816457 26073 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:14.816941 18019 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:14.817209 26073 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.800346 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.818053 25787 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:14.799952 18016 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:14.819343 25787 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:14.819792 25787 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.791149 26316 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:14.820165 26286 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:14.821151 26284 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.820797 25787 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.806795 26084 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.805096 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:14.805975 26137 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.806571 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:14.806746 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:14.824806 26073 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.825455 18021 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:25:14.822394 26284 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.823369 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:14.823442 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:14.817659 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:14.829229 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48068.9us
W20260709 12:25:14.774350 17914 rpcz_store.cc:269] 0709 12:24:22.084287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.084640 (+   353us) service_pool.cc:224] Handling call
0709 12:24:23.337719 (+1253079us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16621090 00007f6c166bff3a 00007f6c1c475e9c 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:25:14.773407 (+51435688us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:06.733370 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:06.853942 (+120572us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:06.854007 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:25:06.854030 (+    23us) tablet.cc:662] Decoding operations
0709 12:25:06.855171 (+  1141us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:06.855310 (+   139us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:06.855352 (+    42us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:25:06.856252 (+   900us) tablet.cc:701] Row locks acquired
0709 12:25:06.856328 (+    76us) write_op.cc:260] PREPARE: finished
0709 12:25:06.856558 (+   230us) write_op.cc:270] Start()
0709 12:25:06.856763 (+   205us) write_op.cc:276] Timestamp: P: 1783599906856474 usec, L: 0
0709 12:25:06.856821 (+    58us) op_driver.cc:348] REPLICATION: starting
0709 12:25:06.955640 (+ 98819us) log.cc:844] Serialized 3256 byte log entry
0709 12:25:07.836192 (+880552us) op_driver.cc:464] REPLICATION: finished
0709 12:25:08.345107 (+508915us) write_op.cc:301] APPLY: starting
0709 12:25:08.367905 (+ 22798us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:08.533032 (+165127us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:08.533177 (+   145us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:08.995649 (+462472us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:09.160989 (+165340us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 12:25:09.170494 (+  9505us) write_op.cc:312] APPLY: finished
0709 12:25:09.252813 (+ 82319us) log.cc:844] Serialized 81 byte log entry
0709 12:25:09.350042 (+ 97229us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:09.365723 (+ 15681us) write_op.cc:454] Released schema lock
0709 12:25:09.383535 (+ 17812us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:09.418211 (+ 34676us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:09.418250 (+    39us) write_op.cc:454] Released schema lock
0709 12:25:10.028834 (+610584us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
Metrics: {"mutex_wait_us":96218,"reported_tablets":31,"spinlock_wait_cycles":2374210816,"child_traces":[["op",{"apply.queue_time_us":432993,"apply.run_cpu_time_us":26911,"apply.run_wall_time_us":1084381,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37615,"cfile_init":1,"lbm_read_time_us":571,"lbm_reads_lt_1ms":4,"num_ops":8,"prepare.queue_time_us":267499,"prepare.run_cpu_time_us":28429,"prepare.run_wall_time_us":393019,"raft.queue_time_us":321608,"raft.run_cpu_time_us":34039,"raft.run_wall_time_us":427651,"replication_time_us":955055,"spinlock_wait_cycles":11708288,"thread_start_us":726581,"threads_started":5,"wal-append.queue_time_us":280821,"wal-append.run_cpu_time_us":38567,"wal-append.run_wall_time_us":3152882}]]}
I20260709 12:25:14.831779 26351 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:14.831893 26134 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.832322 26351 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.832854 26351 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.833459 26134 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:14.833902 26134 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:14.834535 26358 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.835112 26134 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.836731 26134 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.837061 18812 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:14.838794 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:14.839836 26332 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:14.840385 26332 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:14.841569 26332 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.842742 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.843529 26094 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.844190 26094 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.844501 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.846369 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.847285 18814 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:14.850030 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.845629 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.851933 26257 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.858232 25787 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.859280 18807 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:14.866418 26328 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:14.776865 26318 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:14.867487 26318 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.867905 26318 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:14.870666 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53145.7us
I20260709 12:25:14.871852 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:14.872980 26359 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:14.873718 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:14.878314 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:14.879537 18810 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:14.855365 26094 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:14.856626 26205 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.873878 26359 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.886227 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:14.887516 26094 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:14.888147 26094 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.888634 26094 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.890837 26276 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.892782 26084 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.893544 26359 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:14.895112 26286 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:14.895603 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:14.895684 26286 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.897377 26286 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.899678 26286 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.861474 26351 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:14.902150 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 136816us
I20260709 12:25:14.903306 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:14.904345 26137 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:14.830804 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:14.855955 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:14.915092 18090 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:14.920614 18811 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
W20260709 12:25:14.926759 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.943650 18807 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:14.947978 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92160.8us
I20260709 12:25:14.837683 26358 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:14.954993 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 159967us
I20260709 12:25:14.956351 26338 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.957170 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:14.957912 25675 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:14.958556 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:14.959551 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62063.3us
I20260709 12:25:14.960667 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:14.966200 26301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.589s	user 0.026s	sys 0.048s
W20260709 12:25:14.966796 26301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.652s	user 0.026s	sys 0.049s
I20260709 12:25:14.970285 18808 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:14.970789 18808 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:14.971850 18808 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:14.974869 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 68655.3us
I20260709 12:25:14.976081 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:14.978720 26159 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:14.979851 26159 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:14.980454 26159 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:14.983245 26246 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:14.984856 26037 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:14.984676 25675 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:14.966041 26205 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.984850 25974 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.973531 18808 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:14.950999 26320 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.509s	user 0.014s	sys 0.000s
I20260709 12:25:14.969085 17920 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 12 to 15, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:14.998862 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53073.3us
I20260709 12:25:14.991887 26297 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:15.004307 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.004092 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:15.005119 18811 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:15.005975 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:14.974948 26276 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:14.953563 26075 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.001230 17920 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 16 to 17, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 17 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.013310 26260 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:14.990681 26351 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:15.013581 17920 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:15.013597 26349 consensus_queue.cc:1175] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" responder_term: 18 status { last_received { term: 17 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:15.016984 17920 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.999300 26359 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:15.023741 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65306.5us
I20260709 12:25:15.009092 26358 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:15.012048 26037 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:15.014453 26297 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:15.021179 26330 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:15.019932 26260 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:15.041164 26303 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.040962 26297 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:14.988456 26159 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.059406 26246 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:15.068115 18167 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.17-21.17]
W20260709 12:25:15.073482 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 83654.2us
I20260709 12:25:15.044565 26037 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:15.047073 26358 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:15.078186 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77034.2us
W20260709 12:25:15.085418 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 305574us
I20260709 12:25:15.093652 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.094326 26297 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.094655 18024 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:15.096992 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.081591 25675 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.094367 26260 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.112002 18166 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:25:15.109789 17920 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 8 to 15. New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.153052 18811 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:15.065658 25812 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.155161 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:15.158407 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.163257 17920 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 9 to 17. New cstate: current_term: 17 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:25:15.112046 26123 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.010s	sys 0.060s
W20260709 12:25:15.171932 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 211377us
I20260709 12:25:15.172921 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.175140 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:15.169406 18780 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.695s (SENT)
I20260709 12:25:15.161167 26286 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:15.183048 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:15.183986 17920 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 13 to 18, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 18 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:15.191890 18807 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 13. Current term is 14. Ops: []
I20260709 12:25:15.185565 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:15.185971 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.162951 26338 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:15.186720 26227 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:15.187121 26037 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.169531 26276 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.195909 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:15.160322 26137 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.192800 18811 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 14: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:15.173810 26123 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.010s	sys 0.061s
I20260709 12:25:15.221144 18808 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:15.187729 26332 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:15.190320 25974 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:15.204394 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48141.9us
I20260709 12:25:15.204195 26358 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.206473 18165 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:15.210007 25787 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:15.234035 26276 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.254665 26351 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:25:15.254904 18168 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:15.256225 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:15.256685 26276 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:15.256896 18168 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:15.259589 26094 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.200001 26205 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:15.263314 26205 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.263770 26205 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:15.265141 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.205786 26227 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.266543 18092 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:15.268554 26227 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:15.236213 26075 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:15.238708 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75126.9us
I20260709 12:25:15.239565 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:15.241096 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.242065 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:15.243409 18171 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 23 index: 20. (index mismatch)
I20260709 12:25:15.243564 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.244746 25812 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:15.252388 18164 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:15.267943 26137 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.235086 26303 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:15.253270 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.246541 17920 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 21, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 21 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:15.272897 18170 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:15.273967 18093 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:15.288547 26137 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:15.289949 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:15.291029 18164 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:15.292125 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:15.293113 18164 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:25:15.277755 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:15.294833 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:15.295674 26276 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:15.296207 26276 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.296110 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:15.296700 26276 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.302269 18811 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:15.302495 25787 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.303179 25787 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.304388 26073 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:15.306877 18811 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:25:15.310138 18171 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:15.308172 25819 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:15.312216 25819 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.308821 26075 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:15.309103 18808 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:15.311335 26328 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:15.318825 18169 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:15.318800 18172 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:15.255230 26351 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:15.321837 18780 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.721s (SENT)
I20260709 12:25:15.335146 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:15.338726 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.338173 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.342198 18093 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.328612 18097 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [19.16-21.17]
W20260709 12:25:15.334093 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.350471 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:15.351771 18164 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:15.354830 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:15.359915 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.366345 17920 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.374994 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:15.389369 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73226.8us
I20260709 12:25:15.390404 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.403030 18171 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:15.404034 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.354605 26303 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.393180 26257 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.536s	user 0.012s	sys 0.003s
I20260709 12:25:15.432528 17920 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 16, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 16 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.427650 18024 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:15.431463 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:15.439615 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.457505 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53579us
I20260709 12:25:15.458408 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.398646 26359 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:15.430472 26284 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:25:15.473495 17920 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 18, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 18 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:15.405962 26349 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:15.476970 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:15.479132 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:15.498569 18808 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:15.499480 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.502156 18093 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:15.478641 18170 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:15.412253 26378 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:15.425163 25812 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:15.326706 26190 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.105s
I20260709 12:25:15.456955 26286 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:15.512302 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.479630 26351 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.432514 25787 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:15.487078 26318 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.619s	user 0.010s	sys 0.004s
W20260709 12:25:15.490243 26371 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.223s	user 0.000s	sys 0.003s
W20260709 12:25:15.526865 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172601us
W20260709 12:25:15.492359 26376 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.229s	user 0.001s	sys 0.001s
I20260709 12:25:15.528679 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.529927 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:15.518435 26094 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:15.530797 18808 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:15.503829 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.532145 18097 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:15.478561 26246 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:15.513146 18090 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:15.512795 17920 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.514983 26257 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:15.514204 26359 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:15.513875 26137 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:15.511978 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.523519 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.524108 18172 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:15.517212 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.481036 26159 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
W20260709 12:25:15.491070 26375 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.233s	user 0.001s	sys 0.001s
W20260709 12:25:15.505548 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 183804us
I20260709 12:25:15.532909 18810 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260709 12:25:15.512795 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 197385us
I20260709 12:25:15.445958 26303 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.533005 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:15.541179 18092 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:15.550424 26351 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:25:15.551023 17920 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 6 to 17, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 17 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:15.553247 17920 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 18, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 18 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.541567 26352 raft_consensus.cc:993] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:25:15.578436 18096 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:25:15.578970 17920 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 21, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 21 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:15.584625 18090 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:15.528553 26318 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.559381 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:15.587091 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.541285 26246 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:15.536819 26123 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.593333 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.599576 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.602837 18090 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:15.528595 26378 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.609943 18093 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:15.614073 17920 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 13, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 13 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:15.624390 26297 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.625150 26297 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.634303 18808 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:25:15.638447 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.638844 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.639734 18096 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:15.623512 26352 raft_consensus.cc:1081] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Signalling peer 768723648a9444fda53659e6a8cc3091 to start an election
I20260709 12:25:15.656661 18097 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:15.625710 17920 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 23, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 23 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.614593 26284 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:15.667738 18095 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:15.593228 26137 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.690248 18096 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:15.668016 18090 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:15.670388 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:15.672355 26384 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:25:15.674358 26303 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:15.675354 26349 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:15.680912 18168 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:15.682221 26338 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:25:15.682647 26382 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
W20260709 12:25:15.683503 26380 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
W20260709 12:25:15.684523 26390 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:25:15.699266 18171 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:15.703588 18091 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
W20260709 12:25:15.706194 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174576us
I20260709 12:25:15.739907 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:15.709532 18024 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 22. Ops: [20.12-20.12]
I20260709 12:25:15.710000 26303 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:15.614050 26367 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.073s
W20260709 12:25:15.712894 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47922.8us
I20260709 12:25:15.662850 25819 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:15.745079 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68584.6us
I20260709 12:25:15.748602 25812 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:15.655200 26123 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.770030 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.767664 17920 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 19, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 19 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.778110 17920 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 16 to 19, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 19 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:15.780128 17920 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 17, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 17 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:15.786415 26205 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.008s
I20260709 12:25:15.787043 26205 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:15.790587 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.791947 18138 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.510s (SENT)
I20260709 12:25:15.792456 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:15.798182 18022 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:15.798287 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:15.800079 26389 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:25:15.798013 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.778178 26123 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:15.802536 26392 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
W20260709 12:25:15.806107 26402 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:25:15.808204 26137 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:15.808666 26383 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:15.810334 26378 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:15.811215 18099 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 23 index: 20. (index mismatch)
I20260709 12:25:15.812263 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.813028 18096 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:25:15.810423 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.818276 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.819008 18023 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:25:15.819255 26388 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:15.819813 26388 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:15.820632 25812 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.820891 26388 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.811215 18020 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:15.789410 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.833236 18095 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:15.825191 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:15.800143 18015 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 21. Ops: []
I20260709 12:25:15.818945 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:15.828471 18808 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:25:15.834728 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63243.4us
I20260709 12:25:15.835199 18096 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:25:15.829531 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.830953 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.662303 26286 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:15.773478 26326 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:25:15.841579 18091 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 25 because replica has last-logged OpId of term: 24 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 19.
I20260709 12:25:15.857489 17920 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:15.866882 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.867352 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.865886 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:15.868170 18090 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:15.868278 18092 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 14: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:15.864943 17920 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:15.874874 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45424.2us
I20260709 12:25:15.740325 26257 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:15.865893 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:15.873888 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.879541 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:15.880383 18091 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
W20260709 12:25:15.838671 26379 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.577s	user 0.004s	sys 0.000s
I20260709 12:25:15.883199 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.884296 18019 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:15.884718 18090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45040
I20260709 12:25:15.885706 18090 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:15.886283 18090 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.886293 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.888669 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:15.889424 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.889422 18022 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:15.890759 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56390.7us
I20260709 12:25:15.706809 26359 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.874476 26303 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:15.895179 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.892931 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:15.896927 26388 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:15.876255 17920 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 23, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 23 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:15.869594 26338 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:15.888921 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.904409 18015 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:15.898409 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.865712 26123 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:15.864346 26094 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:15.901999 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:15.904619 26383 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:25:15.948069 26398 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:15.948746 26398 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:15.912540 26385 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 12:25:15.949888 26398 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.892762 26399 raft_consensus.cc:2764] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:25:15.915892 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:15.954531 26190 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:15.955235 26190 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:15.916637 25819 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:15.959267 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:15.918152 18138 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.619s (SENT)
I20260709 12:25:15.960291 18016 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:15.960382 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:15.961374 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:15.961593 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43501.9us
I20260709 12:25:15.962154 26401 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:15.962364 18138 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.622s (SENT)
I20260709 12:25:15.962905 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.963374 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.963629 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.917984 26406 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:15.918157 26284 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:15.919579 17920 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 19, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 19 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:15.932821 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.930375 18166 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:15.975657 18168 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:25:15.976251 18168 pending_rounds.cc:85] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 14
I20260709 12:25:15.900295 26303 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:15.947618 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48250us
I20260709 12:25:15.764518 26246 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:15.939492 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.839313 26227 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:25:15.962937 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:15.972644 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.982831 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.984879 18171 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:15.996171 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:15.998644 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:15.975159 18090 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:15.946590 26094 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:16.004861 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 20, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 20 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:15.976511 18168 pending_rounds.cc:107] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 19.15
W20260709 12:25:15.992545 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 232835us
I20260709 12:25:16.006824 18168 raft_consensus.cc:2889] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.932972 26338 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:15.997742 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:15.995774 18017 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:25:16.009580 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47251.9us
I20260709 12:25:16.010753 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:15.898476 26075 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
I20260709 12:25:16.006146 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:16.014961 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 122587us
I20260709 12:25:16.018404 18021 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:16.011585 18138 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.626s (SENT)
I20260709 12:25:16.027351 18165 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:16.034319 18169 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:16.023059 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:15.925273 26134 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
I20260709 12:25:16.038868 18812 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:25:16.043447 18090 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:15.956351 26190 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.046512 17920 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 14 to 17, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 17 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:16.049523 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:16.045346 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:16.051155 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153625us
I20260709 12:25:16.008208 26318 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:16.058485 18810 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:16.010648 26395 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:16.062083 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:16.045643 26397 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.080s	user 0.000s	sys 0.002s
W20260709 12:25:16.100349 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.966483 26257 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.055470 26399 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:16.212111 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.063759 26378 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:16.217432 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:16.234036 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 225286us
I20260709 12:25:16.239614 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:16.208350 25812 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:16.235041 26351 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.684s	user 0.015s	sys 0.000s
W20260709 12:25:16.270718 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53600.5us
I20260709 12:25:16.270840 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:16.281764 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.065444 26383 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.185s
W20260709 12:25:16.251858 26276 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.955s	user 0.017s	sys 0.000s
W20260709 12:25:16.288807 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 277261us
I20260709 12:25:16.249197 26359 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:16.254901 26367 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:16.295414 18165 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:25:16.298861 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.220419 25675 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.070s
I20260709 12:25:16.309638 18023 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:25:16.283401 26406 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:16.255244 26399 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:16.302181 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.338496 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.299827 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:16.308403 18138 leader_election.cc:336] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.648s (SENT)
I20260709 12:25:16.340487 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.291174 26123 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:16.365128 18022 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:16.302322 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.313206 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:16.366706 18171 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:16.289484 26075 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.290313 26227 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.169456 26303 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.319167 26395 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:16.319739 26420 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.323990 26418 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:16.391258 26418 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.391786 26418 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:16.381410 26075 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:16.393266 26409 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:25:16.324539 26367 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:16.395323 26367 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.397068 26420 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.398480 25819 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:16.398983 25819 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:16.314733 26406 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:16.399953 25819 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.402287 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1662) took 1076 ms. Trace:
I20260709 12:25:16.401974 26406 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.402722 18811 rpcz_store.cc:276] 0709 12:25:15.326193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.326398 (+   205us) service_pool.cc:224] Handling call
0709 12:25:15.345001 (+ 18603us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.375134 (+ 30133us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.375190 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.397201 (+ 22011us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.749106 (+351905us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:15.749166 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.749278 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.401967 (+652689us) raft_consensus.cc:1685] finished
0709 12:25:16.401994 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.402168 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4681,"raft.queue_time_us":22910,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":32229,"thread_start_us":242819,"threads_started":1,"wal-append.queue_time_us":618394}
I20260709 12:25:16.404240 26406 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:16.299110 26276 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.325783 18016 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 14.
I20260709 12:25:16.327209 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:16.406111 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79059.9us
I20260709 12:25:16.329249 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:16.407047 26276 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:16.407595 18167 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:16.330595 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.408036 18094 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:16.345422 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.285202 26094 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.409313 26416 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:25:16.410468 26137 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.373045 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: true dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:16.366307 18169 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:16.407157 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:16.412034 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.377072 18015 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260709 12:25:16.375722 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75990.7us
I20260709 12:25:16.384250 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.413898 26419 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:16.414448 26419 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.414861 26419 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:16.415071 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:16.240875 26338 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:16.390337 26411 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
W20260709 12:25:16.390664 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89644.5us
I20260709 12:25:16.293072 25812 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:16.379825 25675 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:16.409255 26260 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:16.315256 26359 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.410585 18094 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:16.410801 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:16.433624 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.434021 18090 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:16.413436 26383 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:16.416297 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:16.389071 26408 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:25:16.413491 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.336160 26399 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.384588 26395 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:16.420239 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.401546 25819 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:16.374467 26123 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:16.383010 26227 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.430601 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:16.432114 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.434913 18022 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
W20260709 12:25:16.386600 26404 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:25:16.446566 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:16.371757 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.451210 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.387814 26388 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:16.454479 18171 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:16.369966 26413 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.066s
I20260709 12:25:16.421254 26075 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:16.469296 18018 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:25:16.475224 18023 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:16.476550 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.477748 18019 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:16.489969 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.491037 18022 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:16.494360 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.495546 18017 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:16.463167 26123 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.422724 26137 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.486723 26410 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:16.537941 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:16.544637 26404 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:25:16.489619 26303 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:16.457440 26383 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.553081 26383 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:16.554205 18024 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:16.562081 26424 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.562624 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.563455 18018 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:16.563218 26424 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.565258 18018 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:16.565012 25812 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.569613 26424 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:16.599421 26227 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:16.600338 26395 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.587561 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3262) took 1253 ms. Trace:
I20260709 12:25:16.623344 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.632757 18017 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:25:16.625406 18166 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1245) took 1467 ms (client timeout 1587 ms). Trace:
I20260709 12:25:16.615808 18090 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:16.482525 26388 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.588773 26351 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:16.589495 26284 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:16.630318 18170 rpcz_store.cc:276] 0709 12:25:15.333948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.524904 (+190956us) service_pool.cc:224] Handling call
0709 12:25:15.537550 (+ 12646us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.604783 (+ 67233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.604838 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.640734 (+ 35896us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.744909 (+104175us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:15.744971 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.745056 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.583366 (+838310us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b24001cff58. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c13608fb0 00007f6c1627f488 00007f6c1627fbe8 00007f6c16257398 00007f6c1625276a 00007f6c16251bb6
0709 12:25:16.587024 (+  3658us) raft_consensus.cc:1685] finished
0709 12:25:16.587064 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.587453 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":78722,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":71,"spinlock_wait_cycles":71949568,"thread_start_us":113896,"threads_started":2,"wal-append.queue_time_us":130576}
I20260709 12:25:16.562222 26425 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:16.635484 18166 rpcz_store.cc:269] 0709 12:25:15.157928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.164242 (+  6314us) service_pool.cc:224] Handling call
0709 12:25:16.625114 (+1460872us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54001b6054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:16.625305 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":651553536}
I20260709 12:25:16.586714 26205 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:16.639302 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.576030 26137 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:16.614481 26410 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:16.637332 26375 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:25:16.650517 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:16.652576 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.628986 26420 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:16.670324 18166 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:16.638082 26227 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.664383 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 219491us
I20260709 12:25:16.677132 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.641430 26351 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:16.683655 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:16.655607 26395 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:16.642223 26284 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:16.687273 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 263335us
I20260709 12:25:16.689771 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.666574 26425 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:16.820019 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:16.821487 18017 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:16.735783 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:16.823918 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.788s (SENT)
W20260709 12:25:16.825660 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 410707us
W20260709 12:25:16.832823 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 155814us
I20260709 12:25:16.834221 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:16.716799 26417 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.390s	user 0.002s	sys 0.000s
I20260709 12:25:16.837765 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:16.841041 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.840056 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.847162 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:16.848152 18021 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:16.720927 26286 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:16.733312 26388 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:16.858115 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.858793 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.859313 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:16.870893 18170 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:16.871028 18166 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:16.847442 18172 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:16.871731 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.839517 26420 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:16.685416 26419 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:16.881680 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114903us
W20260709 12:25:16.882723 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.883827 18138 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.626s (SENT)
I20260709 12:25:16.884377 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.686268 26227 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.872495 18020 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:16.888207 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.891898 18016 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:16.893411 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:16.894498 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:16.895728 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:16.896451 18807 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:16.896891 18807 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:16.897861 18807 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.908990 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.910753 18807 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.875967 26284 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:16.915537 26284 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:16.918615 26428 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:25:16.919487 26227 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:16.866285 26286 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.922415 26425 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.924467 26286 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:16.925174 18807 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:25:16.927371 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:16.928195 18020 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:16.887356 26438 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:16.930696 18815 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:25:16.934926 18018 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:25:16.948823 18020 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:16.939841 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.868269 26418 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.949760 18017 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:16.960574 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:16.960409 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:16.969274 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:16.970932 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:16.971561 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87770us
I20260709 12:25:16.943631 26420 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:16.938977 26439 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:16.985711 18138 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.657s (SENT)
I20260709 12:25:16.983783 18020 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:16.959676 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:16.885389 26351 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:16.960912 18021 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:25:17.002209 26437 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:25:17.002409 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.003391 18021 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:17.004876 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.006286 26094 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:25:17.006932 26094 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:17.007251 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.007982 18809 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:25:17.009176 26406 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.003s
I20260709 12:25:16.998673 18016 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:25:16.883257 26410 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 21.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.010764 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.014493 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.016549 26422 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:16.995229 18810 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:16.997370 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:16.996223 18807 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 24. Ops: []
W20260709 12:25:17.019467 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.774s (SENT) [suppressed 39 similar messages]
I20260709 12:25:17.019580 26257 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:17.020629 18138 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.877s (SENT)
I20260709 12:25:16.996315 18812 raft_consensus.cc:2368] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26.
I20260709 12:25:17.016974 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:17.021507 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.022995 18813 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:17.023968 26205 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:17.002218 26351 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.026835 26419 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:17.027704 18020 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:17.007987 26094 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.029772 26419 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.030483 26419 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.031690 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:16.996343 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.038336 18017 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:17.024047 18138 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.011s (SENT)
I20260709 12:25:17.022135 26443 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:17.041805 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.044059 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.045014 18021 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:17.042057 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:17.043907 26443 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:17.047955 26359 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:17.048031 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.047043 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.050163 26418 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:16.977340 26440 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:17.052543 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.048997 26443 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.045060 18020 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:17.054705 26303 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:17.084293 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.086800 18020 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:17.084995 18017 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:17.108675 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.109622 18017 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:17.091030 18780 connection.cc:256] RPC call timeout handler was delayed by -1.2115374049999446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
I20260709 12:25:17.116274 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.117323 18017 raft_consensus.cc:2376] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23. Re-sending same reply.
I20260709 12:25:17.119267 18021 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:17.120626 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.121570 18021 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.121872 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:17.124445 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 164095us
I20260709 12:25:17.102854 26443 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:17.127840 18020 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:17.129462 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.106567 26303 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:17.142340 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.149065 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:17.149425 26123 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.543s	user 0.014s	sys 0.001s
I20260709 12:25:17.154726 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.155712 18017 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
W20260709 12:25:17.159225 26432 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.221s	user 0.001s	sys 0.001s
I20260709 12:25:17.172995 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.176126 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:17.176635 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127669us
I20260709 12:25:17.177006 18017 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:17.180053 26338 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:17.149147 26442 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:25:17.142563 26297 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:17.198980 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.203899 18021 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:17.205171 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.212496 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:17.217705 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.218717 18090 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:17.221514 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79888.3us
I20260709 12:25:17.232851 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.234476 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.234913 26134 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.235142 18019 raft_consensus.cc:2376] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26. Re-sending same reply.
I20260709 12:25:17.235507 26134 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:17.250733 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:17.251322 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.238807 18168 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:17.236176 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:17.211786 26381 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.166s	user 0.007s	sys 0.001s
I20260709 12:25:17.275890 18094 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:17.278179 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.154801 26123 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:17.299950 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44552.6us
I20260709 12:25:17.302891 18017 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:17.309973 18090 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:17.312078 18138 leader_election.cc:336] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.877s (SENT)
I20260709 12:25:17.224519 25812 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:17.301796 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.316915 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.314911 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.315359 18092 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 16.
I20260709 12:25:17.254930 26395 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.320379 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:17.323304 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.313283 26134 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:17.316522 26303 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.335054 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.336637 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.337347 18017 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:25:17.338511 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.338675 26419 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:17.340094 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.340795 18019 raft_consensus.cc:2376] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23. Re-sending same reply.
I20260709 12:25:17.335353 18021 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:17.341810 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.343281 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.346249 18021 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:17.346823 18019 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:17.223935 26297 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.090s
I20260709 12:25:17.351588 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:17.352897 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146123us
I20260709 12:25:17.352545 18019 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:17.356772 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:17.355132 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1243) took 2272 ms. Trace:
I20260709 12:25:17.361268 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.343613 26367 raft_consensus.cc:2764] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:25:17.358969 18173 rpcz_store.cc:276] 0709 12:25:15.079527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.083877 (+  4350us) service_pool.cc:224] Handling call
0709 12:25:15.104405 (+ 20528us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.104903 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.104942 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.105613 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.411100 (+305487us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:15.411170 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.411288 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.191596 (+1780308us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:17.344560 (+152964us) raft_consensus.cc:1685] finished
0709 12:25:17.344587 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.349986 (+  5399us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1475,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":153,"spinlock_wait_cycles":8924416,"thread_start_us":304836,"threads_started":1,"wal-append.queue_time_us":834750}
I20260709 12:25:17.360123 26425 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:17.364372 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:17.364650 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.365754 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:17.342628 26123 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:17.369179 18812 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 17: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:17.370182 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53489.7us
I20260709 12:25:17.365306 26349 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:17.371382 25819 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:17.371621 26349 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:17.372006 25819 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:17.372879 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3076) took 1500 ms. Trace:
I20260709 12:25:17.372694 26349 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.374207 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.374511 25819 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:17.376073 18019 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:17.370718 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.380923 18021 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.380697 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:17.373342 18024 rpcz_store.cc:276] 0709 12:25:15.872553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.889316 (+ 16763us) service_pool.cc:224] Handling call
0709 12:25:15.909966 (+ 20650us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.554760 (+644794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.372547 (+817787us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:17.372803 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1480927104}
I20260709 12:25:17.355865 26419 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:17.365371 26458 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:25:17.385108 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.349601 26395 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.389107 18017 raft_consensus.cc:2376] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21. Re-sending same reply.
I20260709 12:25:17.396212 26349 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:17.397464 26395 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:17.398742 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.399013 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.400348 26413 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.400465 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:17.400992 26413 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.401473 26455 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:17.393752 26423 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:17.402230 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.402554 26423 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:17.403198 18813 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:17.403208 26423 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.403581 26413 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:17.383817 26425 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:17.405453 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.405772 26425 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.406575 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.408268 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:17.412633 25819 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:17.414525 18092 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:25:17.416849 18091 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:17.417320 18094 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:17.418736 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.402380 26455 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:17.405794 18810 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:17.419421 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:17.419749 18096 raft_consensus.cc:2410] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:17.420486 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.435307 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.436419 18095 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:17.438536 26419 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:17.438745 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4136) took 1122 ms. Trace:
I20260709 12:25:17.439168 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.439265 18023 rpcz_store.cc:276] 0709 12:25:16.316031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.316295 (+   264us) service_pool.cc:224] Handling call
0709 12:25:17.438499 (+1122204us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:17.438646 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139131136}
I20260709 12:25:17.440188 26419 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.440835 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.441695 18099 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:17.442878 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.443616 18098 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.392266 26094 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.440721 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.451114 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.461367 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.468142 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.472189 18096 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:17.478511 18021 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:17.449859 26422 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:17.456931 26424 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:17.487341 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1502) took 1068 ms. Trace:
I20260709 12:25:17.521390 18811 rpcz_store.cc:276] 0709 12:25:16.419427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.419906 (+   479us) service_pool.cc:224] Handling call
0709 12:25:16.439468 (+ 19562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.843938 (+404470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.844145 (+   207us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.150626 (+306481us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b2c000fafd8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:17.156411 (+  5785us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.321950 (+165539us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:17.322013 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.322095 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.475060 (+152965us) raft_consensus.cc:1685] finished
0709 12:25:17.475087 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.487250 (+ 12163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":166049,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":185,"spinlock_wait_cycles":16806144,"thread_start_us":544798,"threads_started":2,"wal-append.queue_time_us":209707}
I20260709 12:25:17.476336 26260 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.489276 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.457950 26094 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:17.474238 26257 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:17.495100 26414 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.167s	user 0.005s	sys 0.007s
I20260709 12:25:17.498776 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:17.499876 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.501250 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3088) took 1540 ms. Trace:
I20260709 12:25:17.501642 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:17.504276 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.505798 18807 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 17. (term mismatch)
I20260709 12:25:17.546182 18807 pending_rounds.cc:85] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 16
I20260709 12:25:17.546571 18807 pending_rounds.cc:107] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 21.17
I20260709 12:25:17.546890 18807 raft_consensus.cc:2889] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.501276 18812 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:17.513063 18098 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:17.550787 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.522s (SENT)
I20260709 12:25:17.513566 18813 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:17.514487 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:17.515178 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:25:17.484351 26455 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:17.488246 26137 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:17.544801 18015 rpcz_store.cc:276] 0709 12:25:15.960546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.964799 (+  4253us) service_pool.cc:224] Handling call
0709 12:25:16.367708 (+402909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.413431 (+ 45723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.501007 (+1087576us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b54001d2250. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:17.501184 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":908219264}
W20260709 12:25:17.525379 26284 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.609s	user 0.012s	sys 0.006s
I20260709 12:25:17.585641 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.587759 18098 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:25:17.591643 17907 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 76525.7us
I20260709 12:25:17.524765 26424 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:17.588917 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:17.553052 26443 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.602993 18016 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:17.568966 26137 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.605130 26349 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:17.603843 26424 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:17.609519 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.598369 26425 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.018s
I20260709 12:25:17.553946 26422 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.618224 18090 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:17.608454 26094 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:17.622228 26094 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.610932 26137 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.622880 26094 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:17.625447 26094 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:17.630589 26466 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.063s	user 0.000s	sys 0.003s
I20260709 12:25:17.632897 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.634032 18098 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:17.635854 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.640748 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.587s (SENT)
I20260709 12:25:17.612262 26349 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:17.610357 26443 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.667749 26303 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:25:17.683082 26349 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:17.679438 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 227010us
I20260709 12:25:17.689383 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.692557 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.693629 18098 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 25 because replica has last-logged OpId of term: 24 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 19.
I20260709 12:25:17.696432 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.690083 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:17.699236 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.699785 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.700170 18095 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:17.702132 18094 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24. Re-sending same reply.
I20260709 12:25:17.702394 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.703109 18092 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:17.703879 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:17.705982 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.706738 18090 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:17.719035 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.724314 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:17.697512 26440 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:25:17.731616 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:17.733541 26478 raft_consensus.cc:2764] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:25:17.733374 26476 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:17.731849 26440 consensus_queue.cc:1243] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 22.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:17.710074 26419 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:17.709033 26303 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:17.744333 18096 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.20-23.20]
I20260709 12:25:17.711447 26359 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:17.746883 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.713497 26351 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.749513 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.736649 26434 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:17.751832 18020 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:17.753855 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:17.729143 26260 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:17.753686 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.756873 18017 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:17.749940 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.758919 18814 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:17.758042 18023 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:17.759975 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.760448 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.760633 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.761523 18020 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:17.763703 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:17.761137 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:17.767693 18021 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:25:17.768273 18021 pending_rounds.cc:85] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 14
I20260709 12:25:17.768540 18021 pending_rounds.cc:107] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 19.15
W20260709 12:25:17.768668 26471 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:25:17.768796 18021 raft_consensus.cc:2889] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:17.769436 26474 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
W20260709 12:25:17.776932 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 462057us
W20260709 12:25:17.779366 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.783118 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.756086 26434 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.793673 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.788715 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:17.788003 18096 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:17.799810 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.800686 18017 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:17.762826 26413 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:17.805505 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77375.1us
I20260709 12:25:17.752318 26359 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:17.845122 18017 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.853766 18021 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:25:17.805392 26422 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:17.797915 26434 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.845959 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:17.753196 26351 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:17.853379 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.856801 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:17.867123 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:17.867285 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:17.838380 26475 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 12:25:17.864526 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.841658 26413 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:17.880391 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.890781 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:17.862536 26303 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:17.896512 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.909143 18096 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:17.902056 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 113477us
I20260709 12:25:17.911725 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:17.839136 26395 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:25:17.868913 26480 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.920943 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 608890us
I20260709 12:25:17.864526 26094 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:25:17.924136 26094 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:17.924611 26094 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.924985 26094 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.925364 26399 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:17.908352 26423 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.927942 26297 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:17.928514 26297 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:17.879967 26257 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.883126 26413 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.916414 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:17.935561 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:17.875808 26438 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:17.798071 26478 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.087s
I20260709 12:25:17.838809 26479 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.087s
I20260709 12:25:17.920881 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:17.922550 18138 leader_election.cc:336] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.023s (SENT)
I20260709 12:25:17.870677 26123 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.929260 18093 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:17.908327 18017 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:17.886511 26477 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:17.941147 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:17.865054 26424 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:17.942569 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3161) took 1621 ms (client timeout 1704 ms). Trace:
W20260709 12:25:17.946764 18167 rpcz_store.cc:269] 0709 12:25:16.321519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.338135 (+ 16616us) service_pool.cc:224] Handling call
0709 12:25:17.942260 (+1604125us) spinlock_profiling.cc:243] Waited 67.9 ms on lock 0x7b5400050ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:17.942481 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156183296}
W20260709 12:25:17.937453 26475 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:25:17.932078 26297 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.928872 26423 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:17.978053 18137 leader_election.cc:400] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26.
I20260709 12:25:17.982564 18137 leader_election.cc:403] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:25:17.979241 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:17.984333 26482 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:25:17.941941 26438 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:17.988485 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 245922us
I20260709 12:25:17.992152 18096 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:17.919888 26303 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.974490 26425 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:17.977023 26440 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.977788 26463 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:17.982898 26399 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:18.021159 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:17.987615 26438 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:18.028501 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105984us
I20260709 12:25:18.035794 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.029189 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.034432 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:18.034674 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.037266 18166 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:18.037547 18165 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:18.030383 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:18.035851 18138 leader_election.cc:336] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.102s (SENT)
I20260709 12:25:18.039003 18167 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:18.042388 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.043251 18170 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:18.044404 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:18.047004 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 103250us
I20260709 12:25:18.047567 18169 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:18.048280 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:17.948645 26123 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:17.939097 26480 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.053376 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.053707 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.054406 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.055013 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.090803 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:18.091691 18166 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.094321 18165 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:18.096041 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 202239us
I20260709 12:25:18.097311 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:18.096359 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48161.1us
I20260709 12:25:18.020539 26494 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:18.103340 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.104578 18171 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:18.091843 18167 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:18.099704 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:18.094532 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:18.050293 26491 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:18.107770 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:18.054220 18096 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:25:18.121942 18096 raft_consensus.cc:606] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:18.122509 18096 raft_consensus.cc:614] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:18.131377 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:18.135805 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.136437 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:18.137629 18017 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 17: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:18.138185 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:18.042058 26440 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.035121 26425 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:18.176836 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 141052us
I20260709 12:25:18.146356 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:18.177719 18138 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.151s (SENT)
I20260709 12:25:18.178298 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.178377 18167 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:18.099927 26495 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:18.095854 26399 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:18.183187 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.188508 18171 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:18.120672 26423 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:18.186640 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:18.119959 26246 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:18.133031 26395 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:18.103785 26478 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.081s
W20260709 12:25:18.196288 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 143355us
I20260709 12:25:18.199821 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:18.204276 18165 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 17: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:18.192339 18166 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:18.159291 26498 raft_consensus.cc:2764] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:25:18.176090 25812 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:18.207954 18167 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24. Re-sending same reply.
I20260709 12:25:18.219256 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.190088 26424 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:18.221798 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.208396 26457 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:18.193706 26359 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.196846 26423 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.145777 26500 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.103s
I20260709 12:25:18.246599 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.205149 26504 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:18.247483 18167 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:18.223984 26399 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.256845 26399 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.258687 26303 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.224757 26395 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:18.260222 26395 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 22.19, Last appended by leader: 19, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.233933 26497 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.263841 26123 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:18.264818 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 87151.8us
W20260709 12:25:18.265551 18138 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.160s (SENT)
I20260709 12:25:18.265990 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.266966 26499 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.267324 18138 leader_election.cc:336] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.212s (SENT)
I20260709 12:25:18.267772 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.267547 26499 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.249164 26423 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.270886 26205 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:18.271085 26493 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:25:18.271694 26205 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:18.273139 26489 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
W20260709 12:25:18.274938 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.275707 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67833.2us
W20260709 12:25:18.248023 26421 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.763s	user 0.000s	sys 0.011s
I20260709 12:25:18.282536 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3173) took 1957 ms. Trace:
I20260709 12:25:18.288812 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.289033 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:18.290156 18167 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:18.250377 26504 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.275120 26424 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:18.256120 26502 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.312157 26137 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.689s	user 0.014s	sys 0.001s
I20260709 12:25:18.312719 26094 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.313925 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.328361 26480 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:18.331868 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.271142 26425 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.343634 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3268) took 2373 ms. Trace:
I20260709 12:25:18.346119 26297 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.347086 26094 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:18.366147 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 234882us
I20260709 12:25:18.348202 18167 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:18.345028 26434 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.287751 18172 rpcz_store.cc:276] 0709 12:25:16.325075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.882062 (+556987us) service_pool.cc:224] Handling call
0709 12:25:16.898444 (+ 16382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.927551 (+ 29107us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:25:17.070651 (+143100us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.268353 (+197702us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.771640 (+503287us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b18003f8620. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:25:18.099320 (+327680us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.103316 (+  3996us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.108890 (+  5574us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:18.108952 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.109055 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.282067 (+173012us) raft_consensus.cc:1685] finished
0709 12:25:18.282120 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.282411 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":519689,"prepare.queue_time_us":209564,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":6210,"raft.queue_time_us":525344,"raft.run_cpu_time_us":462,"raft.run_wall_time_us":68316,"spinlock_wait_cycles":9055104,"thread_start_us":602919,"threads_started":3,"wal-append.queue_time_us":185378}
I20260709 12:25:18.332652 26499 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:18.365723 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:18.347198 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.330823 26504 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:18.335575 26424 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:18.356472 18164 rpcz_store.cc:276] 0709 12:25:15.970255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.970865 (+   610us) service_pool.cc:224] Handling call
0709 12:25:16.006890 (+ 36025us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.019673 (+ 12783us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.019707 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.042465 (+ 22758us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.296961 (+254496us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:16.297009 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.297114 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.031697 (+1734583us) spinlock_profiling.cc:243] Waited 9.16 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:18.343118 (+311421us) raft_consensus.cc:1685] finished
0709 12:25:18.343145 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.343536 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4766,"raft.queue_time_us":4594,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":16256,"spinlock_wait_cycles":21069696,"thread_start_us":99551,"threads_started":1,"wal-append.queue_time_us":328419}
I20260709 12:25:18.370846 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.371723 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.369650 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.370561 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:18.376207 18809 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:25:18.364941 26509 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.365355 26478 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.380806 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:18.382998 18808 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:25:18.381371 26503 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 12:25:18.386179 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:18.386477 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:18.387157 18164 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.387046 26495 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:18.387565 26094 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:18.391368 26430 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.525s	user 0.000s	sys 0.003s
W20260709 12:25:18.394572 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3168) took 2070 ms (client timeout 1973 ms). Trace:
W20260709 12:25:18.395192 18168 rpcz_store.cc:269] 0709 12:25:16.323640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.490341 (+166701us) service_pool.cc:224] Handling call
0709 12:25:18.394289 (+1903948us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:18.394467 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1600667392}
I20260709 12:25:18.396999 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.398831 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.400822 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:18.400945 26520 raft_consensus.cc:2764] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:25:18.401849 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.402909 18165 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:18.385972 26499 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.384105 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:18.404099 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.380328 26424 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:18.377625 26497 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:18.341012 26502 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.395404 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:18.405750 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.369971 26479 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:25:18.382611 26498 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:18.348771 26137 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.428791 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:18.429088 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48444.1us
W20260709 12:25:18.496024 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.421485 18165 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:25:18.432735 18810 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:25:18.421705 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1467) took 1065 ms. Trace:
I20260709 12:25:18.431255 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.447824 18164 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:18.416091 18092 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:25:18.422232 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4159) took 1494 ms (client timeout 1788 ms). Trace:
I20260709 12:25:18.512395 18171 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [20.16-21.17]
I20260709 12:25:18.463408 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:18.460286 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.515137 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:18.511713 18807 rpcz_store.cc:276] 0709 12:25:17.356356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.356690 (+   334us) service_pool.cc:224] Handling call
0709 12:25:17.485887 (+129197us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.574089 (+ 88202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.421497 (+847408us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54005ad550. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:18.421641 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296125696}
W20260709 12:25:18.518391 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87223.5us
W20260709 12:25:18.513157 18016 rpcz_store.cc:269] 0709 12:25:16.927573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.046629 (+119056us) service_pool.cc:224] Handling call
0709 12:25:18.422059 (+1375430us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:18.422170 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1271373184}
I20260709 12:25:18.519407 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.512529 18092 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:18.521093 18021 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:18.522939 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.419587 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:18.524820 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105331us
I20260709 12:25:18.528357 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:18.529268 18098 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:18.339389 25812 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.534310 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:18.535408 18024 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:18.537370 18814 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:18.533535 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:18.462376 26433 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.522s	user 0.003s	sys 0.001s
W20260709 12:25:18.412900 26492 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.220s	user 0.001s	sys 0.002s
I20260709 12:25:18.562134 18096 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:18.562645 18164 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.20-23.20]
I20260709 12:25:18.465638 26338 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:18.449342 26246 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.566609 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:18.567710 26123 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.580999 26123 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.582110 26395 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:18.582616 26395 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.582978 26395 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:18.416586 26495 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:18.584189 26413 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.636s	user 0.004s	sys 0.008s
I20260709 12:25:18.584487 26495 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.585120 26495 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.584818 26413 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.509714 26303 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.565579 18092 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.420841 26443 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:18.594146 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:18.423822 26503 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:25:18.421125 26472 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.740s	user 0.000s	sys 0.002s
I20260709 12:25:18.469560 26520 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:18.572386 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.029s	sys 0.051s
I20260709 12:25:18.418877 26434 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:18.411310 26399 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:18.425014 26403 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:18.602303 26303 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:18.617313 26518 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:25:18.574644 26499 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.417249 26502 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.422865 26297 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:18.579725 26246 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.426129 26498 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:18.447288 26440 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:18.571264 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51975.9us
I20260709 12:25:18.532265 26419 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.590699 18099 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:18.528539 26463 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:18.405997 26508 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.228s
I20260709 12:25:18.606124 26520 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:18.672945 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 300299us
I20260709 12:25:18.673902 18092 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.576730 26491 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26.
I20260709 12:25:18.613513 26303 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.584261 26123 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:18.600878 26503 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:18.609127 26481 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:18.719172 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.724587 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.606273 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.030s	sys 0.051s
I20260709 12:25:18.426240 26509 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:18.676450 26489 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:25:18.414557 26497 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.533020 26514 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.158s
I20260709 12:25:18.433432 26478 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.713438 26498 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:18.684442 26338 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:18.608322 26403 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:18.779414 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.861701 18166 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:18.866847 26513 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:18.867369 18166 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:18.864238 26491 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.868058 26338 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.867386 26513 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:18.865185 26257 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.872s	user 0.017s	sys 0.000s
I20260709 12:25:18.870347 26257 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:18.879221 26497 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:18.880792 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110679us
I20260709 12:25:18.751127 26137 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.748137 26494 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.690184 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:18.854638 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 331841us
I20260709 12:25:18.868193 26491 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.748741 26520 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:18.799754 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 858632us
I20260709 12:25:18.884234 18168 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:18.885941 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:18.790750 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76050.7us
I20260709 12:25:18.883442 18172 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:18.895916 18017 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:18.898738 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:18.900574 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 210546us
I20260709 12:25:18.883523 18166 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 12:25:18.903744 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.905712 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:18.907371 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:18.904349 25819 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:18.890872 26419 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.893632 26497 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.924275 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:25:18.905910 26494 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.928836 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:18.930995 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1674) took 1183 ms. Trace:
I20260709 12:25:18.958371 18808 rpcz_store.cc:276] 0709 12:25:17.747886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.748057 (+   171us) service_pool.cc:224] Handling call
0709 12:25:17.748402 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.385633 (+637231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.930632 (+544999us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b54005429d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:18.930915 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1458819840}
I20260709 12:25:18.921036 18166 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:18.935634 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:18.943308 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1299) took 1250 ms. Trace:
W20260709 12:25:18.962605 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55302.1us
I20260709 12:25:18.967974 18092 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 17. (term mismatch)
I20260709 12:25:18.961084 18093 rpcz_store.cc:276] 0709 12:25:17.692856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.696338 (+  3482us) service_pool.cc:224] Handling call
0709 12:25:17.708553 (+ 12215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.939575 (+231022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.942961 (+1003386us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400291b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:18.943186 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506931968}
I20260709 12:25:18.968851 18092 pending_rounds.cc:85] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 16
I20260709 12:25:18.969230 18092 pending_rounds.cc:107] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 21.17
I20260709 12:25:18.969692 18092 raft_consensus.cc:2889] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:18.963500 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:18.973701 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:18.970598 26438 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.864s	user 0.013s	sys 0.001s
I20260709 12:25:18.984691 26438 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:18.931839 26491 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:18.979302 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.993248 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3404) took 1224 ms. Trace:
I20260709 12:25:18.746210 26463 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:18.998407 18166 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:18.948283 26521 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:25:19.003155 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:18.994089 18092 rpcz_store.cc:276] 0709 12:25:17.768945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.415525 (+646580us) service_pool.cc:224] Handling call
0709 12:25:18.415945 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.992869 (+576924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.993162 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.004920 18093 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:18.925748 26525 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:25:19.005882 26440 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.005987 18094 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:19.006299 26438 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:18.943806 26478 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:18.967998 26481 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:19.016604 26205 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.744s	user 0.010s	sys 0.003s
I20260709 12:25:19.023118 18091 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:18.936622 26246 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:18.926750 26403 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:18.933215 26359 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.097s
I20260709 12:25:18.923174 26286 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.111s
I20260709 12:25:18.934305 26419 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.035614 26413 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:19.052090 26499 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:19.039543 26519 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
I20260709 12:25:19.040628 26517 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:19.044047 26351 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.067117 26205 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:19.078403 26481 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:18.987474 26502 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:18.972700 26137 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:19.046518 26455 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:25:18.965935 26257 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.092s
W20260709 12:25:19.038045 26423 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.767s	user 0.015s	sys 0.000s
I20260709 12:25:18.948992 26497 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.108783 18092 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:19.170637 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4324) took 1024 ms. Trace:
I20260709 12:25:19.171056 18092 rpcz_store.cc:276] 0709 12:25:18.146054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.052844 (+906790us) service_pool.cc:224] Handling call
0709 12:25:19.103830 (+ 50986us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.170269 (+ 66439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.170565 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:19.110845 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 208127us
I20260709 12:25:19.123140 26499 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:19.053757 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50710.3us
I20260709 12:25:18.999277 26524 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.153s
W20260709 12:25:19.121795 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 140257us
I20260709 12:25:19.056070 26515 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:19.057677 26491 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
W20260709 12:25:19.140437 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 217635us
I20260709 12:25:19.113490 26443 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.111407 26367 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:19.113130 26517 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:19.138172 26395 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.555s	user 0.012s	sys 0.003s
I20260709 12:25:19.177311 18169 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:19.134876 26502 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.237774 18092 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:19.188637 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:19.115976 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.139463 26137 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.183588 26440 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:19.170038 26504 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.770s	user 0.013s	sys 0.002s
W20260709 12:25:19.185070 26495 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.599s	user 0.011s	sys 0.005s
I20260709 12:25:19.231987 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3105) took 1096 ms. Trace:
W20260709 12:25:19.266783 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 150948us
W20260709 12:25:19.278406 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 90975us
W20260709 12:25:19.280264 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65829.4us
I20260709 12:25:19.232695 26517 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:19.183005 26423 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.271852 18015 rpcz_store.cc:276] 0709 12:25:18.135447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.135616 (+   169us) service_pool.cc:224] Handling call
0709 12:25:18.182514 (+ 46898us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.194315 (+ 11801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.194380 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.216276 (+ 21896us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.382198 (+165922us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:18.382249 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.382323 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.231707 (+849384us) raft_consensus.cc:1685] finished
0709 12:25:19.231734 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.231907 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33431,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":11460,"thread_start_us":109718,"threads_started":1,"wal-append.queue_time_us":460769}
I20260709 12:25:19.210074 26524 consensus_queue.cc:1243] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 24.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:19.329442 18092 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:25:19.332417 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143874us
I20260709 12:25:19.269522 26137 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:19.268538 26205 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.248142 26502 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:19.345520 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:19.399991 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 120597us
I20260709 12:25:19.284613 26495 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.350754 26515 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.402748 18811 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:19.406227 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.412324 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:19.417732 18815 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 12:25:19.414168 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134828us
I20260709 12:25:19.249979 26395 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.277333 26504 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.437125 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:19.419211 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:19.441147 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.397173 26338 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.394279 26526 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:19.283111 26440 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.396515 26524 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.456561 26529 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:19.437295 26441 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 2.496s	user 0.004s	sys 0.015s
I20260709 12:25:19.474303 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:19.394722 26351 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 12:25:19.419080 26523 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.412156 26494 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.438423 26257 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:19.404831 26359 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 12:25:19.495586 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3187) took 1449 ms. Trace:
I20260709 12:25:19.455054 26286 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.512629 18170 rpcz_store.cc:276] 0709 12:25:18.046561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.053435 (+  6874us) service_pool.cc:224] Handling call
0709 12:25:19.495324 (+1441889us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54001b5b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:19.495493 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1826540416}
I20260709 12:25:19.475884 26205 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:19.518147 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114972us
I20260709 12:25:19.413472 26541 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260709 12:25:19.397716 26481 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:19.531610 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 115902us
I20260709 12:25:19.487825 26419 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:19.540268 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 194894us
I20260709 12:25:19.472458 26515 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:25:19.470655 26338 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.415580 26499 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.550015 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.502206 26543 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:19.476903 26526 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.522003 26403 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:25:19.484431 26524 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.524401 26286 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.557860 26491 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:19.576723 26424 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.168s	user 0.012s	sys 0.004s
I20260709 12:25:19.499908 26523 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.609246 26549 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:19.610374 26338 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.610194 26549 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:19.614531 26546 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:25:19.614938 26286 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.615679 18095 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:19.616398 26550 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:19.616706 26503 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.617365 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3402) took 1849 ms. Trace:
I20260709 12:25:19.619272 26549 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.567829 26497 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:19.603137 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70315.5us
W20260709 12:25:19.627516 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 232479us
I20260709 12:25:19.628153 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.620224 26495 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.620586 26478 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:19.613418 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:19.636775 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1471) took 1167 ms. Trace:
I20260709 12:25:19.637228 18807 rpcz_store.cc:276] 0709 12:25:18.469215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.613061 (+1143846us) service_pool.cc:224] Handling call
0709 12:25:19.636692 (+ 23631us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.621567 26503 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:19.623319 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1260) took 1514 ms. Trace:
I20260709 12:25:19.644241 18169 rpcz_store.cc:276] 0709 12:25:18.109709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.395011 (+285302us) service_pool.cc:224] Handling call
0709 12:25:19.623076 (+1228065us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b5400031bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:19.623239 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830076160}
I20260709 12:25:19.614528 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3396) took 1903 ms. Trace:
W20260709 12:25:19.594257 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.552628 26529 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:25:19.624936 18095 rpcz_store.cc:276] 0709 12:25:17.768228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.210527 (+442299us) service_pool.cc:224] Handling call
0709 12:25:19.615254 (+1404727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.616099 (+   845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.617112 (+  1013us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54002972d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:19.617296 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3178474880}
I20260709 12:25:19.588303 26538 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.645918 18099 rpcz_store.cc:276] 0709 12:25:17.710943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.711257 (+   314us) service_pool.cc:224] Handling call
0709 12:25:19.614283 (+1903026us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54002972d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:19.614425 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1596522240}
I20260709 12:25:19.625278 26494 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.651620 18173 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:19.626262 26359 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.598896 26423 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:19.629895 26530 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.309s	user 0.000s	sys 0.002s
I20260709 12:25:19.663275 18092 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:19.560813 26543 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.649263 26416 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.669679 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.624408 26424 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.673147 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:19.642484 26504 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:19.676864 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:19.677721 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:19.669032 26535 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.266s	user 0.000s	sys 0.002s
W20260709 12:25:19.682217 26539 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.296s	user 0.000s	sys 0.002s
W20260709 12:25:19.682487 26537 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:25:19.683504 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3384) took 2258 ms. Trace:
I20260709 12:25:19.644086 26403 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:19.702204 18097 rpcz_store.cc:276] 0709 12:25:17.412410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.429404 (+ 16994us) service_pool.cc:224] Handling call
0709 12:25:17.429798 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.446811 (+ 17013us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:17.475675 (+ 28864us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.369809 (+1894134us) spinlock_profiling.cc:243] Waited 7.77 ms on lock 0x7b18002a2440. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:25:19.670604 (+300795us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.670670 (+    66us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:19.670721 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.670793 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.670930 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19123,"prepare.queue_time_us":29078,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":151553,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":17864576,"thread_start_us":260642,"threads_started":2,"wal-append.queue_time_us":2005234}
I20260709 12:25:19.644335 26478 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:19.677721 26523 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.649664 26497 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
W20260709 12:25:19.706512 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 179347us
I20260709 12:25:19.645263 26550 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:25:19.711863 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:19.698462 26395 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:19.732776 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74173.4us
W20260709 12:25:19.734308 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 315220us
I20260709 12:25:19.678260 18095 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 24 index: 20. (index mismatch)
I20260709 12:25:19.738404 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:19.684065 26351 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.695842 26491 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:19.724905 18097 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:19.671442 26359 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.690434 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:19.743320 18092 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:19.699714 26515 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.725085 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:19.660171 26538 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.702193 26284 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:19.727977 26526 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:19.727602 26478 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.731031 26440 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.662472 26494 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.699391 26495 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.735263 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:19.702714 26286 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.703526 26548 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:19.780267 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.780128 26548 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.781082 18016 raft_consensus.cc:2376] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19. Re-sending same reply.
I20260709 12:25:19.780165 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.782807 18021 raft_consensus.cc:2376] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26. Re-sending same reply.
I20260709 12:25:19.783026 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.683961 26423 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:25:19.748169 18091 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:19.695013 26416 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.751456 26403 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:25:19.754815 18165 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:19.755962 26424 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.787477 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3288) took 1210 ms. Trace:
I20260709 12:25:19.787945 18165 rpcz_store.cc:276] 0709 12:25:18.577366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.754058 (+1176692us) service_pool.cc:224] Handling call
0709 12:25:19.754551 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.787135 (+ 32584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.787390 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.766559 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1681) took 1337 ms. Trace:
I20260709 12:25:19.766979 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.735628 26554 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.746140 26523 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:19.776407 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.777302 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.796242 18024 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:25:19.779148 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:19.784201 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49036.5us
I20260709 12:25:19.789933 18809 rpcz_store.cc:276] 0709 12:25:18.428926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.431869 (+  2943us) service_pool.cc:224] Handling call
0709 12:25:18.517771 (+ 85902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.794791 (+277020us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.794848 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.795720 (+   872us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.112050 (+316330us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:19.112090 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.112444 (+   354us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.766284 (+653840us) raft_consensus.cc:1685] finished
0709 12:25:19.766311 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.766486 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":78847,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":455505,"threads_started":2,"wal-append.queue_time_us":379962}
I20260709 12:25:19.769467 26543 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.791972 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3285) took 1252 ms. Trace:
I20260709 12:25:19.801347 18023 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:19.757287 26351 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.809439 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.814868 18097 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:25:19.815414 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.820241 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.804946 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.810241 18166 rpcz_store.cc:276] 0709 12:25:18.540082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.048037 (+507955us) service_pool.cc:224] Handling call
0709 12:25:19.048443 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.079098 (+ 30655us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.079152 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.079867 (+   715us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.243158 (+163291us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:19.569625 (+326467us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.570040 (+   415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.570148 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.791710 (+221562us) raft_consensus.cc:1685] finished
0709 12:25:19.791735 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.791889 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30120,"prepare.queue_time_us":456020,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3414,"raft.queue_time_us":52579,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":15905,"thread_start_us":317720,"threads_started":2,"wal-append.queue_time_us":252401}
I20260709 12:25:19.823939 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:19.825179 18166 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:19.826937 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.832008 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.834233 18021 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:19.835824 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4319) took 2041 ms. Trace:
I20260709 12:25:19.763012 26515 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.769497 26284 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.838140 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.860574 18020 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:19.838383 18024 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:19.838209 18016 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:19.842383 26559 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.804216 26523 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.843842 26420 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.844144 26542 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.844463 26549 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.883314 26549 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.883793 18095 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:19.885020 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:19.804946 26554 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.842383 26551 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:19.846210 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3286) took 1305 ms. Trace:
I20260709 12:25:19.846951 26552 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:19.854103 26351 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.838783 26422 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:25:19.783948 26286 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.830760 26526 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:19.859462 26538 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.861625 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.867426 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3287) took 1325 ms. Trace:
I20260709 12:25:19.802838 26553 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:19.890302 26551 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.783522 26495 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.895678 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:19.836262 18090 rpcz_store.cc:276] 0709 12:25:17.794624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.932052 (+1137428us) service_pool.cc:224] Handling call
0709 12:25:18.932502 (+   450us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:18.962737 (+ 30235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.962774 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:18.999204 (+ 36430us) log.cc:844] Serialized 52 byte log entry
0709 12:25:19.324560 (+325356us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:19.539312 (+214752us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.695036 (+155724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.695133 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.835569 (+140436us) raft_consensus.cc:1685] finished
0709 12:25:19.835595 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.835756 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29703,"prepare.queue_time_us":291034,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":108389,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":21206,"thread_start_us":601230,"threads_started":3,"wal-append.queue_time_us":391886}
I20260709 12:25:19.899098 18023 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:19.886159 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.915975 18016 raft_consensus.cc:2376] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21. Re-sending same reply.
I20260709 12:25:19.866401 18097 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:25:19.875264 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3274) took 1601 ms (client timeout 2053 ms). Trace:
I20260709 12:25:19.810952 26478 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:19.878578 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.882462 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.857285 26416 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.886607 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.890446 18170 rpcz_store.cc:276] 0709 12:25:18.540981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.520935 (+979954us) service_pool.cc:224] Handling call
0709 12:25:19.521372 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.521782 (+   410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.521842 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.539483 (+ 17641us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.626656 (+ 87173us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:19.626705 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.626783 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.845914 (+219131us) raft_consensus.cc:1685] finished
0709 12:25:19.845946 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.846122 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":663,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":10172,"thread_start_us":46845,"threads_started":1,"wal-append.queue_time_us":201252}
I20260709 12:25:19.922379 18018 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:19.922601 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:19.894254 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:19.895215 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:19.917318 18021 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:19.902045 18091 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:25:19.916921 18168 rpcz_store.cc:269] 0709 12:25:18.274181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.396624 (+122443us) service_pool.cc:224] Handling call
0709 12:25:19.875024 (+1478400us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54003ba1d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:19.875191 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1118662528}
I20260709 12:25:19.910320 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.890163 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:19.928138 18166 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:19.920220 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:19.929313 18097 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:19.930044 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:19.933506 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.934012 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.934109 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:19.838491 26524 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.854085 26494 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.935547 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:19.937073 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:19.940311 18091 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:25:19.935448 18170 raft_consensus.cc:2368] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:19.886322 26420 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:19.950613 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:19.951385 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:19.893101 26338 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:25:19.952098 18016 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:19.952627 26420 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.885763 26549 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.954035 26504 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:19.954211 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:19.894997 26422 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:19.876859 26497 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.941776 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:19.921887 26539 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:25:19.888438 26554 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.960983 18090 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:19.925432 26413 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.896831 26286 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.921321 26526 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:19.956533 18015 raft_consensus.cc:2376] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23. Re-sending same reply.
I20260709 12:25:19.921198 26538 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.910830 26553 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:19.999328 26553 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.969728 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:19.972124 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.003196 26403 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:19.898591 26559 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:20.008334 26137 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.008s	sys 0.008s
I20260709 12:25:20.009349 26137 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.011416 26551 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.922715 26542 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.015218 26542 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.017108 26137 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.937333 26548 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.966272 26491 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:19.958786 26549 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:20.018988 26359 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:19.985092 26413 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.021543 26413 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:20.022226 26413 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:20.022655 26413 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.023015 26413 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:20.024667 26523 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:20.028154 26520 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.068s	user 0.013s	sys 0.003s
I20260709 12:25:20.036150 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3398) took 2269 ms. Trace:
W20260709 12:25:20.036375 26517 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.696s	user 0.004s	sys 0.010s
W20260709 12:25:19.973958 26498 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.082s	user 0.009s	sys 0.009s
I20260709 12:25:20.038295 26498 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.040066 26463 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.041079 26498 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:20.045058 26528 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.870s	user 0.001s	sys 0.001s
I20260709 12:25:20.045846 26538 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.048696 26538 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:20.051342 26540 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.516s	user 0.000s	sys 0.002s
I20260709 12:25:20.052970 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:20.053756 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4325) took 1897 ms. Trace:
I20260709 12:25:20.055473 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.055748 26509 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.054304 18093 rpcz_store.cc:276] 0709 12:25:18.156076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.063164 (+907088us) service_pool.cc:224] Handling call
0709 12:25:19.103934 (+ 40770us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.104364 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.104400 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.120129 (+ 15729us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.344559 (+224430us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:19.344638 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.344725 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.050419 (+705694us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:20.053356 (+  2937us) raft_consensus.cc:1685] finished
0709 12:25:20.053394 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.053686 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1249,"raft.queue_time_us":132,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":11669,"spinlock_wait_cycles":9672960,"thread_start_us":171896,"threads_started":1,"wal-append.queue_time_us":345078}
W20260709 12:25:20.057528 26533 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.924s	user 0.002s	sys 0.002s
I20260709 12:25:20.053048 25812 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:20.060704 25812 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:19.974788 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:19.932700 26525 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:25:19.995807 26284 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:19.996731 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:19.959877 26422 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:19.997064 18016 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:25:19.946468 26395 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:19.976311 26497 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:19.993084 26495 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:20.003319 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43581.4us
I20260709 12:25:20.036741 18096 rpcz_store.cc:276] 0709 12:25:17.766650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.208921 (+442271us) service_pool.cc:224] Handling call
0709 12:25:18.209416 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.575350 (+365934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.035803 (+1460453us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:20.036059 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810631424}
I20260709 12:25:20.071580 18097 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:20.028995 26520 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.105520 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.075858 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3289) took 1156 ms. Trace:
I20260709 12:25:20.085064 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.109337 18016 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:20.096653 18166 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:20.072945 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98231.9us
I20260709 12:25:20.099836 18091 raft_consensus.cc:606] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.037111 26517 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.115536 18064 leader_election.cc:400] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:20.116089 18064 leader_election.cc:403] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:25:20.114401 18091 raft_consensus.cc:614] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:20.118080 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:20.118870 18172 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.115172 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:20.120231 18166 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:20.120959 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:20.130218 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.134426 18806 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:20.082341 26551 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:20.140385 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3097) took 2385 ms. Trace:
I20260709 12:25:20.134251 26403 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 12:25:20.104351 26395 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:20.124416 26440 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18.
I20260709 12:25:20.124796 26463 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:20.137279 26523 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:20.140862 18019 rpcz_store.cc:276] 0709 12:25:17.752778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.754945 (+  2167us) service_pool.cc:224] Handling call
0709 12:25:17.756245 (+  1300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.875918 (+119673us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:17.876226 (+   308us) log.cc:844] Serialized 14 byte log entry
0709 12:25:18.087454 (+211228us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:18.087564 (+   110us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:18.087762 (+   198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.495588 (+1407826us) spinlock_profiling.cc:243] Waited 7.55 ms on lock 0x7b18002e3cc0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:20.136507 (+640919us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54002ffb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:20.136577 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.136726 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2076,"prepare.queue_time_us":3410,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":88719,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":13686,"spinlock_wait_cycles":291830784,"thread_start_us":210646,"threads_started":2,"wal-append.queue_time_us":158078}
I20260709 12:25:20.093179 26413 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.124981 26420 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:20.151170 26479 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.727s	user 0.005s	sys 0.028s
I20260709 12:25:20.116778 26504 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:25:20.164314 18166 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:20.189942 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.191460 18166 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:25:20.199760 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69637.6us
W20260709 12:25:20.179116 26479 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.755s	user 0.005s	sys 0.029s
I20260709 12:25:20.182893 26523 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:20.214905 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.146845 26422 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.197001 26515 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.136781 25812 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.220759 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:20.232970 18166 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.179332 26420 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.216872 26463 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:20.261444 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:20.261708 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57057.4us
W20260709 12:25:20.263799 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.253219 26422 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.290200 18812 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:20.259112 26502 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.861s	user 0.012s	sys 0.006s
I20260709 12:25:20.294821 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.254889 26403 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.298826 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.299736 26502 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.297518 26351 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.297739 26498 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.284216 26463 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:25:20.315052 26420 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:20.315845 26569 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:25:20.259613 26575 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:20.322227 26573 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:20.307056 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:20.367728 26403 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.375857 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.331818 26520 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.315735 25812 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:20.302301 26562 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.164s	user 0.001s	sys 0.001s
W20260709 12:25:20.346396 26561 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:25:20.351076 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1255) took 2249 ms. Trace:
I20260709 12:25:20.388736 18171 rpcz_store.cc:276] 0709 12:25:18.101707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.417864 (+316157us) service_pool.cc:224] Handling call
0709 12:25:18.512085 (+ 94221us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:18.516945 (+  4860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.350731 (+1833786us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b5400201cd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:20.350968 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215441280}
I20260709 12:25:20.389309 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.390231 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.390775 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.390945 18171 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:20.391614 18015 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:20.392326 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:20.275609 26574 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:20.395413 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.395220 26574 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.396698 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.361383 18098 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:20.362758 26502 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.363368 26351 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.364313 26498 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.407552 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3290) took 1488 ms. Trace:
I20260709 12:25:20.369642 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.296797 26515 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.371320 26517 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.380550 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.373344 26523 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.399047 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:20.393635 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.402623 18166 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:20.452608 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.455122 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.457376 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:20.468410 26478 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.542s	user 0.007s	sys 0.003s
W20260709 12:25:20.424819 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.499027 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3667465159996937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
I20260709 12:25:20.407982 18168 rpcz_store.cc:276] 0709 12:25:18.919548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.406564 (+1487016us) service_pool.cc:224] Handling call
0709 12:25:20.406972 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.407190 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.407217 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.407264 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:20.407305 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.407372 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.407493 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.238821 26395 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.401227 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:20.440001 26359 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
W20260709 12:25:20.501436 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 100319us
I20260709 12:25:20.502317 18097 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:20.441341 26574 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.442628 26351 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.378780 26575 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.444555 26403 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.377332 26497 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.452569 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:20.469729 26503 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.815s	user 0.008s	sys 0.008s
I20260709 12:25:20.470431 26581 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:20.473999 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.474506 26515 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.450143 26463 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:20.530324 18138 leader_election.cc:336] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.222s (SENT)
I20260709 12:25:20.477331 26498 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.479099 18171 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:25:20.480742 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.407351 26538 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.075s
W20260709 12:25:20.482872 26570 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:25:20.419569 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.484149 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.452651 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.404381 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.491827 18172 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:20.404197 26573 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:20.502545 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:20.533747 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1268) took 1575 ms. Trace:
I20260709 12:25:20.536351 18023 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:20.514314 18811 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:20.520484 18810 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:20.543232 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.508025 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.521271 18022 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:20.521134 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.544977 18096 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:25:20.520743 26525 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:20.522513 26418 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:20.523370 26575 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.525084 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.511605 26455 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:20.562235 26455 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:20.556504 26367 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:20.563408 26455 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:20.526813 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:25:20.499899 26524 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:20.567034 26524 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.476166 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:20.550922 26573 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.546082 26581 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.541528 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:20.541258 18171 rpcz_store.cc:276] 0709 12:25:18.958165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.478319 (+1520154us) service_pool.cc:224] Handling call
0709 12:25:20.478699 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.533374 (+ 54675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.533669 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.520695 26413 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:20.550182 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 112221us
I20260709 12:25:20.553182 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.531088 26359 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:20.557052 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.583369 18015 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.560395 26515 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.550535 26227 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:20.585616 18023 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:20.588181 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46718.2us
I20260709 12:25:20.548604 26498 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.595134 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.596313 18023 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:20.597725 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.599320 18023 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:25:20.600888 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.585240 26359 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:20.606654 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.535454 26463 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.609433 26359 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.605499 26227 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:20.611857 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.623968 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1680) took 2195 ms. Trace:
I20260709 12:25:20.634410 18023 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:20.107275 18165 rpcz_store.cc:276] 0709 12:25:18.919280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.794756 (+875476us) service_pool.cc:224] Handling call
0709 12:25:19.821096 (+ 26340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.821403 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:19.865226 (+ 43823us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.074927 (+209701us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.075263 (+   336us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.075346 (+    83us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:20.075434 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.075549 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.075752 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32023,"prepare.queue_time_us":45698,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":1956,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":24497,"thread_start_us":203493,"threads_started":1}
I20260709 12:25:20.563438 26367 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:20.632447 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.592603 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:20.634241 18806 rpcz_store.cc:276] 0709 12:25:18.428605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.428743 (+   138us) service_pool.cc:224] Handling call
0709 12:25:18.511118 (+ 82375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.135891 (+1624773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.623464 (+487573us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54007823d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:20.623798 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3732604672}
I20260709 12:25:20.643144 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.644614 18023 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:20.599694 26502 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:25:20.660717 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68242.5us
I20260709 12:25:20.667416 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.679790 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.680727 18023 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.681232 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:20.681973 18093 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.682443 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.650892 26438 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:20.655836 26502 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.685676 18096 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:20.603494 26498 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.687325 18097 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:20.659296 26524 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:20.672200 26359 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.693811 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.693602 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.683471 26420 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.701947 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.707185 18167 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:20.708461 26581 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:20.709129 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63972.9us
I20260709 12:25:20.690459 26502 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:20.716890 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.694835 26438 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:20.731230 18095 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:20.732825 18090 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 17.
I20260709 12:25:20.724882 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.714246 26573 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:20.681974 26227 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.728291 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.735117 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:20.740134 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:20.728410 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:20.739971 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.744787 26524 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:20.749645 26524 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.750218 26524 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.750177 18097 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:20.742831 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.739240 26517 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:25:20.752650 26517 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:20.754328 26517 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.757262 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.753206 26478 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:20.757977 26478 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:20.758267 26443 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:20.758726 26443 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.758827 26463 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:20.759155 26443 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:20.759097 26478 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.760782 26478 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.756174 26520 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:20.762173 26520 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:20.762687 26582 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:25:20.762817 26520 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:20.767788 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:20.768674 18170 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
I20260709 12:25:20.769934 26517 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.746345 26438 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.774211 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.740132 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:20.741068 18093 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:20.740875 18168 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:20.786268 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
W20260709 12:25:20.781946 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46953.8us
I20260709 12:25:20.792487 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:20.796669 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.798768 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.799736 18016 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.717384 26420 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.811017 18170 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:25:20.802425 18020 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:20.728135 26503 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:20.812932 18023 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:20.814548 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.808674 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:20.821359 18095 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:20.821977 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1264) took 2657 ms. Trace:
I20260709 12:25:20.822094 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.822593 18173 rpcz_store.cc:276] 0709 12:25:18.164834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.712609 (+547775us) service_pool.cc:224] Handling call
0709 12:25:19.625938 (+913329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.657056 (+ 31118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.794443 (+1137387us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b5400031bd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:20.821882 (+ 27439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2014042496}
I20260709 12:25:20.823537 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.824548 18020 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:20.827139 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.831480 18016 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:20.833705 26367 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.837500 26498 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.842518 26517 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.852563 26553 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:20.853382 26553 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.853861 26553 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:20.852684 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60405.5us
I20260709 12:25:20.857136 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.858347 26542 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:20.858776 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.859649 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.860311 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.861202 26516 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:20.852036 26503 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.846781 26443 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.866106 18023 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:20.867288 26503 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.868083 26443 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.849350 26520 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.871165 26520 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.871505 26573 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:20.871781 26581 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:20.872229 26516 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:20.871572 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44303.5us
I20260709 12:25:20.864670 18095 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:20.875059 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.882673 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4208) took 1075 ms. Trace:
I20260709 12:25:20.883425 18022 rpcz_store.cc:276] 0709 12:25:19.807435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.807677 (+   242us) service_pool.cc:224] Handling call
0709 12:25:19.813057 (+  5380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.544478 (+731421us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.882227 (+337749us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:20.882550 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1627778816}
W20260709 12:25:20.884441 26586 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.137s	user 0.001s	sys 0.002s
I20260709 12:25:20.886368 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.887140 18021 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:20.885452 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.893463 18022 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
W20260709 12:25:20.852403 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1240) took 2381 ms (client timeout 1705 ms). Trace:
W20260709 12:25:20.894316 18017 rpcz_store.cc:269] 0709 12:25:18.439710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.459545 (+ 19835us) service_pool.cc:224] Handling call
0709 12:25:20.820343 (+2360798us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:20.820534 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748511104}
I20260709 12:25:20.895463 18015 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:20.890256 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.898345 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.891781 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:20.846886 26517 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:20.905056 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.873467 26573 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:25:20.907269 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:20.909380 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:20.911152 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.931434 18021 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:20.901891 26395 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:20.911970 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.824170 26463 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:25:20.904513 26440 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:20.938967 26137 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:20.935113 26491 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:20.942628 26137 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.943233 26137 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.906702 26403 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:20.914435 26538 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:25:20.916927 26420 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.919013 26367 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.864796 26553 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.923986 26549 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.950242 26549 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:20.951782 26553 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.952672 26549 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.872702 26524 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.933254 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.953900 18021 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:20.955332 26524 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:20.955721 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4334) took 1287 ms. Trace:
W20260709 12:25:20.956327 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.935690 18018 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:20.962881 26536 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.600s	user 0.000s	sys 0.003s
I20260709 12:25:20.956116 18099 rpcz_store.cc:276] 0709 12:25:19.668167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.669359 (+  1192us) service_pool.cc:224] Handling call
0709 12:25:19.669965 (+   606us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:19.693694 (+ 23729us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.693727 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:20.772148 (+1078421us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:20.772809 (+   661us) log.cc:844] Serialized 52 byte log entry
0709 12:25:20.818974 (+ 46165us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:20.838420 (+ 19446us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.869712 (+ 31292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.869807 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.955320 (+ 85513us) raft_consensus.cc:1685] finished
0709 12:25:20.955455 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.955644 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82189,"prepare.queue_time_us":11358,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":71879,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":8344,"spinlock_wait_cycles":2763136,"thread_start_us":45644,"threads_started":1,"wal-append.queue_time_us":135370}
I20260709 12:25:20.967645 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1259) took 2863 ms. Trace:
I20260709 12:25:20.968189 18164 rpcz_store.cc:276] 0709 12:25:18.104745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.587576 (+482831us) service_pool.cc:224] Handling call
0709 12:25:19.875256 (+1287680us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.875631 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:19.925577 (+ 49946us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.965880 (+1040303us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b1800462400. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287
0709 12:25:20.966014 (+   134us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.966072 (+    58us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:20.966139 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.966242 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.967415 (+  1173us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54003ba1d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:20.967564 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":65588,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":7841,"raft.queue_time_us":59336,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":9564,"spinlock_wait_cycles":2970524672,"thread_start_us":273228,"threads_started":2,"wal-append.queue_time_us":260093}
I20260709 12:25:20.908537 26574 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:25:20.947193 26403 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.936002 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:20.914387 18168 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:20.905049 26497 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.905539 25812 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:20.940150 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:20.915313 26476 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:20.905344 26575 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:20.909485 26541 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:20.985968 26443 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:20.986594 26443 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:20.987028 26443 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:20.922891 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:20.899636 26581 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:25:20.981874 25812 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.988476 25812 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:20.988657 26476 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:20.980039 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:20.998420 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:21.007176 18021 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:25:21.008162 26553 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:21.008628 26559 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:21.009935 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3208) took 1094 ms. Trace:
I20260709 12:25:21.017174 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:20.989861 26499 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:20.982089 26403 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.980954 26504 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:20.996645 26575 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:20.989766 26497 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.017357 18167 rpcz_store.cc:276] 0709 12:25:19.915940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.261079 (+345139us) service_pool.cc:224] Handling call
0709 12:25:21.009603 (+748524us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:21.009811 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1022734592}
I20260709 12:25:20.989099 26476 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:21.020133 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:20.997895 26541 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:21.035038 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.041002 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:21.044489 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.031170 26520 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:21.044487 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.054235 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.056356 18170 raft_consensus.cc:606] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:21.031981 26553 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:21.051282 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:21.058382 18170 raft_consensus.cc:614] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:21.061236 18021 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
W20260709 12:25:21.061798 18137 connection.cc:256] RPC call timeout handler was delayed by -1.214859811000224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
I20260709 12:25:21.033433 26491 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:21.063764 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.037695 26541 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:21.069478 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 137454us
I20260709 12:25:21.069653 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:21.072569 18096 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:21.073568 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:21.075493 18167 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: [19.12-22.14]
I20260709 12:25:21.075995 18164 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:21.046507 26542 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:21.070395 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:21.082098 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.082996 18164 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:21.083942 18095 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: [21.15-22.16]
I20260709 12:25:21.084287 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.082185 18020 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:21.063627 26563 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:21.063619 26463 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.067926 26581 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.072976 26541 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:21.086736 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.094540 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.094795 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:21.089581 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.068292 26520 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:21.086452 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.068632 26499 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.087284 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.102214 18098 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:21.102571 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.102819 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.103142 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:21.095889 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.095986 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.091913 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.096592 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.077790 26574 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:21.098451 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.100530 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.100530 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.099926 26491 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.080550 26476 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.097369 26573 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:21.088558 26563 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:21.074031 26478 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:25:21.105831 18813 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:21.126471 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.119784 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3204) took 1279 ms. Trace:
I20260709 12:25:21.113605 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3203) took 1279 ms. Trace:
I20260709 12:25:21.106076 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:21.140522 18095 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.121032 26443 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.106703 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.107210 18164 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.144008 26443 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:21.127125 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.112650 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:21.105809 18018 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:21.109750 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.110138 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.108284 18173 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:21.189503 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3205) took 1324 ms. Trace:
I20260709 12:25:21.108771 18094 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:21.118310 26476 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:21.204268 18093 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:21.096698 26463 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.138288 18172 rpcz_store.cc:276] 0709 12:25:19.839895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.539786 (+699891us) service_pool.cc:224] Handling call
0709 12:25:20.562419 (+ 22633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.660672 (+ 98253us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:20.661023 (+   351us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.728845 (+ 67822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.728939 (+    94us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:20.729016 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.118352 (+389336us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b180048a420. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:25:21.119481 (+  1129us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:21.119542 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.119706 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4175,"prepare.queue_time_us":348,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":72,"raft.queue_time_us":1963,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":6085,"spinlock_wait_cycles":230300416,"thread_start_us":53905,"threads_started":1,"wal-append.queue_time_us":165869}
I20260709 12:25:21.206367 26478 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:19.898329 18169 rpcz_store.cc:276] 0709 12:25:18.541857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.652019 (+1110162us) service_pool.cc:224] Handling call
0709 12:25:19.652554 (+   535us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.652904 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:19.759557 (+106653us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.866993 (+107436us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.867059 (+    66us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:19.867111 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.867189 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.867353 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":91671,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":3391,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":39098,"thread_start_us":195753,"threads_started":2}
I20260709 12:25:21.214157 26478 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:21.214694 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:21.215152 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.215727 18021 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:21.215914 18169 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:25:21.215471 26478 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.217116 26478 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.082599 26418 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:25:21.079737 26338 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:25:21.206634 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.117396 26497 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:21.121582 26520 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.122406 26502 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.121294 26516 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:21.138437 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:21.217866 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.228395 18021 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:25:21.222235 18091 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:25:21.230716 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:21.231607 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.222445 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1521) took 1418 ms. Trace:
I20260709 12:25:21.223793 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.232908 18090 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:21.122236 18168 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.20-23.20]
I20260709 12:25:21.234167 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.232591 18809 rpcz_store.cc:276] 0709 12:25:19.804152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.804397 (+   245us) service_pool.cc:224] Handling call
0709 12:25:19.830921 (+ 26524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.898629 (+ 67708us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:20.066041 (+167412us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.446401 (+380360us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b180041c620. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:25:20.446614 (+   213us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.446676 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:20.446736 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.221208 (+774472us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b180041c620. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:25:21.222147 (+   939us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b54004a60d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:21.222206 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.222373 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5779,"prepare.queue_time_us":279692,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":266438,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":4460,"spinlock_wait_cycles":249326464,"thread_start_us":268608,"threads_started":2,"wal-append.queue_time_us":153387}
I20260709 12:25:21.208094 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.205711 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.208482 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:21.138435 18098 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:21.235857 18024 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:21.203390 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.210182 26513 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.20, Last appended by leader: 20, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.255738 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.138895 18166 rpcz_store.cc:276] 0709 12:25:19.834435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.375510 (+541075us) service_pool.cc:224] Handling call
0709 12:25:21.113371 (+737861us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b54000438d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:21.113530 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59606400}
I20260709 12:25:21.202827 18171 rpcz_store.cc:276] 0709 12:25:19.864838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.576379 (+711541us) service_pool.cc:224] Handling call
0709 12:25:20.593852 (+ 17473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.594323 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.594357 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:20.602266 (+  7909us) log.cc:844] Serialized 26 byte log entry
0709 12:25:20.903829 (+301563us) spinlock_profiling.cc:243] Waited 7.86 ms on lock 0x7b1800184600. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:25:21.064205 (+160376us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:21.064258 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.064343 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:21.189092 (+124749us) raft_consensus.cc:1685] finished
0709 12:25:21.189126 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.189417 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":633,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":6853,"spinlock_wait_cycles":18073472,"thread_start_us":160121,"threads_started":1,"wal-append.queue_time_us":480215}
I20260709 12:25:21.260718 18095 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:21.261368 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.262282 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.211223 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.211670 26463 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:21.123741 26563 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.126245 26499 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.212954 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:21.267398 18017 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:21.237524 18099 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:21.264113 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:21.204375 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.270824 18094 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:21.273005 18094 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:25:21.227329 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.274242 18172 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:25:21.245196 18164 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:21.276945 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.278914 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.278929 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.281342 18021 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:21.282303 18094 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:21.123179 26595 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:25:21.264734 18169 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:25:21.265228 18093 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:21.122558 26547 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:21.266486 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.277658 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.304040 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.247201 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:21.296547 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.307832 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:21.123265 26517 consensus_queue.cc:1175] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ae9641d5c3184572a102d51a75f670ee" responder_term: 20 status { last_received { term: 20 index: 12 } last_committed_idx: 11 last_received_current_leader { term: 20 index: 12 } }
I20260709 12:25:21.300107 18090 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:21.301765 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.316938 18168 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.317616 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:21.117070 26575 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:21.306226 18098 raft_consensus.cc:2376] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22. Re-sending same reply.
W20260709 12:25:21.309577 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 223725us
I20260709 12:25:21.236353 26338 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:25:21.223474 26502 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:21.311510 18169 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.318392 18021 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.329557 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.330404 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3109) took 1037 ms. Trace:
I20260709 12:25:21.341336 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.338291 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.330735 18018 rpcz_store.cc:276] 0709 12:25:20.293157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.302882 (+1009725us) service_pool.cc:224] Handling call
0709 12:25:21.330334 (+ 27452us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.345278 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.347743 18172 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:21.349638 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.306700 26513 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.107558 26581 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:21.386221 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57015.1us
I20260709 12:25:21.303119 26547 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:21.394325 18024 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:21.405220 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:21.404925 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.406451 18171 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:21.407302 18017 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:21.388386 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:21.412104 18098 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:21.412982 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3107) took 1120 ms. Trace:
I20260709 12:25:21.413625 18017 rpcz_store.cc:276] 0709 12:25:20.292549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.408015 (+1115466us) service_pool.cc:224] Handling call
0709 12:25:21.411728 (+  3713us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.411936 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.412589 (+   653us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:21.412661 (+    72us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:21.412709 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.412780 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.412914 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.406744 18168 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:21.407912 18171 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:21.410329 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:21.429385 26497 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:21.429970 26497 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.430378 26497 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:21.431429 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.433952 18017 raft_consensus.cc:1217] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Deduplicated request from leader. Original: 19.16->[21.17-21.17]   Dedup: 21.17->[]
I20260709 12:25:21.434626 18017 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: [21.17-21.17]
I20260709 12:25:21.435272 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3108) took 1142 ms. Trace:
I20260709 12:25:21.435595 18017 rpcz_store.cc:276] 0709 12:25:20.292872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.415548 (+1122676us) service_pool.cc:224] Handling call
0709 12:25:21.433786 (+ 18238us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.434959 (+  1173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.435194 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.328920 26573 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:21.437729 26573 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.189317 26553 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.446529 26547 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.408804 26491 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:25:21.454728 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.456164 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:21.474115 18018 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:21.475073 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3113) took 1180 ms. Trace:
I20260709 12:25:21.475698 18018 rpcz_store.cc:276] 0709 12:25:20.338715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.381766 (+1043051us) service_pool.cc:224] Handling call
0709 12:25:21.474919 (+ 93153us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:21.482410 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.487428 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77183.4us
W20260709 12:25:21.489317 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.497185 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 190683us
I20260709 12:25:21.328105 26338 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:21.467958 18164 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:21.498118 18021 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:21.508490 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:21.515394 18098 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:25:21.522401 18021 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:25:21.458506 18169 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:21.521354 26542 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:21.525846 26597 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:21.526049 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.526473 26597 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:21.526937 26597 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:21.527057 18166 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:21.528213 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.528867 18166 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:21.531107 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4228) took 1214 ms. Trace:
I20260709 12:25:21.531149 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.531785 18021 rpcz_store.cc:276] 0709 12:25:20.317236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.521627 (+1204391us) service_pool.cc:224] Handling call
0709 12:25:21.522135 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.530799 (+  8664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.531043 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.532577 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:21.522503 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.536463 18098 raft_consensus.cc:2368] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:21.536373 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:21.535280 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4224) took 1648 ms. Trace:
I20260709 12:25:21.538736 18019 rpcz_store.cc:276] 0709 12:25:19.887251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.160416 (+273165us) service_pool.cc:224] Handling call
0709 12:25:20.401882 (+241466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.404768 (+  2886us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.404804 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:20.408045 (+  3241us) log.cc:844] Serialized 26 byte log entry
0709 12:25:20.650926 (+242881us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:20.651012 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.651139 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.755495 (+104356us) raft_consensus.cc:1685] finished
0709 12:25:20.755525 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.519375 (+763850us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400251a50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:21.535192 (+ 15817us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2310,"raft.queue_time_us":35080,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":34166,"spinlock_wait_cycles":522610432,"thread_start_us":141313,"threads_started":1,"wal-append.queue_time_us":199632}
I20260709 12:25:21.539028 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:21.528307 26516 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:21.547842 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.575927 18170 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:21.548216 26479 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:21.550552 18809 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:21.552325 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.584344 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d440327a63754b889f7edd91e1373942"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:21.585323 18170 raft_consensus.cc:606] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:21.585856 18170 raft_consensus.cc:614] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:21.552551 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.553858 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:21.587679 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53922.9us
I20260709 12:25:21.588047 18165 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:21.590004 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.500823 26338 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:21.535566 26517 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:21.559860 26403 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.004s
I20260709 12:25:21.568436 18017 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:21.569653 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1248) took 1037 ms. Trace:
I20260709 12:25:21.412573 26137 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.575929 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1244) took 1088 ms. Trace:
I20260709 12:25:21.578675 18018 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:21.601621 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:21.610581 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4226) took 1302 ms. Trace:
I20260709 12:25:21.611191 18018 rpcz_store.cc:276] 0709 12:25:20.308583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.477027 (+1168444us) service_pool.cc:224] Handling call
0709 12:25:21.542252 (+ 65225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.610162 (+ 67910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.610491 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.609699 18020 rpcz_store.cc:276] 0709 12:25:20.487650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.859280 (+371630us) service_pool.cc:224] Handling call
0709 12:25:21.575709 (+716429us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:21.575841 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":404524288}
I20260709 12:25:21.614465 18020 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:21.538673 26515 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:21.592736 18136 leader_election.cc:400] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:21.625063 18136 leader_election.cc:403] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:21.594656 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:21.599177 18023 rpcz_store.cc:276] 0709 12:25:20.532705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.821719 (+289014us) service_pool.cc:224] Handling call
0709 12:25:21.569390 (+747671us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54001d2254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:21.569569 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99093760}
I20260709 12:25:21.603283 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3111) took 1309 ms. Trace:
I20260709 12:25:21.615417 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3129) took 1218 ms. Trace:
I20260709 12:25:21.532797 26134 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:25:21.574565 26547 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.622151 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.633390 18020 rpcz_store.cc:276] 0709 12:25:20.396518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.613612 (+1217094us) service_pool.cc:224] Handling call
0709 12:25:21.614175 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.615041 (+   866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.615331 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.634896 18098 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:21.635213 18017 rpcz_store.cc:276] 0709 12:25:20.293684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.436490 (+1142806us) service_pool.cc:224] Handling call
0709 12:25:21.567808 (+131318us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:25:21.568111 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.602793 (+ 34682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.603180 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46093056}
I20260709 12:25:21.544670 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:21.659322 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114718us
W20260709 12:25:21.659708 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.660256 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:21.637660 18023 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:25:21.661340 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1242) took 1264 ms. Trace:
I20260709 12:25:21.661679 18023 rpcz_store.cc:276] 0709 12:25:20.396841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.636706 (+1239865us) service_pool.cc:224] Handling call
0709 12:25:21.637301 (+   595us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.661023 (+ 23722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.661277 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:21.651566 26541 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.542s	user 0.013s	sys 0.000s
W20260709 12:25:21.594730 26227 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.039s	sys 0.049s
W20260709 12:25:21.651407 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61600.8us
I20260709 12:25:21.674119 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:21.654870 18095 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:21.655046 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:21.536859 26542 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.665179 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:21.679771 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:21.613682 26516 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:21.624997 26515 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.695439 26515 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.685504 18170 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:21.685624 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.689294 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:21.688448 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:21.689970 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1245) took 1202 ms. Trace:
I20260709 12:25:21.703155 18017 rpcz_store.cc:276] 0709 12:25:20.488054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.658842 (+1170788us) service_pool.cc:224] Handling call
0709 12:25:21.689208 (+ 30366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.689472 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.689513 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:21.689577 (+    64us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:21.689626 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.689702 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.689871 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.617098 26510 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:25:21.677311 26525 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.663573 26541 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.686915 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:21.716725 18815 raft_consensus.cc:2368] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:21.607491 26418 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:25:21.665158 26491 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
W20260709 12:25:21.677058 25812 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.005s
I20260709 12:25:21.724753 18166 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
W20260709 12:25:21.687459 26227 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.040s	sys 0.049s
I20260709 12:25:21.729009 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.609737 26395 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:25:21.668388 26479 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:21.617874 26403 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.859812 26581 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:21.862759 26602 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:21.863394 26602 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:21.864372 26602 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.867036 26504 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:21.867802 26504 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:21.874688 26607 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260709 12:25:21.809576 26499 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:21.808926 18099 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:21.691282 26517 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:21.688414 26503 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:21.817193 18170 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:21.692543 26531 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:21.789073 26529 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:25:21.736696 26525 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.691490 26514 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.111s
I20260709 12:25:21.734508 26538 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:21.897006 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.822229 26524 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:21.823257 26367 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:21.745559 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.925956 18020 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:21.754128 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.910439 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:21.801926 25812 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.890522 26602 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.977329 18170 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:21.989223 26538 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:21.991787 26608 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:25:21.991032 26538 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:21.912604 26601 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.390s	user 0.000s	sys 0.002s
I20260709 12:25:21.998766 18809 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 21 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:25:21.909864 26420 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:22.001608 18170 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:25:22.002691 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:22.005326 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 496981us
I20260709 12:25:22.014545 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:22.017115 18170 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:22.019627 18170 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:22.039158 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:22.039886 18173 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:21.860674 26479 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:21.974593 26367 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.973194 26134 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.045890 18170 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:21.899631 26516 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.21, Last appended by leader: 21, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:21.973115 26438 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:21.980399 26605 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:22.052577 18166 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:22.047077 17908 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.004s	sys 0.025s
W20260709 12:25:22.059121 17908 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.004s	sys 0.025s
I20260709 12:25:21.986974 26524 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:22.010680 26440 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:22.074895 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.17611e+06us
I20260709 12:25:22.079238 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:22.050575 26367 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:22.054122 26549 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:22.089771 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:22.059945 26137 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:22.100988 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1243) took 1703 ms. Trace:
W20260709 12:25:22.103797 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 502244us
I20260709 12:25:22.042398 26420 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:22.107275 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 387176us
I20260709 12:25:22.079944 26440 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:22.110416 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3414) took 1023 ms. Trace:
I20260709 12:25:22.060766 26134 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.093467 26549 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:22.092229 26611 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:22.120143 26603 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.454s	user 0.005s	sys 0.000s
W20260709 12:25:22.117367 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 443315us
W20260709 12:25:22.108589 26497 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.674s	user 0.012s	sys 0.003s
I20260709 12:25:22.085724 26455 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:22.143654 18097 rpcz_store.cc:276] 0709 12:25:21.087008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.087140 (+   132us) service_pool.cc:224] Handling call
0709 12:25:21.087512 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.087962 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.087998 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.088548 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.305091 (+216543us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:21.332862 (+ 27771us) log.cc:844] Serialized 14 byte log entry
0709 12:25:21.333188 (+   326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.333278 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:21.436951 (+103673us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b1800393700. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:22.099748 (+662797us) raft_consensus.cc:1685] finished
0709 12:25:22.110012 (+ 10264us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.110300 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49845,"prepare.queue_time_us":34463,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":4906,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":2977,"spinlock_wait_cycles":49027840,"thread_start_us":216104,"threads_started":1,"wal-append.queue_time_us":313407}
W20260709 12:25:22.170365 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80682.6us
I20260709 12:25:22.101773 18020 rpcz_store.cc:276] 0709 12:25:20.397139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.658741 (+1261602us) service_pool.cc:224] Handling call
0709 12:25:21.687771 (+ 29030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.929223 (+241452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.100546 (+171323us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54001d1d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:22.100884 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":546963328}
I20260709 12:25:22.150136 18172 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:22.094115 26542 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:22.067823 26605 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:22.065037 26438 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.106455 26581 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:22.124856 18778 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:22.433580 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 308831us
I20260709 12:25:22.108069 26541 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:22.165956 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:22.380445 18015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.071s	sys 0.045s
W20260709 12:25:22.437022 18015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.071s	sys 0.053s
W20260709 12:25:22.304253 18021 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.050s	sys 0.078s
I20260709 12:25:22.275451 26479 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:22.438063 18021 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.050s	sys 0.110s
I20260709 12:25:22.158716 26549 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:22.166875 26515 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:22.358716 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4225) took 2471 ms. Trace:
W20260709 12:25:22.296789 26610 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
W20260709 12:25:22.356662 26615 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:25:22.161852 26513 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.756s	user 0.010s	sys 0.004s
I20260709 12:25:22.277418 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:22.278287 26403 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:22.333447 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.007s	sys 0.014s
W20260709 12:25:22.471190 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.009s	sys 0.014s
I20260709 12:25:22.272115 26443 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260709 12:25:22.271663 26497 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.157536 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:22.358887 26606 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.539s	user 0.000s	sys 0.001s
W20260709 12:25:22.380189 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.038s	sys 0.057s
W20260709 12:25:22.484423 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.039s	sys 0.062s
W20260709 12:25:22.484865 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 798001us
I20260709 12:25:22.485970 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:22.359522 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3123) took 2000 ms. Trace:
I20260709 12:25:22.486752 18019 rpcz_store.cc:276] 0709 12:25:20.359555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.540906 (+1181351us) service_pool.cc:224] Handling call
0709 12:25:21.547473 (+  6567us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.547948 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.549031 (+  1083us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.549672 (+   641us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.092274 (+542602us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:22.092333 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.092426 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.359245 (+266819us) raft_consensus.cc:1685] finished
0709 12:25:22.359273 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.359450 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":950,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":480045,"threads_started":1,"wal-append.queue_time_us":786838}
W20260709 12:25:22.381037 26476 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.177s	user 0.009s	sys 0.006s
I20260709 12:25:22.359181 26420 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:22.288743 26524 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.360071 25812 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:22.414601 18097 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:22.604038 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4356) took 1087 ms. Trace:
I20260709 12:25:22.604542 18097 rpcz_store.cc:276] 0709 12:25:21.516713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.169678 (+652965us) service_pool.cc:224] Handling call
0709 12:25:22.171194 (+  1516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.603645 (+432451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.603951 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.353387 26510 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:25:22.412734 26612 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
I20260709 12:25:22.389488 26525 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:22.426362 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.427340 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 388260us
W20260709 12:25:22.424942 18096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.013s	sys 0.011s
W20260709 12:25:22.122565 17989 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.618s (SENT) [suppressed 29 similar messages]
I20260709 12:25:22.439656 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1476) took 1389 ms. Trace:
W20260709 12:25:22.440948 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 275142us
I20260709 12:25:22.444080 18024 rpcz_store.cc:276] 0709 12:25:19.887615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.464498 (+1576883us) service_pool.cc:224] Handling call
0709 12:25:21.465460 (+   962us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.486484 (+ 21024us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.486544 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.501205 (+ 14661us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.900032 (+398827us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:21.900122 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.900241 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.358408 (+458167us) raft_consensus.cc:1685] finished
0709 12:25:22.358446 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.358639 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":149,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8447,"thread_start_us":347745,"threads_started":1,"wal-append.queue_time_us":623468}
I20260709 12:25:22.453703 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1275) took 1365 ms. Trace:
I20260709 12:25:22.456717 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3301) took 1189 ms. Trace:
W20260709 12:25:22.437875 26338 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.817s	user 0.017s	sys 0.000s
I20260709 12:25:22.466279 18018 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:22.466466 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1478) took 1399 ms. Trace:
I20260709 12:25:22.397215 26623 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260709 12:25:22.441520 26478 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:22.474843 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 197527us
I20260709 12:25:22.482117 18807 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 20. Preceding OpId from leader: term: 24 index: 20. (term mismatch)
W20260709 12:25:22.482379 26622 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:25:22.445811 26625 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:22.491071 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:22.486002 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 328594us
W20260709 12:25:22.492805 26624 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.081s	user 0.001s	sys 0.001s
I20260709 12:25:22.482895 26502 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:25:22.544624 26634 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:22.536532 26479 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:22.546953 26626 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:22.546518 26515 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:22.493399 26611 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:22.579792 26563 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.277s	user 0.015s	sys 0.000s
W20260709 12:25:22.562677 26614 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
W20260709 12:25:22.580946 26597 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.053s	user 0.009s	sys 0.007s
I20260709 12:25:22.570006 26497 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:22.572250 26438 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:22.550122 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1255) took 1850 ms. Trace:
I20260709 12:25:22.550122 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1254) took 1858 ms. Trace:
I20260709 12:25:22.531963 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:22.491719 26581 raft_consensus.cc:2764] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:25:22.561715 26503 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:22.583484 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4230) took 2216 ms. Trace:
I20260709 12:25:22.584646 18814 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:22.528668 18169 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:22.527772 18016 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
W20260709 12:25:22.539299 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.525071 26616 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
W20260709 12:25:22.530851 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44966.6us
W20260709 12:25:22.601586 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.573735 26547 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.905s	user 0.016s	sys 0.000s
I20260709 12:25:22.726413 26549 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.772836 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3294) took 1677 ms. Trace:
I20260709 12:25:22.677942 26605 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:22.773475 18171 rpcz_store.cc:276] 0709 12:25:21.095082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.427489 (+332407us) service_pool.cc:224] Handling call
0709 12:25:21.428151 (+   662us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:21.490429 (+ 62278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.490484 (+    55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:21.519564 (+ 29080us) log.cc:844] Serialized 52 byte log entry
0709 12:25:21.980382 (+460818us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:21.980433 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.980516 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.772520 (+792004us) raft_consensus.cc:1685] finished
0709 12:25:22.772547 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.772756 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68909,"raft.queue_time_us":67809,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":30474,"thread_start_us":454678,"threads_started":1,"wal-append.queue_time_us":753583}
I20260709 12:25:22.775300 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:22.776005 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3291) took 1682 ms. Trace:
I20260709 12:25:22.776646 26637 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:22.776521 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3437) took 1458 ms. Trace:
I20260709 12:25:22.777050 26638 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:22.777405 26637 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:22.779127 26637 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.776510 18167 rpcz_store.cc:276] 0709 12:25:21.093319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.363660 (+270341us) service_pool.cc:224] Handling call
0709 12:25:21.364269 (+   609us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:21.364864 (+   595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.364908 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:21.415277 (+ 50369us) log.cc:844] Serialized 52 byte log entry
0709 12:25:21.882281 (+467004us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:21.882330 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.882428 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.775724 (+893296us) raft_consensus.cc:1685] finished
0709 12:25:22.775751 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.775929 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48357,"raft.queue_time_us":17733,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":60481,"thread_start_us":359404,"threads_started":1,"wal-append.queue_time_us":638663}
I20260709 12:25:22.780609 26637 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:22.776721 26137 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:22.777537 18090 rpcz_store.cc:276] 0709 12:25:21.318003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.318315 (+   312us) service_pool.cc:224] Handling call
0709 12:25:21.322264 (+  3949us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.322705 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.322738 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.386962 (+ 64224us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.758187 (+371225us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:21.758247 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.758354 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.774546 (+1016192us) raft_consensus.cc:1685] finished
0709 12:25:22.774577 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.776224 (+  1647us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b54002972d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:22.776451 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5127,"raft.queue_time_us":569,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":32049,"spinlock_wait_cycles":7834880,"thread_start_us":260144,"threads_started":1,"wal-append.queue_time_us":610690}
W20260709 12:25:22.783160 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 660600us
W20260709 12:25:22.784129 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.784571 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:22.785002 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.785372 18807 pending_rounds.cc:85] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 19
I20260709 12:25:22.675770 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3427) took 1549 ms. Trace:
I20260709 12:25:22.785524 18017 rpcz_store.cc:276] 0709 12:25:20.700172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.704789 (+1004617us) service_pool.cc:224] Handling call
0709 12:25:21.787347 (+ 82558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.787837 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.787870 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.788635 (+   765us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.117963 (+329328us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:22.118032 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.118120 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.549838 (+431718us) raft_consensus.cc:1685] finished
0709 12:25:22.549866 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.550037 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":192261,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":14632,"thread_start_us":297688,"threads_started":1,"wal-append.queue_time_us":717606}
I20260709 12:25:22.787985 18099 rpcz_store.cc:276] 0709 12:25:21.126053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.269055 (+143002us) service_pool.cc:224] Handling call
0709 12:25:21.312317 (+ 43262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.898129 (+585812us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.675401 (+777272us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b540005fad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:22.675684 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141489024}
W20260709 12:25:22.788888 18096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.908s	user 0.020s	sys 0.021s
I20260709 12:25:22.789386 18807 pending_rounds.cc:107] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 23.20
I20260709 12:25:22.790160 18807 raft_consensus.cc:2889] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:22.792315 18810 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:22.792752 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:22.781599 18015 rpcz_store.cc:276] 0709 12:25:20.366935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.570920 (+1203985us) service_pool.cc:224] Handling call
0709 12:25:21.571391 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.571827 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.571858 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.572536 (+   678us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.437574 (+865038us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:22.437630 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.437733 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.583201 (+145468us) raft_consensus.cc:1685] finished
0709 12:25:22.583230 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.583402 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3809,"raft.queue_time_us":13350,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":14909,"thread_start_us":864551,"threads_started":1,"wal-append.queue_time_us":921323}
I20260709 12:25:22.795560 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:22.797086 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:22.797922 18164 rpcz_store.cc:276] 0709 12:25:21.267327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.276286 (+  8959us) service_pool.cc:224] Handling call
0709 12:25:22.456508 (+1180222us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400031bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:22.456641 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437098752}
W20260709 12:25:22.602705 26504 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.734s	user 0.017s	sys 0.000s
I20260709 12:25:22.801973 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3135) took 1863 ms. Trace:
I20260709 12:25:22.802608 18024 rpcz_store.cc:276] 0709 12:25:20.938279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.800613 (+1862334us) service_pool.cc:224] Handling call
0709 12:25:22.801127 (+   514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.801405 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.801443 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.801510 (+    67us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:22.801560 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.801642 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.801844 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.786966 18023 rpcz_store.cc:276] 0709 12:25:20.691408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.663977 (+972569us) service_pool.cc:224] Handling call
0709 12:25:21.665402 (+  1425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.690741 (+ 25339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.690774 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.734262 (+ 43488us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.155233 (+420971us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:22.155283 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.155382 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.549910 (+394528us) raft_consensus.cc:1685] finished
0709 12:25:22.549932 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.550052 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24827,"raft.queue_time_us":24969,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":29410,"thread_start_us":327691,"threads_started":1,"wal-append.queue_time_us":633760}
I20260709 12:25:22.783982 18812 rpcz_store.cc:276] 0709 12:25:21.053570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.053738 (+   168us) service_pool.cc:224] Handling call
0709 12:25:22.452884 (+1399146us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54006f2054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:22.453071 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1754059520}
I20260709 12:25:22.787639 18173 rpcz_store.cc:276] 0709 12:25:21.088883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.206234 (+117351us) service_pool.cc:224] Handling call
0709 12:25:22.453489 (+1247255us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b54003016d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:22.453627 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1885341440}
I20260709 12:25:22.785549 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:22.676119 26633 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.131s	user 0.000s	sys 0.002s
W20260709 12:25:22.678797 26620 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.254s	user 0.002s	sys 0.000s
I20260709 12:25:22.786419 18806 rpcz_store.cc:276] 0709 12:25:21.050514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.052164 (+  1650us) service_pool.cc:224] Handling call
0709 12:25:21.067936 (+ 15772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.074837 (+  6901us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.074873 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.080256 (+  5383us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.320479 (+240223us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:21.320552 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.320660 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:21.911529 (+590869us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:22.426189 (+514660us) raft_consensus.cc:1685] finished
0709 12:25:22.426217 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.439579 (+ 13362us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6414,"raft.queue_time_us":10634,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5569,"spinlock_wait_cycles":8894336,"thread_start_us":207056,"threads_started":1,"wal-append.queue_time_us":447316}
W20260709 12:25:22.679695 26602 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.691s	user 0.015s	sys 0.001s
I20260709 12:25:22.772836 26520 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:25:22.846303 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60827.4us
I20260709 12:25:22.862955 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:22.846455 18017 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:22.848155 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:22.847289 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3295) took 1747 ms. Trace:
I20260709 12:25:22.864938 18168 rpcz_store.cc:276] 0709 12:25:21.106449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.432599 (+326150us) service_pool.cc:224] Handling call
0709 12:25:21.433014 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.796213 (+363199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.796270 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.797108 (+   838us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.471900 (+674792us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:22.815148 (+343248us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.846856 (+ 31708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.846953 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.847000 (+    47us) raft_consensus.cc:1685] finished
0709 12:25:22.847015 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.847214 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87265,"prepare.queue_time_us":294124,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":138428,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":1240979,"threads_started":3,"wal-append.queue_time_us":712148}
W20260709 12:25:22.848062 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51072.4us
I20260709 12:25:22.854264 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4357) took 1321 ms. Trace:
I20260709 12:25:22.854264 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4354) took 1338 ms. Trace:
I20260709 12:25:22.869804 18096 rpcz_store.cc:276] 0709 12:25:21.516090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.878919 (+362829us) service_pool.cc:224] Handling call
0709 12:25:21.879368 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.879719 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:21.880120 (+   401us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.853805 (+973685us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.853892 (+    87us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:22.853943 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.854024 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.854184 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63902,"prepare.queue_time_us":831,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":64664,"thread_start_us":909240,"threads_started":1,"wal-append.queue_time_us":689955}
I20260709 12:25:22.855261 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:22.791513 26627 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.786203 26634 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:22.873551 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3138) took 1646 ms. Trace:
I20260709 12:25:22.606273 26510 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:22.792263 26642 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:22.804308 26575 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:22.825474 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44066.4us
I20260709 12:25:22.881469 18017 rpcz_store.cc:276] 0709 12:25:21.227374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.833291 (+1605917us) service_pool.cc:224] Handling call
0709 12:25:22.833877 (+   586us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.873058 (+ 39181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.873452 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.604220 26367 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.814190 26513 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:22.818387 26623 consensus_queue.cc:1243] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 24.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:22.821004 26547 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.885727 18814 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:22.985519 26642 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.861918 26515 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.136s
I20260709 12:25:22.878856 26516 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.133s
I20260709 12:25:22.997577 26575 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:22.859735 26625 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:23.005683 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4350) took 1492 ms. Trace:
W20260709 12:25:23.008601 26636 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260709 12:25:22.930841 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.801209 26597 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:22.813313 26574 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.177s
I20260709 12:25:22.806525 26479 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:22.852500 26605 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:22.847034 26503 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:22.855394 25812 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:23.015864 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1485) took 1355 ms. Trace:
I20260709 12:25:23.131605 18809 rpcz_store.cc:276] 0709 12:25:21.660916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.661751 (+   835us) service_pool.cc:224] Handling call
0709 12:25:23.015570 (+1353819us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:23.015783 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":680781568}
I20260709 12:25:23.016501 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1282) took 1925 ms. Trace:
I20260709 12:25:22.892139 26497 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.040347 18815 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:25:22.988538 26611 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:23.041478 18806 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:25:23.042079 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 249417us
I20260709 12:25:22.820824 26504 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.006505 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4348) took 1494 ms. Trace:
I20260709 12:25:23.163625 18094 rpcz_store.cc:276] 0709 12:25:21.512520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.514003 (+  1483us) service_pool.cc:224] Handling call
0709 12:25:21.514449 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.521753 (+  7304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.521792 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.540341 (+ 18549us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.059943 (+519602us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:22.059991 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.060064 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.006248 (+946184us) raft_consensus.cc:1685] finished
0709 12:25:23.006276 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.006433 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6823,"raft.queue_time_us":8134,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":11361,"spinlock_wait_cycles":48896,"thread_start_us":410631,"threads_started":1,"wal-append.queue_time_us":719083}
W20260709 12:25:23.060026 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52677.9us
I20260709 12:25:23.137344 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:22.860082 26581 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:22.805799 26563 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.000715 26650 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.057587 18091 rpcz_store.cc:276] 0709 12:25:21.513296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.591851 (+ 78555us) service_pool.cc:224] Handling call
0709 12:25:21.596559 (+  4708us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.616248 (+ 19689us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.616302 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.630852 (+ 14550us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.059184 (+428332us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:22.059232 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.059323 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.005393 (+946070us) raft_consensus.cc:1685] finished
0709 12:25:23.005421 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.005594 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19115,"raft.queue_time_us":19292,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":27094,"thread_start_us":394300,"threads_started":1,"wal-append.queue_time_us":668057}
I20260709 12:25:22.862304 26520 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:22.976406 26634 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:22.951107 26627 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:23.107017 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 322478us
I20260709 12:25:23.008091 26134 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:23.109979 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:23.181288 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:23.129348 26631 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:23.129840 26640 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.129891 26542 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.132898 18172 rpcz_store.cc:276] 0709 12:25:21.090895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.351096 (+260201us) service_pool.cc:224] Handling call
0709 12:25:21.430593 (+ 79497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.415598 (+985005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.016144 (+600546us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54001b6054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:23.016414 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1653937920}
W20260709 12:25:23.192217 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82365.3us
I20260709 12:25:23.193019 18019 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:23.134395 18165 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:23.042264 26654 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:22.805523 26338 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:23.167770 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 312676us
I20260709 12:25:23.169149 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4346) took 1657 ms. Trace:
I20260709 12:25:23.194293 18092 rpcz_store.cc:276] 0709 12:25:21.511633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.513950 (+  2317us) service_pool.cc:224] Handling call
0709 12:25:21.514446 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.514869 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.514893 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.531442 (+ 16549us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.059577 (+528135us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:22.059623 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.059701 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.672441 (+612740us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:23.131730 (+459289us) raft_consensus.cc:1685] finished
0709 12:25:23.131760 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.169038 (+ 37278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11104,"raft.queue_time_us":7598,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":17733,"spinlock_wait_cycles":9127424,"thread_start_us":436022,"threads_started":1,"wal-append.queue_time_us":761486}
I20260709 12:25:23.172672 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:23.195262 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:23.168061 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:23.122339 26137 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.069782 26547 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.180346 18164 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:23.136368 26611 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:23.184455 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3132) took 2459 ms. Trace:
I20260709 12:25:23.193256 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:23.206916 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:23.208338 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.218152 18015 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 12:25:23.218644 18015 pending_rounds.cc:85] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 14
I20260709 12:25:23.139309 25812 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.184867 26503 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:23.220515 18015 pending_rounds.cc:107] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 12:25:23.220570 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.222244 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.223830 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.226135 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:23.180864 26549 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:23.217110 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 215841us
I20260709 12:25:23.252470 18015 raft_consensus.cc:2889] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:23.255079 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61932.6us
I20260709 12:25:23.252342 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4359) took 1561 ms. Trace:
I20260709 12:25:23.274222 18813 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:23.201020 26529 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:23.274724 18090 rpcz_store.cc:276] 0709 12:25:21.690734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.782300 (+1091566us) service_pool.cc:224] Handling call
0709 12:25:22.782689 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.782961 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:22.791274 (+  8313us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.251889 (+460615us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.251971 (+    82us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:23.252028 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.252103 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.252270 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":7676,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":119,"thread_start_us":460117,"threads_started":1}
I20260709 12:25:23.276100 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:23.228997 26440 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.072s	user 0.019s	sys 0.003s
I20260709 12:25:23.286055 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:23.288285 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93140.9us
I20260709 12:25:23.283057 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:23.289508 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:23.284017 26497 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.205523 26581 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.210588 26516 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:23.256793 26605 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.219347 26574 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:23.274883 26438 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:23.380738 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 91374.2us
I20260709 12:25:23.395128 18166 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: [20.11-21.12]
I20260709 12:25:23.218065 26611 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:23.250278 26661 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:23.355094 26549 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:23.280556 26529 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:23.408921 26648 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:25:23.409303 26529 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.410241 26642 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:23.410841 26529 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:23.412485 26656 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 12:25:23.358109 26440 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:23.415114 26649 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
I20260709 12:25:23.416553 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4358) took 1825 ms. Trace:
I20260709 12:25:23.417027 18097 rpcz_store.cc:276] 0709 12:25:21.591498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.617980 (+1026482us) service_pool.cc:224] Handling call
0709 12:25:22.699660 (+ 81680us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:22.700461 (+   801us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:22.759052 (+ 58591us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.010371 (+251319us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:23.013793 (+  3422us) log.cc:844] Serialized 52 byte log entry
0709 12:25:23.014007 (+   214us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:23.049209 (+ 35202us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.108145 (+ 58936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.108250 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.416270 (+308020us) raft_consensus.cc:1685] finished
0709 12:25:23.416297 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.416476 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110077,"prepare.queue_time_us":112059,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":113,"raft.queue_time_us":88008,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":24970,"thread_start_us":161718,"threads_started":1,"wal-append.queue_time_us":374802}
I20260709 12:25:23.212495 26137 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.290953 26499 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.286074 26651 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:23.291604 26623 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:25:23.389216 26647 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.266s	user 0.002s	sys 0.000s
I20260709 12:25:23.181686 26650 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.406641 26367 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:23.359483 26575 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:23.356627 26631 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.424669 26661 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.359917 26497 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.407811 25812 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:23.203164 26542 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.359592 26653 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.254352 26515 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.132s
I20260709 12:25:23.433149 26137 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:23.434214 26502 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:23.435794 26664 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.355522 26516 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:23.355630 26663 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:23.403127 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.399192 26652 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:23.200779 26640 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.423120 26517 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:23.464058 18780 connection.cc:256] RPC call timeout handler was delayed by -1.1203110569999808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20030000000000003s for a response
W20260709 12:25:23.471063 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 298460us
I20260709 12:25:23.479138 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.686698 18171 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:23.688134 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.479619 18137 leader_election.cc:400] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:23.407475 26657 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:23.483109 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:23.689872 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 210463us
I20260709 12:25:23.691130 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:23.380036 26581 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.574810 25812 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.441552 26655 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:23.193854 26625 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.729205 25812 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.564447 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:23.564743 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:23.568468 18091 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:23.738026 18092 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:23.439641 26623 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:23.739241 26623 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.219834 26520 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.384325 26134 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20260709 12:25:23.578861 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4231) took 3201 ms. Trace:
W20260709 12:25:23.579813 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 288718us
I20260709 12:25:23.745728 18018 rpcz_store.cc:276] 0709 12:25:20.377984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.612498 (+1234514us) service_pool.cc:224] Handling call
0709 12:25:21.653713 (+ 41215us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.786081 (+1132368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.578318 (+792237us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:23.578737 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867944704}
I20260709 12:25:23.593232 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:23.394685 26605 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:23.653004 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.400743 26574 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.548646 26651 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:23.551805 26575 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:23.552908 26661 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:23.686164 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:23.689729 18167 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
W20260709 12:25:23.559648 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 391623us
I20260709 12:25:23.696561 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:23.684111 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:23.695801 26657 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.781320 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:23.694657 26504 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:23.602154 26516 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:23.734709 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.563733 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:23.594626 26597 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:23.747483 18097 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.740602 26623 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:23.687212 26672 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.686095 26338 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:23.826262 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 140232us
W20260709 12:25:23.799540 26637 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.018s	user 0.023s	sys 0.004s
I20260709 12:25:23.679291 26664 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.791628 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.782542 18018 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:23.730870 26655 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:23.747351 26418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.920s	user 0.030s	sys 0.064s
I20260709 12:25:23.599359 26502 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.799937 18090 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:23.787318 26657 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.749444 26479 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:23.749401 26652 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:23.822162 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 131152us
I20260709 12:25:23.678030 26663 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:23.588156 26653 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:23.825872 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48551.5us
W20260709 12:25:23.834769 26658 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:25:23.789484 26575 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.789124 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:23.840955 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.798806 26520 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.799537 26529 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.700510 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.803951 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:23.851306 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:23.866146 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:23.851256 26627 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:23.867919 26515 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:23.763432 26137 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:25:23.813017 26504 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:23.803594 26547 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:23.867403 26678 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:25:23.873224 25812 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:23.862986 26418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.043s	user 0.031s	sys 0.064s
I20260709 12:25:23.909756 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:23.849212 26675 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.948628 18018 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:23.951376 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:23.914697 18167 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.862702 26655 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:23.920732 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 54741.3us
W20260709 12:25:23.918042 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 110968us
I20260709 12:25:23.922869 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1262) took 2817 ms. Trace:
I20260709 12:25:23.977319 18015 rpcz_store.cc:276] 0709 12:25:21.105294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.795974 (+1690680us) service_pool.cc:224] Handling call
0709 12:25:22.796507 (+   533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.274252 (+477745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.922434 (+648182us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54001095d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:23.922765 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":952087936}
I20260709 12:25:23.932750 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20260709 12:25:23.883389 26679 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:23.949888 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.832031 26672 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.834862 26651 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:23.908460 18017 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:23.992480 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:23.994989 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:23.995673 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 104837us
I20260709 12:25:23.890085 26637 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:25:24.001634 18015 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:24.004591 26664 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:23.944907 26515 consensus_queue.cc:1243] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:23.941931 26137 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:24.013700 18017 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:24.010226 18778 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.004s	sys 0.004s
I20260709 12:25:23.925755 26529 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:24.023363 18778 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.004s	sys 0.004s
W20260709 12:25:24.024029 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 575041us
I20260709 12:25:24.025249 18022 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:23.942308 26504 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.023607 18017 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:24.029157 18022 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:24.030979 26666 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:24.031488 26668 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.103s
W20260709 12:25:24.038311 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.039616 18022 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.043818 18097 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:24.055018 18022 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:24.058909 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.040072 26575 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.067970 26502 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.069507 26653 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.005985 26671 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.151s
I20260709 12:25:24.070273 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:24.071026 26671 consensus_queue.cc:1243] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:24.071681 26671 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:24.072026 26671 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:24.033701 26605 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:24.090200 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.018486 26137 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:24.101758 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 162787us
W20260709 12:25:24.112228 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 134295us
I20260709 12:25:24.074196 26666 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:24.110827 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 360078us
I20260709 12:25:24.079212 26650 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.086760 26581 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:24.111740 26660 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.305s	user 0.001s	sys 0.001s
I20260709 12:25:24.100798 26574 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.132187 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:24.136371 18092 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:24.108904 26640 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.136278 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:24.134380 18091 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:24.139024 18092 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:24.109160 26680 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:24.011919 26652 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.147975 18097 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:25:24.144655 18022 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:25:24.150372 18022 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 11
I20260709 12:25:24.150972 18022 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:25:24.151309 18022 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:24.152384 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3131) took 3429 ms. Trace:
I20260709 12:25:24.178115 18097 raft_consensus.cc:606] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:24.179860 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:24.236337 18094 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:24.152884 18022 rpcz_store.cc:276] 0709 12:25:20.724404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.737232 (+1012828us) service_pool.cc:224] Handling call
0709 12:25:21.817615 (+ 80383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.025096 (+2207481us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b5400048e20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662edc0 00007f6c166b91c0 00007f6c166ba231 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c
0709 12:25:24.151959 (+126863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.152297 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1373262592}
W20260709 12:25:24.212519 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.240386 18092 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:25:24.190660 18099 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:24.250986 18092 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:25:24.262456 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 256159us
I20260709 12:25:24.277954 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:24.135049 26542 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:24.188861 26531 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:25:24.296511 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:24.098246 26671 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:24.306106 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.016s	sys 0.042s
W20260709 12:25:24.330495 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.028s	sys 0.046s
I20260709 12:25:24.332466 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:24.128075 26681 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.131s
I20260709 12:25:24.308746 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:25:24.295928 26677 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.380s	user 0.002s	sys 0.000s
I20260709 12:25:24.149442 26574 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.153s
I20260709 12:25:24.125718 26666 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.301153 26664 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:24.361846 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 236811us
I20260709 12:25:24.174255 26680 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.336921 26663 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:24.336290 26688 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.331375 26418 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:25:24.339306 26693 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:24.420971 26504 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:24.421265 26696 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:25:24.423791 18167 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:24.429628 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.400966 26672 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.369402 26668 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.373463 26653 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:24.380162 26529 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.445261 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:24.478355 18167 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:24.479859 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:24.480774 18167 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
W20260709 12:25:24.483150 26682 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:25:24.494045 26514 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:25:24.494467 26520 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.583s	user 0.013s	sys 0.004s
I20260709 12:25:24.494872 26514 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:24.496387 26514 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:24.498802 26694 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:25:24.500965 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1286) took 1582 ms. Trace:
I20260709 12:25:24.501554 18168 rpcz_store.cc:276] 0709 12:25:22.919092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.919319 (+   227us) service_pool.cc:224] Handling call
0709 12:25:22.997601 (+ 78282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.201267 (+203666us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.201304 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.201970 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.331321 (+1129351us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:24.331392 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.331475 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.500569 (+169094us) raft_consensus.cc:1685] finished
0709 12:25:24.500606 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.500848 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4362,"raft.queue_time_us":176540,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":9250,"thread_start_us":805869,"threads_started":2,"wal-append.queue_time_us":770236}
I20260709 12:25:24.363868 26671 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:24.432086 26681 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.504666 26681 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:24.508453 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.508915 26681 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.509840 26681 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.510550 26681 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.514250 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:24.519915 26680 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.521013 26680 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:24.307996 26650 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.135s
I20260709 12:25:24.428313 26690 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:24.477010 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 207895us
I20260709 12:25:24.522461 26690 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.525070 26690 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.525228 26502 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:24.525563 26655 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.006s
I20260709 12:25:24.526081 26683 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:25:24.527515 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95750.8us
I20260709 12:25:24.526165 26655 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.442466 26688 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.450558 26137 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.513470 26681 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:24.449009 26651 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:24.546268 26514 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:24.533155 26640 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:24.553723 26696 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:25:24.553561 26634 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.354s	user 0.016s	sys 0.002s
W20260709 12:25:24.571317 26657 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.699s	user 0.014s	sys 0.000s
I20260709 12:25:24.567461 26651 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:24.477485 26668 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.521697 26680 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.552907 26650 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:25:24.587445 26650 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:24.587883 26650 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:24.588372 26680 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:24.588969 26650 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.590613 26650 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:24.590878 26666 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:24.591117 26661 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.789s	user 0.008s	sys 0.011s
I20260709 12:25:24.591903 26651 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.592023 26661 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:24.704280 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.701498 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.005s	sys 0.000s
I20260709 12:25:24.678484 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4349) took 3165 ms. Trace:
I20260709 12:25:24.690687 18167 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:24.744634 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.008s	sys 0.000s
I20260709 12:25:24.746477 26671 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.564311 26574 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:24.691167 18171 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.001s	sys 0.001s
I20260709 12:25:24.682718 18090 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.20-23.20]
I20260709 12:25:24.744180 18093 rpcz_store.cc:276] 0709 12:25:21.512905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.530285 (+ 17380us) service_pool.cc:224] Handling call
0709 12:25:21.530670 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.541268 (+ 10598us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.541314 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.008932 (+1467618us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:23.009620 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.547867 (+538247us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:23.547918 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.548019 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.677097 (+1129078us) raft_consensus.cc:1685] finished
0709 12:25:24.677208 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.678060 (+   852us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":42573,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":15289,"spinlock_wait_cycles":11190528,"thread_start_us":417082,"threads_started":1,"wal-append.queue_time_us":733604}
I20260709 12:25:24.756080 26671 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.756875 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:24.759560 26676 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.760764 26676 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.765976 26671 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.766964 26542 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:24.768141 26542 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.553604 26443 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.228s
I20260709 12:25:24.685051 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:24.775743 26680 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:24.775946 18168 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:24.776340 26680 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:24.533560 26502 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.584585 26634 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.778515 26680 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.542250 26520 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:25:24.781330 26680 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.596875 26655 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:24.753274 18171 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.005s	sys 0.003s
I20260709 12:25:24.601680 26668 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:24.682531 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 153819us
I20260709 12:25:24.593751 26651 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.789021 26581 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:24.790585 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:24.792210 26542 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.793138 26581 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:24.595629 26661 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.796013 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:24.799564 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:24.801437 18017 raft_consensus.cc:2410] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:25:24.600366 26688 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.681159 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:24.804563 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3228) took 1326 ms. Trace:
W20260709 12:25:24.808360 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 127806us
I20260709 12:25:24.809031 18022 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:24.809574 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:24.810928 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:24.811810 26574 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:24.811094 26655 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:24.809051 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:24.814073 26655 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.814837 26655 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:24.814044 26574 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.815157 18017 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:25:24.585918 26657 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.816577 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:24.817389 18018 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:24.818465 26683 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.819669 26683 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.789330 26510 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.580749 26640 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.822227 26510 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:24.821977 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3133) took 4096 ms. Trace:
I20260709 12:25:24.811962 18022 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:23.212064 18020 rpcz_store.cc:276] 0709 12:25:20.724753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.415193 (+1690440us) service_pool.cc:224] Handling call
0709 12:25:22.433475 (+ 18282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.433741 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:22.586474 (+152733us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.178236 (+591762us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.178300 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:23.181140 (+  2840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.183889 (+  2749us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.184260 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":140403,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":180878,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":20499,"thread_start_us":735632,"threads_started":3,"wal-append.queue_time_us":538576}
I20260709 12:25:24.817602 18017 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:24.832087 18023 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:25:24.838469 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:24.839191 26661 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:24.839800 26661 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.840240 26661 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:24.823922 18019 rpcz_store.cc:276] 0709 12:25:20.725028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.494948 (+1769920us) service_pool.cc:224] Handling call
0709 12:25:22.664022 (+169074us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.198775 (+534753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.821013 (+1622238us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54002ffb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.821616 (+   603us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1215965952}
I20260709 12:25:22.868340 18098 rpcz_store.cc:276] 0709 12:25:21.533053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.482754 (+949701us) service_pool.cc:224] Handling call
0709 12:25:22.483180 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.483438 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:22.483762 (+   324us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.853925 (+370163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.853978 (+    53us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:22.854030 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.854089 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.854184 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30643,"prepare.queue_time_us":139,"prepare.run_cpu_time_us":355,"prepare.run_wall_time_us":750,"raft.queue_time_us":31327,"thread_start_us":339026,"threads_started":1}
I20260709 12:25:24.781740 26634 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.850301 18019 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:24.851069 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1288) took 1930 ms. Trace:
I20260709 12:25:24.851637 18164 rpcz_store.cc:276] 0709 12:25:22.933911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.934161 (+   250us) service_pool.cc:224] Handling call
0709 12:25:22.934715 (+   554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.203922 (+269207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.850404 (+1646482us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54003ba1d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.850805 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":564348288}
I20260709 12:25:24.859169 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3298) took 3623 ms. Trace:
I20260709 12:25:24.859328 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:24.859711 18165 rpcz_store.cc:276] 0709 12:25:21.235589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.588772 (+353183us) service_pool.cc:224] Handling call
0709 12:25:21.589153 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.199055 (+1609902us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.858497 (+1659442us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b5400383954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.858929 (+   432us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3553171200}
I20260709 12:25:24.860558 18164 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:24.862854 18169 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3223) took 3330 ms (client timeout 3341 ms). Trace:
I20260709 12:25:24.863060 26657 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:24.863401 18169 rpcz_store.cc:269] 0709 12:25:21.531803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.583509 (+ 51706us) service_pool.cc:224] Handling call
0709 12:25:24.862407 (+3278898us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:24.862644 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997329280}
I20260709 12:25:24.864120 18173 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:25:24.865272 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:24.866182 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.867126 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3226) took 1389 ms. Trace:
I20260709 12:25:24.868057 26640 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.861368 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:24.852056 26676 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.871081 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3326) took 1483 ms. Trace:
I20260709 12:25:24.857053 26574 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:24.911043 26517 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:24.911819 26517 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.913331 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:24.915299 26137 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.916471 26137 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:24.879842 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1472) took 4407 ms. Trace:
I20260709 12:25:24.883438 26668 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:24.852371 26581 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:24.885803 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:24.887216 26714 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:24.888037 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1696) took 2448 ms. Trace:
I20260709 12:25:24.890817 26683 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.893409 26438 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:24.805693 18166 rpcz_store.cc:276] 0709 12:25:23.590355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.927280 (+336925us) service_pool.cc:224] Handling call
0709 12:25:23.927721 (+   441us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:23.946207 (+ 18486us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:24.153782 (+207575us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.425284 (+271502us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:24.426090 (+   806us) log.cc:844] Serialized 52 byte log entry
0709 12:25:24.426274 (+   184us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:24.429094 (+  2820us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.803647 (+374553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.803846 (+   199us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.803938 (+    92us) raft_consensus.cc:1685] finished
0709 12:25:24.803964 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.804264 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108985,"prepare.queue_time_us":385338,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":6739,"raft.queue_time_us":350439,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":26882,"thread_start_us":633106,"threads_started":3,"wal-append.queue_time_us":203073}
I20260709 12:25:24.867519 18170 rpcz_store.cc:276] 0709 12:25:23.477153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.477915 (+   762us) service_pool.cc:224] Handling call
0709 12:25:23.478414 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.596630 (+118216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.596672 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.596723 (+    51us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:23.596768 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.866595 (+1269827us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002021d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.866664 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.866929 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":227781,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":5771,"spinlock_wait_cycles":270926720}
I20260709 12:25:24.874583 18172 rpcz_store.cc:276] 0709 12:25:23.386883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.387104 (+   221us) service_pool.cc:224] Handling call
0709 12:25:23.387530 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.809518 (+421988us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.809577 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.809651 (+    74us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:23.809706 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.868795 (+1059089us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54003b9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.868870 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.870237 (+  1367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":969642240}
I20260709 12:25:24.899228 26478 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:24.899832 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:24.901562 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:24.903991 26497 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:24.909140 26515 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:24.911754 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:24.915403 26715 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:24.946616 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:24.948726 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:24.881256 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:24.920737 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:24.883980 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:24.881557 18811 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:25:24.918655 18807 rpcz_store.cc:276] 0709 12:25:20.472471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.472674 (+   203us) service_pool.cc:224] Handling call
0709 12:25:20.473195 (+   521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.790712 (+2317517us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.879012 (+2088300us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.879565 (+   553us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4618647296}
I20260709 12:25:24.927831 18165 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:24.929469 18815 rpcz_store.cc:276] 0709 12:25:22.439297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.439664 (+   367us) service_pool.cc:224] Handling call
0709 12:25:22.460840 (+ 21176us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.133306 (+672466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.887092 (+1753786us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54004b41d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:24.887602 (+   510us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1331744000}
I20260709 12:25:24.941483 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:24.946373 26715 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:24.934545 18168 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:24.957520 18814 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:24.960536 26440 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:24.960239 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:24.961611 26563 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:24.962033 18168 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:24.962284 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:24.950074 18811 pending_rounds.cc:85] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 13
I20260709 12:25:24.963596 18093 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:24.965557 18165 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:24.966822 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3237) took 1174 ms. Trace:
I20260709 12:25:24.973583 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:24.975204 18166 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:24.978536 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:24.980087 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:24.981743 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:24.982977 18094 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:25:24.978931 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93573.5us
I20260709 12:25:24.986359 18023 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:24.987629 26676 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:24.987944 18020 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:24.991523 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:24.967358 18165 rpcz_store.cc:276] 0709 12:25:23.792309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.963965 (+1171656us) service_pool.cc:224] Handling call
0709 12:25:24.964607 (+   642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.966300 (+  1693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.966686 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.937883 26438 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:24.951411 26478 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:24.963590 18811 pending_rounds.cc:107] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:25:25.028182 18811 raft_consensus.cc:2889] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:24.996431 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:25.009703 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.010030 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.010565 18164 raft_consensus.cc:2376] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21. Re-sending same reply.
W20260709 12:25:25.015156 26611 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.574s	user 0.028s	sys 0.001s
I20260709 12:25:25.021288 18020 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:24.987682 26667 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 1.581s	user 0.007s	sys 0.003s
I20260709 12:25:24.962746 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.023247 26438 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.024394 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.024982 26497 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:25.026314 26674 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:25.034139 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:24.944288 26515 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:25.034523 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.037331 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.034731 18170 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:25.025929 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.040601 18093 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.039690 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3232) took 1412 ms. Trace:
I20260709 12:25:25.038374 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.044451 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:25.055698 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60164.8us
I20260709 12:25:25.055379 26497 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.058295 18097 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:25.049610 18015 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.026s	sys 0.018s
W20260709 12:25:25.060859 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51098.7us
I20260709 12:25:25.025836 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.065601 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:25.069165 18015 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.063s	user 0.027s	sys 0.018s
I20260709 12:25:24.975077 26581 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.057793 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:25.085986 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.063720 26674 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:25.091042 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.082561 18815 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:25.048053 26611 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.081007 26438 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.088474 18099 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:25.045382 26668 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.045037 18171 rpcz_store.cc:276] 0709 12:25:23.627267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.999042 (+371775us) service_pool.cc:224] Handling call
0709 12:25:23.999485 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.999730 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:24.783628 (+783898us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.924738 (+141110us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:24.930272 (+  5534us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:24.995039 (+ 64767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.006485 (+ 11446us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.039597 (+ 33112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10761,"prepare.queue_time_us":379622,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":78,"raft.queue_time_us":2263,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":200,"thread_start_us":824950,"threads_started":2,"wal-append.queue_time_us":159881}
I20260709 12:25:25.093461 18094 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:25.100518 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4351) took 3532 ms. Trace:
I20260709 12:25:25.100665 26520 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:25.104714 26650 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.010s
I20260709 12:25:25.107178 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4229) took 4788 ms. Trace:
I20260709 12:25:25.112644 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:25.124368 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.127023 26516 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.303s	user 0.014s	sys 0.011s
I20260709 12:25:25.127516 26640 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:25.127388 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69843.9us
I20260709 12:25:25.132074 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:25.140690 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 79249.2us
I20260709 12:25:25.145989 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.138902 18814 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:25.128208 26640 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.112725 26657 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:25.158766 18168 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:25.100566 26678 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:25.153405 18807 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:25.170332 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3145) took 1812 ms. Trace:
I20260709 12:25:25.171844 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:25.171808 26674 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:25.173041 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 158136us
I20260709 12:25:25.176676 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:25.184803 26687 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:25.186057 26687 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.187933 26678 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.188527 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:25.190645 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.191133 26678 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.194828 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:25.200381 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.148921 26563 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:25.186141 26515 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.202579 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.207938 26137 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.208971 26653 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:25.209406 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:25.127730 26516 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.177548 26640 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.178897 26657 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:25.217708 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46024.9us
I20260709 12:25:25.178570 18015 rpcz_store.cc:276] 0709 12:25:23.358101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.005137 (+647036us) service_pool.cc:224] Handling call
0709 12:25:24.005521 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.137020 (+1131499us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:25.145157 (+  8137us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.153726 (+  8569us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:25.153792 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.169979 (+ 16187us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.170217 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25174,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":152,"thread_start_us":1063645,"threads_started":1}
I20260709 12:25:25.112088 18095 rpcz_store.cc:276] 0709 12:25:21.513611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.863566 (+349955us) service_pool.cc:224] Handling call
0709 12:25:21.864046 (+   480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:21.864560 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.864595 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:21.865283 (+   688us) log.cc:844] Serialized 52 byte log entry
0709 12:25:22.337020 (+471737us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:22.413513 (+ 76493us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.414578 (+  1065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.414701 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.547711 (+133010us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b18004db5a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:22.675204 (+127493us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:24.976441 (+2301237us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b2400278378. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c13608fb0 00007f6c1627f488 00007f6c1627fbe8 00007f6c16257398 00007f6c1625276a 00007f6c16251bb6
0709 12:25:25.027016 (+ 50575us) raft_consensus.cc:1685] finished
0709 12:25:25.027047 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.045873 (+ 18826us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35521,"prepare.queue_time_us":78182,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":7150,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":254,"spinlock_wait_cycles":68070400,"thread_start_us":471206,"threads_started":1,"wal-append.queue_time_us":578729}
I20260709 12:25:25.211318 26563 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.220250 26497 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.224548 26563 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:25.226681 26623 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.402s	user 0.012s	sys 0.011s
I20260709 12:25:25.229779 26611 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:25.119975 18021 rpcz_store.cc:276] 0709 12:25:20.318972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.533406 (+1214434us) service_pool.cc:224] Handling call
0709 12:25:21.576332 (+ 42926us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.597265 (+ 20933us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:21.700184 (+102919us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:25:22.438519 (+738335us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.294971 (+856452us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.383570 (+ 88599us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.389694 (+  6124us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.389892 (+   198us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:23.389926 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.390003 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.978689 (+1588686us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b18001f0f00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:25.106625 (+127936us) raft_consensus.cc:1685] finished
0709 12:25:25.106655 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.106955 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94505,"prepare.queue_time_us":1453313,"prepare.run_cpu_time_us":8653,"prepare.run_wall_time_us":74690,"raft.queue_time_us":377434,"raft.run_cpu_time_us":3195,"raft.run_wall_time_us":240730,"spinlock_wait_cycles":59698816,"thread_start_us":1597488,"threads_started":4,"wal-append.queue_time_us":593273}
W20260709 12:25:25.233999 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.209496 26653 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.223112 26657 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:25.247009 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.253616 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.239784 26634 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:25.258315 18809 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 26 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:25:25.260108 26613 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:25.231645 26650 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.255018 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:25.265110 26650 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.157536 26520 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.266434 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.242311 26653 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.249289 26513 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:25.238916 26713 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:25.270677 18095 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:25.283221 18097 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:25.301513 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.264556 26634 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.306782 18815 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:25.269588 26137 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:25.304358 26538 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.845s	user 0.030s	sys 0.014s
I20260709 12:25:25.268110 26613 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:25.320168 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 143791us
I20260709 12:25:25.301087 26513 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:25.310892 26634 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.326371 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.325083 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:25.333354 18097 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:25.318135 26538 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.042s	user 0.033s	sys 0.016s
I20260709 12:25:25.320626 26613 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:25.338814 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 225698us
I20260709 12:25:25.345050 26529 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.346084 26529 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.348814 26503 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:25.350239 26720 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:25.335906 26517 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.353904 18098 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:25.364202 26596 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:25.364899 26596 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:25.365386 26596 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.366087 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1323) took 1235 ms. Trace:
I20260709 12:25:25.366540 18090 rpcz_store.cc:276] 0709 12:25:24.130566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.130761 (+   195us) service_pool.cc:224] Handling call
0709 12:25:24.681683 (+550922us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.754509 (+ 72826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.365697 (+611188us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b5400410fd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:25.365991 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1265486720}
I20260709 12:25:25.351406 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.367913 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.368618 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:25:25.371714 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 136049us
I20260709 12:25:25.326009 26137 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:25.339813 26655 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.524s	user 0.016s	sys 0.004s
W20260709 12:25:25.382573 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 132086us
I20260709 12:25:25.385290 18093 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:25.386185 18094 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:25.386667 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.386689 18172 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:25.388073 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1538) took 2406 ms. Trace:
I20260709 12:25:25.388893 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.389782 18808 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:25.359505 26478 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.391458 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.392170 18093 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.388573 18806 rpcz_store.cc:276] 0709 12:25:22.982094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.983789 (+  1695us) service_pool.cc:224] Handling call
0709 12:25:23.015768 (+ 31979us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.136792 (+121024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.387649 (+2250857us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b54001d9f50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:25.387860 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70157056}
I20260709 12:25:25.393018 26478 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.391777 18099 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:25:25.394811 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.396456 18099 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:25.337944 26678 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:25:25.365305 26638 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:25.399688 26638 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:25.401037 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.402012 26668 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:25.355803 26720 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:25.403723 18093 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:25:25.404187 26720 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.404345 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.405194 18165 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:25.407308 26517 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.389243 26513 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:25.389134 26721 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:25:25.349557 26529 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.400972 26638 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.416637 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.418846 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:25.409932 26515 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:25.432827 26510 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.610s	user 0.010s	sys 0.006s
I20260709 12:25:25.462591 18811 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 22. (index mismatch)
I20260709 12:25:25.458546 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.468524 26137 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:25.469954 26651 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.590s	user 0.018s	sys 0.000s
W20260709 12:25:25.478608 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60415.9us
W20260709 12:25:25.478761 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.449173 26516 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.410068 26655 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.454782 26713 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:25.494861 18090 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:25.495548 26731 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.496237 26731 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.496635 26728 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:25:25.458271 26529 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:25.509392 18022 raft_consensus.cc:2368] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:25.514403 26731 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.515496 26531 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:25:25.516007 26520 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.516089 26529 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.517472 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:25.489032 26510 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:25.498414 26638 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.532146 26529 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:25.544147 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 135115us
I20260709 12:25:25.545524 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:25.550518 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.564694 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.552590 18810 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:25.578202 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1287) took 2658 ms. Trace:
W20260709 12:25:25.579984 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 60113.3us
I20260709 12:25:25.578051 26520 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:25.578788 18173 rpcz_store.cc:276] 0709 12:25:22.919740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.919911 (+   171us) service_pool.cc:224] Handling call
0709 12:25:24.863399 (+1943488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.864783 (+  1384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.574720 (+709937us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54001b3d50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:25.578100 (+  3380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4319061888}
I20260709 12:25:25.584535 26520 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:25.590654 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:25.558684 26714 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:25.588805 26520 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:25.586900 26531 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 12:25:25.581102 26510 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:25.607239 26529 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:25.609093 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:25.610021 18024 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:25.610302 26438 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:25.526823 26563 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.601094 26714 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:25.624500 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 107184us
W20260709 12:25:25.612494 26661 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.772s	user 0.017s	sys 0.005s
I20260709 12:25:25.624104 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.629035 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.629971 18019 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:25:25.634425 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43877us
W20260709 12:25:25.634943 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.642103 18092 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:25.642795 26655 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.648625 26563 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.652120 26715 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.635802 26661 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.654680 26653 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.657451 26661 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:25.623363 26680 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.841s	user 0.012s	sys 0.007s
I20260709 12:25:24.804229 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:25.661445 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3238) took 1855 ms. Trace:
I20260709 12:25:25.667593 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.668651 18019 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:25.668105 18166 rpcz_store.cc:276] 0709 12:25:23.805703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.976732 (+1171029us) service_pool.cc:224] Handling call
0709 12:25:24.977683 (+   951us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:25.082909 (+105226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.082956 (+    47us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:25.155388 (+ 72432us) log.cc:844] Serialized 78 byte log entry
0709 12:25:25.312411 (+157023us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:25.318529 (+  6118us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.661080 (+342551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.661166 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.661213 (+    47us) raft_consensus.cc:1685] finished
0709 12:25:25.661230 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.661378 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":374729,"prepare.queue_time_us":16468,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":14002,"raft.queue_time_us":375981,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":19208,"thread_start_us":121460,"threads_started":1,"wal-append.queue_time_us":238038}
I20260709 12:25:25.670212 26613 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.671901 26596 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.673094 18093 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:25.675805 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.676470 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.676994 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.677234 18172 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:25.677515 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.678745 18169 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:25.678875 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.679916 26596 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.680423 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.681161 18166 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:25.682441 26653 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.679962 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.685158 18173 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:25.686195 26497 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.686450 26680 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:25.687302 26497 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.660091 26634 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.627727 26714 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.695858 26497 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.696842 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.712240 18809 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.697868 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.700651 26657 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.702334 26602 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.717670 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.689563 26613 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.690668 26713 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.692269 18097 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:25.714185 18024 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:25.702742 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:25.716369 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3336) took 1365 ms. Trace:
W20260709 12:25:25.721450 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.728787 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.729261 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.735076 26675 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:25.736193 26574 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.826s	user 0.014s	sys 0.004s
I20260709 12:25:25.738747 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1533) took 4334 ms. Trace:
I20260709 12:25:25.739280 18813 rpcz_store.cc:276] 0709 12:25:21.404101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.404309 (+   208us) service_pool.cc:224] Handling call
0709 12:25:21.404657 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.281097 (+1876440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.738155 (+2457058us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54005901d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:25.738530 (+   375us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4256503168}
I20260709 12:25:25.743387 26634 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.744374 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.744750 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:25.745187 18021 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
I20260709 12:25:25.746373 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:25.746614 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3114) took 5407 ms (client timeout 2558 ms). Trace:
I20260709 12:25:25.747629 18173 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 18: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:25:25.747262 18016 rpcz_store.cc:269] 0709 12:25:20.339143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.112311 (+773168us) service_pool.cc:224] Handling call
0709 12:25:25.746002 (+4633691us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400251054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:25.746346 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2893084416}
I20260709 12:25:25.749598 18170 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:25.749434 26678 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:25.717664 26602 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.749944 26613 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:25.754215 26510 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.754787 26513 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:25.757019 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3235) took 2004 ms. Trace:
I20260709 12:25:25.757512 18168 rpcz_store.cc:276] 0709 12:25:23.752832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.963323 (+1210491us) service_pool.cc:224] Handling call
0709 12:25:24.963932 (+   609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.163712 (+199780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.756651 (+592939us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:25.756912 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":430026496}
W20260709 12:25:25.758549 26735 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:25:25.759456 26653 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.760229 26653 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.761190 26479 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:25.761128 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.761857 26479 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:25.763048 26479 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.763651 26653 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.764284 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.764783 26479 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:25.765214 18807 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:25.766721 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:25.767161 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.768572 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.770066 26653 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:25.771198 26653 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:25.771623 26653 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.771971 26653 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.774443 26602 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:25.777010 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74529us
I20260709 12:25:25.778928 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.780339 26497 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:25.781337 26497 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:25.782234 26668 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:25.782425 26732 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:25:25.782840 26668 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:25.783430 26668 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.735949 26675 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.749509 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.785653 18809 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:25.779881 18063 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.796602 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.769716 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:25.726519 18164 rpcz_store.cc:276] 0709 12:25:24.351272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.038849 (+687577us) service_pool.cc:224] Handling call
0709 12:25:25.085506 (+ 46657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.125791 (+ 40285us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:25.309381 (+183590us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.715522 (+406141us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.715812 (+   290us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.715884 (+    72us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:25.715973 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.716086 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.716268 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":239564,"prepare.queue_time_us":64667,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":235047,"thread_start_us":285956,"threads_started":2,"wal-append.queue_time_us":197134}
I20260709 12:25:25.800016 18814 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:25.802516 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.804811 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:25.805219 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 67539.8us
I20260709 12:25:25.806018 18813 raft_consensus.cc:2410] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:25:25.806218 26675 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.806357 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.807025 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.807916 26502 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.808661 26502 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:25.809185 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.809912 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.810420 18098 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 25 because replica has last-logged OpId of term: 24 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 19.
I20260709 12:25:25.811595 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.812440 18165 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.813903 18098 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20260709 12:25:25.815145 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.815676 26613 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:25.816044 26640 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.586s	user 0.008s	sys 0.009s
I20260709 12:25:25.816401 26613 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:25.816591 26640 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.817384 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.817517 26657 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.818058 18168 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:25.819177 18169 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:25.819257 26657 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.820359 26657 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.821271 18807 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:25.822764 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.822849 26678 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:25.823726 26678 consensus_queue.cc:1243] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 25.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:25.824496 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.825519 18806 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:25.817616 26613 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.827965 18808 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:25:25.828382 18808 pending_rounds.cc:85] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 17
I20260709 12:25:25.828652 18808 pending_rounds.cc:107] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 12:25:25.828912 18808 raft_consensus.cc:2889] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.830087 26678 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:25.831372 26438 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:25:25.831804 26438 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:25.832965 18094 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:25:25.832847 26438 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.724704 26520 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.834286 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.835021 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:25.826538 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.835778 26657 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.835877 18810 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:25.836357 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.836614 26520 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:25.836977 18172 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.838557 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.839232 18024 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:25.839295 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:25.840231 18019 raft_consensus.cc:2368] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22.
I20260709 12:25:25.840608 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.841490 18023 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:25.841347 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.842449 18020 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:25.843309 26687 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:25.843959 26657 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:25.844194 26499 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:25.844455 26657 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.835891 18091 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:25.844877 26657 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:25.747993 26596 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:25.852104 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:25.857205 18166 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:25.860281 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:25.862216 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.809278 26502 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.835049 18812 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:25.872437 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.853029 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.863794 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.874225 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:25.874549 18170 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:25.843608 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73983.2us
I20260709 12:25:25.845947 26602 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20260709 12:25:25.844811 26499 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.867151 18810 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 19
I20260709 12:25:25.880595 18810 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 25.20
I20260709 12:25:25.867314 18806 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:25.867671 26640 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.868289 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.868880 26679 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:25.863415 26715 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:25.870157 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:25.870791 26707 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.875299 18169 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:25.855257 26596 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.880903 18097 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:25:25.880997 18810 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.876909 18165 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:25.905051 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.905488 18020 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:25.906250 18166 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:25.904134 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:25.913048 18171 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.919567 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:25.926946 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:25.931452 18169 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:25:25.907179 26596 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:25.942595 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.943647 18166 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:25.951689 18172 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:25.957448 18020 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:25.924749 26520 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.920379 26715 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:25.972314 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:25.978376 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.981508 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:25.984313 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.979984 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:25.989243 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.990211 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.967154 26713 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:25.993901 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.914434 26707 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:25.995813 18168 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:26.001246 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:25.937043 26679 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.983376 26559 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.004285 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.004807 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89563.2us
I20260709 12:25:26.006156 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:25.963891 26714 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:25.983400 26520 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.020700 18173 raft_consensus.cc:2410] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:25:26.022441 18015 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 12:25:26.016388 26549 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.483s	user 0.014s	sys 0.018s
I20260709 12:25:26.028945 26642 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.029702 26642 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.032189 26642 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.034369 26499 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:26.028141 26549 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.620s	user 0.014s	sys 0.020s
I20260709 12:25:26.037997 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.038398 26549 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:26.038942 18813 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:26.039680 26549 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.041050 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:26.026065 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:26.042727 26499 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:26.042997 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:26.043210 26499 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:26.043705 26499 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:26.044101 26674 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:26.044862 26674 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:26.046087 26687 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:26.046620 26687 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.047091 26687 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.019841 26510 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:26.048985 26650 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:25.986281 26663 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.223s
I20260709 12:25:26.049031 26510 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:26.050431 26663 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:26.050626 26510 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.050890 26663 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:26.050305 26674 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.052058 26663 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.053557 26663 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:26.023258 26559 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.014288 26502 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:26.065542 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45811.7us
I20260709 12:25:26.066681 18091 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:25:26.070277 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80130.9us
I20260709 12:25:26.103705 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:26.106563 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82251us
I20260709 12:25:26.107105 18094 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:25:26.107576 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:26.110136 18169 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:26.113549 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:26.112274 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.127378 18015 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:26.110002 26713 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.117462 26731 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:26.142347 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:26.137387 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.146528 18098 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:26.157001 26497 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:26.128751 26515 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:26.155365 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:26.170584 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.172472 26720 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.768s	user 0.014s	sys 0.002s
W20260709 12:25:26.172888 26738 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.144s	user 0.000s	sys 0.003s
I20260709 12:25:26.174469 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:26.177024 26559 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.177289 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.173040 26720 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.160828 26666 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.036s
I20260709 12:25:26.167985 26499 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.169647 18023 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:26.207168 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.203420 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76974.6us
I20260709 12:25:26.186465 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.101608 26715 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.197890 18172 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:25:26.183624 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.232390 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 85312.7us
I20260709 12:25:26.205721 18165 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:25:26.225319 18166 raft_consensus.cc:2368] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:26.233155 18170 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:26.222635 26497 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 12:25:26.196082 26520 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.243551 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.245597 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:26.242086 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.246157 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.249302 26574 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.250022 26574 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.252863 26574 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.254109 26499 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.258399 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:26.261018 26707 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.262602 26515 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:26.263156 26515 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.263441 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1551) took 1085 ms. Trace:
I20260709 12:25:26.263998 18814 rpcz_store.cc:276] 0709 12:25:25.178167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.199934 (+ 21767us) service_pool.cc:224] Handling call
0709 12:25:26.263199 (+1063265us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:26.263334 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1084932480}
W20260709 12:25:26.192669 26739 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:25:26.263549 26515 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:26.268100 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.268911 18814 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:26.269241 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.263355 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.270802 26714 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.259080 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.272001 18095 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:26.271906 26679 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:26.273437 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3141) took 2916 ms (client timeout 2290 ms). Trace:
W20260709 12:25:26.273947 18018 rpcz_store.cc:269] 0709 12:25:23.356668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.994629 (+637961us) service_pool.cc:224] Handling call
0709 12:25:26.273159 (+2278530us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b5400109fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:26.273332 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1875229952}
I20260709 12:25:26.274353 26638 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.275023 26638 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.267333 18063 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:26.278549 26743 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:26.279237 26743 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:26.279533 26438 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:26.280380 26743 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:26.284654 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.286983 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.287878 18806 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:25:26.287146 18814 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:26.291785 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:26.282828 26744 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:25:26.294920 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:26.177630 26668 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.300195 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.280124 26638 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.309993 18095 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
W20260709 12:25:26.300004 26653 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.528s	user 0.015s	sys 0.000s
I20260709 12:25:26.321583 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.299719 26679 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:26.283275 26687 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.187597 26713 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.315694 26675 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:25:26.345034 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50265.2us
I20260709 12:25:26.321732 26520 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:26.328131 26679 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:26.337126 26707 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.341012 26731 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.363545 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:26.364283 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.364670 26547 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.368165 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.312448 26714 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:25:26.380642 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.373880 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:26.392346 18171 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:26.335312 26653 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.395434 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.400256 18015 raft_consensus.cc:2368] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:26.402127 18024 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:26.398448 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:26.412565 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4240) took 1616 ms. Trace:
I20260709 12:25:26.367242 26520 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.420431 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:26.348445 26650 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:25:26.434830 18016 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:26.408254 26627 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:26.443192 26514 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.450434 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.452149 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.454025 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:26.462334 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:26.472558 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.474617 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.428978 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:26.484225 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.507047 18090 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:26.484225 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.489742 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.507934 18164 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:26.381105 26731 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.489774 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.397567 26596 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.464473 26547 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.418594 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.495631 18022 rpcz_store.cc:276] 0709 12:25:24.795943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.827655 (+ 31712us) service_pool.cc:224] Handling call
0709 12:25:26.412233 (+1584578us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54001d0954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:26.412440 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261944064}
I20260709 12:25:26.500423 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:26.500955 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80687.2us
I20260709 12:25:26.394604 26707 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.502264 18023 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:26.407960 26668 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.506259 18095 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:25:26.528126 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 129857us
I20260709 12:25:26.516438 18098 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:26.533087 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:26.535579 26403 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:26.535872 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.536154 26403 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:26.536571 26403 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:26.537264 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:26.537849 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.538590 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.535604 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.539985 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.540934 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:26.546962 18093 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:26.551496 18168 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
W20260709 12:25:26.551792 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 190933us
W20260709 12:25:26.529378 26657 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.640s	user 0.015s	sys 0.000s
I20260709 12:25:26.554855 18806 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:26.504601 26714 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:26.560211 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.560431 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:26.561614 18165 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:26.562412 18091 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:26.507303 26627 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.554636 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:26.528599 26513 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:26.565068 26657 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.569911 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.571826 26731 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.503376 26510 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.574712 26653 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.575312 26627 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.579043 18164 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:26.555418 26497 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.580475 18168 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 26 index: 22. (index mismatch)
I20260709 12:25:26.552675 26674 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:26.588811 18095 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:26.594623 18165 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:26.593430 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 199597us
W20260709 12:25:26.592079 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51221.1us
I20260709 12:25:26.552731 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.583001 18173 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:26.592695 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:26.604374 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:26.567442 26720 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.606187 18020 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:26.608825 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.609586 18173 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:26.550787 26650 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:25:26.508090 26679 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.619592 18811 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:26.617743 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.622586 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.627525 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:26.627146 26759 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:26.632658 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46304.7us
I20260709 12:25:26.594146 26687 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.633878 18164 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:26.641307 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:26.640241 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.605736 26627 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:26.586548 26714 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.710500 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.726190 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.727880 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:26.721628 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 292842us
I20260709 12:25:26.675797 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.673480 26443 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:26.620589 26574 raft_consensus.cc:2764] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:25:26.616585 26653 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.704931 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 74266.5us
I20260709 12:25:26.607223 26497 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.704695 18171 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:26.663350 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.764186 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.634680 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:26.708717 26520 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:26.712208 26547 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.710299 26596 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.708413 26707 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:26.721009 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 235704us
I20260709 12:25:26.715354 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.620059 26674 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:26.676884 26754 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
I20260709 12:25:26.782224 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.790899 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.792012 18021 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:26.714334 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:26.705348 26758 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:25:26.708120 26513 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:26.804950 26714 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.751901 26627 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:26.806195 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.650161 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.807770 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.808014 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.807548 26627 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.809062 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:26.734054 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.692711 26757 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:25:26.810220 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.810686 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.811605 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:26.779439 26611 raft_consensus.cc:2764] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:25:26.795423 26520 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:26.814114 26520 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.814543 26520 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:26.815833 26714 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:26.797289 26581 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:26.817318 26581 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:26.814550 26657 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.817929 26514 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:26.818783 18099 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:26.818502 26581 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.819718 26581 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.820344 18099 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:26.779884 26653 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.742934 18806 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:25:26.704569 26751 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.292s	user 0.002s	sys 0.000s
I20260709 12:25:26.753517 18136 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:26.669317 26759 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:26.777140 18019 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:26.777663 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 143085us
I20260709 12:25:26.736728 26679 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.781015 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.785383 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.792992 18091 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:26.879170 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:26.880679 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63835.5us
I20260709 12:25:26.881048 18021 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:25:26.796442 18018 raft_consensus.cc:2368] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22.
I20260709 12:25:26.797545 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.799391 18016 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:26.806674 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:26.809134 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:26.729789 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:26.817385 26714 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:26.746333 18814 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:26.885475 18096 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:26.737635 26731 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.887599 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157881us
I20260709 12:25:26.882388 18024 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
W20260709 12:25:26.888335 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58933.1us
I20260709 12:25:26.887023 18091 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:26.890690 18091 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:26.888504 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:26.893870 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 252639us
I20260709 12:25:26.799080 26713 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:26.895641 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.899506 18024 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:26.825619 26510 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.878849 26613 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:26.903244 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 149850us
I20260709 12:25:26.904054 18094 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:26.904717 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:26.905449 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.906723 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:26.907370 18024 raft_consensus.cc:2376] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24. Re-sending same reply.
I20260709 12:25:26.910843 18024 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 12:25:26.912010 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:26.914273 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 21 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.914690 26759 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:26.915999 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.916344 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.917008 18806 raft_consensus.cc:2410] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:25:26.917810 26613 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:26.919008 26613 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.20, Last appended by leader: 20, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.904162 26596 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:25:26.922366 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.923293 18808 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:26.884984 26714 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:26.929715 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:26.934247 18024 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:26.929870 18096 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:26.928901 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.917865 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.898483 26764 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:26.952632 18019 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:26.951354 18812 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:25:26.899358 26547 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:26.959805 18095 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 12:25:26.959841 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 234922us
I20260709 12:25:26.906855 26653 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:26.962790 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.964399 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.965386 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:26.966256 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.933310 26602 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:26.971264 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.972455 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:26.967480 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:26.967861 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:26.956040 26758 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:25:26.886637 26666 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:25:26.901551 26731 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:26.964191 26764 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:27.015223 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48429.7us
W20260709 12:25:27.017690 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 88075.3us
I20260709 12:25:27.017513 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.018597 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.020067 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.029577 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:26.975364 26513 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:25:27.026644 26772 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:27.030823 18016 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:27.030103 18808 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:26.972767 26515 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.704s	user 0.016s	sys 0.000s
I20260709 12:25:27.053486 18090 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:27.053880 26515 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:27.048698 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 141279us
I20260709 12:25:27.056824 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.057760 18015 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:25:26.964444 26510 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.046800 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.059965 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.067584 18019 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23.
I20260709 12:25:27.066111 18024 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 18.
I20260709 12:25:27.069998 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.073927 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.077612 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.077363 26510 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.077857 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:27.078536 18808 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:27.078202 26764 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.084018 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.085035 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.085855 18019 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:27.086468 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:27.086841 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.087343 18808 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:27.087504 18019 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:25:27.088539 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:27.090034 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55834.9us
W20260709 12:25:27.094023 26403 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.557s	user 0.016s	sys 0.003s
I20260709 12:25:27.094570 26403 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.096306 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.096210 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.097333 18165 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 31 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:25:27.097241 26403 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:27.097767 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.098187 26707 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:27.098307 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.098591 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.098933 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:27.099081 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69602.6us
I20260709 12:25:27.099761 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:27.100296 26771 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:25:27.078552 18016 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:27.085501 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:27.126598 18169 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.088845 18814 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:27.127418 18814 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:27.108079 18808 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:27.110613 26638 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:26.975216 26574 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.091s
I20260709 12:25:27.133078 18172 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:27.130934 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.113523 26731 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:27.117452 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.146545 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:27.106429 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.124514 18171 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:27.147322 18168 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.066684 26602 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:27.141912 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:27.148532 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.148628 18780 leader_election.cc:400] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:27.149550 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:27.104858 26642 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:27.076597 26666 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:27.125379 26707 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.128384 18814 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.154385 18780 leader_election.cc:403] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:27.154541 18806 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:27.155285 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97494.1us
I20260709 12:25:27.163535 18814 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:27.155493 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49215us
I20260709 12:25:27.076797 26770 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:27.168946 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.173040 26403 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:27.039863 26714 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:27.179033 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.180109 26777 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.110s	user 0.000s	sys 0.005s
I20260709 12:25:27.131733 26663 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
W20260709 12:25:27.195623 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78293.6us
I20260709 12:25:27.150624 26713 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:27.202086 18064 leader_election.cc:400] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22.
I20260709 12:25:27.202481 18064 leader_election.cc:403] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
W20260709 12:25:27.198266 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74976.8us
I20260709 12:25:27.135619 26638 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:27.178936 26714 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:27.202898 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.155318 26574 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:27.154078 26602 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:27.163095 26542 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:27.156299 26642 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:27.181244 26707 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:27.158919 26666 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:27.195032 26674 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:27.207320 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.224412 26666 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:27.232612 26642 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.208961 26513 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:27.234402 26642 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.206552 26638 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:27.244129 18168 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:27.211421 26602 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:27.209249 26574 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:27.211266 26663 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:27.210635 26713 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:27.211635 26542 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:27.237764 26756 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.710s	user 0.000s	sys 0.004s
I20260709 12:25:27.224656 26613 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:27.239686 26547 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:27.209156 26443 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:25:27.262998 26443 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:27.263417 26443 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:27.263833 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:27.264707 18024 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:27.264394 26443 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.249981 26513 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:27.267299 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:27.257100 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49865.5us
I20260709 12:25:27.269812 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.270824 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.271811 18024 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:27.274451 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110065us
I20260709 12:25:27.275836 18814 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:25:27.276352 18814 pending_rounds.cc:85] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 14
I20260709 12:25:27.276588 18814 pending_rounds.cc:107] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:25:27.222057 26773 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:25:27.278232 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.279021 18024 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:27.244230 18169 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:27.280287 18165 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:27.282189 26715 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 12:25:27.287156 26715 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:27.287588 26715 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:27.288561 26715 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.255861 26542 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:27.293251 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.293965 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.296207 18016 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:27.296643 18808 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:27.293908 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.294682 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:27.299916 18812 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:27.299717 26790 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:27.271961 26663 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:27.301074 18810 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:27.265761 26443 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.305753 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.303602 18814 raft_consensus.cc:2889] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:27.307293 18813 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: [24.20-24.20]
I20260709 12:25:27.307787 26713 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.307752 26547 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.290604 26650 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:27.310545 26778 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:25:27.304380 26715 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.301357 26663 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:27.305783 26574 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.322131 26663 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.323734 26663 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.322656 26574 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:27.330611 26731 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:27.331918 26581 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.004s
I20260709 12:25:27.332856 26581 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:27.333488 26581 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:27.334817 26581 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.335961 26784 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:27.336778 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.336622 26784 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.337703 26653 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:27.338265 26653 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:27.338573 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.339388 26784 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:27.346360 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.347505 18813 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:27.351147 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3164) took 1356 ms. Trace:
I20260709 12:25:27.351653 18020 rpcz_store.cc:276] 0709 12:25:25.995179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.995349 (+   170us) service_pool.cc:224] Handling call
0709 12:25:27.350863 (+1355514us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b54001d1d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:27.351049 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1374567808}
I20260709 12:25:27.353175 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.355929 26765 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:27.356695 26765 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.359519 26765 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:27.350677 26653 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:27.319813 26520 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.502s	user 0.014s	sys 0.000s
I20260709 12:25:27.362447 26520 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.363260 26510 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:27.365541 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:27.367086 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.368404 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.331374 26731 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.370599 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1268) took 1017 ms. Trace:
I20260709 12:25:27.371217 18023 rpcz_store.cc:276] 0709 12:25:26.353039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.353191 (+   152us) service_pool.cc:224] Handling call
0709 12:25:26.363432 (+ 10241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.535365 (+171933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.370250 (+834885us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:27.370477 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309986560}
I20260709 12:25:27.372712 26765 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:27.375171 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75903us
I20260709 12:25:27.383327 26497 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:27.386260 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.388521 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:27.392092 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54279us
I20260709 12:25:27.396085 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:27.398027 18172 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:27.390473 18019 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:27.391153 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:27.408627 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.408851 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:27.369644 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.413331 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.413619 18806 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:27.414232 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:27.416960 18780 leader_election.cc:400] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:27.418392 26784 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:27.418195 26627 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:27.443229 26497 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:27.444105 26783 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:25:27.442740 18168 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:27.373395 26765 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.451924 18170 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
W20260709 12:25:27.460678 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54887.8us
I20260709 12:25:27.463562 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.468171 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:27.470142 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:27.454777 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45989.4us
I20260709 12:25:27.466565 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.478367 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.476334 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.481225 18021 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:27.400446 26675 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:25:27.485941 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:27.485308 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:27.490782 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 293228us
W20260709 12:25:27.491046 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.476266 26611 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:27.476043 26510 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:27.475149 26773 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:27.497392 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.475543 26770 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.023s
I20260709 12:25:27.504969 26770 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:27.514345 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 345486us
I20260709 12:25:27.515373 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:27.486778 26650 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:27.498483 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:27.516798 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.498947 26510 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.520385 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.496644 26520 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:27.523574 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.497164 26715 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:27.524012 18094 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:27.524493 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:27.522825 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44766.7us
I20260709 12:25:27.523324 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.528908 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.536499 18090 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:27.537266 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.497804 26497 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:25:27.519349 26650 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:27.540200 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63942us
W20260709 12:25:27.554630 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69452.4us
I20260709 12:25:27.540987 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.537673 18098 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:25:27.562590 18172 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:27.563256 18172 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:27.564739 18170 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:25:27.534636 26789 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:25:27.556852 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:27.514488 26773 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.527034 26510 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.577221 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:27.588918 26791 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:27.589557 26791 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:27.589993 26791 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:27.534253 26611 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:27.581617 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.594151 26611 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:27.594676 26611 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:27.595077 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:27.596140 26403 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:27.571743 18172 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 24.20, Last appended by leader: 20, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.597607 26403 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.598343 18172 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:27.598397 26403 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.597513 26574 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.600970 26770 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:27.605605 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.566226 26650 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:27.568020 26731 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:27.601263 26602 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.619956 18169 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:27.630815 26479 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:27.631562 26479 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:27.646440 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48666.6us
I20260709 12:25:27.649137 26581 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.653506 18019 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:27.654093 26765 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:27.654740 26793 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:27.657290 26796 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:27.598681 18064 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.787s (SENT) [suppressed 26 similar messages]
W20260709 12:25:27.703863 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 56328.8us
W20260709 12:25:27.704632 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 167479us
I20260709 12:25:27.705538 18138 leader_election.cc:400] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22.
I20260709 12:25:27.705884 18138 leader_election.cc:403] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:27.695459 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 171100us
I20260709 12:25:27.697482 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.602288 26786 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:27.698606 18017 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:27.586869 26759 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:27.716765 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:27.701804 18168 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:25:27.587958 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.573772 26520 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.096s
W20260709 12:25:27.670440 26679 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.781s	user 0.012s	sys 0.003s
I20260709 12:25:27.632882 26479 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.712942 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:27.700158 26574 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.732167 18019 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:25:27.732419 26542 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.703293 26793 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.701258 26657 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.712533 26497 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:25:27.669855 26770 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:27.721602 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.722172 26663 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22.
I20260709 12:25:27.724000 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.725072 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.733772 18019 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:27.735302 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.735669 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.736050 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.738204 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.748831 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.772500 18096 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.738693 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:27.747720 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:27.742781 18020 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:27.774641 18099 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:27.772691 18090 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:27.753914 18092 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:27.782630 18017 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:27.798465 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 81756.2us
I20260709 12:25:27.810720 18090 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:27.803870 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:27.802388 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 203735us
I20260709 12:25:27.818599 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.819491 18168 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:27.737694 26497 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:27.719439 26796 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:27.816938 18094 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:27.730716 26707 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:27.781195 26666 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.549s	user 0.022s	sys 0.000s
I20260709 12:25:27.744771 26613 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22.
I20260709 12:25:27.824673 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:27.734809 26773 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:27.867733 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.826918 18172 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 25 index: 21. (index mismatch)
I20260709 12:25:27.776719 26602 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:27.867444 18169 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:27.793289 26520 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:25:27.868403 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.735303 26793 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.838841 18167 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:27.828433 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89838.7us
I20260709 12:25:27.885144 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:27.875773 18091 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.735720 26657 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:27.820720 26776 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:27.830345 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.814232 26663 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:27.823748 26547 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.813292 26542 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:27.877632 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0647461479998128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24910000000000002s for a response
W20260709 12:25:27.914290 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.895769 18173 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:27.897097 18091 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:27.892583 26714 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.674s	user 0.003s	sys 0.018s
I20260709 12:25:27.811728 26443 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:25:27.900532 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:27.902951 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.903165 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.892122 26520 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:27.825629 26497 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:27.888887 26403 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:27.912752 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:27.898986 18099 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.925041 18094 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:27.825344 26675 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.073s
I20260709 12:25:27.924090 18090 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:25:27.746735 26770 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.831485 26759 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 12:25:27.802317 26803 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.108s
I20260709 12:25:27.905803 26663 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.945639 18092 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:27.945771 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:27.951983 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.954401 18093 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:27.955034 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 153132us
I20260709 12:25:27.879065 26666 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.964032 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.964077 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:27.897872 26776 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:27.962566 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.985448 18092 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:27.986972 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.968704 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:27.973985 26720 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:25:27.989219 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:27.968701 26497 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:27.997213 26497 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:27.947228 26602 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:28.000676 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:27.962505 26663 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.931699 26679 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:25:27.965332 26678 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.010144 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.010797 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3472) took 1507 ms. Trace:
W20260709 12:25:27.995532 26638 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.743s	user 0.010s	sys 0.010s
I20260709 12:25:28.013521 18091 rpcz_store.cc:276] 0709 12:25:26.503767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.886490 (+382723us) service_pool.cc:224] Handling call
0709 12:25:26.886879 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.875153 (+988274us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b54003555a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1663506f 00007f6c166adf60 00007f6c166af7de 00007f6c166b9b49 00007f6c166ba416 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62
0709 12:25:27.961149 (+ 85996us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.961185 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.961228 (+    43us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:27.961258 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.961333 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.010720 (+ 49387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":427,"raft.queue_time_us":938,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":136,"spinlock_wait_cycles":64720256}
I20260709 12:25:27.923987 26547 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:28.021587 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 92695.2us
I20260709 12:25:28.037671 18091 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:27.990049 26776 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:28.077859 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.088939 26799 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:25:28.062436 26403 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:28.095415 26666 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.098644 26666 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:28.095635 26707 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:28.038600 26638 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.020397 26796 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:28.116467 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.132436 26638 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.139786 26515 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.138190 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1280) took 1123 ms. Trace:
W20260709 12:25:28.134184 26800 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:25:28.174194 18020 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:25:28.174783 18096 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:25:28.146436 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1306) took 1048 ms. Trace:
I20260709 12:25:28.147934 26574 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:28.151983 26611 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.004s
I20260709 12:25:28.019812 26678 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:28.158314 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.169425 18021 rpcz_store.cc:276] 0709 12:25:27.019458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.455393 (+435935us) service_pool.cc:224] Handling call
0709 12:25:27.455851 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.481833 (+ 25982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.137695 (+655862us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:28.138072 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57188736}
I20260709 12:25:28.175798 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:28.176518 18094 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:28.175379 18165 rpcz_store.cc:276] 0709 12:25:27.097580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.132520 (+ 34940us) service_pool.cc:224] Handling call
0709 12:25:27.280054 (+147534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.282351 (+  2297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.146088 (+863737us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540002a3d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:28.146333 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285185280}
I20260709 12:25:28.093212 26547 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.198839 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:28.204190 26627 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.205045 26627 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:28.207405 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4275) took 1184 ms. Trace:
I20260709 12:25:28.207947 18018 rpcz_store.cc:276] 0709 12:25:27.022491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.554898 (+532407us) service_pool.cc:224] Handling call
0709 12:25:27.575705 (+ 20807us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.599214 (+ 23509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.599253 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.599919 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.714664 (+114745us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:27.714966 (+   302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.715062 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.207070 (+492008us) raft_consensus.cc:1685] finished
0709 12:25:28.207101 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.207309 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58867,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"thread_start_us":113388,"threads_started":1,"wal-append.queue_time_us":326659}
I20260709 12:25:28.199836 26542 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.214628 26542 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.198386 26611 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.221339 18020 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:28.226845 18092 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
W20260709 12:25:28.190819 26808 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:25:28.123494 26773 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:28.234426 26650 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.011s
I20260709 12:25:28.247319 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:28.252215 18021 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:28.189471 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:28.254792 18019 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:28.256078 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66464.4us
I20260709 12:25:28.261318 18165 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:25:28.265254 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 326338us
W20260709 12:25:28.267129 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 354455us
I20260709 12:25:28.076967 26520 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.310763 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:28.198872 26731 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:25:28.315202 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:28.317214 18096 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:28.311691 18165 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:28.319619 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:28.320674 18169 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:28.259721 26714 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:28.325877 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:28.330680 18091 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:28.279407 26638 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.342885 26663 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.344444 26687 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:28.345211 26803 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:28.344631 26602 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:28.254520 26773 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:28.365131 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 306145us
I20260709 12:25:28.254281 26650 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:28.363559 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43817.6us
I20260709 12:25:28.366299 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.366458 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:28.317713 26611 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:28.388381 18168 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:28.329810 26714 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:28.313482 26715 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 12:25:28.403646 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3170) took 1395 ms. Trace:
I20260709 12:25:28.356307 26657 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.356882 26542 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.411708 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1275) took 2034 ms. Trace:
I20260709 12:25:28.414018 18015 rpcz_store.cc:276] 0709 12:25:26.376839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.506593 (+129754us) service_pool.cc:224] Handling call
0709 12:25:26.534280 (+ 27687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.058269 (+523989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.411254 (+1352985us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b54001095d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:28.411609 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1203145472}
I20260709 12:25:28.363935 26678 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:28.403815 26801 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.387s	user 0.002s	sys 0.000s
I20260709 12:25:28.423015 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1277) took 1409 ms. Trace:
I20260709 12:25:28.406126 26796 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.404546 18022 rpcz_store.cc:276] 0709 12:25:27.009625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.018259 (+  8634us) service_pool.cc:224] Handling call
0709 12:25:27.018728 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.045739 (+ 27011us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.045777 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.641071 (+595294us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b18002a86e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:25:27.641501 (+   430us) spinlock_profiling.cc:243] Waited 7.55 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:27.713985 (+ 72484us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.916665 (+202680us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:27.916708 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.916795 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.403322 (+486527us) raft_consensus.cc:1685] finished
0709 12:25:28.403353 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.403547 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10153,"raft.queue_time_us":10407,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":18145,"spinlock_wait_cycles":19961600,"thread_start_us":95488,"threads_started":1,"wal-append.queue_time_us":217827}
I20260709 12:25:28.422798 26773 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.359040 26638 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.405282 26714 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:28.411152 26574 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:28.433183 18023 rpcz_store.cc:276] 0709 12:25:27.014100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.372435 (+358335us) service_pool.cc:224] Handling call
0709 12:25:27.372847 (+   412us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:27.373342 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.373382 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:28.239020 (+865638us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b2c00057268. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:28.239828 (+   808us) log.cc:844] Serialized 52 byte log entry
0709 12:25:28.344052 (+104224us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:28.344113 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.344226 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.422637 (+ 78411us) raft_consensus.cc:1685] finished
0709 12:25:28.422665 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.422923 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1238,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":47947008,"thread_start_us":103689,"threads_started":1,"wal-append.queue_time_us":169956}
I20260709 12:25:28.364933 26798 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260709 12:25:28.362965 26803 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:28.437618 18099 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.504s	user 0.013s	sys 0.000s
I20260709 12:25:28.448201 18138 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23.
I20260709 12:25:28.398458 26499 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:25:28.455808 18099 raft_consensus.cc:2410] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:28.455948 18138 leader_election.cc:403] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:28.433071 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128419us
I20260709 12:25:28.449218 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:28.476780 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:28.436568 26513 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.479542 18095 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:28.483215 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:28.494391 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 302312us
I20260709 12:25:28.447554 26773 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.506448 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:28.511965 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:28.453605 26675 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.515225 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:28.517357 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:28.518111 18813 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:28.521526 26542 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:28.522145 26542 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:28.522300 26520 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.530035 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.529928 18813 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
W20260709 12:25:28.531368 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.532771 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.533296 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.539193 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:28.542358 26707 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:28.542865 26707 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:28.543123 18168 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:28.543793 26707 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.544411 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:28.544921 26707 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:28.545287 18168 raft_consensus.cc:2410] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:28.550980 18168 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:28.555064 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3474) took 1074 ms. Trace:
I20260709 12:25:28.555613 18093 rpcz_store.cc:276] 0709 12:25:27.481015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.481180 (+   165us) service_pool.cc:224] Handling call
0709 12:25:28.554821 (+1073641us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b540040c4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:28.554954 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":887515392}
I20260709 12:25:28.530304 26520 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.513360 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.560135 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:28.482900 26794 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.560127 18168 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:28.483837 26759 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:25:28.582182 26807 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.688s	user 0.000s	sys 0.009s
I20260709 12:25:28.508989 26602 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.563126 26675 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.587096 26759 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:28.588773 26759 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:28.505200 26513 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.587805 26794 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.445941 26796 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.506321 26713 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:28.508818 26765 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:28.626789 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122029us
W20260709 12:25:28.621363 26653 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.006s	sys 0.012s
W20260709 12:25:28.618084 26581 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.952s	user 0.016s	sys 0.002s
I20260709 12:25:28.635473 26713 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.647910 26814 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
W20260709 12:25:28.649894 26823 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:25:28.651212 26812 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:25:28.652102 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:28.654099 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.657007 26720 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:28.657557 26720 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:28.658051 26720 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:28.659940 26513 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:28.660082 26638 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.660840 26638 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:28.662353 26841 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:28.663861 26841 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:28.664258 26841 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:28.664291 26837 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:28.666043 26713 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.666644 26611 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:28.667124 26611 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:28.667569 26611 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:28.617273 26826 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:28.670080 26843 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.670682 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.670799 26843 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.671332 26786 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23.
W20260709 12:25:28.678857 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 312443us
W20260709 12:25:28.681643 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 233618us
I20260709 12:25:28.683008 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:28.558732 26773 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:28.636729 26653 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.007s	sys 0.013s
I20260709 12:25:28.680878 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:28.667142 26797 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:28.690824 26806 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.691088 26797 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.691639 26806 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.691802 26797 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.696995 26773 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.698823 26803 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.699800 26831 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:28.700327 26796 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.701664 26839 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.702382 26839 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.706218 26650 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.698865 26713 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:28.685526 26789 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:25:28.674552 26765 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:25:28.724087 26793 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.826s	user 0.013s	sys 0.003s
I20260709 12:25:28.724306 26765 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:28.725626 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.648002 26547 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.643486 26663 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:28.736068 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55276.2us
W20260709 12:25:28.632575 26784 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.148s	user 0.021s	sys 0.002s
I20260709 12:25:28.741868 18093 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:28.729053 26510 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.120s	user 0.008s	sys 0.008s
I20260709 12:25:28.744663 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:28.746310 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63420.7us
I20260709 12:25:28.705139 26839 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.748309 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:28.724977 26765 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:28.736547 26790 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:28.665417 26841 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.770300 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:28.796536 18090 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:28.772521 18099 raft_consensus.cc:2368] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:28.758064 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:28.766083 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:28.755889 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:28.795185 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43588.9us
I20260709 12:25:28.815984 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:28.818354 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.816839 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 304926us
W20260709 12:25:28.816100 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 280977us
I20260709 12:25:28.822155 18814 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:28.748577 26810 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:28.748723 26786 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:28.753623 26784 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.278s	user 0.022s	sys 0.003s
W20260709 12:25:28.771435 26627 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.566s	user 0.012s	sys 0.006s
I20260709 12:25:28.842621 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3380) took 1453 ms. Trace:
I20260709 12:25:28.844812 18173 rpcz_store.cc:276] 0709 12:25:27.389400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.389642 (+   242us) service_pool.cc:224] Handling call
0709 12:25:27.406684 (+ 17042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.921836 (+515152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.842291 (+920455us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b5400030a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:28.842541 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119214976}
I20260709 12:25:28.824589 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:28.854437 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:28.841792 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47054.6us
W20260709 12:25:28.856729 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.858335 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:28.859730 18170 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 26 because replica has last-logged OpId of term: 24 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 19.
I20260709 12:25:28.822798 26798 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:25:28.862113 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106353us
I20260709 12:25:28.864219 18806 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:28.866437 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:28.858186 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:28.724936 26793 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.749598 26547 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.825920 26513 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.758492 26806 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:28.860785 26403 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.885412 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:28.822568 26581 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:28.896518 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 152002us
W20260709 12:25:28.867691 26791 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.277s	user 0.017s	sys 0.000s
I20260709 12:25:28.897866 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.846007 26810 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.882946 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:28.823849 26794 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.826968 26790 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:28.947438 26679 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:28.947677 26790 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.894327 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:28.890767 18090 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:28.948410 26679 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:28.947124 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46128.4us
I20260709 12:25:28.902426 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:28.950080 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64787.1us
I20260709 12:25:28.950356 18018 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:28.951125 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.902904 18806 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:28.844485 26784 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:28.931233 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.775771 26510 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.950454 18023 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:28.953155 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:28.944343 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:28.954607 18169 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:28.954988 18165 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:28.944869 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:28.955969 18022 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:28.774341 26830 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.134s
W20260709 12:25:28.889910 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 327005us
W20260709 12:25:28.959450 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66611us
W20260709 12:25:28.963600 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69367.7us
I20260709 12:25:28.961217 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:28.894991 26574 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:28.822690 26847 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.155s
W20260709 12:25:28.965967 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3258) took 2519 ms (client timeout 1747 ms). Trace:
I20260709 12:25:28.967581 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.979238 26821 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.980082 26821 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.981031 26613 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:28.981539 26613 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:28.982046 26835 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:25:28.982544 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:28.982755 26613 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.951164 26675 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.944934 26806 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:25:28.984408 26613 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:28.984892 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:28.991179 26828 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:25:28.991778 26520 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.863574 26839 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.083s
I20260709 12:25:28.950867 26714 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.862236 26627 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.000149 26627 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.976397 26574 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.001147 26479 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:28.896798 26797 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:28.895056 26811 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:29.004423 26811 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:29.003422 26663 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:29.005584 26811 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.006369 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:28.895434 26653 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.007341 26653 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.008303 26784 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:28.944506 26791 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.011086 18168 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:28.892563 26843 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:28.985041 26806 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:29.001313 26574 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:28.974949 18166 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:28.991045 26863 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:28.976965 18167 rpcz_store.cc:269] 0709 12:25:26.446858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.533149 (+ 86291us) service_pool.cc:224] Handling call
0709 12:25:28.965593 (+2432444us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:28.965762 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2694233344}
I20260709 12:25:28.955600 18019 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 14.
I20260709 12:25:28.868793 26786 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.944722 26810 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:28.952732 26679 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:29.082973 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 118175us
W20260709 12:25:29.085244 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 118885us
I20260709 12:25:29.107321 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.090041 18813 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:29.009256 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:28.966481 26602 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:29.090557 18810 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:28.974124 26674 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:28.973554 26798 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:25:29.100600 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 202901us
I20260709 12:25:29.145491 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:28.943936 26403 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:28.905536 26513 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.089454 18165 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
W20260709 12:25:29.113946 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105091us
I20260709 12:25:29.093604 26863 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.089046 26653 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.090808 26806 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:29.119308 26663 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:29.124162 26642 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:29.124598 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.124963 17913 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 13 to 17. New cstate: current_term: 17 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:29.127040 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3491) took 1163 ms. Trace:
I20260709 12:25:29.127188 26821 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.094408 26826 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:25:29.092042 26794 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
W20260709 12:25:29.132370 26497 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.135s	user 0.015s	sys 0.003s
I20260709 12:25:29.094005 26675 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:25:29.135167 26714 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.134708 26720 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.136623 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:29.136864 26510 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.113281 26547 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:28.972378 26820 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.254s
W20260709 12:25:29.139532 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.143083 18136 leader_election.cc:400] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:29.222069 18813 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:25:29.223374 18812 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:29.148187 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.224659 18090 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:29.090354 26581 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.225775 26581 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.228255 26581 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:29.138525 26707 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.571s	user 0.018s	sys 0.002s
W20260709 12:25:29.138940 26776 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.053s	user 0.012s	sys 0.005s
I20260709 12:25:29.222072 18136 leader_election.cc:403] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:25:29.229341 26776 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.230942 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:29.231806 18813 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:29.136680 26847 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:29.232744 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:29.232772 26776 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.233553 18812 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:29.180482 18092 rpcz_store.cc:276] 0709 12:25:27.963418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.986497 (+ 23079us) service_pool.cc:224] Handling call
0709 12:25:29.126779 (+1140282us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:29.126959 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":535143296}
W20260709 12:25:29.213864 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77328.9us
I20260709 12:25:29.235486 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.174952 26863 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:29.240613 18810 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:29.198751 26663 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:29.164605 26611 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:29.250999 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111422us
I20260709 12:25:29.201001 26819 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.207991 26602 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.250229 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.255111 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:29.253850 18136 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:29.212852 26817 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.223191 26831 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:29.238025 26516 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:25:29.259021 18811 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:29.219741 26835 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:25:29.195535 26642 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.264258 26602 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.273350 26817 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.256161 26806 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.277392 26847 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:29.204401 26798 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:25:29.287959 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:29.290508 18171 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:29.239729 26803 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:29.256879 26820 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:25:29.295404 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.269140 26819 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.380785 26845 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.344108 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3389) took 1095 ms. Trace:
I20260709 12:25:29.375699 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1355) took 1435 ms. Trace:
I20260709 12:25:29.389496 26817 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:29.353571 26759 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.737s	user 0.017s	sys 0.000s
I20260709 12:25:29.286640 26581 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 12:25:29.213696 26765 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.302408 26764 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.520931 26764 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.263650 26707 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:29.264889 26794 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:29.263002 26793 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.357232 26611 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.374962 26786 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.380684 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3270) took 1820 ms. Trace:
I20260709 12:25:29.499629 26770 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:29.537050 18166 rpcz_store.cc:276] 0709 12:25:27.560629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.560821 (+   192us) service_pool.cc:224] Handling call
0709 12:25:28.965812 (+1404991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.091242 (+125430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.380279 (+289037us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54000325d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:29.380549 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3190789376}
I20260709 12:25:29.262297 26776 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:29.358891 26663 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:29.479249 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 218410us
W20260709 12:25:29.482815 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.314389 26811 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:29.495857 18095 rpcz_store.cc:276] 0709 12:25:27.939896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.998322 (+ 58426us) service_pool.cc:224] Handling call
0709 12:25:28.479250 (+480928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.482350 (+  3100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.375514 (+893164us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54002963d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:29.375633 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1104942976}
I20260709 12:25:29.495784 18172 rpcz_store.cc:276] 0709 12:25:28.248687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.248821 (+   134us) service_pool.cc:224] Handling call
0709 12:25:28.249169 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:28.671544 (+422375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:28.671587 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:28.672248 (+   661us) log.cc:844] Serialized 26 byte log entry
0709 12:25:28.886049 (+213801us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:28.886126 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.006597 (+120471us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:29.006657 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.343705 (+337048us) raft_consensus.cc:1685] finished
0709 12:25:29.343734 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:29.344016 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":143965,"raft.queue_time_us":144270,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":638944512,"thread_start_us":211901,"threads_started":1,"wal-append.queue_time_us":574775}
I20260709 12:25:29.373588 26791 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.283857 26821 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.198s
I20260709 12:25:29.502552 18024 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
W20260709 12:25:29.503760 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 358329us
I20260709 12:25:29.502099 26845 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.510646 26765 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.519908 26806 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:29.505260 26581 consensus_queue.cc:1243] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:25:29.529273 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 241464us
I20260709 12:25:29.563918 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.540005 26776 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.484862 26820 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:29.585608 26657 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:29.586608 26657 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:29.588048 26657 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.590083 26811 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:29.590598 26403 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.591691 26811 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.594771 26764 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.596330 26720 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.489528 26803 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.532881 26794 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:29.566651 26869 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.605189 26784 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:29.606645 26821 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:29.607160 26821 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:29.488765 26831 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:25:29.565503 26542 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.607486 26803 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.607858 26831 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:29.608217 26542 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:29.608412 26831 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:29.578436 26877 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.609433 26791 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.609388 26877 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.610178 26791 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.612263 26877 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.612577 26791 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.613267 26791 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:29.573935 26776 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:29.505795 26679 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:29.576808 26868 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:25:29.616680 26679 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.557909 26611 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:29.605189 26794 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.617923 26611 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.618889 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.619163 26794 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:29.620316 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.578433 26817 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:29.605098 26869 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.623804 26869 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:29.625860 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.628253 26845 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.608155 26821 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.631489 26821 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:29.570535 26793 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:29.633486 26403 consensus_queue.cc:1175] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "768723648a9444fda53659e6a8cc3091" responder_term: 20 status { last_received { term: 20 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 20 index: 12 } }
I20260709 12:25:29.635005 26831 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.637627 26831 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.527765 26759 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.602875 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:29.644348 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.016s	sys 0.019s
W20260709 12:25:29.644990 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.016s	sys 0.019s
I20260709 12:25:29.644958 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.645385 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:29.645658 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 501697us
I20260709 12:25:29.647285 26764 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:29.650166 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.672237 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:29.646499 26835 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:29.654798 26759 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:29.679543 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.658224 26657 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:29.620316 26611 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.673316 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:29.675464 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:29.676299 26831 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:29.676669 26674 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:29.676754 26826 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.678452 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:29.681253 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.668567 26866 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:29.661738 26602 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.651300 26720 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:29.644333 26791 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.707075 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.681113 26873 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:29.689024 26817 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:29.715915 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:29.718248 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.719408 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:29.721724 18097 raft_consensus.cc:2376] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22. Re-sending same reply.
I20260709 12:25:29.722820 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.730967 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.723572 18016 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:29.723927 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.725314 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:29.716907 26613 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.732s	user 0.014s	sys 0.005s
I20260709 12:25:29.733688 18094 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:29.736822 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:29.734201 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.737768 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:29.739039 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:29.740180 18091 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:29.740512 26791 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.733094 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.743175 18093 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:25:29.743364 26513 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.744014 26513 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:29.743644 26863 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.744264 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:29.744551 26513 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.745206 18093 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 19: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:29.746279 26866 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.747004 26866 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.736891 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.725239 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:29.720204 26826 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:29.708704 26869 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.741360 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:29.707054 26574 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.603s	user 0.014s	sys 0.000s
I20260709 12:25:29.758783 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:29.709910 26759 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:29.782858 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.715953 26611 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.735797 18020 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:29.754038 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75677.2us
I20260709 12:25:29.766418 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:29.758934 18099 raft_consensus.cc:2376] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25. Re-sending same reply.
I20260709 12:25:29.775916 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.787142 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.774628 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.789073 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:29.763005 26653 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:29.755363 18018 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:29.762261 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.799384 18021 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:29.784346 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:29.716132 26707 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:29.780165 18091 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 15.
I20260709 12:25:29.790169 18172 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
W20260709 12:25:29.765269 26810 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.652s	user 0.009s	sys 0.006s
I20260709 12:25:29.761672 26826 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.754251 26516 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:29.790552 26611 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.793262 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.809114 18015 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:29.810374 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.810604 26611 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.811034 26863 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.811626 26810 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.812151 26720 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:29.812295 26877 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:29.815658 26810 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.817502 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:29.818487 18021 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:29.820286 26542 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:29.820768 26542 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:29.801772 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:29.762368 26869 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.823050 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.824198 26869 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.824398 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.824748 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:29.825338 26516 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:29.755597 26674 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:25:29.825743 26516 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:29.826151 26793 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:29.826256 26516 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:29.826608 26674 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:29.827018 26674 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:29.732131 26817 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.827947 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.828915 18096 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:29.829437 26817 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:29.786293 26602 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:29.830394 26602 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.830977 26542 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.832455 26798 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:29.832908 26602 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:29.834398 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:29.825634 18094 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:29.813524 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.765090 26819 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:29.794500 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.836275 18093 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:29.834004 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:29.785562 26574 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:29.811775 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:29.836844 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:29.837174 18810 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:29.835614 18018 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:29.749779 26866 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:29.847750 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 115840us
I20260709 12:25:29.864162 18021 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:25:29.882790 18016 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:29.878513 18020 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:29.904215 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:29.941159 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115427us
I20260709 12:25:29.941846 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.798300 26479 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.978281 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:29.980876 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.978247 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98489.3us
I20260709 12:25:30.007141 18016 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:29.943771 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:30.009960 18809 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 15: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:29.798908 26653 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:29.833261 26798 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.019826 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:29.989929 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:29.992621 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.022367 18091 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
W20260709 12:25:29.943636 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 68867.9us
I20260709 12:25:30.009024 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.026695 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:29.881119 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.028128 18171 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:29.836336 26873 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:30.031663 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.033254 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.022565 18018 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:29.944875 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.040403 18172 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:29.983368 26542 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:30.051687 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.030191 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.066812 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.061020 18164 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:30.053419 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.032492 18815 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 15.
I20260709 12:25:30.059329 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.079838 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.089709 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.090480 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.090814 18167 raft_consensus.cc:2368] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:25:30.017658 26770 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:30.101476 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:30.103694 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.828276 26674 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.054510 26638 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.115005 18165 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:30.114815 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.050715 26876 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.054234 26547 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:30.061573 26720 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:29.983083 26765 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.092s
I20260709 12:25:30.009034 26714 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 12:25:30.009200 26866 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.072s
I20260709 12:25:30.127264 18169 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:30.018409 26819 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260709 12:25:30.092133 26657 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:30.148183 18168 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:30.148595 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:30.082254 26784 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:30.150787 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.057350 26794 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.073s
I20260709 12:25:30.149422 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.115238 26770 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:30.129307 26776 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.515s	user 0.012s	sys 0.001s
W20260709 12:25:30.194656 26879 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:30.158552 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:30.138952 26806 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.565s	user 0.013s	sys 0.002s
I20260709 12:25:30.082928 26613 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:25:30.200946 26613 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:25:30.201161 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:30.201802 26613 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:30.202643 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:30.154261 26657 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.138521 26876 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.203447 26657 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:30.205266 26876 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:30.165036 26574 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:30.165403 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.207844 18168 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:30.130165 26853 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:25:30.215982 18169 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
W20260709 12:25:30.176252 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.177556 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.178247 26873 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:30.128793 26653 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.182600 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.183105 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1580) took 1202 ms. Trace:
W20260709 12:25:30.223366 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45940.2us
I20260709 12:25:30.185446 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.192683 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.225504 18096 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:30.194255 26776 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.073848 26826 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.071s
I20260709 12:25:30.210803 18017 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:30.171963 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.234777 18167 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:30.217016 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:30.229053 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 162792us
I20260709 12:25:30.157552 26720 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:30.219246 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.208839 26817 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.223583 18808 rpcz_store.cc:276] 0709 12:25:28.980886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.983950 (+  3064us) service_pool.cc:224] Handling call
0709 12:25:28.984365 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.208833 (+224468us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:29.208894 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.218089 (+  9195us) log.cc:844] Serialized 26 byte log entry
0709 12:25:29.534199 (+316110us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:29.534905 (+   706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.182574 (+647669us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400781754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:30.182638 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.182693 (+    55us) raft_consensus.cc:1685] finished
0709 12:25:30.182709 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.182994 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4099,"raft.queue_time_us":4382,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":18552,"spinlock_wait_cycles":485920512,"thread_start_us":265449,"threads_started":1,"wal-append.queue_time_us":349710}
W20260709 12:25:30.236094 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 169360us
I20260709 12:25:30.217448 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.215065 26547 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:25:30.208489 26784 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:30.246100 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.250550 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.219808 26674 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:30.253147 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.237802 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.272655 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.274344 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.274739 18023 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:30.226224 26513 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:30.277014 18020 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [20.12-23.13]
I20260709 12:25:30.276734 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.289327 18168 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:30.239035 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:30.279961 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.240597 26574 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:30.292961 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.302785 18090 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:30.218838 26819 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:25:30.301542 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.250227 26786 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.275184 26653 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:30.318197 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.319183 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.319890 18096 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:30.276744 26793 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:30.321004 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:30.293902 26707 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:30.291829 26547 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:30.308866 26817 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:30.352370 26817 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.352957 26817 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:25:30.312395 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 73490.2us
I20260709 12:25:30.226593 26873 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.328907 18167 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:30.362534 26873 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:30.363346 18094 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:30.308040 26574 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.363817 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.364787 26574 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:30.326004 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.371721 26796 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:30.372411 26796 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:30.372212 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51329.2us
I20260709 12:25:30.373203 26796 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:30.365526 18167 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:30.385594 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.387161 18169 raft_consensus.cc:2410] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 11.
I20260709 12:25:30.390810 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.391667 18090 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
W20260709 12:25:30.396212 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70389.8us
W20260709 12:25:30.399832 26887 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:25:30.400308 26810 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:30.400836 26810 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.401224 26810 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:30.327382 26714 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:30.323539 26890 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.341926 26831 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:30.350073 26707 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.417235 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:30.435240 18021 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:30.415784 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.373986 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.438225 18022 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:30.420928 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 221751us
I20260709 12:25:30.431329 18169 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 12:25:30.327791 26786 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.396322 26873 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:30.425690 18096 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:30.432866 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:30.431903 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 224857us
I20260709 12:25:30.446940 18015 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:30.447154 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.410252 26714 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:30.448151 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74368.5us
I20260709 12:25:30.453408 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.453588 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.458000 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.459981 18808 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:30.435176 26831 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.440491 26707 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:30.476012 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:30.467060 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.485623 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.464112 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.487248 18096 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:30.446659 26663 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:30.450922 26873 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:30.458012 26714 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:30.456537 26793 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:30.497572 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.462230 26791 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:30.506800 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.508647 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.495337 26663 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.478747 26891 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.502525 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.478891 26831 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:30.409597 26720 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.519974 26663 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:30.504750 26889 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:30.413743 26784 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.513736 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.535844 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.547326 18015 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:30.504726 26793 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.494228 26547 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.561332 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.557276 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.523541 26791 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:30.602057 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.574134 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.523940 26776 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:30.603303 18096 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:30.583096 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 107199us
W20260709 12:25:30.566511 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.603359 26878 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:30.605697 18064 leader_election.cc:400] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:25:30.605888 26878 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.607928 26819 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.608453 26878 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:30.608655 26900 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:30.605041 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.610572 26776 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:30.611032 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.611234 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.611780 18165 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:30.612408 26776 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:30.613173 18165 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:30.615373 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.616677 18094 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:30.618194 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1287) took 1764 ms. Trace:
I20260709 12:25:30.608521 26819 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.618722 18024 rpcz_store.cc:276] 0709 12:25:28.853980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.854180 (+   200us) service_pool.cc:224] Handling call
0709 12:25:30.617985 (+1763805us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:30.618122 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1488746368}
I20260709 12:25:30.622426 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.623237 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.623862 26513 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:30.624327 26513 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:30.624894 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.624918 26513 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:30.626571 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.627260 18097 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:30.630271 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.625734 18024 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:30.632252 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.632962 18024 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:30.633267 18023 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:30.638339 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.638922 18096 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:30.643513 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.624698 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.645563 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.645761 18018 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:30.648295 26806 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:30.649233 26806 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:30.650712 26806 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.657243 26819 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:30.647326 26764 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.660426 26764 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.662887 26764 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:30.663430 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.665279 18165 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:30.666180 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.666939 18811 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:30.647542 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.545491 26889 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:30.565218 26793 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:30.524428 26574 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.657723 18015 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:30.647147 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.693985 18096 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:30.701205 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.688621 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.689007 18090 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:30.691882 18097 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
W20260709 12:25:30.703418 26898 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:25:30.716025 18097 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 12:25:30.716478 18094 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:30.702615 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:30.689739 26443 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.20, Next index: 21, Last known committed idx: 17, Time since last communication: 0.014s
I20260709 12:25:30.704458 26889 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.722347 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.713140 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:30.706049 26892 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
W20260709 12:25:30.725209 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.741577 18097 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
W20260709 12:25:30.719358 26882 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.378s	user 0.002s	sys 0.000s
I20260709 12:25:30.711506 26403 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:30.748636 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.757030 18808 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:30.749085 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:30.750003 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.759529 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.751138 26837 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:30.752398 26542 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.680s	user 0.005s	sys 0.010s
I20260709 12:25:30.746006 26806 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:25:30.723109 26821 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.091s	user 0.008s	sys 0.010s
I20260709 12:25:30.775283 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.784983 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.781268 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.785902 18167 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:30.778115 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:30.786583 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:30.787453 18017 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:30.788309 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:30.786124 18814 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:30.793627 18090 pending_rounds.cc:85] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:25:30.794094 18090 pending_rounds.cc:107] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:25:30.794425 18090 raft_consensus.cc:2889] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:30.799278 18808 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:25:30.800977 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 192860us
I20260709 12:25:30.738468 26905 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.782691 26497 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.781912 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.802230 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:30.782049 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.806674 18022 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:30.807945 18096 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:25:30.808146 18808 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 12:25:30.783006 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:30.789100 26898 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:25:30.790812 26893 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.273s	user 0.001s	sys 0.002s
I20260709 12:25:30.841044 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:30.806674 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:30.784514 26786 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:30.846798 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:30.784157 26806 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:30.853296 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.792683 26890 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:30.770756 26764 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.067s
I20260709 12:25:30.807257 26759 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:30.779204 26403 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.839797 26679 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:30.876277 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:30.877528 18815 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:30.783958 26837 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.856254 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:30.863926 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.866564 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:30.872961 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:30.790484 26819 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:25:30.879469 18808 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:30.879707 18022 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:30.746943 26791 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.850346 26821 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:30.884593 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.876266 26679 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.906061 26679 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:30.907385 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:30.908442 18021 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:30.888688 26896 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
W20260709 12:25:30.887953 26911 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:25:30.890990 26821 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:30.935539 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.937307 18018 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:30.937767 18815 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:25:30.938776 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:30.938956 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:30.907645 26403 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:30.901351 26841 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:30.914533 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52724.4us
I20260709 12:25:30.870545 26888 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:30.909693 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.826879 26497 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.891559 18165 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:30.990962 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.992000 18165 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:30.993079 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:30.854344 26878 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
W20260709 12:25:30.889443 26903 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:25:30.935667 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.860028 26811 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:25:30.884564 26890 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:30.942345 26894 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.388s	user 0.002s	sys 0.000s
I20260709 12:25:30.971837 26574 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:30.972293 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:31.025689 18173 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:30.962172 26806 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:30.965772 26516 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.139s	user 0.014s	sys 0.002s
I20260709 12:25:30.975142 26837 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:30.965772 26657 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.762s	user 0.010s	sys 0.001s
I20260709 12:25:30.978247 18022 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:30.968241 26513 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.979547 26707 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:30.911320 26891 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:30.980330 26679 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.940382 26821 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:31.039067 26516 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:30.982043 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3398) took 1177 ms. Trace:
W20260709 12:25:30.983047 26770 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.792s	user 0.014s	sys 0.000s
I20260709 12:25:31.058368 26770 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:30.991135 26653 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.666s	user 0.012s	sys 0.002s
I20260709 12:25:31.062371 26657 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:31.064041 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.994350 18165 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:30.995165 18021 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:31.006356 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:30.939767 18097 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:30.941370 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:30.943255 26901 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:31.617657 18019 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
W20260709 12:25:30.959141 26902 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:25:31.041575 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:31.001107 26888 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:31.033317 26714 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.003s
I20260709 12:25:31.058089 18166 rpcz_store.cc:276] 0709 12:25:29.804329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.805369 (+  1040us) service_pool.cc:224] Handling call
0709 12:25:29.805741 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.814958 (+  9217us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:30.081892 (+266934us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.402490 (+320598us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.403763 (+  1273us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.404090 (+   327us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:30.404133 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.404208 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.981737 (+577529us) raft_consensus.cc:1685] finished
0709 12:25:30.981764 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.981947 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38067,"prepare.queue_time_us":435905,"prepare.run_cpu_time_us":2673,"prepare.run_wall_time_us":12240,"raft.queue_time_us":27875,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":4700,"thread_start_us":496303,"threads_started":2,"wal-append.queue_time_us":356850}
W20260709 12:25:31.529395 26841 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.002s
I20260709 12:25:31.678382 18099 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:31.531801 18168 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
W20260709 12:25:31.524840 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.679445 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3414) took 1480 ms. Trace:
I20260709 12:25:31.025472 26900 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:25:31.273741 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 338182us
I20260709 12:25:31.687835 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:31.026441 26663 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.504s	user 0.009s	sys 0.005s
W20260709 12:25:31.689729 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3194) took 1262 ms (client timeout 1672 ms). Trace:
W20260709 12:25:31.522495 26817 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.169s	user 0.031s	sys 0.016s
I20260709 12:25:31.092602 26837 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:31.074154 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:31.538384 26707 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.716602 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3404) took 1678 ms. Trace:
I20260709 12:25:31.077296 26820 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.743355 26820 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:31.561002 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:31.548839 18813 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.333s	user 0.023s	sys 0.001s
W20260709 12:25:31.746255 18813 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.531s	user 0.027s	sys 0.004s
I20260709 12:25:31.746999 26497 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.747531 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1744) took 1914 ms. Trace:
I20260709 12:25:31.706723 26837 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:31.748381 26837 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:31.567233 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:31.002568 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:31.752409 18017 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:31.606129 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3191) took 1179 ms. Trace:
I20260709 12:25:31.754069 18017 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:31.597472 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:31.706118 26663 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.754954 18173 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:31.755506 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3422) took 1078 ms. Trace:
I20260709 12:25:31.755862 18173 rpcz_store.cc:276] 0709 12:25:30.677005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.032941 (+355936us) service_pool.cc:224] Handling call
0709 12:25:31.755426 (+722485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.609735 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3418) took 1366 ms. Trace:
I20260709 12:25:31.748219 18813 rpcz_store.cc:276] 0709 12:25:29.833193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.833342 (+   149us) service_pool.cc:224] Handling call
0709 12:25:29.833642 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.833913 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:30.215374 (+381461us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.746949 (+1531575us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.747044 (+    95us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:31.747136 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.747242 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.747434 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":352740,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":1912109,"threads_started":2,"wal-append.queue_time_us":727182}
W20260709 12:25:31.611558 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:31.621451 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.753954 18015 rpcz_store.cc:276] 0709 12:25:30.426566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.573721 (+147155us) service_pool.cc:224] Handling call
0709 12:25:31.100917 (+527196us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:31.606022 (+505105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":124004096}
I20260709 12:25:31.654415 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:31.642141 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 641657us
W20260709 12:25:31.571144 26479 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.540s	user 0.014s	sys 0.018s
W20260709 12:25:31.530347 26895 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.822s	user 0.003s	sys 0.000s
I20260709 12:25:31.679944 18168 rpcz_store.cc:276] 0709 12:25:30.198717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.652392 (+453675us) service_pool.cc:224] Handling call
0709 12:25:31.531021 (+878629us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:25:31.531305 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.679078 (+147773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.679363 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25140992}
I20260709 12:25:31.688405 26806 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:31.522563 26759 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.649s	user 0.005s	sys 0.010s
W20260709 12:25:31.617657 26810 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.149s	user 0.021s	sys 0.009s
I20260709 12:25:31.689733 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:31.284659 18778 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.050s	sys 0.073s
W20260709 12:25:31.794116 18778 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.015s	user 0.057s	sys 0.081s
W20260709 12:25:31.794586 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.01578e+06us
W20260709 12:25:31.795403 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.691075 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:31.797504 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4314) took 1010 ms. Trace:
I20260709 12:25:31.797925 18020 rpcz_store.cc:276] 0709 12:25:30.786664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.690763 (+904099us) service_pool.cc:224] Handling call
0709 12:25:31.797411 (+106648us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.799391 18020 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
W20260709 12:25:31.691529 18019 rpcz_store.cc:269] 0709 12:25:30.427725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.449175 (+ 21450us) service_pool.cc:224] Handling call
0709 12:25:31.689450 (+1240275us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b5400108bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:31.689645 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":635481344}
I20260709 12:25:31.691717 18021 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:31.702651 18806 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:25:31.618152 26793 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.917s	user 0.023s	sys 0.004s
I20260709 12:25:31.704334 18165 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
W20260709 12:25:31.653475 26674 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.344s	user 0.009s	sys 0.017s
I20260709 12:25:31.707017 18096 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:31.461293 26720 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.001s	sys 0.010s
I20260709 12:25:31.711030 18097 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:25:31.814097 18022 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:31.814435 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3522) took 1026 ms. Trace:
I20260709 12:25:31.814935 18097 rpcz_store.cc:276] 0709 12:25:30.787702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.885407 (+ 97705us) service_pool.cc:224] Handling call
0709 12:25:31.814348 (+928941us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.813810 26909 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:31.816237 26909 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:31.615650 26890 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:31.713568 26790 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.535053 26443 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:31.817763 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4412) took 1638 ms (client timeout 1533 ms). Trace:
I20260709 12:25:31.717296 18171 rpcz_store.cc:276] 0709 12:25:30.038594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.038902 (+   308us) service_pool.cc:224] Handling call
0709 12:25:30.039301 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.208033 (+168732us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:30.208067 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.241576 (+ 33509us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.479371 (+237795us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:30.479416 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.659520 (+180104us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:30.659578 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.704194 (+1044616us) raft_consensus.cc:1685] finished
0709 12:25:31.704219 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.716481 (+ 12262us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":7852,"spinlock_wait_cycles":386810624,"thread_start_us":172434,"threads_started":1,"wal-append.queue_time_us":401249}
I20260709 12:25:31.818138 26443 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:31.818244 18099 rpcz_store.cc:269] 0709 12:25:30.178898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.185107 (+  6209us) service_pool.cc:224] Handling call
0709 12:25:31.817497 (+1632390us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:31.817679 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3341410048}
I20260709 12:25:31.618065 26901 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.819896 18099 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:31.820097 26901 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:31.820642 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4430) took 1003 ms. Trace:
I20260709 12:25:31.723734 26679 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:31.724653 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3514) took 2011 ms. Trace:
I20260709 12:25:31.727316 26657 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:31.727807 26821 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.673s	user 0.018s	sys 0.019s
I20260709 12:25:31.687988 26714 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.703567 26878 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.716140 26900 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:31.729461 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 297) took 37071 ms (client timeout 15000 ms). Trace:
I20260709 12:25:31.834082 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3408) took 1749 ms. Trace:
I20260709 12:25:31.732110 26784 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.784180 26642 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:31.814109 26720 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s	user 0.002s	sys 0.010s
I20260709 12:25:31.733878 26907 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.735493 26547 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:31.736240 26853 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.823225 26891 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.671435 26888 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:31.670414 26542 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:31.758147 18167 rpcz_store.cc:276] 0709 12:25:30.242944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.058657 (+815713us) service_pool.cc:224] Handling call
0709 12:25:31.608773 (+550116us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.609115 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.609157 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.609227 (+    70us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:31.609291 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.609377 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.609554 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.899412 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3421) took 1280 ms. Trace:
I20260709 12:25:31.900027 18167 rpcz_store.cc:276] 0709 12:25:30.618883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.897944 (+1279061us) service_pool.cc:224] Handling call
0709 12:25:31.898622 (+   678us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.898920 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.898951 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.899011 (+    60us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:31.899067 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.899145 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.899320 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:31.761514 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.539815 17913 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 10 to 19, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 19 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.903738 17913 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 16 to 20, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 20 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:31.783746 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:31.787725 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:31.906442 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4429) took 1089 ms. Trace:
I20260709 12:25:31.906599 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4433) took 1054 ms. Trace:
I20260709 12:25:31.907255 18090 rpcz_store.cc:276] 0709 12:25:30.852531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.779653 (+927122us) service_pool.cc:224] Handling call
0709 12:25:31.906510 (+126857us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.908880 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:31.908648 17913 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:31.909790 18090 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:31.911504 17913 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 18, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 18 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:31.792382 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 191640us
I20260709 12:25:31.804491 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4308) took 1019 ms. Trace:
I20260709 12:25:31.806409 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3287) took 1264 ms. Trace:
I20260709 12:25:31.747704 26497 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.711457 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:31.817343 26790 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.912925 18018 rpcz_store.cc:276] 0709 12:25:30.784944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.689352 (+904408us) service_pool.cc:224] Handling call
0709 12:25:31.804394 (+115042us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.913030 18165 rpcz_store.cc:276] 0709 12:25:30.542131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.618625 (+1076494us) service_pool.cc:224] Handling call
0709 12:25:31.704054 (+ 85429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.806058 (+102004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.806337 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.913265 26497 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.906927 18092 rpcz_store.cc:276] 0709 12:25:30.817149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.783337 (+966188us) service_pool.cc:224] Handling call
0709 12:25:31.906364 (+123027us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.914912 18165 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:31.915079 26790 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.915387 18092 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:25:31.820679 26443 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.915664 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1326) took 1190 ms. Trace:
I20260709 12:25:31.654954 26830 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:31.822062 18091 rpcz_store.cc:276] 0709 12:25:29.712924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.023289 (+310365us) service_pool.cc:224] Handling call
0709 12:25:30.073462 (+ 50173us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.220135 (+146673us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.220174 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.235155 (+ 14981us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.401286 (+166131us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:25:30.519812 (+118526us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:30.519864 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.519949 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.724281 (+1204332us) raft_consensus.cc:1685] finished
0709 12:25:31.724321 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.724571 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8005,"raft.run_cpu_time_us":2587,"raft.run_wall_time_us":312362,"spinlock_wait_cycles":33478912,"thread_start_us":358056,"threads_started":2,"wal-append.queue_time_us":418870}
I20260709 12:25:31.823931 18099 rpcz_store.cc:276] 0709 12:25:30.817522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.819395 (+1001873us) service_pool.cc:224] Handling call
0709 12:25:31.819715 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.820369 (+   654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.820579 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.916117 18165 rpcz_store.cc:276] 0709 12:25:30.725281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.914312 (+1189031us) service_pool.cc:224] Handling call
0709 12:25:31.914706 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.915369 (+   663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.915598 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:31.825104 18812 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:25:31.918147 18091 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 12:25:31.828809 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4428) took 1012 ms. Trace:
I20260709 12:25:31.783360 26479 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:31.841084 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:31.794598 26759 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:31.919932 18024 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:31.920161 18815 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:31.915977 26497 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.716140 26817 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.823704 26663 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.806125 26841 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:31.896768 18023 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:31.536916 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3513) took 1824 ms. Trace:
I20260709 12:25:31.934863 18093 rpcz_store.cc:276] 0709 12:25:29.712759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.008890 (+296131us) service_pool.cc:224] Handling call
0709 12:25:30.058740 (+ 49850us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.059047 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.059074 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.086035 (+ 26961us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.405284 (+319249us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:30.405323 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.405395 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.536476 (+1131081us) raft_consensus.cc:1685] finished
0709 12:25:31.536638 (+   162us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.536807 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":256465,"threads_started":1,"wal-append.queue_time_us":397225}
I20260709 12:25:31.918382 18812 pending_rounds.cc:85] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 15
I20260709 12:25:31.936563 18812 pending_rounds.cc:107] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:25:31.936960 18812 raft_consensus.cc:2889] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:31.831807 26821 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.827967 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:31.733201 26889 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.939121 26889 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.817474 26793 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.940348 26821 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:31.916371 26830 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.942265 26714 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:31.886576 26784 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.943632 26830 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:31.945179 26784 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:31.931676 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.929623 26542 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.932734 26479 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.817343 26810 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:31.947137 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 235825us
I20260709 12:25:31.947911 18170 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:31.846158 26900 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.949939 18020 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:25:31.839169 18164 rpcz_store.cc:276] 0709 12:25:30.084857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.087091 (+  2234us) service_pool.cc:224] Handling call
0709 12:25:30.126976 (+ 39885us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.137067 (+ 10091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.137112 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.137806 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.474391 (+336585us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:30.476883 (+  2492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.476978 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.529524 (+1052546us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:31.821934 (+292410us) raft_consensus.cc:1685] finished
0709 12:25:31.821962 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.833973 (+ 12011us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8786,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":14635,"spinlock_wait_cycles":27484032,"thread_start_us":303791,"threads_started":1,"wal-append.queue_time_us":418318}
I20260709 12:25:31.958458 18815 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:31.832849 26921 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.952097 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:31.987761 18099 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:31.961025 18096 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:31.945287 26793 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:31.930676 17913 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 18, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 18 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:31.831995 26826 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.103s
I20260709 12:25:31.967976 18098 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:31.969326 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:32.003710 18169 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 24. Ops: []
I20260709 12:25:31.773586 26873 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.981691 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3193) took 1554 ms. Trace:
I20260709 12:25:32.012449 18024 rpcz_store.cc:276] 0709 12:25:30.427372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.635979 (+208607us) service_pool.cc:224] Handling call
0709 12:25:30.636309 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.920461 (+1284152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.981383 (+ 60922us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540024e854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:31.981614 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2951407488}
I20260709 12:25:32.013116 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:31.985373 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.014472 18093 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:31.864790 26878 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.999622 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:31.982177 26900 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:31.924475 26907 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.055954 26542 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.057659 18015 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:32.002197 26926 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:32.023301 26770 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.059293 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3192) took 1632 ms. Trace:
I20260709 12:25:32.059864 26770 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:32.060457 26770 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.060559 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4432) took 1242 ms. Trace:
I20260709 12:25:32.060987 26770 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:31.981058 26663 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:25:31.971844 26909 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.026311 26790 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:32.024423 17913 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 20, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 20 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:32.027777 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1297) took 1192 ms. Trace:
I20260709 12:25:31.927983 26853 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:31.962800 26841 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:32.031411 26796 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.658s	user 0.012s	sys 0.016s
I20260709 12:25:32.033705 26810 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:32.034523 26928 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:25:31.925909 26547 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:32.035003 26831 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.494s	user 0.019s	sys 0.009s
I20260709 12:25:32.035696 26516 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.038966 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3281) took 1911 ms. Trace:
I20260709 12:25:32.079504 18169 rpcz_store.cc:276] 0709 12:25:30.128063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.443886 (+315823us) service_pool.cc:224] Handling call
0709 12:25:30.444166 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.004299 (+1560133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.038582 (+ 34283us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400201cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:32.038885 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3586084224}
W20260709 12:25:32.040078 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1363) took 1589 ms (client timeout 1624 ms). Trace:
W20260709 12:25:32.081276 18098 rpcz_store.cc:269] 0709 12:25:30.451053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.492779 (+ 41726us) service_pool.cc:224] Handling call
0709 12:25:32.039786 (+1547007us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400259754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:32.040001 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3329401088}
I20260709 12:25:32.070960 26790 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:32.082898 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1294) took 1249 ms. Trace:
W20260709 12:25:32.041083 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1578) took 3061 ms (client timeout 2241 ms). Trace:
I20260709 12:25:32.042963 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1300) took 1176 ms. Trace:
I20260709 12:25:32.041734 18167 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:32.043749 26638 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:32.044075 18093 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:25:32.044180 26513 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.045349 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3411) took 1910 ms. Trace:
I20260709 12:25:32.095671 18170 rpcz_store.cc:276] 0709 12:25:30.135184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.648535 (+513351us) service_pool.cc:224] Handling call
0709 12:25:31.123466 (+474931us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:25:31.566221 (+442755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.970138 (+403917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.045054 (+ 74916us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54003e2bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:32.045275 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":910709376}
I20260709 12:25:31.974444 26479 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.049098 26889 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:31.989591 26642 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:25:31.927127 26891 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:31.919443 18097 rpcz_store.cc:276] 0709 12:25:30.816789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.815968 (+999179us) service_pool.cc:224] Handling call
0709 12:25:31.816378 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.828171 (+ 11793us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.828228 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.828311 (+    83us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:31.828370 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.828485 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.828711 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11329,"raft.queue_time_us":15901,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":50962}
I20260709 12:25:32.105175 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:32.105597 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.106087 18164 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:32.107302 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.107694 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:32.107929 18097 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:32.059383 26926 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.025749 26878 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.061025 18090 rpcz_store.cc:276] 0709 12:25:30.818262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.912004 (+1093742us) service_pool.cc:224] Handling call
0709 12:25:31.912432 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.947229 (+ 34797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.947270 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.947320 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:31.947352 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.060298 (+112946us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b5400280354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:32.060352 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.060493 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79401856}
I20260709 12:25:32.059926 18023 rpcz_store.cc:276] 0709 12:25:30.427028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.635149 (+208121us) service_pool.cc:224] Handling call
0709 12:25:30.635565 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.934408 (+1298843us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.058921 (+124513us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:32.059210 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2899747584}
I20260709 12:25:32.064303 18020 rpcz_store.cc:276] 0709 12:25:30.834942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.800317 (+965375us) service_pool.cc:224] Handling call
0709 12:25:31.800628 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.971749 (+171121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.027428 (+ 55679us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54001d2254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:32.027699 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":342184192}
I20260709 12:25:31.996441 26921 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.008620 18099 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:32.073701 26909 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:31.990052 26497 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:25:32.082780 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.121112 18098 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:32.084090 18022 rpcz_store.cc:276] 0709 12:25:30.866128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.041117 (+174989us) service_pool.cc:224] Handling call
0709 12:25:32.042706 (+1001589us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540022f954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:32.042872 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338944768}
I20260709 12:25:32.121596 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:32.043501 26817 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:32.083473 18806 rpcz_store.cc:269] 0709 12:25:28.980115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.005986 (+ 25871us) service_pool.cc:224] Handling call
0709 12:25:32.040821 (+3034835us) spinlock_profiling.cc:243] Waited 2.7 s on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:32.040958 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6198956928}
W20260709 12:25:32.135509 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.136003 18170 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:25:32.137513 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.138521 18093 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:32.083349 18015 rpcz_store.cc:276] 0709 12:25:30.833487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.760190 (+926703us) service_pool.cc:224] Handling call
0709 12:25:31.796432 (+ 36242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.058364 (+261932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.082595 (+ 24231us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:32.082825 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":599978752}
I20260709 12:25:32.087417 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1322) took 1386 ms. Trace:
I20260709 12:25:32.094909 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:32.104492 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.144047 18167 rpcz_store.cc:276] 0709 12:25:30.701007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.901362 (+1200355us) service_pool.cc:224] Handling call
0709 12:25:32.041500 (+140138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.084225 (+ 42725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.087157 (+  2932us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400201cd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:32.087343 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321119104}
I20260709 12:25:32.088946 17913 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 7 to 19, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 19 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:32.110536 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:32.147462 18018 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 17.
I20260709 12:25:32.114465 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.148777 18090 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:32.117877 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.151392 18099 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:32.152699 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.153509 18097 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:32.088253 26638 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:32.074869 26796 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.166529 18097 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:32.156772 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.079030 26831 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.157567 18090 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:32.113655 26926 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:32.172459 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
W20260709 12:25:32.176227 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 415162us
I20260709 12:25:32.158516 18096 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
W20260709 12:25:32.158785 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69290.5us
W20260709 12:25:32.161175 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.015s	sys 0.024s
W20260709 12:25:32.178376 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.015s	sys 0.024s
W20260709 12:25:32.171185 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 329172us
I20260709 12:25:32.173745 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:32.180842 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80740.8us
I20260709 12:25:32.182227 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.183741 18099 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:32.127701 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:32.183758 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.190133 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.196257 18016 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:31.838480 17920 rpcz_store.cc:269] 0709 12:24:54.657468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.414371 (+9756903us) service_pool.cc:224] Handling call
0709 12:25:31.729157 (+27314786us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:17.051412 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:17.052227 (+   815us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:17.052294 (+    67us) write_op.cc:435] Acquired schema lock
0709 12:25:17.052314 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:17.229679 (+177365us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:17.229807 (+   128us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:17.229885 (+    78us) tablet.cc:685] Acquiring locks for 26 operations
0709 12:25:17.232351 (+  2466us) tablet.cc:701] Row locks acquired
0709 12:25:17.232402 (+    51us) write_op.cc:260] PREPARE: finished
0709 12:25:17.232608 (+   206us) write_op.cc:270] Start()
0709 12:25:17.232793 (+   185us) write_op.cc:276] Timestamp: P: 1783599917232536 usec, L: 0
0709 12:25:17.232829 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 12:25:17.233702 (+   873us) log.cc:844] Serialized 10219 byte log entry
0709 12:25:17.862064 (+628362us) op_driver.cc:464] REPLICATION: finished
0709 12:25:18.278988 (+416924us) write_op.cc:301] APPLY: starting
0709 12:25:18.279168 (+   180us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:18.953242 (+674074us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:18.953402 (+   160us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:20.111769 (+1158367us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b180036eec0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:20.912215 (+800446us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:20.912911 (+   696us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=0,mrs_lookups=0
0709 12:25:20.912978 (+    67us) write_op.cc:312] APPLY: finished
0709 12:25:20.913605 (+   627us) log.cc:844] Serialized 228 byte log entry
0709 12:25:21.003745 (+ 90140us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:21.037455 (+ 33710us) write_op.cc:454] Released schema lock
0709 12:25:21.047730 (+ 10275us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:21.095649 (+ 47919us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:21.097304 (+  1655us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3261,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":351686,"apply.run_cpu_time_us":64662,"apply.run_wall_time_us":2826842,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112963,"num_ops":26,"prepare.queue_time_us":421930,"prepare.run_cpu_time_us":14436,"prepare.run_wall_time_us":338545,"raft.queue_time_us":97324,"raft.run_cpu_time_us":17897,"raft.run_wall_time_us":326005,"replication_time_us":571852,"spinlock_wait_cycles":36985088,"thread_start_us":650611,"threads_started":5,"wal-append.queue_time_us":362135,"wal-append.run_cpu_time_us":41677,"wal-append.run_wall_time_us":2542908}]]}
I20260709 12:25:32.083855 26790 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:32.201944 18063 connection.cc:256] RPC call timeout handler was delayed by -1.6437497789997906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22410000000000002s for a response
W20260709 12:25:32.199390 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71916.9us
I20260709 12:25:32.201352 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:32.228647 18137 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:32.228706 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:32.224318 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3521) took 1450 ms. Trace:
I20260709 12:25:32.210578 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:32.169368 26497 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:32.211968 18097 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:32.200842 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:32.223596 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:32.212684 18778 connection.cc:256] RPC call timeout handler was delayed by -1.5378224979999686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260709 12:25:32.224903 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:32.239360 18778 connection.cc:256] RPC call timeout handler was delayed by -1.4840621650000685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20260709 12:25:32.242245 26821 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:32.242961 26821 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.243484 26821 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:32.245968 26542 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:32.246526 26542 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.246446 26759 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:32.247224 26542 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:32.200920 26817 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:25:32.236649 26638 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.239426 26901 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.236897 26679 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:32.251505 26679 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:32.251629 26479 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.252434 26901 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.250178 26796 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.230528 18094 rpcz_store.cc:276] 0709 12:25:30.773935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.783005 (+  9070us) service_pool.cc:224] Handling call
0709 12:25:30.783359 (+   354us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:30.803515 (+ 20156us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:30.866087 (+ 62572us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.292765 (+426678us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:31.658389 (+365624us) log.cc:844] Serialized 52 byte log entry
0709 12:25:31.658676 (+   287us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:31.658733 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.658858 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.224024 (+565166us) raft_consensus.cc:1685] finished
0709 12:25:32.224052 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.224245 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19642,"prepare.queue_time_us":63196,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":3735,"raft.queue_time_us":38248,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":18984,"thread_start_us":374932,"threads_started":1,"wal-append.queue_time_us":1085477}
I20260709 12:25:32.223091 26936 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:32.259719 26921 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.263131 18098 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:32.240746 18778 connection.cc:256] RPC call timeout handler was delayed by -1.4666325100001814s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
W20260709 12:25:32.270777 18778 connection.cc:256] RPC call timeout handler was delayed by -1.3856325100005051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20670000000000002s for a response
W20260709 12:25:32.271347 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0962116120003884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
W20260709 12:25:32.271799 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0886325100004797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2397s for a response
W20260709 12:25:32.272240 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0094325100003516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2485s for a response
I20260709 12:25:32.273370 26951 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:32.274333 26951 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.278337 26951 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.279080 26831 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.279702 26547 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.280520 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:32.281569 26841 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.263447 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.268425 26542 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.284104 26542 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.285054 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:32.285770 18806 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:32.286237 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:32.198616 26907 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:32.234658 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:32.227022 26810 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:32.252846 26679 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.225211 26945 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.307682 26853 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:32.307682 26945 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.226289 26707 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.310292 26707 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.226029 26663 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:32.311743 26663 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.307842 26907 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:32.315596 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50336.7us
I20260709 12:25:32.315908 26542 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:32.315547 17913 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 16 to 18, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 18 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:32.312908 26707 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:32.292610 26784 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:32.315212 26820 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.006s
I20260709 12:25:32.224946 26933 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:32.298496 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60516.4us
W20260709 12:25:32.306982 26837 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.551s	user 0.008s	sys 0.009s
I20260709 12:25:32.334453 26937 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.306718 26810 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.337461 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:32.363371 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56378us
I20260709 12:25:32.342278 18172 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:25:32.365280 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:32.337693 18806 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:32.373752 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.367514 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:32.368669 18810 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.019s	sys 0.005s
I20260709 12:25:32.370147 26929 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:32.371434 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3290) took 1828 ms. Trace:
W20260709 12:25:32.372731 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 144153us
W20260709 12:25:32.372924 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.006s	sys 0.012s
I20260709 12:25:32.335865 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:32.336364 26784 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.380463 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.335295 26542 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.334740 26897 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:32.381803 18810 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.019s	sys 0.005s
I20260709 12:25:32.342640 26707 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:32.385848 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:32.444725 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64411.4us
I20260709 12:25:32.380656 18098 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:25:32.445629 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.366318 26810 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:32.451437 17913 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:32.447795 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.338785 26796 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:25:32.449640 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:32.365953 26811 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.331600 26907 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:32.418669 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.332909 26940 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.115s
I20260709 12:25:32.399056 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.467348 18098 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:32.490199 18098 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:32.401799 18168 rpcz_store.cc:276] 0709 12:25:30.543161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.778955 (+1235794us) service_pool.cc:224] Handling call
0709 12:25:31.779394 (+   439us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:31.785603 (+  6209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.785651 (+    48us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:31.805486 (+ 19835us) log.cc:844] Serialized 78 byte log entry
0709 12:25:32.170815 (+365329us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:32.369851 (+199036us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.370421 (+   570us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.370735 (+   314us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.370947 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.371066 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.371120 (+    54us) raft_consensus.cc:1685] finished
0709 12:25:32.371138 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.371350 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5606,"raft.queue_time_us":41931,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"thread_start_us":445259,"threads_started":2,"wal-append.queue_time_us":382926}
W20260709 12:25:32.431542 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 94423.5us
I20260709 12:25:32.433089 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.381443 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.381992 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:32.428596 26841 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:32.379236 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:32.495458 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 116402us
I20260709 12:25:32.496052 26841 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:32.464660 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:32.391561 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:32.476544 18092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.014s	sys 0.014s
W20260709 12:25:32.410848 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.006s	sys 0.013s
W20260709 12:25:32.499087 18092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.015s	sys 0.014s
W20260709 12:25:32.494597 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 76775.8us
W20260709 12:25:32.479318 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 244708us
I20260709 12:25:32.454937 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3419) took 1841 ms. Trace:
I20260709 12:25:32.484539 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.502233 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:32.502588 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:32.504024 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:32.504818 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:32.503824 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 167936us
I20260709 12:25:32.503069 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:32.504187 18171 rpcz_store.cc:276] 0709 12:25:30.613214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.818681 (+1205467us) service_pool.cc:224] Handling call
0709 12:25:31.819081 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.819343 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:32.214121 (+394778us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.453793 (+239672us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.454566 (+   773us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:32.454635 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.454721 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.454861 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":623177,"thread_start_us":618015,"threads_started":2}
W20260709 12:25:32.480268 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 217009us
I20260709 12:25:32.440307 26784 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:32.376466 26934 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:32.508273 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 116920us
I20260709 12:25:32.437886 26897 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:32.511260 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.342549 26837 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.443274 26663 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:32.443210 26707 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:32.510185 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:32.375548 26933 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.434903 26936 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.462929 26513 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:32.518327 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 298) took 16281 ms (client timeout 15000 ms). Trace:
I20260709 12:25:32.384696 26929 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:32.430464 26951 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:32.454625 26924 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:32.629608 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3203) took 1736 ms. Trace:
I20260709 12:25:32.807723 18019 rpcz_store.cc:276] 0709 12:25:30.893092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.802781 (+909689us) service_pool.cc:224] Handling call
0709 12:25:31.803132 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.803382 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:32.215316 (+411934us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.628786 (+413470us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.628891 (+   105us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:32.628990 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.629283 (+   293us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.629510 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61694,"prepare.queue_time_us":958159,"raft.queue_time_us":304854,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":26346,"thread_start_us":715869,"threads_started":3}
I20260709 12:25:32.808954 18813 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:32.515169 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 149965us
I20260709 12:25:32.653525 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:32.674901 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.434774 26817 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.499334 26497 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:32.364356 26937 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:32.693763 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 184349us
W20260709 12:25:32.837142 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 328583us
I20260709 12:25:32.838135 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:25:32.699991 17920 rpcz_store.cc:269] 0709 12:25:16.253387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.454172 (+16200785us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:32.509308 (+ 55136us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:32.614416 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.442365 26843 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:32.514612 26443 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:32.448426 26888 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:32.503069 26841 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.186s
I20260709 12:25:32.577379 26831 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:32.512068 26866 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:32.606276 26889 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:32.678450 26707 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:32.835598 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 232993us
I20260709 12:25:32.838629 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:32.840204 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:32.714389 26513 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:32.866241 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.483578 26811 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.717679 26929 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.890462 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:32.896535 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:32.884810 26516 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:25:32.897017 26956 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 12:25:32.901882 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 394321us
I20260709 12:25:32.901201 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:32.901153 18808 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:32.918089 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:32.918375 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 412025us
I20260709 12:25:32.910352 18806 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:32.720469 26936 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:32.889096 26924 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:32.922531 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 282) took 15719 ms (client timeout 15000 ms). Trace:
W20260709 12:25:32.911902 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 512976us
I20260709 12:25:32.924140 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:32.931313 17920 rpcz_store.cc:269] 0709 12:25:17.190746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.900698 (+15709952us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:32.909981 (+  9283us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:32.920020 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:32.936082 26953 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.597s	user 0.002s	sys 0.000s
I20260709 12:25:32.842537 26817 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:33.048584 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 125703us
W20260709 12:25:32.918442 26948 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.122s	user 0.001s	sys 0.001s
I20260709 12:25:32.910321 26513 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:32.915365 26929 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:33.104243 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 292136us
I20260709 12:25:33.111014 18806 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:33.105893 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:33.046579 26798 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:33.056866 26889 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:33.132481 26900 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.079s	user 0.008s	sys 0.009s
I20260709 12:25:33.116125 26921 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:33.131270 26956 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:25:33.146767 26960 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 12:25:32.910573 26897 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.055644 26954 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.235s
I20260709 12:25:33.115720 26796 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:25:33.179177 26975 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 12:25:33.186650 26933 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.187271 26878 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:33.187582 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3423) took 1103 ms. Trace:
I20260709 12:25:33.187762 26878 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:33.187992 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.188043 18169 rpcz_store.cc:276] 0709 12:25:32.084667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.085255 (+   588us) service_pool.cc:224] Handling call
0709 12:25:32.085701 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.086300 (+   599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.086331 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:32.103666 (+ 17335us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.375780 (+272114us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:32.375843 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.375969 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.187307 (+811338us) raft_consensus.cc:1685] finished
0709 12:25:33.187338 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.187509 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":141,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":126,"thread_start_us":199878,"threads_started":1,"wal-append.queue_time_us":1065230}
I20260709 12:25:33.189160 26443 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:33.194311 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3288) took 2651 ms. Trace:
I20260709 12:25:33.194923 18166 rpcz_store.cc:276] 0709 12:25:30.542453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.671132 (+1128679us) service_pool.cc:224] Handling call
0709 12:25:31.671689 (+   557us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.948818 (+277129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.948854 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.965576 (+ 16722us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.376154 (+410578us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:32.376996 (+   842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.377089 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.193946 (+816857us) raft_consensus.cc:1685] finished
0709 12:25:33.193977 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.194216 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112844,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":19496,"thread_start_us":651123,"threads_started":2,"wal-append.queue_time_us":1208120}
I20260709 12:25:33.197347 26907 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:33.198549 18812 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:33.198948 26975 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:33.199189 26793 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.199572 26975 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:33.199759 26793 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.201110 26831 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:33.178076 26900 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.204442 26889 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.147140 26542 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:33.208552 18806 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:33.217314 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:33.082409 26497 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:33.225524 17989 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.501s (SENT) [suppressed 13 similar messages]
W20260709 12:25:33.227447 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.227605 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:33.201622 26831 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.239315 18811 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 25 index: 21. (index mismatch)
W20260709 12:25:33.241369 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 315062us
I20260709 12:25:33.227293 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:33.236871 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 312860us
I20260709 12:25:33.243870 18807 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:33.245301 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:33.246915 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:33.248466 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.253868 26924 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:33.254557 26924 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:33.255055 26924 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:33.223029 26770 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.160s	user 0.012s	sys 0.004s
I20260709 12:25:33.256606 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:25:33.265681 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:33.267040 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 161316us
I20260709 12:25:33.260027 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:33.270371 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3289) took 2727 ms. Trace:
I20260709 12:25:33.228456 26954 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:33.270980 18173 rpcz_store.cc:276] 0709 12:25:30.542735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.756698 (+1213963us) service_pool.cc:224] Handling call
0709 12:25:31.757059 (+   361us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:31.757520 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.757544 (+    24us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:31.843145 (+ 85601us) log.cc:844] Serialized 78 byte log entry
0709 12:25:32.367637 (+524492us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:32.367710 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.367826 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.270027 (+902201us) raft_consensus.cc:1685] finished
0709 12:25:33.270060 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.270279 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":64063,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":429941,"threads_started":1,"wal-append.queue_time_us":591212}
I20260709 12:25:33.256150 26831 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:33.299492 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:33.200594 26975 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.272311 26837 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:33.301896 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.348119 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1589) took 3305 ms. Trace:
W20260709 12:25:33.317803 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 397852us
I20260709 12:25:33.293457 26954 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:33.358311 18136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.002s	sys 0.011s
W20260709 12:25:33.359843 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60940.4us
W20260709 12:25:33.360069 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 134598us
I20260709 12:25:33.351096 18809 rpcz_store.cc:276] 0709 12:25:30.042643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.042841 (+   198us) service_pool.cc:224] Handling call
0709 12:25:30.071718 (+ 28877us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.072765 (+  1047us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:31.072486 (+999721us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b18002ee880. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:25:31.843469 (+770983us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.380633 (+537164us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:32.430836 (+ 50203us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.431009 (+   173us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:32.431049 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.431146 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.310296 (+879150us) raft_consensus.cc:1685] finished
0709 12:25:33.347618 (+ 37322us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.347977 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":451,"prepare.queue_time_us":114084,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":13911,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":15875,"spinlock_wait_cycles":24454400,"thread_start_us":686987,"threads_started":2,"wal-append.queue_time_us":623544}
I20260709 12:25:33.377377 18019 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:33.274053 26542 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
I20260709 12:25:33.263954 26798 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.055s
I20260709 12:25:33.274453 26770 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.379941 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:33.380366 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:33.380775 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:33.378834 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:33.383450 18136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.002s	sys 0.011s
I20260709 12:25:33.381781 18020 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:33.399336 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 895354us
I20260709 12:25:33.296442 26516 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:33.309662 26793 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.399214 18097 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:33.400231 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:33.404912 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:33.360131 26939 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.007s	sys 0.006s
I20260709 12:25:33.279031 26841 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.094s
I20260709 12:25:33.410188 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.439759 18023 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:33.430730 18015 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 12:25:33.441067 18015 pending_rounds.cc:85] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 10
I20260709 12:25:33.441435 18015 pending_rounds.cc:107] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:25:33.429090 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.428785 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:33.387143 26821 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.143s	user 0.013s	sys 0.005s
I20260709 12:25:33.441731 18015 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:33.444001 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:33.445238 18809 raft_consensus.cc:2410] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:33.446206 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4437) took 1496 ms. Trace:
I20260709 12:25:33.446758 18091 rpcz_store.cc:276] 0709 12:25:31.949752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.950033 (+   281us) service_pool.cc:224] Handling call
0709 12:25:31.983414 (+ 33381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.983971 (+   557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.984019 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.995877 (+ 11858us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.468400 (+472523us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:32.468451 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.468529 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.445840 (+977311us) raft_consensus.cc:1685] finished
0709 12:25:33.445875 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.446104 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":179,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8782,"thread_start_us":423450,"threads_started":1,"wal-append.queue_time_us":1286576}
W20260709 12:25:33.463433 26961 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:25:33.465737 26897 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:33.466601 26974 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.467341 26542 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:33.318557 26980 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.126s
I20260709 12:25:33.411196 26982 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:33.392997 26888 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:33.435286 26939 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.007s	sys 0.006s
I20260709 12:25:33.378769 26985 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:33.458773 26516 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.386319 26921 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:33.430912 26866 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:33.445861 26810 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.962s	user 0.009s	sys 0.005s
I20260709 12:25:33.435302 26900 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.301465 26878 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.492724 18023 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:33.567534 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.568526 18022 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:33.568771 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.500705 18097 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:33.569811 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.570926 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:33.571663 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:33.469291 26989 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.531304 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.531728 18024 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:33.528101 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44133.7us
I20260709 12:25:33.473899 26817 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:33.541673 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.567782 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.506534 26982 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:33.612871 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 346229us
I20260709 12:25:33.450961 26821 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.606523 18019 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:33.526520 26954 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.537081 26985 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.608635 18020 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:33.536435 26937 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.614442 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:33.647722 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.607704 26810 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.652603 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:33.625525 18019 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:33.546712 26891 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.627938 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.637583 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.666198 18015 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:33.603571 26986 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.623308 26982 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:33.655519 18807 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:33.607975 26990 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.648699 18020 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:33.632077 26981 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:33.619091 26969 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:33.618937 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.625536 26907 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.678059 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.639266 26878 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:33.629613 26985 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:33.528576 26974 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.687458 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:33.682286 26979 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.693754 26990 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.694123 26969 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.706545 18807 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:33.686726 26909 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.566s	user 0.009s	sys 0.008s
W20260709 12:25:33.711599 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97302.6us
I20260709 12:25:33.685743 26891 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:33.699847 26954 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.698781 26873 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:33.751924 18807 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:33.619803 26989 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:33.744087 26806 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.949s	user 0.018s	sys 0.001s
I20260709 12:25:33.755621 18022 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:33.759445 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:33.732769 26542 raft_consensus.cc:2764] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:25:33.683418 26986 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.764401 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:33.769550 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:33.766521 18811 raft_consensus.cc:2376] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25. Re-sending same reply.
I20260709 12:25:33.773023 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1749) took 1965 ms. Trace:
I20260709 12:25:33.775839 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:33.773567 18810 rpcz_store.cc:276] 0709 12:25:31.807594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.807746 (+   152us) service_pool.cc:224] Handling call
0709 12:25:31.808143 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.808544 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.808576 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.809157 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.439033 (+629876us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:32.439100 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.439214 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.772743 (+1333529us) raft_consensus.cc:1685] finished
0709 12:25:33.772768 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.772942 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":137504,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":14596,"thread_start_us":629380,"threads_started":1,"wal-append.queue_time_us":1809682}
I20260709 12:25:33.780097 18811 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:33.702060 26907 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.784355 18015 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:33.765479 26891 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:33.764411 26984 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:33.822450 26984 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.823693 26811 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:33.824487 18098 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:33.825037 26817 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:33.825495 26817 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:33.826136 26926 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.623s	user 0.014s	sys 0.004s
I20260709 12:25:33.799557 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.800750 18024 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:33.806011 26971 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.812893 26989 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:33.814532 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4446) took 1740 ms. Trace:
I20260709 12:25:33.816048 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3523) took 1976 ms. Trace:
W20260709 12:25:33.818804 26992 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:25:33.726455 26979 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:33.838457 18015 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:33.832656 26707 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.913s	user 0.015s	sys 0.000s
I20260709 12:25:33.837508 18093 rpcz_store.cc:276] 0709 12:25:32.074455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.139391 (+ 64936us) service_pool.cc:224] Handling call
0709 12:25:32.173661 (+ 34270us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.180731 (+  7070us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.180773 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.207156 (+1026383us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:33.207681 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.425458 (+217777us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:33.425523 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.425649 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.814073 (+388424us) raft_consensus.cc:1685] finished
0709 12:25:33.814115 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.814445 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6527,"raft.queue_time_us":6765,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":26247,"spinlock_wait_cycles":5591424,"thread_start_us":150196,"threads_started":1,"wal-append.queue_time_us":508318}
W20260709 12:25:33.845492 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.839082 18092 rpcz_store.cc:276] 0709 12:25:31.839634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.916212 (+ 76578us) service_pool.cc:224] Handling call
0709 12:25:31.916695 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.916959 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.916988 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.917604 (+   616us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.501044 (+583440us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:32.501248 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.501609 (+   361us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.815731 (+1314122us) raft_consensus.cc:1685] finished
0709 12:25:33.815764 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.815963 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":582475,"threads_started":1,"wal-append.queue_time_us":1826309}
I20260709 12:25:33.839756 18808 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:33.866854 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.866854 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:33.879333 18015 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:33.879447 18024 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:33.870761 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:33.880617 18019 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:33.825577 26984 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:33.870723 18064 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.930s (SENT)
I20260709 12:25:33.794946 26542 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
W20260709 12:25:33.898612 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 185831us
I20260709 12:25:33.826697 26926 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:33.911605 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.525s (SENT)
I20260709 12:25:33.912441 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:33.884599 26837 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.888962 26821 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:33.838398 26971 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:33.927448 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56788.3us
I20260709 12:25:33.928037 18015 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:33.932502 18806 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:33.935599 27009 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:33.935990 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4442) took 1979 ms (client timeout 1541 ms). Trace:
I20260709 12:25:33.936412 27009 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:33.936542 18096 rpcz_store.cc:269] 0709 12:25:31.956260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.991162 (+ 34902us) service_pool.cc:224] Handling call
0709 12:25:33.935741 (+1944579us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:33.935901 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":331706624}
I20260709 12:25:33.938267 26977 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:33.939512 26794 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:33.940016 26794 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:33.826586 26817 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.888134 26707 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.941035 26794 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:33.954480 18024 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:33.930887 26806 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:25:33.929533 26837 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:34.006340 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3185) took 3598 ms (client timeout 2146 ms). Trace:
W20260709 12:25:34.008718 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.931249 26810 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.945685 26996 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:25:34.011000 18016 rpcz_store.cc:269] 0709 12:25:30.407711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.407835 (+   124us) service_pool.cc:224] Handling call
0709 12:25:34.006050 (+3598215us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54001fc554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:34.006241 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4012039936}
I20260709 12:25:33.969686 27009 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:33.981132 26951 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.211s	user 0.010s	sys 0.004s
I20260709 12:25:33.970376 26794 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:34.006559 26926 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:34.092760 26817 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:34.028865 27008 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:25:34.058851 26513 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.928s	user 0.008s	sys 0.008s
I20260709 12:25:34.013870 26996 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:34.125344 26784 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.482s	user 0.018s	sys 0.000s
W20260709 12:25:34.140713 27019 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:25:34.053884 26951 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:34.124214 26929 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.983s	user 0.013s	sys 0.003s
I20260709 12:25:34.263480 26974 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:33.965145 26977 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.161s
I20260709 12:25:33.927639 26891 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.262496 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.022943 26984 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.012579 26899 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:34.285737 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 229877us
I20260709 12:25:34.290366 18815 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:34.293529 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:34.274943 26513 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.295964 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.297043 26513 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.297871 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1596) took 1081 ms. Trace:
I20260709 12:25:34.286597 26929 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.298319 18808 rpcz_store.cc:276] 0709 12:25:33.216595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.216929 (+   334us) service_pool.cc:224] Handling call
0709 12:25:34.297610 (+1080681us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b54005f1c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:34.297754 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1268159232}
I20260709 12:25:34.300724 26929 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:34.298125 18810 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:25:34.301994 27002 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:34.302243 26983 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:34.303014 26983 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:34.304579 26924 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.049s	user 0.004s	sys 0.022s
I20260709 12:25:34.304154 26983 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.305163 26924 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.306305 26996 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 26.22, Last appended by leader: 22, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.280670 26784 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:34.309252 26985 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.580s	user 0.023s	sys 0.000s
I20260709 12:25:34.316345 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:34.297082 26945 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:34.317742 27009 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:34.318580 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.320540 26707 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.388939 18091 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:34.319800 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:34.329080 27017 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.358s	user 0.000s	sys 0.001s
W20260709 12:25:34.330804 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3275) took 4252 ms (client timeout 1930 ms). Trace:
I20260709 12:25:34.333647 27002 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:25:34.336134 26878 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.003s
W20260709 12:25:34.336808 27011 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.361s	user 0.001s	sys 0.005s
W20260709 12:25:34.283438 26831 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.964s	user 0.019s	sys 0.001s
I20260709 12:25:34.338917 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.400956 26831 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.402094 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:34.403041 26955 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.052s	user 0.003s	sys 0.001s
I20260709 12:25:34.403335 18811 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:34.403888 26831 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:34.339442 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.406049 26971 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.406458 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:34.407446 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68686.7us
I20260709 12:25:34.408479 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.412633 26516 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:34.413378 26516 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.341338 26937 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:34.413919 26516 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:34.414206 26937 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.417191 26937 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.338954 26638 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:34.349319 26988 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.727s	user 0.002s	sys 0.004s
W20260709 12:25:34.353472 26873 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.008s
I20260709 12:25:34.288133 26907 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.290308 26899 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:34.361688 26479 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:34.365049 26513 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.370781 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:34.433535 18024 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:34.374718 26998 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.374480 26443 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:25:34.377074 26954 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.003s
W20260709 12:25:34.378957 27022 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:25:34.378649 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:34.292927 26984 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.289434 26542 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:34.380134 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:34.373059 26982 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.677s	user 0.011s	sys 0.002s
I20260709 12:25:34.390117 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:34.390591 18810 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
W20260709 12:25:34.392838 18172 rpcz_store.cc:269] 0709 12:25:30.078701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.079058 (+   357us) service_pool.cc:224] Handling call
0709 12:25:34.330396 (+4251338us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b540029dbd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:34.330664 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5198992256}
I20260709 12:25:34.453261 18808 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:34.396646 18807 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:34.398023 27009 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:34.420637 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1292) took 3588 ms. Trace:
I20260709 12:25:34.409443 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.424738 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.393764 26985 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.392315 26996 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:34.372596 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:34.462903 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90387.9us
I20260709 12:25:34.437290 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.391209 26990 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.427189 27002 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:34.426312 26831 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:34.450204 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.470079 18092 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:34.430581 26900 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:34.456764 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47409.4us
I20260709 12:25:34.423601 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:34.454372 18017 rpcz_store.cc:276] 0709 12:25:30.832677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.755974 (+923297us) service_pool.cc:224] Handling call
0709 12:25:31.756339 (+   365us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:31.756723 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.756750 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:31.757519 (+   769us) log.cc:844] Serialized 52 byte log entry
0709 12:25:32.499485 (+741966us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:32.503299 (+  3814us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.775163 (+271864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.775291 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.205538 (+430247us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b48000801d8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:25:34.402112 (+1196574us) spinlock_profiling.cc:243] Waited 8.78 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:34.420285 (+ 18173us) raft_consensus.cc:1685] finished
0709 12:25:34.420321 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.420540 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":630425,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":231196,"raft.run_cpu_time_us":1487,"raft.run_wall_time_us":512486,"spinlock_wait_cycles":50261632,"thread_start_us":1016217,"threads_started":3,"wal-append.queue_time_us":1498507}
I20260709 12:25:34.466552 18098 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:25:34.451524 26542 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.476938 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:34.475171 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51752us
I20260709 12:25:34.485319 18096 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:25:34.486904 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.457896 27009 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:34.481135 26542 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:34.490742 26924 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:34.468408 26899 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.460403 26982 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.481530 26900 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.473722 26831 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.493444 26831 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:34.461926 26873 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:34.479164 26974 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:34.482002 27002 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:34.440987 26513 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.439317 26479 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.498848 18095 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:25:34.503273 18092 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.19-23.19]
I20260709 12:25:34.468158 26954 raft_consensus.cc:2764] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:25:34.440775 26998 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.522889 18018 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:25:34.497884 18093 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:25:34.533646 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.540015 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:34.514729 27016 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.542s	user 0.003s	sys 0.006s
I20260709 12:25:34.517639 26969 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.520004 26933 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:34.516705 26967 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:34.590795 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.19056e+06us
I20260709 12:25:34.581604 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.512739 26963 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:34.493765 26982 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:34.518190 26770 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:34.517781 26873 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:34.580278 27010 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.490765 26784 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.578666 26679 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:34.592044 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:34.520095 26984 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:34.602192 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.604197 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.517388 26935 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:34.490823 27009 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.617466 18096 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:34.595844 26966 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:34.505082 26542 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.579329 26976 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:34.590260 26933 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:34.653669 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61774us
I20260709 12:25:34.590531 26967 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.642643 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.643067 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3540) took 1238 ms. Trace:
I20260709 12:25:34.654912 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:34.657075 18097 rpcz_store.cc:276] 0709 12:25:33.404699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.572325 (+167626us) service_pool.cc:224] Handling call
0709 12:25:33.604347 (+ 32022us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.615066 (+ 10719us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:33.615133 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.636396 (+ 21263us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.898169 (+261773us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:33.898240 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.898350 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:34.513736 (+615386us) spinlock_profiling.cc:243] Waited 8.21 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:34.642770 (+129034us) raft_consensus.cc:1685] finished
0709 12:25:34.642795 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.642996 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":85541,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":37968,"spinlock_wait_cycles":18887040,"thread_start_us":222070,"threads_started":1,"wal-append.queue_time_us":325214}
W20260709 12:25:34.647419 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.71499e+06us
I20260709 12:25:34.660421 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.662796 18093 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:34.663993 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.665274 18097 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:34.641016 26638 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.644779 26784 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:34.680073 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:34.680990 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.682443 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:34.656034 26933 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:34.609620 26963 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.499053 27002 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.640255 26679 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:34.703840 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174643us
I20260709 12:25:34.699551 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:34.634601 27010 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.681731 26784 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.649099 26516 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:34.695185 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 173063us
I20260709 12:25:34.704713 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.725009 18096 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:34.668560 26820 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:25:34.709403 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.725904 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:34.728157 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.669661 26976 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:34.720075 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2217b6329873485ead60419dc5acbff0"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:25:34.714398 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.87619e+06us
I20260709 12:25:34.666133 26967 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:34.684567 26663 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:34.707197 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:34.656492 26707 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:25:34.702085 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:34.722877 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.689064 26935 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:34.759533 18093 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:34.759725 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52616.4us
I20260709 12:25:34.729089 18173 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:34.761835 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.762598 18173 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:34.729228 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.686894 26934 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:25:34.779493 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.781297 18098 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:34.651275 26966 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.736003 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.736624 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:34.722473 26784 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:34.782830 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.782985 18167 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:34.783922 26966 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.688443 26964 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:25:34.738138 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.784265 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.784940 18165 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:34.785580 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.783017 18016 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:34.789249 26516 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.736645 26497 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:34.790190 26497 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.791091 26831 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.792748 26497 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:34.741932 26679 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.741802 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:34.743079 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1756) took 2241 ms (client timeout 1929 ms). Trace:
I20260709 12:25:34.701035 26873 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:34.732276 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:34.748109 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:34.793736 18815 rpcz_store.cc:269] 0709 12:25:32.501678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.502741 (+  1063us) service_pool.cc:224] Handling call
0709 12:25:34.742828 (+2240087us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b5400557154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:34.742989 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4103795968}
I20260709 12:25:34.683409 26638 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:34.765663 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 63688.7us
I20260709 12:25:34.750177 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:34.768841 26976 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.770562 26479 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.774019 26889 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:34.774935 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3208) took 1230 ms. Trace:
I20260709 12:25:34.775606 26998 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.826165 18022 rpcz_store.cc:276] 0709 12:25:33.544233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.569406 (+ 25173us) service_pool.cc:224] Handling call
0709 12:25:34.774735 (+1205329us) spinlock_profiling.cc:243] Waited 89.4 ms on lock 0x7b5400251a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:34.774862 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205652224}
I20260709 12:25:34.776710 18169 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:25:34.776969 26963 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.778618 26542 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.778754 26936 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.730525 18166 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: [25.21-25.21]
I20260709 12:25:34.784104 26962 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:34.786415 26513 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.786273 18098 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:34.850682 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:34.851665 18815 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:34.851598 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.794364 18168 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:34.852030 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:34.852914 18811 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:34.721313 26878 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:34.804500 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:34.804862 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:34.860677 18807 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:34.805414 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:34.797809 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.810241 18167 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:34.862460 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:34.811520 18017 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:34.810865 18093 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
W20260709 12:25:34.865345 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.866078 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:34.796465 26638 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:34.834553 18171 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:34.867779 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:34.868744 18166 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:34.868880 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.840924 18016 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:34.841110 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:34.746663 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:34.862241 18096 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 19.
I20260709 12:25:34.863971 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.874248 18167 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:34.874917 18810 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:34.881656 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.882743 18168 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:25:34.885087 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:34.886039 18815 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:34.886404 18170 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: [23.11-23.11]
I20260709 12:25:34.797868 26663 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:34.831213 26889 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:34.886222 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:34.833415 26820 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:34.783962 26987 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:34.885282 26964 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:34.887642 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.893865 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.904521 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:34.905078 18171 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:34.905396 18093 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:25:34.876366 27030 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:25:34.897603 18167 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:34.898352 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.859602 26878 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.905190 18170 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:34.884095 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.906174 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:34.907598 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 161020us
I20260709 12:25:34.916862 18168 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:34.832553 26770 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:25:34.832180 26982 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:25:34.861159 26479 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:25:34.899946 26889 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:34.917972 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:34.921409 18171 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:34.898006 26820 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:34.904232 26964 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:34.894306 26831 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.909466 18165 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:34.906620 26638 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.915683 26878 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:34.943043 18166 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:34.931928 26820 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:34.920825 26987 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:34.921221 27034 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:34.955076 18812 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:34.955022 18813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1752) took 2675 ms (client timeout 1516 ms). Trace:
W20260709 12:25:34.959116 18813 rpcz_store.cc:269] 0709 12:25:32.279819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.280104 (+   285us) service_pool.cc:224] Handling call
0709 12:25:34.954613 (+2674509us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:34.954893 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1001168128}
I20260709 12:25:34.945046 26638 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:34.970777 27027 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:25:34.976337 27029 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.217s	user 0.001s	sys 0.001s
I20260709 12:25:34.970804 26638 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:34.975271 27034 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:34.984678 27034 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:35.002650 27038 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:25:35.009874 26979 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.010768 26987 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.011418 26967 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.013796 26967 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.015911 26933 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.019806 26933 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:34.898741 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138099us
W20260709 12:25:35.083830 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 211424us
I20260709 12:25:35.089656 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:35.092057 18016 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:35.093303 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.094554 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.095901 18020 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:35.099434 18022 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:35.102734 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.105718 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:35.107388 18023 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.111006 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.113499 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:35.114949 18017 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:35.118623 18020 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:35.118808 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.101377 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.135950 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.137406 18093 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.146592 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.155716 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.160948 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:34.917510 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.168469 18173 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:35.169337 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.177143 26516 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.178825 26516 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.183682 26516 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.185075 26516 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.169898 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.186445 26516 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:34.958137 26663 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.209278 26770 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.210582 26759 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.211449 26770 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.212874 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.214290 26983 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.210372 26933 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:35.209517 26936 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:25:35.226689 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:35.232430 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:35.235954 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.237767 26976 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:35.238065 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3534) took 1858 ms (client timeout 1944 ms). Trace:
I20260709 12:25:35.238842 27034 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:35.253113 18172 raft_consensus.cc:606] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.253741 27034 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.238410 26542 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.258536 27034 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:35.258932 26443 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:35.259621 26443 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:35.259205 26542 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.261590 26891 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.262739 26891 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.263271 26638 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:35.263825 26638 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.264374 26638 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.268918 26873 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:35.269594 26873 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.270071 26873 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.271456 26891 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.210201 26996 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:35.268043 18017 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:35.253867 26976 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.283860 18023 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:35.211505 26759 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:35.290477 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.277182 26996 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.284181 26976 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.287505 26770 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.308791 18093 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:35.310370 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:35.251821 18099 rpcz_store.cc:269] 0709 12:25:33.379750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.380050 (+   300us) service_pool.cc:224] Handling call
0709 12:25:35.237454 (+1857404us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b54002968d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:35.237760 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1209600384}
I20260709 12:25:35.301297 26996 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:35.261587 26443 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.323338 26759 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.324313 26542 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.326294 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.330056 18023 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:35.331859 26542 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.336970 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.326622 26970 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:35.349247 26970 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:35.370162 26542 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.371765 18093 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:35.375142 27059 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:35.382388 18168 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:35.408156 27010 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.385031 18813 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:35.409768 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.410701 18168 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:35.412079 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.412952 18168 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:35.394882 26837 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:35.398002 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.414650 18017 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
W20260709 12:25:35.415733 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 246192us
I20260709 12:25:35.399027 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.400456 18092 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:35.402143 26811 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.418274 27059 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:35.419060 27059 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:35.420451 27059 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:35.400373 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 482784us
I20260709 12:25:35.422684 26516 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.417732 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.406109 26970 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.426627 26970 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:35.406746 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.386284 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:35.418622 26811 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.400337 26889 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.431763 18019 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:35.432348 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.433152 26811 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:35.433969 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:35.434487 26889 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.434811 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.373462 26542 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.458472 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.450867 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.469049 18090 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:35.465925 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50196.4us
I20260709 12:25:35.477295 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.484892 26971 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:35.485774 26971 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:35.485808 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.486961 18099 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.487288 26971 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:35.497984 18811 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.622s	user 0.015s	sys 0.006s
I20260709 12:25:35.498642 18811 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:35.499861 18809 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: [24.20-25.21]
I20260709 12:25:35.504827 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:35.505941 26542 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:35.506583 26613 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:35.507797 26516 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:35.508296 26516 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:35.513124 26516 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.21, Last appended by leader: 21, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.515779 26516 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:35.518172 26900 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.521651 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.522579 18099 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
W20260709 12:25:35.497810 26820 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.531s	user 0.013s	sys 0.004s
I20260709 12:25:35.525247 26866 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:35.521648 18063 connection.cc:256] RPC call timeout handler was delayed by -1.1618399839999256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
I20260709 12:25:35.536686 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.537968 18099 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:35.540246 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.540248 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.527005 26866 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.518698 26638 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.547763 26866 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.558868 18090 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 16: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:35.559491 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.548152 26900 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.562156 26907 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:35.567250 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.561512 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1385) took 1079 ms. Trace:
I20260709 12:25:35.562440 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:35.569576 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.549649 26613 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:35.570678 18090 rpcz_store.cc:276] 0709 12:25:34.479925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.482828 (+1002903us) service_pool.cc:224] Handling call
0709 12:25:35.559390 (+ 76562us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.570385 18168 raft_consensus.cc:2376] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28. Re-sending same reply.
I20260709 12:25:35.537061 26820 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.577332 26613 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.579138 26985 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.583351 26542 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:35.585186 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.597934 18169 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:35.573007 26907 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:35.586447 26784 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.803s	user 0.008s	sys 0.012s
I20260709 12:25:35.587018 26900 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:35.588877 26878 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.637s	user 0.008s	sys 0.012s
I20260709 12:25:35.593711 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.599938 18814 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: []
W20260709 12:25:35.603540 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.605202 26907 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.606945 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.605508 26638 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.608090 18099 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:35.609272 18172 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:35.608908 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.613301 18811 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:35.613672 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:35.616305 18806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1597) took 2371 ms (client timeout 1607 ms). Trace:
I20260709 12:25:35.622278 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:35.618227 26784 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.622782 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:35.623208 18810 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26. Re-sending same reply.
I20260709 12:25:35.622318 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.624258 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1312) took 1252 ms. Trace:
I20260709 12:25:35.625006 18018 rpcz_store.cc:276] 0709 12:25:34.371299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.384033 (+ 12734us) service_pool.cc:224] Handling call
0709 12:25:34.384636 (+   603us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:34.537381 (+152745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.623678 (+1086297us) spinlock_profiling.cc:243] Waited 96.8 ms on lock 0x7b54001d2254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:35.624029 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222658432}
I20260709 12:25:35.627633 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.626171 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:35.619235 18806 rpcz_store.cc:269] 0709 12:25:33.223352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.223641 (+   289us) service_pool.cc:224] Handling call
0709 12:25:35.594243 (+2370602us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54005901d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:35.594531 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1532838784}
I20260709 12:25:35.629168 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.630111 18169 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:35.631441 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.632535 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:35.633289 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1598) took 2406 ms. Trace:
W20260709 12:25:35.634045 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 67071us
I20260709 12:25:35.633862 18812 rpcz_store.cc:276] 0709 12:25:33.226321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.226527 (+   206us) service_pool.cc:224] Handling call
0709 12:25:34.954809 (+1728282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.959161 (+  4352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.632763 (+673602us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54006f0ed0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:35.633009 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3875794176}
I20260709 12:25:35.636016 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.637099 18173 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:35.637256 18090 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:25:35.638159 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1388) took 1155 ms. Trace:
I20260709 12:25:35.640980 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:35.641045 18090 rpcz_store.cc:276] 0709 12:25:34.482998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.595969 (+1112971us) service_pool.cc:224] Handling call
0709 12:25:35.636814 (+ 40845us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.637778 (+   964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.638072 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.643743 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.644069 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.644767 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:35.646436 18172 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:35.648136 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.646392 18812 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:35.646365 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:35.650158 26985 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:35.652835 26613 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.652998 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.657153 26714 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:35.657739 26831 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:35.658200 26899 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:35.658314 26831 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.656344 18810 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 16: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:35.657733 18090 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:35.643803 26984 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:35.660998 26841 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:35.661087 26984 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:35.661758 26841 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:35.662981 26984 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.626017 26784 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.616920 26983 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:35.663240 26841 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:35.667606 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.673875 26841 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.668464 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.677495 18169 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:35.682106 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:35.621585 26971 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:35.683313 26971 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:35.685009 26900 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.684773 26971 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.686693 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.687709 18096 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:35.689379 18096 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:35.689177 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.658885 26831 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.690251 18169 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.670637 18807 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:25:35.671427 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.668862 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:35.695818 27061 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.185s	user 0.002s	sys 0.003s
I20260709 12:25:35.670130 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:35.698848 26820 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.699074 26986 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.695897 26985 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:25:35.700353 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.671217 18811 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:35.701247 26986 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:35.702229 26986 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:35.668663 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:35.705771 27063 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:25:35.706694 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3539) took 2320 ms. Trace:
I20260709 12:25:35.701556 18168 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:35.703580 26986 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.708518 26986 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.707196 18094 rpcz_store.cc:276] 0709 12:25:33.386277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.447330 (+ 61053us) service_pool.cc:224] Handling call
0709 12:25:33.484903 (+ 37573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.630395 (+145492us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:33.690189 (+ 59794us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.788081 (+ 97892us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:33.788207 (+   126us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:33.788294 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.402618 (+1614324us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b18001a4b80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419
0709 12:25:35.706064 (+303446us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540035d9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:35.706128 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.706438 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17464,"prepare.queue_time_us":194120,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":17663,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7924,"spinlock_wait_cycles":347659392,"thread_start_us":93248,"threads_started":2,"wal-append.queue_time_us":198861,"wal-append.run_cpu_time_us":3169,"wal-append.run_wall_time_us":1383038}
I20260709 12:25:35.683063 18097 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:35.710449 26770 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.711467 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3508) took 5999 ms (client timeout 1739 ms). Trace:
I20260709 12:25:35.711187 26770 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:35.712169 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.700366 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:35.714105 18778 connection.cc:256] RPC call timeout handler was delayed by -1.0164364240004033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
I20260709 12:25:35.715319 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1390) took 1209 ms. Trace:
I20260709 12:25:35.715899 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.716265 27009 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:35.714208 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.722154 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:35.712006 18095 rpcz_store.cc:269] 0709 12:25:29.711904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.066414 (+1354510us) service_pool.cc:224] Handling call
0709 12:25:35.710892 (+4644478us) spinlock_profiling.cc:243] Waited 2.75 s on lock 0x7b540035edd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:35.711179 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6329519872}
I20260709 12:25:35.719686 18096 rpcz_store.cc:276] 0709 12:25:34.483748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.688538 (+1204790us) service_pool.cc:224] Handling call
0709 12:25:35.689093 (+   555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.692668 (+  3575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.692988 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.727315 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.729995 26900 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.730232 26875 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.731289 26900 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.672541 26984 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:35.741693 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.731352 26875 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.741895 18173 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:35.739022 18167 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:25:35.742853 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:35.743386 26638 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:35.621073 26878 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.655645 27034 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.758783 18095 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260709 12:25:35.757529 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:35.817690 26853 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.818053 27010 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.818648 26853 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.818804 27010 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.822479 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.824498 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.822117 26976 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:35.826720 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:35.836027 26998 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:35.836716 26998 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:35.837195 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 94486.9us
I20260709 12:25:35.838124 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3555) took 1331 ms. Trace:
I20260709 12:25:35.838605 18095 rpcz_store.cc:276] 0709 12:25:34.506600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.747786 (+1241186us) service_pool.cc:224] Handling call
0709 12:25:35.756624 (+  8838us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.815648 (+ 59024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.838008 (+ 22360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.838742 26996 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.817884 18096 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 26 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:25:35.815299 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:35.748528 26770 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.857540 18168 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:35.736822 26873 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.863987 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3561) took 1285 ms. Trace:
I20260709 12:25:35.868302 18093 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:25:35.863454 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.867579 18094 rpcz_store.cc:276] 0709 12:25:34.579132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.756412 (+1177280us) service_pool.cc:224] Handling call
0709 12:25:35.818771 (+ 62359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.819089 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:35.819490 (+   401us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.863168 (+ 43678us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:35.863234 (+    66us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:35.863656 (+   422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.863760 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.863922 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2425,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":124,"thread_start_us":42928,"threads_started":1}
I20260709 12:25:35.811189 26983 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:35.869158 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4449) took 1160 ms. Trace:
I20260709 12:25:35.878033 26998 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.878767 18093 rpcz_store.cc:276] 0709 12:25:34.708822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.748496 (+1039674us) service_pool.cc:224] Handling call
0709 12:25:35.868014 (+119518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.868814 (+   800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.869083 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:35.873181 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61453us
I20260709 12:25:35.879528 26998 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.875270 26878 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:35.881495 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:35.882694 26853 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:35.887357 18172 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 19: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:35.890765 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.893703 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.894699 18168 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:35.897970 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.904251 18090 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:35.891717 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:35.913129 18096 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:35.863430 26907 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.914853 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 176852us
I20260709 12:25:35.915409 18098 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:35.923991 18099 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:35.882877 26996 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.927593 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:35.932137 18167 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:35.933369 18096 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:35.933444 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54639.1us
I20260709 12:25:35.935534 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.935837 18090 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:35.940760 26613 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:25:35.941757 26613 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:35.867370 26638 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.944060 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:35.946892 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 105329us
I20260709 12:25:35.944110 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:35.954020 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.955102 18167 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:25:35.956822 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.960203 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.961444 18168 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:35.914229 26770 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:25:35.967571 18173 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:35.968683 26873 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.965803 26907 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.970698 26875 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.972235 26907 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.973979 18164 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:35.975454 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3298) took 1241 ms. Trace:
I20260709 12:25:35.975864 18164 rpcz_store.cc:276] 0709 12:25:34.734263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.734423 (+   160us) service_pool.cc:224] Handling call
0709 12:25:35.975140 (+1240717us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:35.975364 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2512171904}
I20260709 12:25:35.977023 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:35.979230 26770 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:35.980018 26770 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:35.986688 26873 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:35.987864 26937 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:35.988250 26873 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:35.990219 26873 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:35.992741 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.993777 18167 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
W20260709 12:25:35.986850 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49618.6us
I20260709 12:25:35.939183 26983 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:35.997176 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:35.988517 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.977385 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:35.999395 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.007438 26900 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:35.943068 26853 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:25:36.008239 26875 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:36.008486 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.008776 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:36.009089 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.008953 26875 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:36.009356 26853 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:36.010097 26990 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:36.010460 26875 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:36.011994 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.000175 26976 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:36.020375 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.027159 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:36.027400 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.028592 18169 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:36.021970 18091 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:36.033521 18167 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:36.036096 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.043164 18023 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:36.053398 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 109509us
I20260709 12:25:36.037441 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.022920 26990 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.063292 18172 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:36.064215 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:36.014410 26853 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:36.065672 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99532.7us
I20260709 12:25:36.029855 26613 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:36.069375 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.069746 18168 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:36.071902 18093 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:36.071086 18018 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:36.084358 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.085711 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:36.086381 27080 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:25:36.087761 27034 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:36.106038 26970 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.655s	user 0.009s	sys 0.009s
I20260709 12:25:36.067049 26990 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:36.110180 17989 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.501s (SENT)
I20260709 12:25:36.110935 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:36.111809 27076 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:36.113080 27076 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:36.113720 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:36.116555 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1758) took 2758 ms. Trace:
W20260709 12:25:36.069203 27073 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.080s	user 0.000s	sys 0.002s
W20260709 12:25:36.118710 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.113744 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.117259 18814 rpcz_store.cc:276] 0709 12:25:33.358073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.358849 (+   776us) service_pool.cc:224] Handling call
0709 12:25:35.598094 (+2239245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.621859 (+ 23765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.116160 (+494301us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54005901d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:36.116450 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5141080960}
W20260709 12:25:36.126480 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 245088us
I20260709 12:25:36.070663 26613 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.132566 18023 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:36.134793 18167 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:36.140288 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.136827 26542 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.006s
W20260709 12:25:36.142488 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59723.8us
I20260709 12:25:36.138471 26613 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:36.146927 18017 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:36.147755 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:36.149449 26866 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.585s	user 0.016s	sys 0.000s
I20260709 12:25:36.150151 26866 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.151803 18167 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:36.152277 18167 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:36.154114 18090 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:25:36.154436 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.153328 18167 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.160152 26770 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:36.164031 18097 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:36.158084 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:36.159654 18018 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:36.155293 18015 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:36.164535 26638 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:36.157524 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:36.165251 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:36.167018 18172 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:36.172756 18095 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:36.143348 26970 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:36.176452 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:36.153468 26866 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:36.134290 27076 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:36.177757 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.195330 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:36.196627 18023 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:36.188043 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50385.1us
W20260709 12:25:36.200177 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.204159 18173 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:36.206257 18173 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:36.208076 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:36.226477 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48773us
I20260709 12:25:36.227885 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.231943 18099 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:25:36.136626 27080 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:25:36.182338 26542 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:36.241664 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1387) took 1757 ms (client timeout 1614 ms). Trace:
I20260709 12:25:36.242190 26853 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.243561 26853 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.247671 18172 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:36.249470 18172 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:36.249975 26841 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.575s	user 0.007s	sys 0.008s
I20260709 12:25:36.250260 26970 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:36.266785 18167 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:36.194051 27076 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:36.242380 18092 rpcz_store.cc:269] 0709 12:25:34.480567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.539598 (+ 59031us) service_pool.cc:224] Handling call
0709 12:25:36.208678 (+1669080us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54002959d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:36.237841 (+ 29163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":844168960}
W20260709 12:25:36.256887 26986 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.548s	user 0.015s	sys 0.000s
I20260709 12:25:36.257920 26866 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:36.258570 27077 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:36.237172 26983 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.20, Last appended by leader: 20, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:36.259486 27072 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.209s	user 0.000s	sys 0.002s
I20260709 12:25:36.240164 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:36.241770 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106706us
I20260709 12:25:36.266965 18168 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:36.268498 18164 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260709 12:25:36.252549 26831 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.000s
I20260709 12:25:36.293296 18173 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:25:36.293334 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 53256.2us
I20260709 12:25:36.294157 18173 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:36.279141 26866 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.279475 27077 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.300482 26866 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.300312 26899 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.005s
I20260709 12:25:36.303205 26542 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:36.326470 18023 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:36.304219 27077 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:36.291944 26831 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.325086 26542 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.307705 26990 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.337055 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:36.305892 26970 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.341270 18173 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:36.347774 26984 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.601s	user 0.015s	sys 0.000s
I20260709 12:25:36.351533 27079 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.352345 27079 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.346516 26542 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:36.355567 26516 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.839s	user 0.014s	sys 0.003s
I20260709 12:25:36.308185 26964 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:36.387512 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.398814 18093 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:36.370781 26984 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.405212 18173 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:36.408829 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.410364 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4459) took 1247 ms. Trace:
I20260709 12:25:36.414007 18095 rpcz_store.cc:276] 0709 12:25:35.162964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.931887 (+768923us) service_pool.cc:224] Handling call
0709 12:25:36.167279 (+235392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.178945 (+ 11666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.401754 (+222809us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400280350. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:36.410234 (+  8480us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515804928}
W20260709 12:25:36.427004 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121303us
I20260709 12:25:36.401115 27086 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:36.436914 27086 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.440388 27086 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:36.428967 26984 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.442507 26964 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.443187 26831 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.444007 27082 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:36.444639 26984 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.444751 27082 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:36.445940 27085 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 12:25:36.446126 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 133215us
I20260709 12:25:36.446252 27082 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.447882 26841 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.448577 26841 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.450846 18807 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:36.451197 26841 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:36.452467 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:36.454651 26990 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.458949 27079 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.464073 18811 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:25:36.449411 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.466867 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:36.468012 18810 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:36.456440 26970 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.474547 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:36.486637 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4456) took 1764 ms. Trace:
W20260709 12:25:36.478705 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.481067 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3440) took 1568 ms. Trace:
I20260709 12:25:36.483249 26613 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.476112 26443 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.497555 18173 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:25:36.497941 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3305) took 1607 ms. Trace:
I20260709 12:25:36.493198 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.498508 18165 rpcz_store.cc:276] 0709 12:25:34.890568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.935637 (+ 45069us) service_pool.cc:224] Handling call
0709 12:25:34.940767 (+  5130us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.941104 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:35.305211 (+364107us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.497173 (+1191962us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.497302 (+   129us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:36.497390 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.497492 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.497696 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1272207,"prepare.queue_time_us":285726,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":103,"thread_start_us":204029,"threads_started":2,"wal-append.queue_time_us":181087}
I20260709 12:25:36.500093 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.500888 18173 pending_rounds.cc:85] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 11
I20260709 12:25:36.501324 18173 pending_rounds.cc:107] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:25:36.501590 26984 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:36.501737 18173 raft_consensus.cc:2889] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:36.502803 26984 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.503309 26984 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:36.503650 27079 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:36.504606 27079 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:36.490113 18166 rpcz_store.cc:276] 0709 12:25:34.912520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.944433 (+ 31913us) service_pool.cc:224] Handling call
0709 12:25:34.945138 (+   705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.945493 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:35.299491 (+353998us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.479954 (+1180463us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b1800310300. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:25:36.480220 (+   266us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.480335 (+   115us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:36.480442 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.480602 (+   160us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.480803 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":339716,"prepare.queue_time_us":281763,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":156,"spinlock_wait_cycles":11541888,"thread_start_us":115209,"threads_started":2,"wal-append.queue_time_us":138354}
I20260709 12:25:36.487221 18098 rpcz_store.cc:276] 0709 12:25:34.722291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.828694 (+1106403us) service_pool.cc:224] Handling call
0709 12:25:35.868018 (+ 39324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.922760 (+ 54742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.486273 (+563513us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b540011dd50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:36.486513 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65157504}
W20260709 12:25:36.491641 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.502225 27002 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.021s
W20260709 12:25:36.523479 27091 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:25:36.495857 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:36.533157 18167 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 12:25:36.533921 18167 pending_rounds.cc:85] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 14
I20260709 12:25:36.536571 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.487270 26866 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.541683 18167 pending_rounds.cc:107] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:25:36.542277 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:36.543737 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94492.1us
I20260709 12:25:36.548609 18167 raft_consensus.cc:2889] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:36.551149 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:36.559417 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123202us
I20260709 12:25:36.560164 18016 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:36.551534 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:36.544580 27002 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:36.562117 18020 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
W20260709 12:25:36.563539 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.560751 18019 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:36.571376 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3452) took 1247 ms. Trace:
I20260709 12:25:36.552121 26971 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.574488 18023 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
W20260709 12:25:36.579618 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.580268 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.577780 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3566) took 1439 ms (client timeout 1708 ms). Trace:
I20260709 12:25:36.574179 18167 rpcz_store.cc:276] 0709 12:25:35.323800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.151226 (+827426us) service_pool.cc:224] Handling call
0709 12:25:36.151602 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.571006 (+419404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.571291 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:36.583537 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:36.582492 18091 rpcz_store.cc:269] 0709 12:25:35.138727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.211906 (+ 73179us) service_pool.cc:224] Handling call
0709 12:25:36.577502 (+1365596us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b5400296dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:36.577687 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1836443520}
I20260709 12:25:36.594620 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:36.596551 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.564591 27002 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.553049 27079 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.575371 26998 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:36.611110 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47835.2us
I20260709 12:25:36.569993 27098 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:25:36.613420 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.614276 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.614451 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.615458 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:36.615752 18165 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: [23.11-23.11]
I20260709 12:25:36.632495 18168 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:36.616744 18172 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:36.618600 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:36.605976 26971 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.605855 26900 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:36.625994 27035 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:36.627904 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.20276e+06us
W20260709 12:25:36.649206 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54778.7us
I20260709 12:25:36.653407 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:36.661127 18167 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:36.607530 27002 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:36.662670 26945 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:25:36.687080 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 116746us
I20260709 12:25:36.689450 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:36.658481 26900 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:36.696353 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.719566 18168 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:36.656282 26998 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.691187 27086 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:36.723101 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.732375 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:36.704836 26900 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:36.724968 26714 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:36.743093 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.750007 26837 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:25:36.755009 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3313) took 1601 ms. Trace:
W20260709 12:25:36.756014 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.693598 27077 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.763269 18169 rpcz_store.cc:276] 0709 12:25:35.153218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.049663 (+896445us) service_pool.cc:224] Handling call
0709 12:25:36.073579 (+ 23916us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.122564 (+ 48985us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:36.150125 (+ 27561us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.695663 (+545538us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.719376 (+ 23713us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.754407 (+ 35031us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.754501 (+    94us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:36.754598 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.754707 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.754907 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":384491,"prepare.queue_time_us":140931,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":3966,"raft.queue_time_us":320184,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":15232,"thread_start_us":220549,"threads_started":2,"wal-append.queue_time_us":242739}
I20260709 12:25:36.775784 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3438) took 1745 ms. Trace:
I20260709 12:25:36.754762 27009 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.764966 27086 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:36.779578 18171 rpcz_store.cc:276] 0709 12:25:34.841295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.927692 (+ 86397us) service_pool.cc:224] Handling call
0709 12:25:34.928195 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.928534 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:35.250098 (+321564us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.448166 (+198068us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.637446 (+189280us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b1800386d40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:35.667792 (+ 30346us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:36.534554 (+866762us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.545266 (+ 10712us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:36.555406 (+ 10140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.568129 (+ 12723us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.586467 (+ 18338us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1106057,"prepare.queue_time_us":294844,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":147,"raft.queue_time_us":1114883,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":9450,"spinlock_wait_cycles":21469824,"thread_start_us":77170,"threads_started":2,"wal-append.queue_time_us":167570,"wal-append.run_cpu_time_us":7542,"wal-append.run_wall_time_us":1239706}
I20260709 12:25:36.805081 26970 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260709 12:25:36.823094 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.831799 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3437) took 1982 ms. Trace:
W20260709 12:25:36.834012 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.835423 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.838315 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.838344 26663 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:36.840505 26663 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:36.805176 26841 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:25:36.841549 26841 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:36.842033 26841 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:36.841692 26663 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.843142 26841 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.844597 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3576) took 1228 ms. Trace:
I20260709 12:25:36.844698 26841 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:36.819582 27088 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260709 12:25:36.850304 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 137884us
I20260709 12:25:36.850683 26998 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.851516 26998 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:36.852535 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:36.859215 27077 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:36.860533 27086 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.861907 26983 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.862934 26983 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.863909 26542 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:36.865653 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 198175us
I20260709 12:25:36.845189 18090 rpcz_store.cc:276] 0709 12:25:35.615679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.987184 (+371505us) service_pool.cc:224] Handling call
0709 12:25:35.987653 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.154634 (+166981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.844220 (+689586us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54004119d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:36.844510 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381964672}
I20260709 12:25:36.873777 18164 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:36.882877 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62240.6us
I20260709 12:25:36.813093 27098 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:36.795879 27009 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:36.892171 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 162890us
I20260709 12:25:36.895577 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.876045 26837 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:36.904521 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:36.908274 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55952.7us
W20260709 12:25:36.910661 17907 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 910f8c8d787e4abbb092413c1f90ebf3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:36.912911 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:36.913159 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.913918 18167 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:36.914628 18173 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:36.944346 27120 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:36.832545 18170 rpcz_store.cc:276] 0709 12:25:34.830873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.914110 (+ 83237us) service_pool.cc:224] Handling call
0709 12:25:34.915883 (+  1773us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.916144 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:35.288344 (+372200us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.551460 (+1263116us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.696649 (+145189us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.725801 (+ 29152us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.768096 (+ 42295us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.775826 (+  7730us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:36.785873 (+ 10047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.802777 (+ 16904us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.812888 (+ 10111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1206611,"prepare.queue_time_us":331859,"prepare.run_cpu_time_us":280,"prepare.run_wall_time_us":289,"raft.queue_time_us":1222376,"raft.run_cpu_time_us":542,"raft.run_wall_time_us":35853,"spinlock_wait_cycles":36347392,"thread_start_us":568034,"threads_started":3,"wal-append.queue_time_us":444209,"wal-append.run_cpu_time_us":1795,"wal-append.run_wall_time_us":1001798}
I20260709 12:25:36.946130 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.950682 18169 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: [17.14-19.15]
I20260709 12:25:36.904012 26837 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.958101 26837 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:36.953629 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:36.954952 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.956475 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:36.965997 18780 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.543s (SENT)
W20260709 12:25:36.949131 27094 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.264s	user 0.002s	sys 0.000s
I20260709 12:25:36.964497 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.917397 26971 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:36.900683 26984 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:36.976307 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:36.977631 27076 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:36.978840 26991 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:36.979828 26821 raft_consensus.cc:2764] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election decision vote started in defunct term 24: won
I20260709 12:25:36.985427 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:36.987484 18095 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:36.989148 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:36.990675 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:36.992188 18170 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:25:36.993453 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:36.998899 18172 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:36.965457 26983 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.003593 18091 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:37.004324 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 137855us
I20260709 12:25:36.969030 27113 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:36.982496 26970 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:37.018690 27076 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:37.052088 26970 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:37.018690 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:37.053411 26970 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:37.022502 27104 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
W20260709 12:25:37.026018 27103 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.273s	user 0.002s	sys 0.000s
I20260709 12:25:37.026535 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:37.027201 27112 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:25:37.027582 17988 leader_election.cc:336] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.929s (SENT)
I20260709 12:25:37.030609 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.031451 26984 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:37.032156 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66200.1us
W20260709 12:25:37.064368 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:37.033414 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77049us
I20260709 12:25:37.035804 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:36.991875 27035 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:37.036971 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 130443us
W20260709 12:25:37.037667 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 128257us
I20260709 12:25:37.040998 27010 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.041566 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.045146 27009 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:37.046084 26443 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.520s	user 0.014s	sys 0.002s
I20260709 12:25:37.013523 27098 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:37.048612 27081 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.625s	user 0.004s	sys 0.000s
I20260709 12:25:36.991648 26991 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:37.091238 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:37.095134 18173 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:37.098831 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.099189 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:37.096428 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.101764 18815 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 25 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:25:37.101799 18172 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:37.102912 18164 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:25:37.097329 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:37.087476 26878 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:37.118422 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.121956 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:37.122884 18172 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:37.116133 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:37.131107 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:37.145649 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53647.8us
I20260709 12:25:37.147663 27116 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:37.148545 27116 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:37.117462 26966 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.150673 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.150558 26966 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.151927 18810 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:25:37.101620 26714 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:37.159621 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.159749 26714 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.160681 18806 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:37.161446 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.161803 18137 leader_election.cc:400] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:37.162326 18137 leader_election.cc:403] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
W20260709 12:25:37.162952 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62357.2us
I20260709 12:25:37.164981 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:37.166177 27035 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
W20260709 12:25:37.168044 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 132358us
I20260709 12:25:37.172827 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:37.173337 18022 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:25:37.173731 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:37.173744 18016 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:37.174530 18015 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.174669 18017 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 12:25:37.178397 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:37.179214 18023 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:37.182418 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:37.183295 18018 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:37.187739 26991 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.226648 18172 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:37.236893 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135980us
I20260709 12:25:37.172906 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.250844 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:37.259852 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 167890us
I20260709 12:25:37.201226 26443 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:37.268554 18806 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
W20260709 12:25:37.250949 27002 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.001s
I20260709 12:25:37.266821 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:37.269656 18020 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.270011 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:37.270834 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 358017us
I20260709 12:25:37.266484 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:37.272307 18016 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:25:37.265328 18023 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
W20260709 12:25:37.273491 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.266408 26878 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:37.252401 27035 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:37.274205 26945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.005s	sys 0.026s
I20260709 12:25:37.284829 18166 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
W20260709 12:25:37.282245 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 117390us
I20260709 12:25:37.279608 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.278029 18173 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:37.298543 18810 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:37.302644 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.305286 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:37.309677 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 136908us
W20260709 12:25:37.310122 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.149659 27116 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 26.22, Last appended by leader: 22, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.275434 27002 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:37.367424 27101 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.667s	user 0.006s	sys 0.003s
I20260709 12:25:37.368912 27002 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.318182 26542 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.318572 18813 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:25:37.319263 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 81268.9us
I20260709 12:25:37.323320 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:37.326289 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.326808 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:37.327157 18171 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:37.319291 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.385493 18171 raft_consensus.cc:606] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.386070 18171 raft_consensus.cc:614] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:25:37.385948 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 196961us
I20260709 12:25:37.319307 26796 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:37.330300 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.330463 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.387676 18808 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:37.331547 18093 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1391) took 1286 ms (client timeout 1641 ms). Trace:
I20260709 12:25:37.334293 18016 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:37.344784 18018 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
W20260709 12:25:37.345584 27102 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.632s	user 0.005s	sys 0.003s
W20260709 12:25:37.358142 27096 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.752s	user 0.004s	sys 0.005s
I20260709 12:25:37.369568 18094 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:37.372062 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.282028 27124 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.282410 27009 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:37.315403 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 43830.2us
I20260709 12:25:37.373063 18020 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:25:37.288225 26945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.006s	sys 0.027s
I20260709 12:25:37.395159 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:37.388386 18093 rpcz_store.cc:269] 0709 12:25:36.046587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.072937 (+ 26350us) service_pool.cc:224] Handling call
0709 12:25:37.331298 (+1258361us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:37.331460 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":546672000}
I20260709 12:25:37.387584 18166 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:25:37.392045 18806 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:37.381204 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:37.393725 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67072.5us
I20260709 12:25:37.399042 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.389520 26796 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:37.395882 26900 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.619s	user 0.012s	sys 0.002s
I20260709 12:25:37.407042 26987 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:37.407810 26987 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:37.409636 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:37.410503 18170 raft_consensus.cc:606] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.414304 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1621) took 1144 ms. Trace:
I20260709 12:25:37.414662 18170 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:37.415323 18170 raft_consensus.cc:606] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.415632 18015 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:37.416347 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.417340 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.415814 18170 raft_consensus.cc:614] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:37.394431 27009 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.404927 26796 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:37.428736 27010 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:37.429517 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 106354us
I20260709 12:25:37.429293 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:37.396757 26945 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:37.398561 27002 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.414870 18814 rpcz_store.cc:276] 0709 12:25:36.270086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.270307 (+   221us) service_pool.cc:224] Handling call
0709 12:25:36.281258 (+ 10951us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.359038 (+ 77780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:36.359091 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.409320 (+ 50229us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.555208 (+145888us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:36.555261 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.556725 (+  1464us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:37.329301 (+772576us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:37.413917 (+ 84616us) raft_consensus.cc:1685] finished
0709 12:25:37.413947 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.414198 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":481,"raft.queue_time_us":17661,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":118,"spinlock_wait_cycles":9682688,"thread_start_us":179861,"threads_started":2,"wal-append.queue_time_us":226902}
I20260709 12:25:37.437949 27010 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.440615 27010 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.441180 27034 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:37.441687 27034 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:37.442179 27034 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.442116 27035 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:37.443646 27035 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:37.444602 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.445326 27035 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:37.446151 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.445384 18813 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:37.405172 26878 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:37.418277 27082 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:37.452139 18091 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:37.452553 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69427.3us
I20260709 12:25:37.453939 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:37.427368 27077 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.460789 18812 raft_consensus.cc:2410] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 26 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:37.418869 26663 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:37.433939 27009 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:37.476693 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92027.9us
I20260709 12:25:37.477857 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.478937 18093 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:25:37.480351 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:37.480763 18090 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:25:37.446003 26983 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.400718 26542 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:25:37.409307 26987 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.483362 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.485085 18092 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:37.491222 26907 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:37.492564 26663 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:37.493093 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1608) took 2109 ms (client timeout 2147 ms). Trace:
W20260709 12:25:37.493588 18815 rpcz_store.cc:269] 0709 12:25:35.383432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.385649 (+  2217us) service_pool.cc:224] Handling call
0709 12:25:37.492854 (+2107205us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:37.493021 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3842398464}
I20260709 12:25:37.495150 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.495910 18815 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:25:37.496073 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.497685 18814 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:25:37.493803 26663 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 24.20, Last appended by leader: 20, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.452622 27035 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:37.497781 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.428076 26900 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.428025 26837 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.508205 18808 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:37.398803 27124 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.612476 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.565915 27035 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.569198 26900 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:37.593174 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 129661us
I20260709 12:25:37.561575 18806 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:37.538717 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:37.582190 26990 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.658259 18165 raft_consensus.cc:606] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
W20260709 12:25:37.567842 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172487us
I20260709 12:25:37.660266 26990 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.567040 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:37.567921 27119 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:37.608623 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 154803us
I20260709 12:25:37.449199 27002 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.513135 27077 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.500835 26970 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:37.530215 26984 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:37.526301 27109 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:37.526851 26907 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:37.511842 26983 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.749040 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.749979 18091 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:37.472139 26964 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:25:37.613623 18170 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:37.606002 26837 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.754926 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.615892 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:37.615906 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.756168 18171 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:37.592428 26945 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.758411 18097 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:37.627540 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:37.759498 18095 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:37.630761 27076 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.005s
I20260709 12:25:37.636788 26998 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:37.640836 27124 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.649384 27105 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:37.770051 27105 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:37.770506 27105 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:37.771519 27105 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:37.773247 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 157437us
I20260709 12:25:37.773860 27077 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.774343 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.776125 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:37.775817 27002 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.649842 26821 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:37.650326 27113 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.778613 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.779547 18094 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:37.779269 27113 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.779624 26990 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.652554 27116 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.782251 27116 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.649407 26966 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.660486 26878 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:25:37.661772 26811 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.014s
W20260709 12:25:37.738489 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64846.4us
W20260709 12:25:37.739905 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 172947us
W20260709 12:25:37.743811 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:37.747219 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98309.3us
W20260709 12:25:37.747327 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 161623us
I20260709 12:25:37.788357 26831 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:37.791079 27077 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:37.791502 18095 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:37.793129 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.793640 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:37.793967 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.794445 27105 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:37.795020 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:37.795174 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:37.796095 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.796947 26853 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:37.797448 26853 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.797658 26900 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:37.797863 26853 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:37.798142 27113 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:37.798363 26900 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:37.764762 27035 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.799510 26900 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.800731 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.802294 26811 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:37.802906 26811 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:37.803386 27119 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:37.746518 26984 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.754155 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.615106 18090 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:37.755966 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.601871 26796 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.757398 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4460) took 2587 ms. Trace:
I20260709 12:25:37.753086 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.811149 18807 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:37.625900 27009 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:37.758404 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.739367 27120 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:37.765401 18092 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:25:37.738188 27134 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.739365 26970 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:37.768214 18809 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:37.673585 27034 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.761376 27135 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:25:37.652681 26991 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:37.785995 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.828562 18097 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 16.
I20260709 12:25:37.802372 27080 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:37.804782 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:37.808702 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.767829 27109 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:37.831007 18094 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:37.818750 18095 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:37.757326 26899 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.049s
W20260709 12:25:37.847154 27106 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:25:37.766044 26998 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:37.803951 26811 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.816517 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3582) took 1294 ms. Trace:
I20260709 12:25:37.848675 18098 rpcz_store.cc:276] 0709 12:25:36.522019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.742334 (+1220315us) service_pool.cc:224] Handling call
0709 12:25:37.816433 (+ 74099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.849218 26811 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:37.850036 18097 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:37.783427 26983 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.850517 27116 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.830054 26970 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:37.845166 26907 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:37.856897 27034 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:37.858196 26613 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.859367 27080 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:37.860131 26983 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.860575 27080 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.860761 26983 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.819463 26837 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:37.863363 27138 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.864178 27138 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:37.866022 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.844338 26990 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:37.866730 26990 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.867178 26990 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:37.868299 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67683.8us
I20260709 12:25:37.868494 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:37.869270 18097 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:37.848980 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98142.7us
I20260709 12:25:37.870271 26967 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:37.870167 26821 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:37.834309 27109 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:37.871766 27109 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.872110 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.827766 27134 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.874228 26837 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:37.875361 27079 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:37.876076 27079 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:37.877218 27079 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:37.826527 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 174711us
I20260709 12:25:37.835307 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.879720 18099 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:37.880369 18093 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:37.837314 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:37.881363 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44140.9us
I20260709 12:25:37.881659 26613 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:37.882318 27009 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:37.804455 26866 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.812459 26984 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.819613 27120 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.883461 27120 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:37.810491 18096 rpcz_store.cc:276] 0709 12:25:35.163607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.938608 (+775001us) service_pool.cc:224] Handling call
0709 12:25:35.938960 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.978778 (+ 39818us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:36.176699 (+197921us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.423893 (+247194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.423966 (+    73us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:36.424023 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.357353 (+933330us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:37.750842 (+393489us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:37.750918 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.751054 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15058,"prepare.queue_time_us":40842,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":92,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":6030,"spinlock_wait_cycles":316507392,"thread_start_us":337648,"threads_started":2,"wal-append.queue_time_us":326049}
I20260709 12:25:37.886180 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.882467 26613 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:37.829735 26878 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:37.906101 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.908602 26613 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.906272 26837 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:37.910841 26983 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.909693 18813 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
W20260709 12:25:37.750388 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75390.3us
I20260709 12:25:37.895213 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.907672 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:37.907274 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.901274 18807 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
W20260709 12:25:37.916347 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.926622 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:37.928596 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:37.929618 18063 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:37.869247 26966 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.071s
I20260709 12:25:37.947091 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.948165 18815 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:37.953059 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:37.954033 18812 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:37.905701 26796 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:37.959215 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.969166 18807 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 25 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
W20260709 12:25:37.972440 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45909.4us
W20260709 12:25:37.973031 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 103447us
I20260709 12:25:37.974673 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:37.934027 18097 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:37.960354 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:37.977939 18093 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:37.961012 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.979313 18815 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:25:37.947979 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:37.946012 26986 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:37.983876 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76680.5us
I20260709 12:25:37.981210 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:37.907488 27135 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:25:37.949153 26837 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:37.993083 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:37.959829 26875 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:37.951092 26866 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:37.980979 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:37.995026 26811 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:37.995864 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:37.959199 27113 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:37.999931 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75299us
I20260709 12:25:37.908339 26998 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:37.951021 27119 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:25:37.962657 27009 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:25:38.022185 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:37.993561 26991 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:38.040792 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:38.028584 26866 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:38.028957 27133 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:38.041400 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48452.6us
I20260709 12:25:38.004096 27116 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:38.037530 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:38.049491 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51432.8us
I20260709 12:25:38.001801 26986 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:38.005795 27010 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.073519 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:38.068522 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93932.2us
I20260709 12:25:37.986618 27082 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:25:38.034075 26811 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:38.072645 26866 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:38.045958 26907 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:38.085250 18097 raft_consensus.cc:1217] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Deduplicated request from leader. Original: 17.10->[19.11-23.13]   Dedup: 19.11->[21.12-23.13]
I20260709 12:25:38.087356 18097 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [19.11-23.13]
I20260709 12:25:38.070818 27077 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:38.077291 27124 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:38.072866 26899 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:38.043020 26971 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:25:38.098138 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:38.104099 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.084460 26937 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:38.084793 27119 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:38.124567 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.126251 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.127214 18015 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:38.136502 18064 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:38.099794 26837 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:38.097054 27077 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.135641 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.153873 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:38.150035 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.073714 27133 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.153728 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:38.163928 18093 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:38.145973 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:38.164945 18022 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:38.166366 18170 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 27 index: 23. (index mismatch)
W20260709 12:25:38.168326 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108441us
I20260709 12:25:38.169581 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:38.119544 26875 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:38.076964 27010 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.143587 26983 raft_consensus.cc:2764] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:25:38.177315 18016 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:38.183619 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:38.107328 26796 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:25:38.184396 18807 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:38.145360 27135 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:38.163619 27077 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:38.144804 26837 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:38.181913 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.131799 26984 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.119153 27124 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.195124 18092 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:38.136837 26853 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:38.200413 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123413us
I20260709 12:25:38.198869 18093 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:38.202883 18015 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:25:38.182032 26971 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:38.212029 27124 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:38.214130 27145 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
W20260709 12:25:38.214435 27140 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 12:25:38.215121 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1623) took 1749 ms (client timeout 2141 ms). Trace:
I20260709 12:25:38.182981 27113 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:38.215592 18809 rpcz_store.cc:269] 0709 12:25:36.466047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.474085 (+  8038us) service_pool.cc:224] Handling call
0709 12:25:38.214770 (+1740685us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400590e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:38.214987 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2945824640}
I20260709 12:25:38.216827 26875 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.188868 26811 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.224190 27010 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.226115 27134 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:38.226687 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.227707 27134 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:38.228576 26853 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.229211 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:38.229394 26983 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:38.230613 26983 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:38.231817 26984 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.232141 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.233111 26984 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:38.233670 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.233392 26837 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.234665 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.235610 26984 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:38.235496 27109 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.190869 26866 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.216006 27113 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:38.214109 18807 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:38.243018 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.234855 26837 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:38.251492 26990 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.253155 27138 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.255033 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:38.246657 27146 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:25:38.247172 18808 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260709 12:25:38.269857 18808 pending_rounds.cc:85] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 15
I20260709 12:25:38.270268 18808 pending_rounds.cc:107] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:25:38.270525 18808 raft_consensus.cc:2889] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:38.267161 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:38.259661 18812 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:38.217675 18809 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 19. (term mismatch)
I20260709 12:25:38.280646 18809 pending_rounds.cc:85] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 18
I20260709 12:25:38.266316 18092 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:38.176136 26986 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.288580 26990 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:38.290138 27144 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:25:38.290566 26663 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:38.291056 26663 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:38.292044 26663 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.293215 27109 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.295665 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:38.296679 18806 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:38.306761 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.307801 27002 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:38.309896 27002 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:38.310339 27002 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:38.310715 27002 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:38.311493 27120 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.281260 18809 pending_rounds.cc:107] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 23.19
I20260709 12:25:38.287117 27152 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:38.289975 26542 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:25:38.274344 27143 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
W20260709 12:25:38.283838 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108276us
I20260709 12:25:38.286460 27151 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:38.297670 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:38.249665 26998 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:38.323066 18809 raft_consensus.cc:2889] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:38.316525 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49544.4us
I20260709 12:25:38.329432 26991 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:38.293236 26663 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:38.330705 26853 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:38.322466 26542 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:38.360713 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.361744 18015 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 10.
I20260709 12:25:38.362753 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.343221 27116 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:38.386144 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.389803 18022 raft_consensus.cc:2368] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:38.349424 27151 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:38.402211 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.404793 18016 raft_consensus.cc:2368] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29.
I20260709 12:25:38.360320 26853 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.407866 18093 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:25:38.360562 26983 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.410435 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:38.425315 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.359519 27156 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:38.406570 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:38.489641 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43493.8us
I20260709 12:25:38.459167 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.350363 27109 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
W20260709 12:25:38.495210 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 148226us
I20260709 12:25:38.491078 18020 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:38.402333 27080 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.499807 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.493372 18022 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:38.494597 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.501343 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.503703 18096 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: [17.14-19.15]
I20260709 12:25:38.505971 18098 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:25:38.446753 26853 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:38.517521 18811 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1774) took 1363 ms (client timeout 1750 ms). Trace:
I20260709 12:25:38.518007 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.518963 18164 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:38.509450 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:38.520519 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.521371 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:38.448266 26964 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:38.522449 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:38.512223 18171 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:25:38.521555 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.523900 18170 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:38.522179 18168 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:38.513736 18096 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:38.515880 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.525424 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.525333 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:38.526409 18172 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
W20260709 12:25:38.518353 18811 rpcz_store.cc:269] 0709 12:25:37.154591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.161052 (+  6461us) service_pool.cc:224] Handling call
0709 12:25:38.512824 (+1351772us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b540059fe54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:38.517450 (+  4626us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1938716928}
I20260709 12:25:38.528080 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:38.528818 18166 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:38.531529 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.533025 18173 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:38.533536 18173 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:38.534767 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.535440 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.528582 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.486954 26811 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:38.543855 26811 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:38.545980 27158 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:25:38.546525 26811 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:38.524824 27034 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:38.547629 27034 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.537985 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:38.552047 27034 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:38.542363 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:38.523070 27102 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:25:38.554440 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:38.555209 18166 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:38.552867 18173 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.556110 18173 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:38.543107 26971 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:38.541242 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.508075 27156 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.557655 18022 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 24 because replica has last-logged OpId of term: 23 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 10.
I20260709 12:25:38.560120 27156 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.563273 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 97) took 1067 ms. Trace:
I20260709 12:25:38.563762 18165 rpcz_store.cc:276] 0709 12:25:37.495506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.515987 (+ 20481us) service_pool.cc:224] Handling call
0709 12:25:37.927138 (+411151us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000f94a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c16634520 00007f6c166b1652 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:38.562870 (+635732us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b54000f94a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166325e7 00007f6c166b1c68 00007f6c166b184a 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c
0709 12:25:38.563181 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508929792}
W20260709 12:25:38.552430 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.570197 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:25:38.570392 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.554461 18169 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:38.579381 18173 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260709 12:25:38.579591 18098 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:38.580359 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71067.5us
I20260709 12:25:38.581934 26878 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:38.587285 26878 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:38.588493 26878 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.590937 27149 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.591604 27149 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.593055 27126 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:38.593538 27126 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:38.594112 27151 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.594465 27126 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.595741 27126 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:38.596191 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:38.597941 26878 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:38.602188 27133 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:38.598075 26542 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.603638 26542 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:38.605224 27149 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:38.610250 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.611856 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:38.538102 27080 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.614328 27080 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:38.616923 26991 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:38.617399 26945 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:38.618727 26991 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:38.619860 27076 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:38.620158 26907 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:38.620659 26907 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:38.621760 26907 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.622390 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:38.623533 26907 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:38.626183 27076 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:38.627316 27076 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:38.627831 27076 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:38.628563 27150 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:38.629037 27150 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:38.633213 26971 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:38.635399 18173 pending_rounds.cc:85] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 14
I20260709 12:25:38.636207 18173 pending_rounds.cc:107] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:25:38.636987 18173 raft_consensus.cc:2889] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:38.637066 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.639611 18168 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:25:38.583227 27161 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:25:38.642020 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:38.637697 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.650787 18170 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:38.655975 18016 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:38.662536 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.506305 27120 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:38.664381 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.671012 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:38.671813 18813 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:38.672251 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:38.673200 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:38.674728 18806 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: [25.18-27.19]
I20260709 12:25:38.675853 18018 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:25:38.676903 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.685240 18020 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:38.677881 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:38.686739 18811 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:38.688321 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.669526 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:38.694478 18167 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:38.696399 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:38.697379 18809 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:38.709426 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:38.710544 18098 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:38.721717 18814 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:25:38.663288 26837 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.725067 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.663167 27116 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.663457 27034 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:38.735462 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:38.714265 26970 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.863s	user 0.012s	sys 0.003s
I20260709 12:25:38.745851 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.751248 18806 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:38.752367 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.750690 27034 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.761399 27034 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:38.769052 18024 raft_consensus.cc:2368] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:38.693128 27148 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.105s
I20260709 12:25:38.540262 26964 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.669364 27156 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.086s
I20260709 12:25:38.666250 26987 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.088s
I20260709 12:25:38.806437 26987 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:38.793744 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:38.628842 27076 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.721279 26986 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:25:38.725975 27138 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 12:25:38.830565 18806 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:25:38.832996 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.748381 27124 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
W20260709 12:25:38.770714 26984 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.535s	user 0.013s	sys 0.000s
W20260709 12:25:38.852187 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.755148 27116 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.807000 26987 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:38.749495 26837 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.824862 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:38.862244 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.840240 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:38.859823 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.680681 27150 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:38.779938 26970 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.753082 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:38.878352 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.879590 18090 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:38.504766 26983 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:38.892730 18096 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:38.877456 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:38.894079 18091 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:38.895150 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.877100 18094 raft_consensus.cc:2376] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18. Re-sending same reply.
I20260709 12:25:38.906883 18022 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:38.907213 18090 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:38.912931 18021 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:38.876227 26984 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:38.925091 18020 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:38.916601 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:38.930125 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.938441 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:38.970655 18024 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:38.941537 18099 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:38.942785 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:38.945989 18090 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 27 index: 23. (index mismatch)
I20260709 12:25:38.907069 26967 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 12:25:38.856523 26875 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:25:38.973728 18811 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
W20260709 12:25:38.994706 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78233.4us
I20260709 12:25:38.995075 18020 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:25:38.996215 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:38.949671 27119 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.810s	user 0.016s	sys 0.000s
I20260709 12:25:39.012108 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:39.019116 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:39.024520 18090 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:39.030993 18019 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:39.033659 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:39.037855 18812 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:25:39.038465 18812 pending_rounds.cc:85] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 13
I20260709 12:25:39.038739 18812 pending_rounds.cc:107] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:25:39.039063 18812 raft_consensus.cc:2889] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:39.039445 18812 pending_rounds.cc:107] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:25:39.039724 18812 raft_consensus.cc:2889] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:39.002632 27124 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:39.003489 27105 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:39.052089 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.050495 18167 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:39.010751 26967 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.054560 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.012090 27119 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:39.063015 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.022289 26991 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:39.077766 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.032038 27157 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:39.050341 26966 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:39.087863 18021 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
W20260709 12:25:39.056070 27077 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.857s	user 0.010s	sys 0.007s
I20260709 12:25:39.096133 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:39.065413 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:39.051978 27105 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:39.016033 27098 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.068s
I20260709 12:25:39.079869 27148 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:39.115152 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.109782 18167 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:39.092314 26899 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:25:39.117640 26899 consensus_queue.cc:1243] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:39.119747 18171 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:39.033958 27138 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.060s
W20260709 12:25:39.095202 27002 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.784s	user 0.011s	sys 0.006s
I20260709 12:25:39.091933 27133 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:39.009173 26987 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.132501 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.133379 18169 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:25:39.133778 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54262us
I20260709 12:25:39.134562 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.127986 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:39.092304 27134 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:39.071640 26967 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.135308 18165 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:39.130174 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.138424 18171 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:39.135926 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.140883 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.141696 18169 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:39.149452 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.146968 18166 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:39.152791 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:39.157474 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.158594 18165 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:39.125058 27157 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:39.139482 26542 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.535s	user 0.016s	sys 0.004s
W20260709 12:25:39.160703 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95405.5us
I20260709 12:25:39.103029 26875 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:25:39.166839 18168 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:39.100205 27077 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.191829 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.150139 26837 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:39.209532 27173 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
W20260709 12:25:39.211510 27177 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:25:39.133500 27002 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.230088 18097 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:39.235826 18018 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:39.163904 26983 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.187987 26970 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.191493 27184 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:39.248286 18098 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.003s
I20260709 12:25:39.253898 18098 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 17.
I20260709 12:25:39.163033 27148 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.068s
I20260709 12:25:39.255859 18018 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:39.190444 27181 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:25:39.260363 26967 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:39.267745 27113 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.029s	user 0.011s	sys 0.007s
I20260709 12:25:39.268484 27113 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.269934 18815 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:39.274065 27184 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.276993 27184 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:39.286948 27010 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.287714 27010 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.288810 27157 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.240833 27105 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.290660 27010 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.291100 27105 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:39.241371 26837 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
W20260709 12:25:39.293598 27126 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.007s
W20260709 12:25:39.260363 27174 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260709 12:25:39.297030 26866 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:39.252589 26964 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:39.297760 26866 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.299552 27098 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:39.301590 18170 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:39.302318 18022 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:39.251689 18813 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.532s	user 0.016s	sys 0.002s
I20260709 12:25:39.309578 18813 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:39.311017 18810 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:39.318701 18019 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:39.323644 18165 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
W20260709 12:25:39.327965 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 136226us
W20260709 12:25:39.331292 18809 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.618s	user 0.019s	sys 0.010s
I20260709 12:25:39.332154 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:39.337759 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.355469 18809 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:39.337940 18019 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:39.350766 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.354280 18171 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:39.364861 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:39.367287 18809 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:39.281560 27034 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:39.385234 18807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.011s	sys 0.014s
W20260709 12:25:39.385668 18807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.011s	sys 0.014s
W20260709 12:25:39.386493 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.389053 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:39.383908 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51900.1us
I20260709 12:25:39.392782 18169 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:39.349728 26964 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:39.415773 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.417748 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.421648 18165 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:39.419627 18019 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:39.426112 26964 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.388865 26945 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:39.388363 27156 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.428743 26964 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:39.428540 27156 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.430913 27156 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:39.431038 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.431767 18020 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:39.431885 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.432723 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.433148 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.433983 18017 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:39.434497 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:39.435081 18017 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:39.435997 18018 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260709 12:25:39.436337 18017 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:39.438035 27002 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.394001 27138 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.439296 27138 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.442632 26866 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:39.444077 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:39.450227 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.377964 27077 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:39.390908 26663 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.032s	user 0.007s	sys 0.017s
I20260709 12:25:39.451800 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1548) took 1344 ms. Trace:
I20260709 12:25:39.455658 18164 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:25:39.456146 18024 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:39.394455 27184 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.458324 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 371480us
I20260709 12:25:39.459489 18022 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [22.16-23.17]
I20260709 12:25:39.464677 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:39.460599 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:39.447151 26998 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.098s	user 0.017s	sys 0.002s
W20260709 12:25:39.419334 27169 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
I20260709 12:25:39.468407 18809 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:39.466885 18021 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:39.468940 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.469799 18019 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:39.462960 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.475337 18024 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:39.471803 18020 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:39.452891 27183 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:39.470293 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:39.462409 18023 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.009s	sys 0.019s
W20260709 12:25:39.487921 18023 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.009s	sys 0.019s
I20260709 12:25:39.422366 26991 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:25:39.462018 27138 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:39.491786 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:39.494408 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 101957us
I20260709 12:25:39.508720 18808 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:39.473335 27077 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:39.514171 27076 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.012s
I20260709 12:25:39.527081 27076 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:39.527737 27076 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:39.530269 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.477730 26866 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:25:39.522241 27188 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.018s
I20260709 12:25:39.535233 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.468132 27184 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.500685 27134 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:39.510727 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.551059 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:39.512656 18019 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:39.554327 18019 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:39.539215 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.541s (SENT) [suppressed 37 similar messages]
I20260709 12:25:39.570513 27133 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:39.528962 27076 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.580886 27189 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.498675 26984 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:25:39.582247 27189 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.582998 27076 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:39.585274 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:39.585397 27189 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.586378 18024 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:39.592271 18097 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:25:39.594764 18809 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:39.595507 27034 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:39.596698 18019 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.597529 27184 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:39.598492 27113 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:39.599138 26853 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.063s	user 0.009s	sys 0.014s
I20260709 12:25:39.599555 26866 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:25:39.602679 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.624361 18019 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:39.607913 26984 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:39.608863 27077 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:39.613047 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:39.618373 18090 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:39.618856 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62622.7us
I20260709 12:25:39.607379 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1648) took 1042 ms. Trace:
I20260709 12:25:39.637751 18815 rpcz_store.cc:276] 0709 12:25:38.564813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.578576 (+ 13763us) service_pool.cc:224] Handling call
0709 12:25:38.578954 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.270410 (+691456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.607056 (+336646us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54005f1c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:39.607288 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588465920}
I20260709 12:25:39.639644 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:39.635435 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 96254.6us
I20260709 12:25:39.616194 27134 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:39.658931 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:39.653789 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:39.668968 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.630834 26984 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:39.670955 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:39.661203 18019 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:39.687147 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.657859 27034 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.625138 26853 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:39.679178 26966 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.565s	user 0.014s	sys 0.006s
I20260709 12:25:39.657651 26866 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:39.715088 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44213.6us
I20260709 12:25:39.668912 26899 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:39.667984 27193 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:39.668763 26991 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:39.669057 27151 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:39.654300 27154 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.085s
I20260709 12:25:39.657547 26998 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:25:39.722251 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:39.731484 18015 raft_consensus.cc:2368] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24.
I20260709 12:25:39.714375 26986 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:39.721164 27190 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:39.688038 26983 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:25:39.746224 26983 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:39.724491 26899 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.689950 27109 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:25:39.723428 26966 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:39.739413 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:39.756883 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:39.739427 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67121.6us
I20260709 12:25:39.758599 18138 leader_election.cc:400] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:39.760450 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.762686 26998 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:39.762881 26542 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:39.763437 26998 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:39.763700 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.765352 27116 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:39.729756 27193 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.767628 27183 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.768404 27193 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:39.768180 27183 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.752959 26899 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:39.770006 27133 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.770639 27183 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.770596 27133 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.773115 27133 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.721606 27034 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.776888 27034 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:39.726334 27113 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:25:39.783782 26866 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.765102 26998 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.795245 18172 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:39.721271 27079 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:39.742741 27190 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.742802 26796 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:39.727031 26991 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.731766 27151 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.749943 27080 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.21, Next index: 22, Last known committed idx: 17, Time since last communication: 0.002s
I20260709 12:25:39.800606 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.826660 26821 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:39.801285 18090 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:39.835071 27205 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.806701 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.804555 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.822990 18780 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:39.824381 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.828933 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.831511 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.848977 27188 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:39.849467 27188 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.849877 27188 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:39.790308 26984 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:39.803769 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47000us
I20260709 12:25:39.849011 18169 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:39.852157 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:39.854321 18809 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: [24.20-26.21]
I20260709 12:25:39.844945 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.855836 18090 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:39.791687 27133 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.844381 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84035.9us
I20260709 12:25:39.856920 27133 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.859488 27133 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.860545 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:39.861250 18813 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:39.863476 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.865655 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.854149 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:39.866894 18167 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:25:39.867697 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:39.859398 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:39.869022 18169 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:39.802131 27113 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:39.763440 26542 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:39.842375 26796 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:39.868825 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:39.874555 18168 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:25:39.838434 26991 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:39.838037 27151 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:39.835862 26613 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:39.826980 27190 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:39.739044 27077 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.878248 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.820387 27109 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:39.887578 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.870209 27034 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.741338 27134 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.21, Last appended by leader: 21, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.837615 27116 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:25:39.897627 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.899588 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.921445 18168 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:39.915769 18171 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:39.836505 27079 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:39.926083 18813 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:39.910425 26821 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:39.879380 26966 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:25:39.928315 18171 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 12:25:39.942586 18171 pending_rounds.cc:85] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 12
I20260709 12:25:39.936895 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:39.890543 27113 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:39.940837 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 81539.7us
I20260709 12:25:39.942881 18171 pending_rounds.cc:107] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 12:25:39.948827 18171 raft_consensus.cc:2889] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:39.871762 27205 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:39.961396 18167 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:25:39.964243 18167 pending_rounds.cc:85] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 15
I20260709 12:25:39.964679 18167 pending_rounds.cc:107] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:25:39.965000 18167 raft_consensus.cc:2889] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:39.903779 26986 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:39.945884 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.980520 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:39.983899 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:39.984678 18171 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
W20260709 12:25:39.989869 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53081.8us
I20260709 12:25:39.923882 27034 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.007944 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:39.884333 27193 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:25:40.014804 18091 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:39.960430 26853 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.981669 26966 consensus_queue.cc:1243] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 29.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:25:40.007859 26663 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:40.031365 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:40.032464 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:39.992303 26986 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:40.039745 27205 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:40.073179 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.082302 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 133246us
I20260709 12:25:40.083683 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:40.088069 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 197196us
I20260709 12:25:40.097734 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:40.098749 18167 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:40.097460 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47682.6us
I20260709 12:25:40.069917 26853 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.060078 27116 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.014215 26796 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.127625 18018 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:40.088276 27109 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:25:40.137343 18018 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 12:25:40.138006 18018 pending_rounds.cc:85] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 10
I20260709 12:25:40.138330 18018 pending_rounds.cc:107] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:25:40.138686 18018 raft_consensus.cc:2889] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.142692 18021 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 11
I20260709 12:25:40.143265 18021 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:25:40.143633 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.143647 18021 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.143143 18024 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:25:40.134238 27204 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.258s	user 0.002s	sys 0.000s
I20260709 12:25:40.151921 18017 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:40.173704 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:40.176029 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3496) took 1034 ms. Trace:
I20260709 12:25:40.184000 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.190900 18167 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:40.158739 27079 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:40.136101 26986 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.163753 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3604) took 1527 ms. Trace:
W20260709 12:25:40.165928 27210 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.231s	user 0.000s	sys 0.002s
I20260709 12:25:40.167420 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:40.167968 27224 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
W20260709 12:25:40.169262 27214 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:40.110713 26853 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.196084 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:40.199308 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.184618 18166 rpcz_store.cc:276] 0709 12:25:39.142149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.160783 (+ 18634us) service_pool.cc:224] Handling call
0709 12:25:39.194943 (+ 34160us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:39.494068 (+299125us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.494104 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:39.498430 (+  4326us) log.cc:844] Serialized 26 byte log entry
0709 12:25:39.714721 (+216291us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:39.714780 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.714864 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.175666 (+460802us) raft_consensus.cc:1685] finished
0709 12:25:40.175697 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.175917 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6822,"raft.queue_time_us":66256,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"thread_start_us":361020,"threads_started":2,"wal-append.queue_time_us":144322}
I20260709 12:25:40.112026 27116 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.204710 18173 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:25:40.231771 18173 pending_rounds.cc:85] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 18
I20260709 12:25:40.232124 18173 pending_rounds.cc:107] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:25:40.232390 18173 raft_consensus.cc:2889] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.154495 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:40.113230 26899 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.188712 26967 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:40.218016 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 134466us
I20260709 12:25:40.235095 18138 leader_election.cc:400] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:40.235643 18138 leader_election.cc:403] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:40.219159 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127393us
I20260709 12:25:40.202454 26966 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.216902 27186 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:40.214514 18095 rpcz_store.cc:276] 0709 12:25:38.636275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.638818 (+  2543us) service_pool.cc:224] Handling call
0709 12:25:38.662566 (+ 23748us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.888922 (+226356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.888981 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:38.889861 (+   880us) log.cc:844] Serialized 26 byte log entry
0709 12:25:38.966120 (+ 76259us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:38.966178 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.282975 (+316797us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b5400214654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:39.283027 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.162879 (+879852us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b54002e66a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662cdbe 00007f6c16637930 00007f6c166378d8 00007f6c16637858 00007f6c16637813 00007f6c16636a9c 00007f6c1c5222f6 00007f6c1c51caa8 00007f6c166737cb 00007f6c166762c0
0709 12:25:40.163432 (+   553us) raft_consensus.cc:1685] finished
0709 12:25:40.163457 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.163669 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":116,"spinlock_wait_cycles":593989888,"thread_start_us":54060,"threads_started":1,"wal-append.queue_time_us":60426}
I20260709 12:25:40.217157 27193 consensus_queue.cc:1243] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 25.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
W20260709 12:25:40.235309 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80935us
I20260709 12:25:40.248333 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:40.218564 18167 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:40.239063 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.218897 26663 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:40.251919 26663 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.243014 18166 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:40.254417 18166 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:40.262831 18168 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:40.259735 26967 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:40.264638 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:40.264478 26967 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.256693 26966 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.265985 27105 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.006s
I20260709 12:25:40.267587 26899 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:40.268272 26966 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:40.269543 27217 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:25:40.271752 27200 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.515s	user 0.001s	sys 0.002s
I20260709 12:25:40.213267 27227 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:25:40.273089 27225 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:40.278803 27216 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:25:40.279006 26853 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:40.289134 18173 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:25:40.293661 27150 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:40.294155 27133 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:40.294641 27133 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:40.295576 27133 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 26.21, Last appended by leader: 21, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.297086 27133 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:40.300329 27116 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:39.454070 18807 rpcz_store.cc:276] 0709 12:25:38.108012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.258646 (+150634us) service_pool.cc:224] Handling call
0709 12:25:38.282983 (+ 24337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.283275 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:38.460321 (+177046us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.834825 (+374504us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b18003729a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:25:39.451233 (+616408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:39.451348 (+   115us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:39.451431 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.451540 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.451713 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119311,"prepare.queue_time_us":351660,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":13159,"raft.queue_time_us":67170,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":2712,"spinlock_wait_cycles":15610752,"thread_start_us":657967,"threads_started":2}
I20260709 12:25:40.304262 27109 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:40.305076 27109 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:40.261778 27113 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:40.307255 18166 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.275998 27188 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:40.313572 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78647.9us
I20260709 12:25:40.316008 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.322737 26613 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:40.322060 17920 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 17 to 20, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 20 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:40.323385 26613 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:40.323980 17920 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 16 to 23, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:40.177615 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:40.327620 26966 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:25:40.333838 18166 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:40.325590 17920 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 17 to 19, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 19 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:25:40.394227 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78308.2us
I20260709 12:25:40.395606 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.396761 26966 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:40.403683 27138 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:40.404317 27138 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:40.408078 26987 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:40.397336 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.415644 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.338615 27035 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:40.431113 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:40.435410 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 307819us
I20260709 12:25:40.439079 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:40.440200 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.441944 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.443869 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:40.454993 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 206758us
I20260709 12:25:40.456040 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:40.458523 27186 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:40.458791 18092 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:40.459247 27226 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:40.459508 27186 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:40.459945 27226 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:40.460048 27186 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:40.461120 27226 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.462989 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:40.465343 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1327) took 1068 ms. Trace:
I20260709 12:25:40.466748 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:40.467777 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:40.470821 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:40.487363 18093 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:40.474890 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.476716 27224 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:40.418409 26966 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:40.461225 27186 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.483614 18171 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 12:25:40.502815 18171 pending_rounds.cc:85] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 12
I20260709 12:25:40.503273 18171 pending_rounds.cc:107] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:25:40.503600 18171 raft_consensus.cc:2889] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:40.478772 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3346) took 1954 ms (client timeout 1838 ms). Trace:
I20260709 12:25:40.432582 26821 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:40.370141 26991 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:40.507063 18164 rpcz_store.cc:269] 0709 12:25:38.524119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.524273 (+   154us) service_pool.cc:224] Handling call
0709 12:25:40.478405 (+1954132us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b540007bf54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:40.478657 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079056640}
I20260709 12:25:40.485357 18022 rpcz_store.cc:276] 0709 12:25:39.397171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.402400 (+  5229us) service_pool.cc:224] Handling call
0709 12:25:39.459050 (+ 56650us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:39.464510 (+  5460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.465085 (+1000575us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b540024ca50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:40.465266 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128907136}
W20260709 12:25:40.460100 27203 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.686s	user 0.002s	sys 0.002s
I20260709 12:25:40.512563 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:40.515821 18814 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260709 12:25:40.482796 27193 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:40.483028 26998 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:40.522753 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84031.3us
I20260709 12:25:40.419050 27151 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.505286 27113 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.517987 18814 pending_rounds.cc:85] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 16
I20260709 12:25:40.405560 27138 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.21, Last appended by leader: 21, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.529271 18814 pending_rounds.cc:107] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 12:25:40.531728 18814 raft_consensus.cc:2889] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.530004 27224 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:40.535140 27224 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.484314 26853 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:25:40.515471 27222 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:40.438051 27109 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.524420 27231 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:40.535601 27224 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:40.562917 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 119155us
I20260709 12:25:40.586109 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:25:40.593993 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.558617 27138 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:40.531212 26796 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:25:40.529089 26998 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.601205 18814 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:40.559732 27231 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:40.555114 27002 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:25:40.616765 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.589365 27034 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:40.536736 27113 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.623920 27113 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:40.596851 26663 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:40.625236 26663 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:40.626050 26663 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:40.618646 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:40.634162 26966 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:40.594825 27185 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:40.635782 27185 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.639983 27185 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:40.640605 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:40.646209 26966 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:40.646803 26966 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.647306 26966 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:40.651067 27035 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:40.656009 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:40.657645 27035 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:40.658604 27035 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:40.659413 27035 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:40.668469 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.676745 18165 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:40.547047 27222 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.710553 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.713972 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:40.715173 18173 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:40.611119 26853 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.094s
I20260709 12:25:40.683333 26821 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:40.613790 27193 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.091s
I20260709 12:25:40.687992 27116 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.735808 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:40.736795 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.737025 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.689013 27188 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:40.758653 18021 raft_consensus.cc:2368] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29.
I20260709 12:25:40.759717 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:40.761248 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:40.762061 18093 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:25:40.736227 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:40.760639 18018 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:40.780224 27246 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.781059 27246 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.785521 18024 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:25:40.718611 27035 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.784323 27246 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:40.763640 27034 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.822219 18166 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:25:40.829308 18166 pending_rounds.cc:85] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 16
W20260709 12:25:40.832459 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 215187us
I20260709 12:25:40.832296 18166 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:25:40.835616 18166 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:40.820423 26986 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.000s	sys 0.007s
I20260709 12:25:40.786623 27116 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.845890 27116 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:40.818207 26991 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:40.847816 26964 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 29 current term: 29 is higher.
I20260709 12:25:40.866021 27189 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:40.866858 27189 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.872144 27189 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:40.850544 26986 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.000s	sys 0.008s
I20260709 12:25:40.834825 27151 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:40.877708 27256 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:25:40.876117 27253 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:25:40.876673 27230 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 12:25:40.884786 18093 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:40.780331 27231 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.868212 27035 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:40.830220 27034 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.864775 27116 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:40.884186 26853 consensus_queue.cc:1243] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 23.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:40.911849 27266 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.912762 27266 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:40.914331 27257 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.326s	user 0.000s	sys 0.003s
I20260709 12:25:40.915804 27266 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:40.916787 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3360) took 1034 ms. Trace:
I20260709 12:25:40.917395 18171 rpcz_store.cc:276] 0709 12:25:39.882119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.985330 (+103211us) service_pool.cc:224] Handling call
0709 12:25:40.483008 (+497678us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.504111 (+ 21103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.916508 (+412397us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b540007bf50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:40.916698 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1143565696}
I20260709 12:25:40.918931 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:40.919773 18171 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:40.921738 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:40.904321 26998 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:40.942833 26966 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:40.953985 27250 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.092s	user 0.000s	sys 0.002s
I20260709 12:25:40.957580 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:40.959694 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79484.3us
I20260709 12:25:40.945108 27189 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.988094 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:40.990046 18173 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:40.946944 26971 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:40.928747 27116 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:40.934048 26663 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:40.984807 27235 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:25:41.005977 27240 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.087s
W20260709 12:25:41.008564 27270 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:25:41.009578 27268 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:41.012991 18015 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4325) took 2626 ms (client timeout 1823 ms). Trace:
I20260709 12:25:41.013381 27185 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:41.014850 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60326.2us
I20260709 12:25:41.020200 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1553) took 2268 ms. Trace:
W20260709 12:25:41.032783 18015 rpcz_store.cc:269] 0709 12:25:38.386725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.387941 (+  1216us) service_pool.cc:224] Handling call
0709 12:25:41.012598 (+2624657us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:41.012817 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2857231104}
W20260709 12:25:41.006106 27263 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:25:41.036317 18810 rpcz_store.cc:276] 0709 12:25:38.751950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.782136 (+ 30186us) service_pool.cc:224] Handling call
0709 12:25:39.310732 (+528596us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.323831 (+ 13099us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.019845 (+1696014us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b5400557150. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:41.020089 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1155238272}
I20260709 12:25:41.029230 27184 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.038981 27193 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.039551 27268 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:40.995018 27189 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.063077 27034 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:41.065040 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.066430 18165 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:25:41.016368 26971 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:41.107327 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76761.5us
I20260709 12:25:41.110846 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:41.107327 27113 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.042730 27185 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.115849 27113 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.119145 27269 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.104545 27188 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:41.150475 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.076323 27193 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.152451 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.070773 27184 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:41.149995 27190 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.229s	user 0.015s	sys 0.004s
W20260709 12:25:41.156329 27150 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.862s	user 0.014s	sys 0.005s
I20260709 12:25:41.156706 27113 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:41.163327 27275 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
W20260709 12:25:41.163488 27271 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:41.174407 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:41.176249 18019 raft_consensus.cc:2368] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24.
I20260709 12:25:41.178040 18809 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:25:41.119884 27269 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.181879 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:41.186795 18016 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:41.187561 18016 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:41.161911 26966 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.195773 27267 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.193966 27034 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:41.197566 27286 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:25:41.151125 27188 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.202062 27150 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:41.223995 27150 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:41.202392 27252 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.398s	user 0.003s	sys 0.000s
I20260709 12:25:41.203550 26971 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.227147 27150 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.227505 27116 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:41.204417 27254 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.619s	user 0.001s	sys 0.001s
I20260709 12:25:41.208101 26663 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.215019 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:41.214649 18015 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:41.210403 26613 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.886s	user 0.018s	sys 0.000s
I20260709 12:25:41.175609 27190 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.180151 27183 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.223165 27188 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:41.229152 18198 heartbeater.cc:646] Failed to heartbeat to 127.16.102.62:42293 (7 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.102.62:42293 timed out after 15.000s (SENT)
W20260709 12:25:41.220006 27224 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.639s	user 0.015s	sys 0.000s
I20260709 12:25:41.248709 27224 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.251516 27224 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.240901 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.258844 27285 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:41.259545 27285 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.259987 27285 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:41.263391 27185 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.231315 18016 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.241726 26613 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.267773 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:41.245971 27190 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.246532 27183 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.266206 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:41.244768 18094 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:41.278075 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:41.291000 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74486us
I20260709 12:25:41.293769 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:41.278724 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126383us
I20260709 12:25:41.282765 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.305723 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.319103 18096 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:41.241308 27150 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:41.281054 27148 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:41.320760 18138 leader_election.cc:400] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29.
I20260709 12:25:41.333006 18097 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:25:41.284925 27211 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.285187 27224 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:41.287201 27079 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:41.343772 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 197070us
I20260709 12:25:41.343376 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.287564 26899 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.354975 18814 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 27 index: 22. (index mismatch)
I20260709 12:25:41.229712 27267 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.359023 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.382328 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.332518 26900 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.386636 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.386919 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.337303 27034 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:41.396524 26875 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:41.334626 27286 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:41.398305 27286 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.399132 26613 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:41.402983 27288 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:41.392144 27268 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.394924 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:41.404460 18093 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:41.404680 26900 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.394292 27267 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.378839 26899 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:41.407146 18809 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1562) took 1546 ms (client timeout 1735 ms). Trace:
I20260709 12:25:41.407913 26900 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.408758 26899 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.396202 27289 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:41.396883 27034 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.410135 27034 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:41.411005 27280 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 12:25:41.415392 18807 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:25:41.415896 18807 pending_rounds.cc:85] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 14
I20260709 12:25:41.416141 18807 pending_rounds.cc:107] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:25:41.416392 18807 raft_consensus.cc:2889] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:41.417361 26821 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:41.418027 26821 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:41.419504 18016 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:41.419152 26821 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.420579 26821 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:41.423221 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:41.426442 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105870us
I20260709 12:25:41.427313 27286 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.428399 18138 leader_election.cc:400] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24.
I20260709 12:25:41.428867 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:41.394934 27290 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:41.431098 27189 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:41.395463 26987 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.987s	user 0.010s	sys 0.009s
I20260709 12:25:41.429337 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:41.407670 18809 rpcz_store.cc:269] 0709 12:25:39.860521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.867292 (+  6771us) service_pool.cc:224] Handling call
0709 12:25:41.406896 (+1539604us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540064e454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:41.407079 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3010691328}
I20260709 12:25:41.443272 18093 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:41.350997 27150 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:41.358385 27080 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.448608 18167 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 21. (term mismatch)
I20260709 12:25:41.458961 18167 pending_rounds.cc:85] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 20
I20260709 12:25:41.459452 18167 pending_rounds.cc:107] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 31.21
I20260709 12:25:41.459847 18167 raft_consensus.cc:2889] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:41.459038 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72216.7us
I20260709 12:25:41.400368 27193 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.382861 27224 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:41.477720 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.480876 18091 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:41.484627 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:41.445127 26899 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:41.376605 27079 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.460820 26878 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.511662 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.512673 18092 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:41.515367 18090 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: [27.23-27.23]
W20260709 12:25:41.519665 27251 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.938s	user 0.003s	sys 0.001s
I20260709 12:25:41.479163 27134 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.524657 27134 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:41.528292 27133 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.231s	user 0.018s	sys 0.003s
I20260709 12:25:41.479498 27184 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.470252 27290 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:41.534737 27290 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.535264 27290 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:41.527420 27150 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.21, Last appended by leader: 21, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.534011 26878 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.468081 27080 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.540115 27080 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.542723 27269 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.549455 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:41.511818 26899 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.554716 18091 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:41.368920 27211 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.481997 26853 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.492918 27193 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:41.571856 18095 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:25:41.566516 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:41.573742 18090 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:41.579491 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.527508 27134 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.560920 27035 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.558346 26878 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:41.605978 17989 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.007s	sys 0.015s
I20260709 12:25:41.606402 18097 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:25:41.608587 17989 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.007s	sys 0.016s
W20260709 12:25:41.609083 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 508774us
I20260709 12:25:41.611727 18172 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:41.616528 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.617439 18172 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:41.618036 18016 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:41.621882 18169 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:41.623963 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:41.576259 27185 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.624881 18172 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:41.638428 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:41.638811 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.640069 18090 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:41.641652 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:41.642808 18090 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:41.647459 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.637168 17989 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:41.658201 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.658990 18164 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:41.586599 27193 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:41.668372 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:41.653759 26971 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.671095 18171 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [26.21-26.21]
I20260709 12:25:41.673657 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.678742 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.679870 18166 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:41.679939 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.674841 18168 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:41.578135 26853 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.654178 27286 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.657961 27269 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:41.657378 26970 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.691537 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.667497 27262 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:41.617242 27035 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.597003 27224 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.661263 27185 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.705567 18172 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:41.758455 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:41.759708 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:41.687651 27268 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.759975 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:41.760641 18172 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:41.731666 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3500) took 2478 ms. Trace:
I20260709 12:25:41.771931 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:41.767705 18170 rpcz_store.cc:276] 0709 12:25:39.253534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.302577 (+ 49043us) service_pool.cc:224] Handling call
0709 12:25:39.303012 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:39.361659 (+ 58647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.361699 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:40.846301 (+1484602us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b18003e6c20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:25:41.265467 (+419166us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b2c001268f8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:25:41.286593 (+ 21126us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.560626 (+274033us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:41.560702 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.567174 (+  6472us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:41.731224 (+164050us) raft_consensus.cc:1685] finished
0709 12:25:41.731255 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.731528 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65275,"raft.queue_time_us":70967,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":5354,"spinlock_wait_cycles":43597824,"thread_start_us":201956,"threads_started":1,"wal-append.queue_time_us":375749}
W20260709 12:25:41.649521 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6563148960003673s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18380000000000002s for a response
I20260709 12:25:41.773160 18809 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
W20260709 12:25:41.734385 27138 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.119s	user 0.018s	sys 0.000s
W20260709 12:25:41.729593 27245 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.144s	user 0.000s	sys 0.003s
I20260709 12:25:41.725346 27079 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.687033 26971 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.784523 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:41.785985 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.789527 27185 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.732697 27267 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:41.721158 27193 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.792096 27267 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.794731 27267 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.785475 18811 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:41.720629 26853 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.726161 27262 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:41.804540 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:41.714574 26899 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.810057 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:41.816856 18809 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 17.
W20260709 12:25:41.811305 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.716888 27211 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.813923 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.819847 18092 raft_consensus.cc:2410] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:25:41.817953 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:41.821218 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:41.815922 27138 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:41.697120 27286 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.785800 26971 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:41.707064 27269 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.777909 26821 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.832257 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:41.844120 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:41.846153 26613 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:41.838243 27211 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 12:25:41.848289 26613 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:41.850389 26613 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:41.855594 27286 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.856606 27002 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:41.860090 27269 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.863111 26971 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:41.863867 26971 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.866122 26983 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:41.834800 27289 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.869730 27289 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:41.830668 27138 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:41.835366 27285 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.575s	user 0.009s	sys 0.009s
I20260709 12:25:41.838739 26899 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:41.832175 27079 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.832410 27133 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:41.832877 27034 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.765635 27290 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.885206 18170 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:41.716228 26970 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.898291 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.906299 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:41.961130 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 274028us
I20260709 12:25:41.875670 27138 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:41.982527 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3517) took 1036 ms. Trace:
I20260709 12:25:41.879148 26899 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.995389 18095 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:41.906111 27290 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:41.956897 27105 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:41.886083 27133 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:41.876968 27285 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.011173 27133 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:41.999553 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:41.963043 27269 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.012404 27269 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.013005 18172 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:41.891763 26821 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.015182 27269 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.016935 26821 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.017045 27268 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:41.880519 27079 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.017786 27268 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.018558 27268 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.019167 27285 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.019685 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.020498 27222 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.021195 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.973344 26967 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:42.019172 26970 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.023001 18092 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:42.024482 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.027184 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1366) took 2036 ms. Trace:
I20260709 12:25:42.027581 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.027752 18167 rpcz_store.cc:276] 0709 12:25:39.990957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.256258 (+265301us) service_pool.cc:224] Handling call
0709 12:25:40.318347 (+ 62089us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:41.460606 (+1142259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.026893 (+566287us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540007c454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:42.027106 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2598015360}
I20260709 12:25:42.029228 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:42.031304 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.031219 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.032374 18167 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:42.032711 27138 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.033505 27138 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.034943 18167 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 21. Preceding OpId from leader: term: 27 index: 22. (index mismatch)
I20260709 12:25:42.036319 27138 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.036433 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:42.037802 18090 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:42.039541 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.041869 18170 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 28 because replica has last-logged OpId of term: 27 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 22.
I20260709 12:25:42.038805 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:41.973173 26971 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.037469 27289 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:42.049032 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.984591 27034 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.006580 26899 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:42.052803 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.054384 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.075433 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.089733 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:42.104327 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.007851 26853 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 12:25:42.106299 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.116088 18090 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:42.048327 27237 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:42.122568 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 263201us
I20260709 12:25:42.123210 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.124221 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:42.042093 27105 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.122231 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:41.886701 27262 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.132467 17920 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:42.126133 27265 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:25:42.126959 18016 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:42.134692 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1805) took 1665 ms. Trace:
I20260709 12:25:42.144373 18807 rpcz_store.cc:276] 0709 12:25:40.469628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.469806 (+   178us) service_pool.cc:224] Handling call
0709 12:25:41.414940 (+945134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:41.416881 (+  1941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.134340 (+717459us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b540064e450. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:42.134586 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2172510976}
I20260709 12:25:42.136492 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.145994 18024 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:42.135807 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.141255 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.129401 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:42.143314 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.147650 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:42.153589 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:42.154615 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:42.157227 18015 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:42.094256 27236 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:25:42.171331 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:42.180339 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:42.135804 27298 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:25:42.127207 27273 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:25:42.151182 27309 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:42.188370 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.188699 27309 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.162305 27002 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:42.191244 27002 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.191797 27002 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.194630 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.195045 27309 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.195856 18015 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:25:42.196893 27134 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:42.197783 26613 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:42.198330 26613 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.198777 26613 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.198689 26983 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:42.164323 27246 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:42.203517 27246 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:42.204213 27246 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:42.209669 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.213364 18093 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:42.219573 27105 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.224964 27231 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:42.227453 27183 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:42.126448 27237 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.224637 27133 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.233765 27133 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.240641 26542 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:41.983278 18173 rpcz_store.cc:276] 0709 12:25:40.945759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.991030 (+ 45271us) service_pool.cc:224] Handling call
0709 12:25:40.991602 (+   572us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.028263 (+ 36661us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:41.046073 (+ 17810us) log.cc:844] Serialized 14 byte log entry
0709 12:25:41.427208 (+381135us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b1800392020. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:25:41.668555 (+241347us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.681904 (+ 13349us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.682160 (+   256us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:41.962857 (+280697us) log.cc:844] Serialized 14 byte log entry
0709 12:25:41.969601 (+  6744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.969763 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:41.982133 (+ 12370us) raft_consensus.cc:1685] finished
0709 12:25:41.982175 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.982435 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64497,"prepare.queue_time_us":69568,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":3490,"raft.queue_time_us":85378,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":20989,"spinlock_wait_cycles":8353152,"thread_start_us":462807,"threads_started":2,"wal-append.queue_time_us":677451}
I20260709 12:25:42.253295 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:42.254204 18814 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:42.255775 26542 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.266716 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.267611 18015 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:42.233750 27237 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:42.262387 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82215.1us
I20260709 12:25:42.290256 26542 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:42.292191 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:42.299458 26542 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:42.268649 27154 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.268914 27105 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.268435 27133 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:42.309149 26542 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:42.313508 26542 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:42.291913 27035 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.322667 26542 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:42.324221 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:42.326501 26542 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:42.327005 26542 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.327366 26542 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.310086 27193 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.316814 27133 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.344173 27236 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.345060 27236 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.350462 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.354538 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.329353 27002 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.362018 27002 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.316570 27154 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.347470 27133 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:42.327392 27298 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.019s
I20260709 12:25:42.369357 27236 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.372434 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.373376 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.376014 18171 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:42.321421 27105 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.319777 27297 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:25:42.327221 27134 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:25:42.346469 27193 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.404285 18170 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
W20260709 12:25:42.384088 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 262005us
W20260709 12:25:42.383452 27319 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:25:42.421439 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.431493 27290 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.432550 27290 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.377363 26542 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.348023 27035 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.438548 27035 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.440055 27290 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.441468 27035 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.394848 27134 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:42.447701 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:42.454736 26542 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.462160 18814 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:42.466616 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:42.467480 18023 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.416728 26998 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:42.472868 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.395087 27268 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.503808 18024 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
W20260709 12:25:42.504853 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 57264.8us
I20260709 12:25:42.482519 27211 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:42.527671 27105 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.531437 27260 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.532295 27260 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.492473 26998 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.537078 26998 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.508651 27211 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.542605 27211 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.547780 26899 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:42.551738 27318 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.226s	user 0.001s	sys 0.001s
W20260709 12:25:42.564087 27330 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:25:42.565351 26984 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:42.566147 27321 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 12:25:42.566706 27320 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 12:25:42.569499 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.577332 18023 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:25:42.570155 18090 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 18: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:42.580065 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:42.585510 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.588270 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:42.570135 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:42.566905 26613 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.568079 26899 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.589214 18814 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:42.592670 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:42.594894 27333 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:42.606870 27212 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:42.607514 27212 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.607932 27212 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:42.607529 27246 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.588894 27134 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:25:42.614250 27193 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.615374 27246 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.620893 27133 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.601804 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.595132 26613 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.605082 27034 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.603235 27105 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:42.630316 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3624) took 2161 ms. Trace:
I20260709 12:25:42.634614 18096 rpcz_store.cc:276] 0709 12:25:40.468502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.469973 (+  1471us) service_pool.cc:224] Handling call
0709 12:25:40.470324 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:41.323169 (+852845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.628232 (+1305063us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b5400051ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:42.630233 (+  2001us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1951460096}
I20260709 12:25:42.631042 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.636461 18167 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:42.637586 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.638360 18167 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.641561 18099 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:42.640096 27034 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.646390 27034 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.647426 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3636) took 1255 ms. Trace:
I20260709 12:25:42.647898 18097 rpcz_store.cc:276] 0709 12:25:41.392215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.393984 (+  1769us) service_pool.cc:224] Handling call
0709 12:25:41.394401 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:41.608673 (+214272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.647136 (+1038463us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54002c9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:42.647353 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":466702208}
I20260709 12:25:42.651692 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.625069 27268 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.631978 27193 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.642498 27105 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.657691 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.662075 18015 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:42.663473 27105 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.663386 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.658327 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.668251 18172 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.652613 18814 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 18.
W20260709 12:25:42.673460 27335 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:25:42.674434 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.676221 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:42.677469 27311 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.484s	user 0.001s	sys 0.003s
I20260709 12:25:42.677909 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.688791 18018 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:42.690497 18018 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:42.684422 27211 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.693656 27134 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.694418 27134 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:42.695093 27332 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.114s	user 0.002s	sys 0.000s
W20260709 12:25:42.698232 27324 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
I20260709 12:25:42.701964 27211 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.704218 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:42.706152 27185 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:42.708498 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:42.714490 27205 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:42.714880 27149 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:42.715119 27205 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:42.716285 27205 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.716964 18812 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:42.718303 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.719769 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.720481 18812 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:42.721496 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.722548 18814 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:42.723027 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:42.669198 27193 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:42.681555 27325 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:25:42.684695 27260 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.657218 27133 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.686679 18167 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:42.750164 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.750612 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.750999 18811 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:42.751605 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:25:42.687959 18098 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:42.709726 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1408) took 1227 ms. Trace:
I20260709 12:25:42.710497 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.753526 18813 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:25:42.711405 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.711800 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.753162 18091 rpcz_store.cc:276] 0709 12:25:41.482739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.484179 (+  1440us) service_pool.cc:224] Handling call
0709 12:25:42.709522 (+1225343us) spinlock_profiling.cc:243] Waited 44 ms on lock 0x7b54003a7854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:42.709651 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101239040}
I20260709 12:25:42.711462 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.713259 18806 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:42.679387 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:42.724465 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:42.740815 18172 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:42.744239 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:42.759099 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.759881 18166 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26. Re-sending same reply.
I20260709 12:25:42.753201 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1413) took 1260 ms. Trace:
I20260709 12:25:42.761216 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.761430 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.762271 18166 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:42.756039 18810 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 20.
I20260709 12:25:42.761361 18095 rpcz_store.cc:276] 0709 12:25:41.492664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.151513 (+658849us) service_pool.cc:224] Handling call
0709 12:25:42.179831 (+ 28318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.198832 (+ 19001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.198889 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:42.199645 (+   756us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.352044 (+152399us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:42.352105 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.352189 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.697508 (+345319us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:42.752866 (+ 55358us) raft_consensus.cc:1685] finished
0709 12:25:42.752890 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.753126 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18328,"raft.queue_time_us":20084,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":161,"spinlock_wait_cycles":30864640,"thread_start_us":151694,"threads_started":1,"wal-append.queue_time_us":247054}
I20260709 12:25:42.755336 18809 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:42.715515 27149 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:42.760313 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.731671 27134 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:42.818900 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.764665 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.731781 27105 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:42.822872 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.825218 18172 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:42.827730 27105 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:42.831267 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.832012 18166 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.833721 27105 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.835649 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:42.836512 18171 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:42.838948 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4505) took 1230 ms. Trace:
I20260709 12:25:42.839551 18093 rpcz_store.cc:276] 0709 12:25:41.608771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.214647 (+605876us) service_pool.cc:224] Handling call
0709 12:25:42.215043 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.215462 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.215496 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:42.216158 (+   662us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.414213 (+198055us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:42.414266 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.414358 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.680774 (+266416us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:42.838541 (+157767us) raft_consensus.cc:1685] finished
0709 12:25:42.838583 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.838864 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1536,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":29485952,"thread_start_us":197491,"threads_started":1,"wal-append.queue_time_us":264056}
I20260709 12:25:42.842000 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.843429 27193 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:42.843511 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.845875 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1671) took 1583 ms. Trace:
I20260709 12:25:42.844331 18170 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:42.849020 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.850569 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.846511 18808 rpcz_store.cc:276] 0709 12:25:41.262361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.262552 (+   191us) service_pool.cc:224] Handling call
0709 12:25:41.262960 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.263425 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:41.693162 (+429737us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.057015 (+363853us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:42.060410 (+  3395us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.060637 (+   227us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:42.060678 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.060768 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.676795 (+616027us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:42.845474 (+168679us) raft_consensus.cc:1685] finished
0709 12:25:42.845608 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.845774 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11,"prepare.queue_time_us":216348,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":73,"raft.queue_time_us":12592,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":46433,"spinlock_wait_cycles":24470912,"thread_start_us":791356,"threads_started":2,"wal-append.queue_time_us":499203}
I20260709 12:25:42.853060 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.764825 26984 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:42.853860 18093 raft_consensus.cc:2376] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23. Re-sending same reply.
I20260709 12:25:42.851856 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.856220 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:42.857491 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.858006 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:42.858609 18166 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:42.866758 18170 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:42.873880 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.875139 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:42.830359 27262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.004s	sys 0.019s
I20260709 12:25:42.873320 18171 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:42.878616 18166 raft_consensus.cc:2410] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 26 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:42.880115 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.884892 18170 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:42.885324 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:42.883652 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.886104 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.886838 18171 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:42.887502 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.887972 27154 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:42.893860 27289 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:42.888253 18171 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: [22.12-24.13]
W20260709 12:25:42.899286 27308 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.698s	user 0.001s	sys 0.002s
W20260709 12:25:42.901396 27314 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.703s	user 0.001s	sys 0.001s
I20260709 12:25:42.904515 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3372) took 1262 ms. Trace:
I20260709 12:25:42.905041 18165 rpcz_store.cc:276] 0709 12:25:41.641924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.642120 (+   196us) service_pool.cc:224] Handling call
0709 12:25:41.672418 (+ 30298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.673030 (+   612us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:41.691110 (+ 18080us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.095549 (+404439us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:42.096981 (+  1432us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.097187 (+   206us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:42.097550 (+   363us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.097846 (+   296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.097937 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.900642 (+802705us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:42.904220 (+  3578us) raft_consensus.cc:1685] finished
0709 12:25:42.904251 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.904432 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22700,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":3999,"raft.queue_time_us":11054,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":22519,"spinlock_wait_cycles":12817792,"thread_start_us":305448,"threads_started":1,"wal-append.queue_time_us":430901}
I20260709 12:25:42.890708 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:42.866045 26984 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:42.893450 18094 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3620) took 2433 ms (client timeout 1584 ms). Trace:
W20260709 12:25:42.881299 27262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.004s	sys 0.019s
I20260709 12:25:42.911346 18170 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
W20260709 12:25:42.911338 18094 rpcz_store.cc:269] 0709 12:25:40.457978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.462615 (+  4637us) service_pool.cc:224] Handling call
0709 12:25:42.890904 (+2428289us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b5400235d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:42.891064 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1795669248}
I20260709 12:25:42.923611 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:42.924556 18166 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:25:42.924801 18164 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:42.924976 18166 pending_rounds.cc:85] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 15
I20260709 12:25:42.925246 18166 pending_rounds.cc:107] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:25:42.925486 18166 raft_consensus.cc:2889] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:42.929883 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82150.1us
I20260709 12:25:42.913260 26984 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:42.936093 18164 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:42.937582 27342 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:42.923483 27290 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:42.934782 27149 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:42.996963 18164 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:42.998760 18809 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:43.006706 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.010002 18164 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:25:43.022214 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 103062us
I20260709 12:25:43.024370 18164 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:43.026743 18815 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:43.027153 18164 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:42.999781 27105 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.035840 27193 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:43.042973 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.044354 18815 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:43.045644 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.046416 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.047183 18812 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:43.047345 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.049536 18815 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:43.049705 18810 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:43.054694 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.034464 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:43.068462 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4489) took 3268 ms (client timeout 1500 ms). Trace:
I20260709 12:25:43.060302 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:43.069172 18098 rpcz_store.cc:269] 0709 12:25:39.800106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.800271 (+   165us) service_pool.cc:224] Handling call
0709 12:25:43.068137 (+3267866us) spinlock_profiling.cc:243] Waited 2.85 s on lock 0x7b54004114d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:43.068363 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6551079168}
I20260709 12:25:43.098541 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:43.101704 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:43.109480 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:43.086900 27079 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.085956 27290 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:43.111032 27328 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.524s	user 0.003s	sys 0.000s
I20260709 12:25:43.111107 27290 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.112552 27290 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.065647 26984 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.116195 26984 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.117287 27148 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:43.118103 27148 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.118561 27148 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:43.055079 27105 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.125212 27105 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.109609 27348 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:43.125931 27189 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:43.126497 27348 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.126749 27189 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.127100 27348 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.127449 27189 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:43.130314 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:43.131402 18172 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:43.132467 27212 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.524s	user 0.006s	sys 0.008s
I20260709 12:25:43.125195 18095 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:43.140967 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:43.165254 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.167196 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.169167 18098 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:43.170755 18094 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:43.144812 27150 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:25:43.133271 27212 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:43.184453 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 461469us
I20260709 12:25:43.185611 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:43.187703 27264 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:43.188392 27264 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:43.149417 27113 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:25:43.191619 27273 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.193732 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:43.196511 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 144897us
I20260709 12:25:43.186182 26796 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:43.189438 27264 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.209476 26796 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:43.226496 26796 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:43.227469 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1416) took 1666 ms. Trace:
I20260709 12:25:43.227998 18090 rpcz_store.cc:276] 0709 12:25:41.560895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.117043 (+556148us) service_pool.cc:224] Handling call
0709 12:25:43.227242 (+1110199us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b5400259754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:43.227394 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":972146048}
W20260709 12:25:43.230113 17987 connection.cc:256] RPC call timeout handler was delayed by -1.4994630370001687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1584s for a response
I20260709 12:25:43.231163 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:43.233211 27222 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.234115 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.233922 27222 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.209496 27193 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.026s
I20260709 12:25:43.235805 27193 consensus_queue.cc:1243] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 25.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:43.235999 18020 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:43.237241 27222 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.238305 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.239008 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.239288 18021 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:43.241075 18016 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:43.201067 27273 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.238363 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.233242 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.237587 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.266229 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.270095 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.276525 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.277642 18016 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:43.278285 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84675us
I20260709 12:25:43.286264 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.286978 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.281504 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:43.286204 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:43.276930 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 129298us
I20260709 12:25:43.307180 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:43.290700 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 132520us
I20260709 12:25:43.305262 27212 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.310026 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.263149 27126 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.315097 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.290839 26971 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:43.312355 26991 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:43.321135 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.275451 27231 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:25:43.315661 27126 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.325525 18018 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:43.333432 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.342007 18018 raft_consensus.cc:2376] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22. Re-sending same reply.
I20260709 12:25:43.317025 27222 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:25:43.332391 27246 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 12:25:43.358822 27272 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:43.315380 27225 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:25:43.323776 26991 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.359346 27272 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:43.359790 26991 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:43.360469 27272 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.314077 26899 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:25:43.324201 26971 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:43.334022 18016 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:43.364006 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.364791 18016 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:43.323451 27268 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 12:25:43.347811 26970 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:43.384251 27126 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.385640 27268 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:43.386373 27268 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:43.387624 27268 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.322635 27348 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.357640 27185 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:43.394330 27185 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.394845 27185 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:43.399899 27273 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.409547 27297 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:43.348374 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.410185 27297 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:43.411194 27297 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.411701 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.412640 27297 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:43.412881 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44690.5us
I20260709 12:25:43.356732 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:43.415313 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58727us
I20260709 12:25:43.416733 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:43.361990 27272 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:43.417615 27246 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:43.417855 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.418552 27126 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:43.342432 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.420221 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.345160 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.422147 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.423040 18090 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:43.422833 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.365453 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.423935 18091 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:43.365902 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.425467 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.426653 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.427410 18016 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:43.428225 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.362704 26971 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:43.421610 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.429740 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:43.430490 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:43.430612 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.431630 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.382473 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:43.370487 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54350.9us
I20260709 12:25:43.431829 18172 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:43.393240 27348 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.444065 18094 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:43.444615 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:43.433745 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:43.441607 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.447783 18016 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:43.449522 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.436338 18017 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:43.389611 27268 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:43.464959 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:43.468089 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.460119 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.447553 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:43.455499 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:43.463639 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.477900 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:25:43.479638 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 343360us
I20260709 12:25:43.483551 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.476763 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:43.433506 27290 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.482575 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:43.330590 27148 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.511922 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:43.514427 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.518355 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:43.526507 18091 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:43.608162 18094 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:43.610821 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.507704 26984 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:43.433334 27272 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.092s
I20260709 12:25:43.448890 27222 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 12:25:43.465621 27035 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.144s
I20260709 12:25:43.463529 27126 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.144s
I20260709 12:25:43.464247 27105 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.142s
I20260709 12:25:43.529688 27348 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:43.443946 27225 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 12:25:43.510591 27010 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:43.532907 27289 raft_consensus.cc:2764] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election decision vote started in defunct term 22: won
I20260709 12:25:43.616912 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.656662 18164 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:43.655277 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.512533 27278 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.137s
I20260709 12:25:43.703250 18091 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 18.
I20260709 12:25:43.664673 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.658159 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:43.644930 27193 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:43.708325 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 400891us
I20260709 12:25:43.649170 27317 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.648205 27109 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:43.694697 27353 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:25:43.712402 26613 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.662403 27365 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:43.657691 26984 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:43.714969 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:43.724532 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:43.713409 27193 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.738511 18171 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:25:43.738135 18164 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:43.619205 27189 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.702868 27010 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.712675 27105 consensus_queue.cc:1243] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:25:43.672469 27079 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.740414 18094 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:43.731941 26613 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.729375 27365 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:43.751385 27327 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.271s	user 0.008s	sys 0.000s
I20260709 12:25:43.728454 27109 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:43.752951 26796 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.526s	user 0.005s	sys 0.009s
I20260709 12:25:43.753616 27364 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:43.759169 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.784016 18091 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:43.759193 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:43.710994 27113 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:25:43.743309 27193 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:43.728724 27126 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:25:43.773475 27105 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:43.724608 27317 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:43.786602 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 174170us
I20260709 12:25:43.787945 18091 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:43.785279 18015 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:43.769166 27366 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:43.765093 27010 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:43.822402 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:43.823259 18094 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:43.823846 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:43.820039 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 301799us
I20260709 12:25:43.773674 27035 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
W20260709 12:25:43.836792 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 324925us
W20260709 12:25:43.817518 27358 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.085s	user 0.001s	sys 0.001s
I20260709 12:25:43.802377 27105 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.850445 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:43.835933 27148 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.839797 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:43.784878 26796 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.849730 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:43.774161 27079 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.820029 27366 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:43.868747 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:43.871124 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:43.830489 27290 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.884191 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3318) took 1523 ms. Trace:
I20260709 12:25:43.861006 27105 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:43.884845 18019 rpcz_store.cc:276] 0709 12:25:42.360461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.360626 (+   165us) service_pool.cc:224] Handling call
0709 12:25:42.361052 (+   426us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:42.361503 (+   451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.361535 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:42.362333 (+   798us) log.cc:844] Serialized 52 byte log entry
0709 12:25:42.446724 (+ 84391us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:42.446773 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.446855 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.724172 (+1277317us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:43.876404 (+152232us) raft_consensus.cc:1685] finished
0709 12:25:43.876429 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.884083 (+  7654us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7398,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"spinlock_wait_cycles":9108224,"thread_start_us":83900,"threads_started":1,"wal-append.queue_time_us":117381}
I20260709 12:25:43.888228 27278 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:43.895447 27189 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.895841 27278 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:43.896235 27079 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:43.873857 27366 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:43.878048 26998 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:43.806025 26984 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:43.904423 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.877242 27212 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:43.920260 18015 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:43.923306 18019 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:43.873840 27148 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:43.920454 27364 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:43.931357 18019 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:43.953153 18015 pending_rounds.cc:85] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 17
I20260709 12:25:43.953768 18015 pending_rounds.cc:107] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:25:43.954156 18015 raft_consensus.cc:2889] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:43.960131 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3537) took 1213 ms. Trace:
I20260709 12:25:43.924397 27264 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:43.919667 27113 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:43.929441 27126 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:43.985987 27370 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:25:43.988503 26796 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.990232 26613 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.868881 27365 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 27.23, Last appended by leader: 23, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:43.997084 27363 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:43.997704 27363 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:43.998013 27376 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:43.998682 27363 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.000437 27363 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:43.978739 18170 rpcz_store.cc:276] 0709 12:25:42.746787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.915862 (+169075us) service_pool.cc:224] Handling call
0709 12:25:42.916490 (+   628us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.917054 (+   564us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.917102 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.120412 (+203310us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b18004a5f60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a
0709 12:25:43.121024 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:25:43.283548 (+162524us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:43.283712 (+   164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.283809 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.943500 (+659691us) raft_consensus.cc:1685] finished
0709 12:25:43.943539 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.960019 (+ 16480us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":159,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":3930,"spinlock_wait_cycles":35742848,"thread_start_us":161935,"threads_started":1,"wal-append.queue_time_us":541081}
W20260709 12:25:44.005864 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 269485us
I20260709 12:25:44.009891 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.027058 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:43.939337 27290 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:43.981235 27264 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:43.929510 27212 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.046617 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.060230 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.004606 27278 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:43.940018 27148 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:44.097501 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51035.2us
I20260709 12:25:44.098618 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.061792 27376 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:44.086129 27289 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:44.094887 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 226256us
I20260709 12:25:44.110563 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:44.105314 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 265680us
W20260709 12:25:44.107785 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 170755us
I20260709 12:25:44.108755 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.126768 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.157266 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:44.157938 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47499.5us
I20260709 12:25:44.158980 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.126286 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:44.160331 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 289258us
I20260709 12:25:44.101588 27383 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:44.169005 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60373.7us
I20260709 12:25:44.170101 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.170239 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.169498 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.171842 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.172679 18093 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:44.173323 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:44.122690 27113 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:44.171995 18097 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:44.107625 27289 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:44.179316 18099 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:44.112136 27376 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:44.179320 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:44.186658 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:44.187340 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.187723 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.112517 27317 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:44.106024 27290 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:44.167119 26613 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:44.107923 27380 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:44.168051 26971 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.734s	user 0.011s	sys 0.003s
W20260709 12:25:44.168503 27368 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.310s	user 0.002s	sys 0.000s
I20260709 12:25:44.181282 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.190240 27289 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:44.196314 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.162900 27382 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:44.230645 27382 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:44.231523 27382 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:44.231933 27382 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:44.232329 27382 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:44.202852 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.236060 18099 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:44.174998 27236 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:44.208292 18807 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:44.208729 27386 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:44.211426 26991 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.851s	user 0.008s	sys 0.008s
I20260709 12:25:44.238523 26991 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.241181 26991 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:44.180943 27079 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
W20260709 12:25:44.219883 27268 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.742s	user 0.013s	sys 0.002s
I20260709 12:25:44.196348 27376 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:44.204250 27185 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.809s	user 0.016s	sys 0.002s
I20260709 12:25:44.226454 18811 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:44.242969 18018 raft_consensus.cc:2368] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:44.217602 26971 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.235257 27148 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:44.238942 27348 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:44.228029 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:44.245105 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63906.1us
I20260709 12:25:44.249302 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.248956 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:44.267367 27297 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.854s	user 0.008s	sys 0.010s
I20260709 12:25:44.196998 27290 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:44.246153 26991 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:44.268530 27290 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:44.269915 27236 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:44.270789 27267 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:25:44.004773 27367 consensus_queue.cc:1175] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ae9641d5c3184572a102d51a75f670ee" responder_term: 28 status { last_received { term: 27 index: 23 } last_committed_idx: 23 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:25:44.275164 27371 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.332s	user 0.000s	sys 0.001s
I20260709 12:25:44.123492 27364 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.275534 27105 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.173913 27383 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.268009 26971 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.280117 27383 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:44.200260 27388 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:44.281179 27105 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.284519 27212 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.161746 27264 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.285789 27264 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:44.290025 27375 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.349s	user 0.000s	sys 0.001s
I20260709 12:25:44.251675 27348 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:44.297758 27348 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.252187 27185 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.302533 18090 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:44.241228 27366 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.305395 27366 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:44.307430 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58586us
I20260709 12:25:44.308561 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.309471 26714 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:44.310148 26714 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:44.310698 26714 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:44.312625 27236 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:44.312942 27185 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:44.313557 27236 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:44.314072 27348 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:44.314194 27236 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:44.317165 27236 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.315783 27267 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:44.319072 27267 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:44.320132 27267 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.329376 27109 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:44.345140 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 143363us
W20260709 12:25:44.347059 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.347440 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.355046 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:44.355978 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106761us
I20260709 12:25:44.356873 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.357343 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.355086 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:44.321624 27267 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:44.378367 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:44.379531 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:44.358599 27236 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:44.399704 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.361071 27034 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:44.400863 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43652.2us
W20260709 12:25:44.401202 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 242286us
I20260709 12:25:44.407673 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.408756 27366 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:44.409269 27366 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.409649 27366 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:44.420177 27193 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.612s	user 0.010s	sys 0.002s
I20260709 12:25:44.420598 27366 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.406713 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:44.360538 27387 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:44.423336 27133 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.423594 27387 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:44.424047 27133 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.426379 27113 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:44.376423 27389 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:44.429352 27389 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.429791 27389 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:44.432044 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.432808 27349 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:44.434383 27126 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.380160 27381 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:25:44.435949 27381 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:44.436321 27381 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:44.435138 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4398) took 1108 ms. Trace:
I20260709 12:25:44.420845 27193 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.407799 18098 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 17. Current term is 18. Ops: [15.14-17.15]
I20260709 12:25:44.452102 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.439888 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.442298 18015 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
W20260709 12:25:44.438170 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 281001us
I20260709 12:25:44.453733 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.449756 18018 rpcz_store.cc:276] 0709 12:25:43.329927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.422577 (+ 92650us) service_pool.cc:224] Handling call
0709 12:25:44.434857 (+1012280us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b540024e354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:44.435048 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884149120}
I20260709 12:25:44.454237 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.453687 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:44.455489 18091 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:44.436461 27317 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:44.462633 18099 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:44.376102 27109 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:25:44.452534 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.396471 26613 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:44.462062 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:44.458761 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:44.466455 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59964.5us
I20260709 12:25:44.402655 26991 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.471822 18098 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:25:44.472848 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.475085 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.475327 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.477106 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:44.483280 18018 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:44.472713 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:44.489459 18806 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:44.490470 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:44.461102 27212 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:44.495543 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.466028 27376 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:44.486320 27398 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:44.451141 27113 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:25:44.464686 27298 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:25:44.469672 27297 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:25:44.460927 27290 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.513190 18023 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:44.527343 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.469308 27079 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:25:44.529899 26613 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.530941 27366 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:44.511303 27034 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.516304 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.508116 26899 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.498018 26986 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:25:44.534797 27289 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.535166 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.522848 27298 consensus_queue.cc:1243] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 26.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
W20260709 12:25:44.537202 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82926.4us
I20260709 12:25:44.539486 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.540671 18170 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:44.541801 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.542146 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.542546 18807 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:44.543443 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.543854 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.536053 18168 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:44.545001 18813 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:44.507829 27148 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:44.522181 27133 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.552309 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.553061 18815 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:44.553452 18810 raft_consensus.cc:2410] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:44.538102 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.555130 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.555768 18169 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:44.499878 27399 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:44.544147 18173 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:44.499769 27405 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:25:44.546559 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.546782 18091 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:44.563819 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.549295 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.565768 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.552309 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.560062 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.567332 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:44.560719 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44861.7us
I20260709 12:25:44.562197 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.569732 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.570502 18813 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:44.572039 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.576480 18169 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:44.510963 27400 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:44.578003 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:44.578768 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 24 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:44.579149 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.581609 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.541662 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.583356 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.583788 18170 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:44.585553 18813 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:44.586366 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.588778 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.589485 18164 raft_consensus.cc:2376] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26. Re-sending same reply.
I20260709 12:25:44.487779 27388 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:25:44.592775 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.593755 18173 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 28 because replica has last-logged OpId of term: 27 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 22.
I20260709 12:25:44.580157 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:25:44.597855 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58815.7us
I20260709 12:25:44.599561 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.529969 27193 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:44.604056 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.607539 18169 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:44.537645 27113 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:44.609125 18021 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:44.568487 26714 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.545873 27034 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:44.615293 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.617442 18166 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:44.626806 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:44.636890 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49236.9us
I20260709 12:25:44.638345 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.639824 18807 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:44.646008 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.645519 17920 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 10 to 26, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 26 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:44.647688 17913 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 285) took 50199 ms (client timeout 15000 ms). Trace:
W20260709 12:25:44.615198 27010 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.779s	user 0.008s	sys 0.007s
I20260709 12:25:44.640250 18138 leader_election.cc:400] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:44.651842 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.654458 18138 leader_election.cc:403] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:44.648025 17920 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 13 to 23, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 23 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.610523 27405 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:44.656118 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.621869 27388 consensus_queue.cc:1243] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:25:44.610586 27113 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:44.650615 27189 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:44.661224 27010 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.667131 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:44.636215 27154 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:44.668563 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.676164 18166 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:44.685963 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.636790 27364 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:44.695043 27364 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.695564 27133 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:44.696447 27133 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:44.663143 17920 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:44.641472 27034 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.683482 27405 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:44.694252 27298 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:25:44.700194 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52790.1us
I20260709 12:25:44.704457 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:44.707435 18095 raft_consensus.cc:2368] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24.
I20260709 12:25:44.650095 27193 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:25:44.710541 27034 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.710027 17920 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.653970 27148 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 17, Time since last communication: 0.049s
I20260709 12:25:44.715070 27289 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.716750 26714 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.718600 27405 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.688511 27189 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.598347 27400 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.688094 27105 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:25:44.742748 17920 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 18 to 22, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 22 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:44.720110 27290 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:44.710390 26899 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:44.750100 27408 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.136s
I20260709 12:25:44.703223 27154 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:44.685706 27376 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:25:44.740079 27317 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:44.731206 27364 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:44.762040 18168 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:44.751605 27298 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:44.757529 27413 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:25:44.757725 27403 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.191s	user 0.002s	sys 0.000s
W20260709 12:25:44.764669 27382 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.004s
I20260709 12:25:44.763819 27298 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:44.767086 27402 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 12:25:44.767544 17920 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:44.769801 27382 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:44.770897 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64490.6us
W20260709 12:25:44.774001 27411 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:25:44.707340 27113 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.751861 26899 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:44.775636 27400 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:44.776161 27365 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:44.792244 17920 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 20 to 23. New cstate: current_term: 23 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:44.794031 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.772038 27317 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:44.805787 18811 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:44.754667 27154 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.807613 17920 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 13 to 20, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 20 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:44.790745 27010 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:44.817793 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 161750us
I20260709 12:25:44.806586 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.813422 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.806255 27400 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.826715 17920 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 18 to 21, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 21 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:44.829905 17920 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 19 to 21, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 21 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:44.832438 17920 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:25:44.835143 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 196248us
I20260709 12:25:44.841599 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:44.842046 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.842649 18166 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:44.844282 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3551) took 1401 ms. Trace:
I20260709 12:25:44.820676 27113 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:44.848047 17920 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 21, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 21 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:44.844770 18172 rpcz_store.cc:276] 0709 12:25:43.442825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.444066 (+  1241us) service_pool.cc:224] Handling call
0709 12:25:44.436429 (+992363us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b5400301028. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b65bebc 00007f6c1b69108c 00007f6c1b67b93b 00007f6c1b67b3d2 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c
0709 12:25:44.436589 (+   160us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:44.463258 (+ 26669us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:44.463295 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:44.476247 (+ 12952us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.608466 (+132219us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:44.608534 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.608659 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.834619 (+225960us) raft_consensus.cc:1685] finished
0709 12:25:44.834647 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.844201 (+  9554us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7849,"raft.queue_time_us":39311,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":11258,"spinlock_wait_cycles":62980480,"thread_start_us":90345,"threads_started":1,"wal-append.queue_time_us":145175}
W20260709 12:25:44.648312 17913 rpcz_store.cc:269] 0709 12:24:54.448014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.151625 (+1703611us) service_pool.cc:224] Handling call
0709 12:25:44.647375 (+48495750us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:34.022664 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:34.102969 (+ 80305us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:34.103053 (+    84us) write_op.cc:435] Acquired schema lock
0709 12:25:34.103081 (+    28us) tablet.cc:662] Decoding operations
0709 12:25:34.263699 (+160618us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:34.263818 (+   119us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:34.263848 (+    30us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:25:34.264933 (+  1085us) tablet.cc:701] Row locks acquired
0709 12:25:34.264999 (+    66us) write_op.cc:260] PREPARE: finished
0709 12:25:34.265225 (+   226us) write_op.cc:270] Start()
0709 12:25:34.265416 (+   191us) write_op.cc:276] Timestamp: P: 1783599934265145 usec, L: 0
0709 12:25:34.265461 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:25:34.397316 (+131855us) log.cc:844] Serialized 4031 byte log entry
0709 12:25:35.027106 (+629790us) op_driver.cc:464] REPLICATION: finished
0709 12:25:35.038924 (+ 11818us) write_op.cc:301] APPLY: starting
0709 12:25:35.039052 (+   128us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:35.419677 (+380625us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:35.419925 (+   248us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:35.711135 (+291210us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:35.711576 (+   441us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 12:25:35.711642 (+    66us) write_op.cc:312] APPLY: finished
0709 12:25:35.712244 (+   602us) log.cc:844] Serialized 97 byte log entry
0709 12:25:35.714057 (+  1813us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:35.714460 (+   403us) write_op.cc:454] Released schema lock
0709 12:25:35.714538 (+    78us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:35.739365 (+ 24827us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:35.741199 (+  1834us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3763730,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":11406,"apply.run_cpu_time_us":27561,"apply.run_wall_time_us":676286,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45987,"num_ops":10,"prepare.queue_time_us":467072,"prepare.run_cpu_time_us":37244,"prepare.run_wall_time_us":684279,"raft.queue_time_us":36911,"raft.run_cpu_time_us":3022,"raft.run_wall_time_us":14216,"replication_time_us":761480,"thread_start_us":628382,"threads_started":4,"wal-append.queue_time_us":278922,"wal-append.run_cpu_time_us":13087,"wal-append.run_wall_time_us":2041072}]]}
I20260709 12:25:44.816133 27395 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:44.829795 27272 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.935s	user 0.005s	sys 0.019s
I20260709 12:25:44.808000 27415 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
W20260709 12:25:44.855840 27272 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.006s	sys 0.019s
W20260709 12:25:44.864342 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57848.1us
I20260709 12:25:44.826789 27417 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:44.853410 27211 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:25:44.878858 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.872509 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:44.876106 17920 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 16 to 22, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 22 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:44.837142 27400 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:44.898754 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.871672 27422 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:44.904755 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:44.908103 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.908910 27418 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:44.913014 27399 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:44.919130 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.920707 18813 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:44.836802 26899 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.922574 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:44.923496 18813 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:44.859197 27389 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:44.930431 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.935315 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.936132 18091 raft_consensus.cc:2368] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:44.847536 27317 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:44.940459 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:44.941882 18017 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4396) took 1607 ms (client timeout 1724 ms). Trace:
I20260709 12:25:44.942778 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.944530 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.945314 18091 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 25 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:25:44.937737 17920 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 18 to 23, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 23 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:44.938795 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:44.951699 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.952739 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
W20260709 12:25:44.942387 18017 rpcz_store.cc:269] 0709 12:25:43.326385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.411264 (+ 84879us) service_pool.cc:224] Handling call
0709 12:25:44.933941 (+1522677us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:44.934073 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54219520}
I20260709 12:25:44.953475 18166 raft_consensus.cc:606] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.953974 18166 raft_consensus.cc:614] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:44.954849 17920 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 18 to 21, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 21 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.956312 17920 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.960500 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.961634 27382 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:44.943828 27416 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.095s	user 0.000s	sys 0.004s
I20260709 12:25:44.968528 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.931457 27408 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:44.929538 27422 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:44.978919 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.939531 27399 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:44.963184 18091 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:44.992470 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.993607 18091 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 26 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 20.
I20260709 12:25:44.962720 27189 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:44.995932 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:44.996556 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3554) took 1526 ms. Trace:
I20260709 12:25:44.997526 17920 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 20, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 20 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.970638 27427 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:45.007400 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.000855 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:44.992976 27418 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:45.015017 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:44.995324 27236 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.006s
I20260709 12:25:45.016489 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.019024 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:45.020493 18136 leader_election.cc:400] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 20 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:45.021477 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.024231 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.027510 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:44.988804 27408 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.030511 17920 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.034789 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.038789 18810 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:45.013870 27427 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:45.030580 27407 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:45.059716 18814 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:44.994396 27399 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.046191 27034 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:45.062493 27399 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.045511 27298 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.042634 27289 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:45.054641 17920 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 21 to 22, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 22 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:45.064657 17920 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 15 to 17, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 17 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:45.066318 17920 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 16 to 21, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 21 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:25:45.053179 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:45.063359 27298 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.068138 18164 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:25:45.069624 27298 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.070353 17920 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 22 to 28, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 28 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:45.070865 27408 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:45.072232 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.071911 17920 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 21 to 23, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 23 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:45.031195 27348 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:45.072917 17988 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.925s (SENT) [suppressed 31 similar messages]
I20260709 12:25:45.022964 27418 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:45.074714 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:45.075945 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:45.024995 27420 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.024811 27424 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:45.078135 18810 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:45.078260 18092 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27. Re-sending same reply.
I20260709 12:25:45.079018 18807 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:45.051591 27124 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:45.077028 27298 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.088763 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.094619 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:45.096089 18164 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:45.096457 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.097498 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.098198 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:45.099259 18164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:45.100226 18164 raft_consensus.cc:606] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.100881 18164 raft_consensus.cc:614] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:45.043503 27421 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:45.102958 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.072960 27400 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:45.059154 27407 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:45.075244 27427 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:45.114260 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 89875.7us
I20260709 12:25:45.105901 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.107837 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.111783 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.094240 27124 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:45.117386 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:45.127383 18166 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:45.104056 27421 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:45.143811 18164 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
W20260709 12:25:45.148904 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114236us
W20260709 12:25:45.152875 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47084.1us
I20260709 12:25:45.159793 27262 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:45.160631 27262 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:45.163929 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.103858 27298 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.166683 18166 pending_rounds.cc:85] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 20
I20260709 12:25:45.170590 18166 pending_rounds.cc:107] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 26.21
I20260709 12:25:45.171247 18166 raft_consensus.cc:2889] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:45.177100 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:45.165359 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:45.145953 27423 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.214s	user 0.000s	sys 0.002s
I20260709 12:25:45.136444 27421 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.162158 27262 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.181152 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.181653 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:45.178177 18165 raft_consensus.cc:606] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.193282 18165 raft_consensus.cc:614] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:45.127110 27407 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:45.195580 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 122663us
I20260709 12:25:45.179527 27348 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:45.094450 27422 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:45.199327 27247 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 4.617s	user 0.014s	sys 0.009s
I20260709 12:25:45.192345 27154 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:45.214578 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:45.207408 27443 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:45.211752 27290 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:45.242934 27262 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:45.209604 27407 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:45.275985 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:45.176298 27124 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.169348 27418 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.277400 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:45.284740 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89761.3us
I20260709 12:25:45.288725 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:45.269809 27290 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:45.289249 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.292833 18165 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:45.294380 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:45.295871 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:45.296512 18164 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:45.301177 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:45.303651 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.304596 18093 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:45.278453 27389 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:45.265730 27429 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:25:45.309120 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:45.311820 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.307950 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:45.294088 27124 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:45.298750 27399 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:45.305176 27113 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.328281 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.329442 18093 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:45.330768 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:45.311376 27380 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:45.326061 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:45.324014 27434 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:25:45.335223 18093 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:45.331025 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:45.318931 27407 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:45.342208 27407 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:45.342995 27407 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:45.335544 27290 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.321982 27365 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.336375 26984 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:45.377907 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3332) took 2110 ms. Trace:
W20260709 12:25:45.381522 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55597.1us
I20260709 12:25:45.390146 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:45.370946 27439 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:25:45.359747 27105 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.603s	user 0.016s	sys 0.000s
W20260709 12:25:45.347201 27448 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:25:45.381572 18020 rpcz_store.cc:276] 0709 12:25:43.267017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.274998 (+  7981us) service_pool.cc:224] Handling call
0709 12:25:43.275384 (+   386us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:43.350486 (+ 75102us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:43.350955 (+   469us) log.cc:844] Serialized 14 byte log entry
0709 12:25:43.761436 (+410481us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:43.779436 (+ 18000us) log.cc:844] Serialized 52 byte log entry
0709 12:25:43.779653 (+   217us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:43.921328 (+141675us) log.cc:844] Serialized 14 byte log entry
0709 12:25:43.925017 (+  3689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.377341 (+1452324us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:45.377396 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.377540 (+   144us) raft_consensus.cc:1685] finished
0709 12:25:45.377556 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.377772 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2856,"prepare.queue_time_us":3701,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":85,"raft.queue_time_us":6387,"raft.run_cpu_time_us":511,"raft.run_wall_time_us":9680,"spinlock_wait_cycles":171191552,"thread_start_us":535179,"threads_started":2,"wal-append.queue_time_us":522470,"wal-append.run_cpu_time_us":4455,"wal-append.run_wall_time_us":1395341}
W20260709 12:25:45.437644 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47647.5us
I20260709 12:25:45.353034 27290 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.356462 27400 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:45.441661 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:45.361133 26984 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:45.336578 27298 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:45.466562 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.460258 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.419624 27429 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:45.394214 27451 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:45.460254 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:45.500109 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58574us
W20260709 12:25:45.500325 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.005808 18167 rpcz_store.cc:276] 0709 12:25:43.469966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.470222 (+   256us) service_pool.cc:224] Handling call
0709 12:25:43.470722 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:43.471081 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:43.511085 (+ 40004us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.432316 (+921231us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b1800481ae0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:44.874472 (+442156us) spinlock_profiling.cc:243] Waited 9.07 ms on lock 0x7b1800481ae0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:25:44.877710 (+  3238us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.996146 (+118436us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:44.996217 (+    71us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:44.996269 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.996344 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.996485 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1275981,"prepare.queue_time_us":1411844,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":104,"raft.queue_time_us":18490,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":7257,"spinlock_wait_cycles":31208576,"thread_start_us":117958,"threads_started":1}
W20260709 12:25:45.506429 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46309.6us
W20260709 12:25:45.470826 27372 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.528s	user 0.002s	sys 0.002s
I20260709 12:25:45.511515 27408 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:45.356462 27365 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.447196 27105 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21.
I20260709 12:25:45.446960 27400 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.487829 26984 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:45.504206 27413 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:45.462603 26986 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:45.505461 27435 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.420711 27427 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.456104 27392 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:45.556532 27392 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:45.557468 27395 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:45.558106 27395 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:45.558008 27427 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.523952 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:45.525916 27289 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:45.514344 27298 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 12:25:45.522444 27413 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:45.535517 27421 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.518366 27400 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:45.555624 27365 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:45.560207 27450 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:25:45.516363 27429 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:45.465142 27381 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.514487 27437 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:45.523972 26986 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:45.595115 27413 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.597074 27413 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:45.600379 27421 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:45.513931 27348 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:25:45.606978 27348 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:45.607432 27348 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:45.605544 27317 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.611390 27381 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:45.617987 27193 consensus_queue.cc:1175] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "910f8c8d787e4abbb092413c1f90ebf3" responder_term: 21 status { last_received { term: 21 index: 15 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []" } } last_received_current_leader { term: 21 index: 15 } }
I20260709 12:25:45.591755 26986 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.608767 27348 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.629245 27460 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:45.644121 27460 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:45.558000 27392 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.645458 27407 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.645233 27460 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.652814 27407 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:45.654011 27429 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.658685 27460 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:45.472769 18021 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4387) took 2185 ms (client timeout 1996 ms). Trace:
W20260709 12:25:45.704592 18021 rpcz_store.cc:269] 0709 12:25:43.287387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.430329 (+142942us) service_pool.cc:224] Handling call
0709 12:25:45.472434 (+2042105us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b5400251a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:45.472645 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1888989056}
I20260709 12:25:45.509102 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:45.717485 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 208311us
W20260709 12:25:45.721117 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.721333 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:45.728578 18093 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:45.741091 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:45.740923 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 216899us
W20260709 12:25:45.745561 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.748775 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:45.753647 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:45.754933 18098 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:45.760838 27126 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:45.762403 27002 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.762456 27126 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.767549 27113 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:45.768860 27113 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.770519 27002 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:45.771144 27002 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:45.772035 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.775563 27113 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.778493 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.779282 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:45.779927 18167 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:45.771569 27002 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:45.777045 27113 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:45.784904 18808 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23. Ops: [22.16-22.16]
W20260709 12:25:45.780210 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.784550 27126 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.791448 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.792927 18169 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
W20260709 12:25:45.785400 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.765532 26714 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.799885 27034 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:45.799762 26714 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.800508 27034 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:45.801685 27380 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:45.768622 26875 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:25:45.818766 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:45.813788 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:45.822597 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.825009 27420 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:45.825763 27415 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.003s	sys 0.019s
W20260709 12:25:45.826438 27415 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.003s	sys 0.019s
I20260709 12:25:45.826746 27421 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:45.828003 27421 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:45.834301 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.826154 26714 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.846585 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3347) took 1999 ms. Trace:
I20260709 12:25:45.848239 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:45.849596 27388 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:45.850838 27388 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.854496 27437 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.856009 27348 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.855715 27437 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.858172 27407 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:45.859294 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:45.860668 18019 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:45.826146 27420 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:45.864099 27461 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:25:45.865235 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:45.785043 26899 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:45.868515 18020 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:45.865923 18017 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:45.873363 27268 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.874439 27268 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:45.649909 27392 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:45.875677 27388 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.876552 27388 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.878366 27419 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:45.877700 27388 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:45.866848 18019 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:45.877722 27268 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:45.849151 27034 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.18, Last appended by leader: 18, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:45.886402 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72896.4us
I20260709 12:25:45.983989 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:45.992502 27437 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:45.889679 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.889778 27399 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:45.992967 18017 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:45.993156 27435 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.996565 18170 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:45.989346 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4512) took 1822 ms (client timeout 1954 ms). Trace:
W20260709 12:25:45.997509 18095 rpcz_store.cc:269] 0709 12:25:44.167148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.167662 (+   514us) service_pool.cc:224] Handling call
0709 12:25:45.885851 (+1718189us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b5400052154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:45.989040 (+103189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1193160448}
I20260709 12:25:45.991027 27421 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:45.993422 27399 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.005293 27399 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:45.881556 27322 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:46.016479 27365 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.021414 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.018213 27272 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:46.045969 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.060545 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.058305 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.062816 27289 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.070953 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4532) took 1058 ms. Trace:
I20260709 12:25:46.071434 27289 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.071583 18091 rpcz_store.cc:276] 0709 12:25:45.000255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.011548 (+ 11293us) service_pool.cc:224] Handling call
0709 12:25:45.011933 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.012177 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:45.025919 (+ 13742us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.790000 (+764081us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b18003ea880. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:25:46.057788 (+267788us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:46.057907 (+   119us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:46.058023 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.058164 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.058431 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13668,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":5496,"spinlock_wait_cycles":12967808,"thread_start_us":252185,"threads_started":1}
W20260709 12:25:46.074503 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.076046 27289 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.027603 27418 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:46.078329 18136 leader_election.cc:400] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24.
I20260709 12:25:46.085290 18136 leader_election.cc:403] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:46.021466 27109 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:46.094187 27418 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.095685 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.096266 27388 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.096944 27420 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:46.103430 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.110337 27380 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:46.129459 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.110710 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.111006 27433 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:46.114038 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:45.847504 18022 rpcz_store.cc:276] 0709 12:25:43.846836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.173428 (+326592us) service_pool.cc:224] Handling call
0709 12:25:44.186717 (+ 13289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:44.186956 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:44.383734 (+196778us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.616033 (+232299us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.738659 (+1122626us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b18004e3520. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:25:45.739226 (+   567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:45.810188 (+ 70962us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:45.823503 (+ 13315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.845949 (+ 22446us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.846239 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1168008,"prepare.queue_time_us":343151,"prepare.run_cpu_time_us":142,"prepare.run_wall_time_us":18100,"raft.queue_time_us":1144612,"raft.run_cpu_time_us":618,"raft.run_wall_time_us":3596,"spinlock_wait_cycles":2589440,"thread_start_us":364148,"threads_started":2,"wal-append.queue_time_us":272656,"wal-append.run_cpu_time_us":2970,"wal-append.run_wall_time_us":1144590}
I20260709 12:25:46.117240 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.078398 27437 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.131675 18098 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:46.137167 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.084275 27407 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:46.088757 27322 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:46.146513 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.148258 18017 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:46.153566 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.121845 27365 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:46.161895 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.162216 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.087910 27272 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.166061 18019 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:46.166706 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.170064 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.171528 27424 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:46.171274 27322 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.172448 18024 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:25:46.175956 26986 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:46.177958 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.180316 18170 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:46.167183 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:46.187479 18020 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:46.149284 27420 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24.
W20260709 12:25:46.188432 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.189805 18023 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:25:46.190691 26984 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.665s	user 0.013s	sys 0.009s
I20260709 12:25:46.190626 27399 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.191323 27109 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.192245 18090 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:46.233865 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.194675 27149 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.195576 27365 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.195827 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.185567 27380 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.189903 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1394) took 1304 ms. Trace:
I20260709 12:25:46.189808 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.169975 18016 raft_consensus.cc:2376] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25. Re-sending same reply.
W20260709 12:25:46.205173 27262 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.929s	user 0.015s	sys 0.007s
I20260709 12:25:46.206136 27264 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.239217 18017 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:46.242053 26984 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.245235 27380 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.236526 18172 rpcz_store.cc:276] 0709 12:25:44.841446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.849715 (+  8269us) service_pool.cc:224] Handling call
0709 12:25:44.871810 (+ 22095us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:44.872236 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:45.309370 (+437134us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.788737 (+479367us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b1800398740. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:25:45.790063 (+  1326us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.790463 (+   400us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:46.082766 (+292303us) log.cc:844] Serialized 52 byte log entry
0709 12:25:46.083106 (+   340us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:46.083169 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.105797 (+ 22628us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:46.144060 (+ 38263us) raft_consensus.cc:1685] finished
0709 12:25:46.144095 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.144306 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30110,"prepare.queue_time_us":413512,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":132,"raft.queue_time_us":43534,"raft.run_cpu_time_us":684,"raft.run_wall_time_us":8951,"spinlock_wait_cycles":5143936,"thread_start_us":552280,"threads_started":2,"wal-append.queue_time_us":289710}
I20260709 12:25:46.164368 27113 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:46.245247 27399 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.247526 27348 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.266319 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.267194 18019 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:46.268299 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.276355 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.241614 27109 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.187857 27437 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.149848 27407 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.288237 27379 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.294312 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.313570 18021 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:25:46.314189 18021 pending_rounds.cc:85] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 13
W20260709 12:25:46.295394 27317 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.687s	user 0.010s	sys 0.017s
I20260709 12:25:46.257540 27002 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.327327 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.314464 18021 pending_rounds.cc:107] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 15.14
I20260709 12:25:46.330596 18021 raft_consensus.cc:2889] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.252971 27149 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.335469 18021 pending_rounds.cc:107] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260709 12:25:46.336010 18021 raft_consensus.cc:2889] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.337010 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.333899 18167 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:46.341223 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 263190us
I20260709 12:25:46.331626 26986 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:46.274686 27264 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:46.340029 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93178.5us
I20260709 12:25:46.310462 27113 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.353734 18019 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:46.357986 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:46.359234 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.360054 18020 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:25:46.364418 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.334074 27399 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.368743 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.372313 18017 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:46.383222 27289 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.388844 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.343793 27348 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.402917 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:46.417932 27413 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.743s	user 0.010s	sys 0.012s
I20260709 12:25:46.418807 27035 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.419668 27035 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.421296 27113 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.376390 27420 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.423539 18098 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:46.384308 27289 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.350235 27379 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.433350 27193 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:46.435858 27407 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.438612 26984 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.438338 27420 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.438879 18167 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:46.439040 26986 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.439754 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.440156 27418 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:46.446408 27460 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.786s	user 0.009s	sys 0.012s
I20260709 12:25:46.466221 18019 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:46.472303 18020 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:46.417944 27190 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.392513 27399 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.510669 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:46.522487 27113 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.522756 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:46.523358 18023 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:46.523308 27113 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.526209 27190 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.528939 27190 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.498174 27193 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:46.530998 27395 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.972s	user 0.012s	sys 0.012s
I20260709 12:25:46.530218 18166 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:46.531672 27395 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.532819 27193 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.536378 27002 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.537616 27289 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:46.563366 27466 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.216s	user 0.002s	sys 0.000s
W20260709 12:25:46.563567 27290 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.116s	user 0.013s	sys 0.007s
I20260709 12:25:46.564725 18168 raft_consensus.cc:606] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:46.565382 18168 raft_consensus.cc:614] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:46.548307 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:46.566372 18095 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:46.567448 27462 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.704s	user 0.007s	sys 0.001s
I20260709 12:25:46.569984 27348 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:46.482636 27400 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.702s	user 0.017s	sys 0.006s
I20260709 12:25:46.573357 27400 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.538568 27113 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.552162 18172 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:46.553053 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.576303 27400 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.576763 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45622 (request call id 101) took 1301 ms. Trace:
I20260709 12:25:46.554334 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:46.556255 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:46.555259 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.556649 18166 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:46.564851 27471 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
I20260709 12:25:46.582866 27272 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.583784 27190 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.584678 27437 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.586946 27420 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.587813 27149 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.590467 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.559762 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4535) took 1347 ms. Trace:
I20260709 12:25:46.550845 27035 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.596604 18168 rpcz_store.cc:276] 0709 12:25:45.275396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.275775 (+   379us) service_pool.cc:224] Handling call
0709 12:25:46.576472 (+1300697us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540029e0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b1853 00007f6c1b67f60a 00007f6c14c057d4 00007f6c14c05762 00007f6c14c05691 00007f6c14c0560c 00007f6c14c048c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:46.576619 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2961144704}
W20260709 12:25:46.597625 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44053.5us
I20260709 12:25:46.610687 18164 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:25:46.621371 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104246us
W20260709 12:25:46.621987 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 210276us
W20260709 12:25:46.624878 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.622503 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.631785 18019 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:46.585619 27264 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.629516 18090 rpcz_store.cc:276] 0709 12:25:45.211998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.376106 (+164108us) service_pool.cc:224] Handling call
0709 12:25:45.376579 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:46.281847 (+905268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.559375 (+277528us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54000523d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:46.559688 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1852489088}
I20260709 12:25:46.621178 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:46.642201 27034 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:46.645134 27381 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.033s	user 0.018s	sys 0.000s
I20260709 12:25:46.647553 18093 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:46.657475 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:46.662032 18097 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:46.672642 18171 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:46.674023 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.674896 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:46.678084 27422 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.678793 27422 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.659584 27035 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:46.681421 27035 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:46.682967 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46167.1us
I20260709 12:25:46.684005 18023 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:46.699327 18164 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:46.636824 27149 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:46.632237 27190 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.707041 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:46.713064 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 368662us
I20260709 12:25:46.714426 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.718148 18170 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:46.719050 18097 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:46.721719 27381 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:46.722564 27381 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:46.723644 27381 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:46.725010 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.729336 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:46.730190 27469 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:46.730625 18168 raft_consensus.cc:606] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:46.731248 18168 raft_consensus.cc:614] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:46.731005 27469 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.735059 27272 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:46.736645 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.737756 18169 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:46.738077 18166 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:25:46.738524 18166 pending_rounds.cc:85] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 13
I20260709 12:25:46.738811 18166 pending_rounds.cc:107] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:25:46.738907 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.739835 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:46.740417 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.740772 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:46.741443 18169 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:46.742766 27379 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.743065 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.743947 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.741506 18097 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:46.742429 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.742460 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:46.752025 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.740535 18172 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:46.739105 18166 raft_consensus.cc:2889] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.741688 18173 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
W20260709 12:25:46.759339 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50498.1us
I20260709 12:25:46.762421 18099 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:46.742547 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:46.776548 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69616.5us
I20260709 12:25:46.777518 18167 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:46.766165 18165 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:46.780115 18017 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
W20260709 12:25:46.788544 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.790995 27149 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.792637 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:46.781304 27400 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.794096 18092 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:46.794400 27400 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.797260 27437 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:46.798516 26986 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.802826 18090 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:46.804572 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:46.799391 27437 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.801029 27035 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.806870 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:46.814495 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.817709 18099 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:46.834635 18097 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:25:46.799479 27190 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.835001 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.837145 18095 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:46.837646 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.836253 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.813410 27437 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:46.850607 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.851733 18097 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:46.853009 18016 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:46.853264 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.854465 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.855221 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.855784 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:46.856165 18167 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:46.856472 27002 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.856832 27399 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:46.857481 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.858484 18091 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:25:46.858369 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.859402 18173 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:46.860236 18165 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:46.856647 18092 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:46.861953 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48786.3us
I20260709 12:25:46.864526 18167 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:46.866269 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:46.866972 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:46.867923 27474 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:46.869228 26986 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:46.869356 27424 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.869736 26986 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:46.869973 27424 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.871189 26986 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.872766 26986 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:46.822324 27420 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:46.874372 27420 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.876014 18092 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:46.877053 27420 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.822502 27035 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:46.873445 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:46.878100 27035 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.878829 18815 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:25:46.879047 18809 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:46.879215 27474 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:46.872897 27424 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.880210 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:46.880793 27035 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.880590 27474 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:46.881220 27424 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:46.881342 27474 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:46.881958 27424 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:46.883038 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.883183 27424 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.884712 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.886063 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:46.882592 27474 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.887195 27109 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:46.888568 27474 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:46.890448 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:46.891649 27109 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:46.895704 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.897336 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.899757 27422 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.900084 18094 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:46.858373 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.903146 18166 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:25:46.911755 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.912878 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.907421 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.900542 27109 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.914682 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3408) took 2364 ms. Trace:
I20260709 12:25:46.921988 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.914173 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:46.829926 27395 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:46.922019 27109 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.929184 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:46.929584 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:46.934762 18809 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:46.931535 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.936865 18021 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 27 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:46.922773 18171 rpcz_store.cc:276] 0709 12:25:44.536634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.536785 (+   151us) service_pool.cc:224] Handling call
0709 12:25:44.537182 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:46.673271 (+2136089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.823111 (+149840us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:46.900863 (+ 77752us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4851140352}
I20260709 12:25:46.857322 27002 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:46.943953 18020 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:46.944415 18023 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:46.942839 27399 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.950945 27399 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:46.955116 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.957432 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:46.962237 18099 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:46.963627 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:46.910362 27400 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:46.985556 18095 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:46.984477 18090 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:46.999356 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:47.009533 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:47.010468 18173 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:47.011063 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.012840 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:46.963496 27422 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:47.017097 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:47.020653 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:47.062072 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45093.8us
I20260709 12:25:47.054205 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:47.064530 18171 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:47.067414 18165 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:47.013940 27400 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.012398 27035 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:47.075743 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.077023 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.089375 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.089228 18091 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:47.048270 27348 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:25:47.019876 26986 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
I20260709 12:25:47.100364 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.102207 26986 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:47.102761 26986 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:47.104085 26986 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.104650 27149 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.097002 27035 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.098646 18809 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:47.108831 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:47.115023 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:47.099658 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.119747 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.098726 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.099315 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.098637 27289 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:47.127462 18806 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:47.128439 18812 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:47.082562 27422 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.098848 27473 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.105809 26986 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:47.141036 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.145033 27149 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:47.156109 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:47.157053 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.083653 27400 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.117949 27264 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:25:47.183017 27002 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.158344 27473 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.171645 27399 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.179111 27109 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.152849 27149 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.170125 27365 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.166515 18810 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:47.194978 27365 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.166505 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:47.196138 18811 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.196643 27149 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.121909 27379 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:25:47.198163 27418 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:47.198273 27400 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:47.174049 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:47.198911 27418 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.167529 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.190094 27469 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:47.179366 27476 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.107s	user 0.001s	sys 0.001s
I20260709 12:25:47.186230 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:47.167809 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51474.3us
I20260709 12:25:47.199875 27399 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:47.202297 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:47.204770 18173 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:25:47.207150 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55794.2us
I20260709 12:25:47.203750 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:47.170320 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.202533 26899 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:47.201503 27418 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.213234 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:47.228526 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.228526 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.231765 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:47.232610 18812 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:47.230376 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.230706 18093 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:47.192268 27035 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:47.178465 27422 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:47.235945 18165 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:25:47.235941 18808 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:47.263954 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44795.5us
W20260709 12:25:47.266136 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53040.4us
I20260709 12:25:47.229857 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:47.211117 27365 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:47.231652 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.034s	sys 0.027s
I20260709 12:25:47.213387 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:47.267998 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54743.6us
I20260709 12:25:47.214123 27379 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:47.269049 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.035s	sys 0.028s
W20260709 12:25:47.280929 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 604365us
W20260709 12:25:47.281018 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 322729us
I20260709 12:25:47.284811 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.217358 27474 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:47.219431 27322 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.027s	sys 0.079s
I20260709 12:25:47.287635 27379 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.287977 27399 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:47.218441 27264 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
W20260709 12:25:47.294478 27322 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.028s	sys 0.080s
I20260709 12:25:47.323184 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:47.289255 27392 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.203s	user 0.032s	sys 0.001s
W20260709 12:25:47.323709 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.228168 26899 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.235723 18813 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:47.289669 27474 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:47.363359 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:47.300814 27034 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.658s	user 0.017s	sys 0.000s
I20260709 12:25:47.320204 27149 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:47.319782 27379 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:47.326390 27365 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:47.320163 27399 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:47.382812 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47438.8us
I20260709 12:25:47.382869 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:47.374913 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80504.4us
W20260709 12:25:47.420463 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71925.8us
I20260709 12:25:47.416087 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:47.423599 18166 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:47.324182 26986 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:25:47.361742 27474 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.383553 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:47.387805 27388 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.010s	sys 0.028s
I20260709 12:25:47.429771 18165 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:47.373061 27149 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.380458 27469 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.384162 27399 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:47.380697 27365 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:47.436853 27388 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.013s	sys 0.028s
I20260709 12:25:47.428736 18090 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:47.434760 27317 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.449276 27149 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.449880 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:47.459692 18169 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.013s	sys 0.021s
W20260709 12:25:47.480944 18169 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.013s	sys 0.021s
I20260709 12:25:47.451859 27365 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:47.480218 18165 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
W20260709 12:25:47.467885 27462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.008s	sys 0.011s
I20260709 12:25:47.489471 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.493418 18095 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
W20260709 12:25:47.486383 18170 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.018s	sys 0.050s
W20260709 12:25:47.496728 18170 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.019s	sys 0.050s
I20260709 12:25:47.496833 18778 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:47.445179 27002 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:25:47.500372 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.500200 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:47.485520 27424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.019s	sys 0.046s
I20260709 12:25:47.502439 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:47.507680 18091 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:47.486311 27475 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:25:47.493767 27462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.009s	sys 0.012s
I20260709 12:25:47.509436 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:47.513684 18018 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.013s	sys 0.044s
W20260709 12:25:47.514258 18018 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.014s	sys 0.044s
I20260709 12:25:47.514475 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:47.515564 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4419) took 1408 ms. Trace:
W20260709 12:25:47.508394 27424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.019s	sys 0.047s
I20260709 12:25:47.455660 27418 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 12:25:47.475419 27317 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.464074 27469 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.511302 27268 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:25:47.512616 27475 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.516398 27388 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:25:47.540299 18173 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:25:47.504411 27113 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:47.504208 27109 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:47.474247 27482 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:25:47.505321 26986 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:25:47.504451 27460 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:25:47.520274 18018 rpcz_store.cc:276] 0709 12:25:46.107671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.163536 (+ 55865us) service_pool.cc:224] Handling call
0709 12:25:46.191865 (+ 28329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:46.192321 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:46.355549 (+163228us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.514867 (+1159318us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.515071 (+   204us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:47.515131 (+    60us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:47.515205 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.515304 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.515478 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":457983,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":4101,"raft.queue_time_us":41145,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"thread_start_us":1263300,"threads_started":3,"wal-append.queue_time_us":598983}
W20260709 12:25:47.561764 27381 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.010s	sys 0.042s
I20260709 12:25:47.557178 27473 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.580267 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.580933 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:47.562670 27475 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:47.560832 27002 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:47.563143 27380 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.004s	sys 0.035s
I20260709 12:25:47.566174 27109 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:47.601104 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 306233us
W20260709 12:25:47.577222 27381 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.010s	sys 0.043s
I20260709 12:25:47.577473 27035 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.615365 18811 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:47.620734 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:47.624006 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.594429 27113 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:47.632259 18164 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:47.613366 27380 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.005s	sys 0.035s
I20260709 12:25:47.595742 27422 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.615483 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:47.601234 26899 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:47.654886 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52622.4us
I20260709 12:25:47.612829 26986 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:47.616057 27424 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:47.623265 27034 raft_consensus.cc:2764] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:25:47.639890 27035 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.637631 27113 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.596243 27002 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:47.689508 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.669788 27473 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:47.690284 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.697132 27483 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:47.697862 27484 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:25:47.673530 26986 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:47.680200 27348 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:47.699621 27289 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.530s	user 0.010s	sys 0.005s
I20260709 12:25:47.670406 27422 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:47.718806 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 209498us
I20260709 12:25:47.684742 27113 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:47.730011 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 155363us
I20260709 12:25:47.883710 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.884615 18090 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:47.884732 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:47.885329 27400 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:47.886063 27400 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.886966 27379 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.887844 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:47.705509 27473 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:47.889026 27473 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:47.890192 27379 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.890575 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.732782 27317 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.732750 27483 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.892385 27400 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.893918 27483 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.734896 27469 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.723737 27381 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:47.894954 27381 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.895370 27381 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:47.767786 27437 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.833s	user 0.016s	sys 0.000s
I20260709 12:25:47.898288 27437 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.900676 27437 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:47.767908 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3673) took 1221 ms. Trace:
I20260709 12:25:47.771451 27002 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.730773 27495 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:47.901724 27495 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:47.902228 27495 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:47.779980 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:47.903427 18165 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:47.786715 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3569) took 1052 ms. Trace:
I20260709 12:25:47.790820 27264 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:47.905808 27264 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.904903 27399 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.907217 18813 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:47.908349 27264 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.791790 27419 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:47.909032 27419 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:47.910116 27419 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.911567 27419 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:47.912515 27002 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:47.680578 27424 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.913968 27002 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:47.914402 27002 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:47.915436 27424 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.915601 27002 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.917146 27002 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:47.919322 27514 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:47.920042 27514 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:47.920907 27399 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.796859 27422 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.716707 27497 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:47.922632 27497 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:47.923069 27497 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.924561 27381 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:47.809379 27486 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.110s	user 0.002s	sys 0.000s
W20260709 12:25:47.811683 27493 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.112s	user 0.002s	sys 0.000s
I20260709 12:25:47.927230 27381 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:47.824141 27035 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:47.726030 27289 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:47.928615 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1436) took 1290 ms. Trace:
I20260709 12:25:47.904232 18164 rpcz_store.cc:276] 0709 12:25:46.734326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.766678 (+ 32352us) service_pool.cc:224] Handling call
0709 12:25:46.789851 (+ 23173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:47.645281 (+855430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.786314 (+141033us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b5400032ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:47.786614 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1936593792}
W20260709 12:25:47.857441 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 233566us
I20260709 12:25:47.862864 27509 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:25:47.864014 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:47.864153 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 435122us
I20260709 12:25:47.930505 27509 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:47.901279 18098 rpcz_store.cc:276] 0709 12:25:46.546827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.554053 (+  7226us) service_pool.cc:224] Handling call
0709 12:25:46.554511 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.555224 (+   713us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:46.977464 (+422240us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.466961 (+489497us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.467263 (+   302us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:47.492927 (+ 25664us) log.cc:844] Serialized 26 byte log entry
0709 12:25:47.501294 (+  8367us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:47.501341 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.501430 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.767554 (+266124us) raft_consensus.cc:1685] finished
0709 12:25:47.767592 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.767807 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7136,"prepare.queue_time_us":807597,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":32641,"raft.queue_time_us":6604,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":15682,"spinlock_wait_cycles":3968,"thread_start_us":857691,"threads_started":2,"wal-append.queue_time_us":626885}
I20260709 12:25:47.929107 18096 rpcz_store.cc:276] 0709 12:25:46.637660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.648204 (+ 10544us) service_pool.cc:224] Handling call
0709 12:25:46.702717 (+ 54513us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:46.823015 (+120298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:46.823053 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:46.944716 (+121663us) log.cc:844] Serialized 52 byte log entry
0709 12:25:47.473250 (+528534us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:47.810685 (+337435us) spinlock_profiling.cc:243] Waited 9.06 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:47.871420 (+ 60735us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.871722 (+   302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.871840 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.928048 (+ 56208us) raft_consensus.cc:1685] finished
0709 12:25:47.928202 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.928394 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88901,"prepare.queue_time_us":26605,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":43350,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":21744,"spinlock_wait_cycles":20826880,"thread_start_us":847716,"threads_started":2,"wal-append.queue_time_us":710691}
W20260709 12:25:47.864856 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 284722us
I20260709 12:25:47.869104 27322 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:25:47.874406 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 184174us
I20260709 12:25:47.933197 27322 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:47.933378 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.933482 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:47.934885 27483 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:47.936800 27483 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:47.936702 27322 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.879765 27474 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.937546 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:47.937925 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.938318 27322 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:47.939059 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:47.939914 27474 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.939442 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:47.940725 27474 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:47.903056 27475 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.941511 27348 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:47.942051 27348 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.942456 27348 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.942569 27035 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:47.880693 18166 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:47.881440 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.943552 27475 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.944037 18093 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
W20260709 12:25:47.882443 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 454992us
I20260709 12:25:47.945322 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:47.945803 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.946070 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:47.946573 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3582) took 1093 ms. Trace:
I20260709 12:25:47.947398 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:47.948055 18091 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.948319 27420 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:47.921110 27514 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.950337 27420 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:47.946944 18170 rpcz_store.cc:276] 0709 12:25:46.853572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.860332 (+  6760us) service_pool.cc:224] Handling call
0709 12:25:46.860697 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:46.861069 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:47.502395 (+641326us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.946148 (+443753us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:47.946209 (+    61us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:47.946292 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.946376 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.946512 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91685,"prepare.queue_time_us":711500,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2499,"raft.queue_time_us":91928,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":101,"spinlock_wait_cycles":81152,"thread_start_us":950856,"threads_started":2,"wal-append.queue_time_us":323193}
I20260709 12:25:47.951582 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.951848 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3584) took 1097 ms. Trace:
I20260709 12:25:47.952256 18093 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:47.953298 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:47.916189 27424 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:47.793009 26986 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.954110 18096 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:47.912446 27149 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:47.955101 26986 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:47.957122 27149 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:47.932014 27509 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 27.23, Last appended by leader: 23, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.950842 27420 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:47.699615 27109 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:47.957703 27290 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.882990 18094 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: [27.22-27.22]
I20260709 12:25:47.961797 27517 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:47.952391 18169 rpcz_store.cc:276] 0709 12:25:46.854429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.863387 (+  8958us) service_pool.cc:224] Handling call
0709 12:25:46.863751 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:46.864082 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:47.487121 (+623039us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.940420 (+453299us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.951365 (+ 10945us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:47.951436 (+    71us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:47.951507 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.951605 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.951774 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82059,"prepare.queue_time_us":746121,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":120,"raft.queue_time_us":74912,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"thread_start_us":977987,"threads_started":2,"wal-append.queue_time_us":374138}
I20260709 12:25:47.962548 27517 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.963135 27517 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.994809 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.996068 27424 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:47.996381 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:47.996621 27424 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.966428 27365 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:47.967499 27483 consensus_queue.cc:1175] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "910f8c8d787e4abbb092413c1f90ebf3" responder_term: 21 status { last_received { term: 18 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: [20.15-20.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:47.970245 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.014698 18170 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:47.971336 27437 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:47.971709 18169 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:48.018234 18091 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:47.972486 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.019268 18171 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
W20260709 12:25:47.985137 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.990509 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:47.969898 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:47.915603 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:47.991679 27109 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:48.003973 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45524.7us
I20260709 12:25:47.997046 27424 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:48.007776 27348 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:48.010201 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71240.2us
W20260709 12:25:48.012188 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 66978.1us
I20260709 12:25:48.035423 17987 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.010978 27379 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:47.997368 27400 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:48.026177 27501 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:48.024384 27437 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.026790 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:48.030161 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 90811.5us
I20260709 12:25:48.024792 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.039710 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:48.044323 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:48.044484 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.046334 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.046607 18813 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:48.043309 27501 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:48.048908 27501 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.049257 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.050773 18810 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:48.054104 27519 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:48.054239 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.054801 27519 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.055469 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:48.048928 18170 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: [23.11-25.12]
I20260709 12:25:48.055305 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.058406 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.060210 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.061726 27149 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:48.064889 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.066556 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.066987 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.067293 18166 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:48.067773 27500 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:48.068421 27500 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:48.069506 27500 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.070096 27317 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:48.070595 27317 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.070998 27317 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:48.072111 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.072952 18170 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:25:48.074043 18091 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:48.074920 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.077914 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.043447 27483 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:48.041716 27400 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.088826 27400 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:48.088857 27483 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:48.078699 18172 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.090314 27483 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.085561 27513 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:48.082336 26986 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:48.078567 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.093528 27513 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.095001 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.065750 18165 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:48.080359 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.096942 26986 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:48.097146 27513 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.097483 26986 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:48.055091 27365 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.081642 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.081887 27475 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:48.082654 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 167200us
I20260709 12:25:48.082414 27109 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:48.067517 27392 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:25:48.080173 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.097998 26986 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.072904 27419 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:48.055114 27437 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:48.055213 27519 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:48.085959 27474 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:48.087190 27348 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.088339 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.100906 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.105553 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:48.107722 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:48.109524 18172 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:48.056092 27514 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:48.117753 18166 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:48.083473 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.121939 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:48.125573 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53581.3us
I20260709 12:25:48.126528 27272 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:48.125466 27379 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:25:48.144614 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43781.5us
I20260709 12:25:48.147676 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.136512 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.126446 27513 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:48.119519 27422 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:25:48.152889 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.157063 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.158474 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.159250 18813 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.128696 27190 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:25:48.168923 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.146687 27034 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:48.174712 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.176409 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.177011 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.146272 27272 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:48.170284 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.131994 27501 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.152329 27513 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.154625 27109 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:48.156988 27475 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:48.162431 27418 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:48.188771 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.183311 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.182725 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.156845 27348 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:48.163411 27514 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:48.147080 27473 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.184365 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:48.170800 27483 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:48.201231 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.180598 27419 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:48.181416 27482 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.187155 27513 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.195802 27435 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:48.203655 27264 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:48.188791 27475 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.194830 27388 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:48.188462 27109 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:48.220636 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50406.6us
I20260709 12:25:48.194770 27348 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.237280 27424 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:48.241171 27524 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:48.228645 27419 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:48.207451 27483 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.206880 27035 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:48.230490 27482 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.265985 27473 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.208436 27418 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:48.272866 27504 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:48.273625 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114360us
W20260709 12:25:48.242237 27520 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
W20260709 12:25:48.274843 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 96923.2us
I20260709 12:25:48.231293 27435 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.277376 18164 raft_consensus.cc:2368] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:25:48.187364 27497 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.278219 27388 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:48.275691 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 170197us
I20260709 12:25:48.274006 27348 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:48.298868 27501 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.288455 27419 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.313442 27420 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.291059 27424 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.315284 27474 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.315914 27474 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.272027 27109 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:48.255431 27475 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:48.293977 27483 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:48.291807 27290 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:48.292454 27272 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.319919 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:48.310330 18170 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:48.300359 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:48.301970 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.304415 18813 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
W20260709 12:25:48.307024 18780 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.637s (SENT)
I20260709 12:25:48.319087 27504 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:48.328738 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 171734us
I20260709 12:25:48.335863 18096 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:25:48.306133 27435 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:48.308698 27497 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.339738 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:48.299738 27482 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.347079 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.354488 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:48.355430 18172 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:48.356004 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.327471 27420 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.362946 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.363790 18169 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:48.364650 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:48.369652 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:48.275146 27517 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.302325 27418 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.378897 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.379213 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:48.379827 18168 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:48.380628 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:48.380982 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80741.7us
I20260709 12:25:48.383380 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.300434 27519 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.389530 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.397037 27519 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.344053 27504 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.400245 27504 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.379041 27381 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:48.366106 27509 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:48.403505 27495 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.501s	user 0.014s	sys 0.006s
I20260709 12:25:48.404239 27495 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.340008 27190 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:25:48.405618 18812 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:48.367657 27473 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:48.364859 27474 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.414891 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.371292 27497 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:48.417934 27525 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:48.311167 27514 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.420550 27495 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:48.390204 27399 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:25:48.363807 27422 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:25:48.311285 27388 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.390623 27420 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:48.461647 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.438217 18169 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:25:48.390235 27365 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:48.391032 27379 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:48.406641 27419 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.464484 27419 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.465297 27322 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:48.465783 27322 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:48.466774 27322 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.470376 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.471165 18808 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.449069 27002 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:48.475996 27002 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.478919 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.443161 27525 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:48.443696 27113 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.560s	user 0.011s	sys 0.009s
I20260709 12:25:48.483809 18096 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:48.458074 27509 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:48.459822 27317 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.467278 27419 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.454066 27400 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.482208 18164 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:48.489476 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.443650 27482 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:48.497014 18170 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:48.498775 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.500027 18169 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:25:48.516402 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.522809 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.524827 18166 pending_rounds.cc:85] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 14
I20260709 12:25:48.525260 18166 pending_rounds.cc:107] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:25:48.528370 18166 raft_consensus.cc:2889] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:48.526089 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.533135 18173 raft_consensus.cc:2368] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:48.447304 27348 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.494572 27509 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:48.538983 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 169455us
I20260709 12:25:48.491132 27525 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.544719 26986 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.496156 27113 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.547366 26986 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.548404 18172 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:25:48.531910 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.514698 27420 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:48.578872 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.579918 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.527202 27482 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:48.587023 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 235560us
I20260709 12:25:48.588589 18167 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:25:48.585489 18021 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:48.593103 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.594312 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.595319 18023 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 30: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:48.597340 27517 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.573983 27420 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:48.599262 27420 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.606218 18022 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:48.618031 18096 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:48.618481 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.571996 27113 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:48.619215 18023 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:48.626817 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.627175 27035 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:48.627820 27035 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.605149 27002 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.629930 27482 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.630998 27400 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.625923 27422 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:25:48.636384 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.637145 18096 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:25:48.642684 18093 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:25:48.571923 27513 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.646816 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.648082 18023 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
W20260709 12:25:48.648579 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.650514 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.652040 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.653898 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.655473 18093 raft_consensus.cc:2410] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:48.658311 18091 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:48.658680 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.650331 27514 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:25:48.660723 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.662061 27540 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.662743 27540 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.663839 18812 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:48.661490 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.682610 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:48.696905 27272 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.685271 27109 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.697633 18811 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:25:48.701767 27109 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.698123 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.724705 18806 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
W20260709 12:25:48.725898 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 418871us
I20260709 12:25:48.734172 27317 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.737440 27540 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.719208 27483 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.740504 18018 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:48.719305 27513 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.743649 27483 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.746882 27525 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.747732 27290 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:48.748478 18165 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:48.748437 27290 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.749234 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.750502 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.751641 18165 raft_consensus.cc:2410] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:48.753523 18164 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:25:48.752830 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.756237 18172 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:48.752818 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.758414 18165 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:48.758780 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.759552 18093 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:25:48.760990 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.759583 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.764998 27290 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:48.766330 27290 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:48.766774 27290 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:48.767174 27290 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:48.722159 27541 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:25:48.773511 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.653463 27509 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.776950 27509 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:48.780815 18168 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
W20260709 12:25:48.802353 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.814818 27482 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:48.815701 27482 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.816382 27322 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.818363 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.819720 18018 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 26 because replica has last-logged OpId of term: 25 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 10.
I20260709 12:25:48.821373 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.571599 27435 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:48.823012 18165 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:48.835654 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 211203us
I20260709 12:25:48.853371 18096 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:48.792806 27290 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.859692 18172 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:48.856317 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:48.873998 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.873890 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 100510us
I20260709 12:25:48.875919 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.879953 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:48.882988 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.884995 18097 raft_consensus.cc:2368] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26.
I20260709 12:25:48.890357 18806 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:48.884069 18021 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:48.892362 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.892256 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:48.851303 27348 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:48.894701 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:48.902419 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.852209 27289 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:48.906116 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 307828us
I20260709 12:25:48.908838 18808 raft_consensus.cc:2368] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:48.906316 26986 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.801613 27475 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:48.916280 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.916847 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.916049 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.918108 18812 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:48.896512 27420 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:48.920902 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.921074 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:48.927681 18810 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:48.918244 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:48.922574 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.929734 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:48.931972 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:48.933388 18098 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:48.937896 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:48.938536 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.942991 18809 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:48.945544 18093 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:48.946798 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:48.947649 18091 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:48.951704 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:48.954674 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.954766 18170 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 27 index: 23. Preceding OpId from leader: term: 29 index: 24. (index mismatch)
I20260709 12:25:48.956563 18171 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:48.924571 27272 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.965476 18019 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:48.927241 27289 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:48.965890 18020 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [22.11-23.12]
I20260709 12:25:48.973356 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:48.973992 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.944156 27379 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:48.930836 27540 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:48.943431 27420 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:48.932587 27482 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:48.971302 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:48.999477 18171 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.973356 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:48.981127 18096 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:25:48.982723 18137 leader_election.cc:400] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:48.975546 18168 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:48.992092 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.992929 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:49.020246 18137 leader_election.cc:403] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.032859 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:48.922010 26986 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.041317 18020 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 33: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:48.964454 27500 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:49.034838 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:49.034946 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 55816.8us
I20260709 12:25:49.044992 18017 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 26 because replica has last-logged OpId of term: 25 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 11.
I20260709 12:25:48.979676 27272 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:49.037377 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49712.1us
W20260709 12:25:49.048606 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59019.2us
I20260709 12:25:49.048909 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.048786 27419 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:49.049613 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:49.050226 27419 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:49.050745 27419 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:49.052029 27109 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:49.051748 27419 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.056032 18164 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 26 index: 12. (index mismatch)
I20260709 12:25:49.060551 27109 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:49.071201 27035 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.071492 27109 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.074412 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:48.987114 27379 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.107429 18810 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:49.119436 18099 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 12. (term mismatch)
I20260709 12:25:49.120074 18099 pending_rounds.cc:85] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 11
I20260709 12:25:49.120429 18099 pending_rounds.cc:107] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 23.12
I20260709 12:25:49.120800 18099 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:49.089918 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.122870 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:49.123764 18168 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:49.118088 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.125592 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.052691 27539 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:49.053619 27527 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.134318 18095 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:49.063169 27483 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.138520 18015 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 19: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:49.132236 18091 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:25:49.117879 27379 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:49.161633 18023 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:49.095077 27500 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:49.160343 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.165663 18020 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:49.168067 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.168936 18023 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:49.172982 18169 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.655s	user 0.009s	sys 0.000s
I20260709 12:25:49.173420 18169 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:49.053701 27540 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.182821 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.187512 18023 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:49.188959 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.190640 18015 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:49.191563 18780 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.753s (SENT)
I20260709 12:25:49.192034 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:49.209877 18812 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:25:49.211886 27548 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:25:49.212474 18021 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:49.213585 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.214303 18021 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:49.218250 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:49.219666 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.221567 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.160249 27392 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.160820 27317 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.223212 18021 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:49.161530 27525 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:49.224689 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.226876 18020 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:49.230031 18016 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:49.232715 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.233440 18021 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.234754 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.159511 27527 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.237566 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.163316 27483 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.238837 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.243299 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:49.246483 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.251816 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.231618 27290 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.254678 18020 raft_consensus.cc:2376] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24. Re-sending same reply.
W20260709 12:25:49.265286 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.269382 18017 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:25:49.271173 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89410.3us
I20260709 12:25:49.272030 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:49.272223 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.238824 27435 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.273360 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.274434 18024 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:49.222926 27420 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.276170 18021 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:49.230913 27392 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.277025 18015 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:49.278282 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.280879 18021 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.262398 27113 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:49.286443 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.233196 27317 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.258106 27475 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.288008 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.291260 18171 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:49.277015 27289 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.290647 18015 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 18.
I20260709 12:25:49.295352 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.296873 18024 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:49.277295 27511 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:49.300037 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.297081 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.303359 18020 raft_consensus.cc:2368] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27.
I20260709 12:25:49.307510 18021 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:49.309043 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.310055 18021 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:49.311157 18018 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:25:49.312896 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.314189 18018 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:49.314291 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.314525 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.315174 18024 raft_consensus.cc:2368] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:49.317973 18021 raft_consensus.cc:2410] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:25:49.318495 18024 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:25:49.324303 27317 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.273913 27500 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:49.326275 27322 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.509s	user 0.013s	sys 0.005s
I20260709 12:25:49.331135 27392 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.332865 27527 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.308799 27511 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:49.334944 18022 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
W20260709 12:25:49.303347 27549 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.174s	user 0.002s	sys 0.000s
I20260709 12:25:49.370301 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:49.372541 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:49.373716 18169 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:49.387523 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.344882 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:49.396763 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53880.5us
I20260709 12:25:49.302489 27272 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.388509 18020 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
W20260709 12:25:49.433717 27551 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.566s	user 0.010s	sys 0.005s
I20260709 12:25:49.434181 27379 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.434880 27540 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.316498 27529 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:49.436604 18019 raft_consensus.cc:2368] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:49.436965 27379 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.437228 27527 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.438110 27527 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.442327 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:49.443293 18172 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:49.334812 27511 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:49.454869 27509 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.677s	user 0.010s	sys 0.004s
I20260709 12:25:49.456019 27509 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:49.456782 27509 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:49.457669 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.459334 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:49.464586 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.467316 27553 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.604s	user 0.005s	sys 0.006s
I20260709 12:25:49.461401 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:49.475600 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:49.476665 18169 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:49.465421 26986 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.479202 18166 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:49.441454 27527 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.481164 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:49.487157 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.488806 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:49.489003 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.491820 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:49.492656 18172 raft_consensus.cc:2368] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27.
I20260709 12:25:49.494067 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.494949 18015 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:49.493311 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:49.469174 27322 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:49.504740 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:49.508617 18812 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:49.499893 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:49.503835 18169 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:49.510200 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.511102 18015 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:49.513536 18811 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:49.493789 27504 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:49.530596 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.531894 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.532809 18018 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.484184 27475 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:49.539834 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:49.540884 18017 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:49.546993 18020 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:49.543268 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43501.8us
I20260709 12:25:49.553758 17989 leader_election.cc:400] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27.
I20260709 12:25:49.554410 17989 leader_election.cc:403] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.555364 18024 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:49.529518 27322 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:49.529947 27540 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:49.565653 18017 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:49.566385 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 290059us
I20260709 12:25:49.569403 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:49.570708 27509 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.572088 27419 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:49.585944 27562 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:49.567281 27540 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.588372 27540 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:49.590829 27035 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:49.566815 27322 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:49.597725 27483 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:49.598713 27483 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.547971 27513 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 12:25:49.597899 27475 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:25:49.599259 27483 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.616297 27483 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:49.620352 27555 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.753s	user 0.014s	sys 0.005s
I20260709 12:25:49.622689 27529 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.623255 27529 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:49.626484 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57212.9us
I20260709 12:25:49.628508 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:49.628844 27562 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:49.629441 18094 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:49.629626 27562 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:49.630782 27562 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:49.630867 18095 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:25:49.631536 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:49.631888 27562 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:49.632841 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:49.633654 18093 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:49.636052 18091 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:49.637244 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:49.639038 27552 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.777s	user 0.005s	sys 0.002s
I20260709 12:25:49.632817 27562 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.642629 27562 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:49.643249 27550 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.778s	user 0.005s	sys 0.009s
I20260709 12:25:49.643599 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:49.644713 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:49.647759 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:49.653714 27504 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:49.550995 27503 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:49.654193 18019 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:49.658354 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:49.662027 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 96054.8us
W20260709 12:25:49.678666 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82186.9us
I20260709 12:25:49.679178 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:49.689709 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121908us
I20260709 12:25:49.694788 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:49.695783 18171 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:49.541193 26986 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:25:49.702636 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:49.707106 18016 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:25:49.720363 18780 leader_election.cc:400] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27.
I20260709 12:25:49.720785 18780 leader_election.cc:403] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.710672 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:49.722559 18019 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:49.723683 18172 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:49.724120 18167 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:49.663336 27317 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.706774 27483 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:49.698503 27497 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:25:49.662119 27113 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:25:49.740725 18018 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:49.757292 18022 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:49.772197 27435 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:49.708532 27509 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:25:49.708703 27514 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:25:49.739981 27558 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:49.738744 27500 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
I20260709 12:25:49.753281 27504 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:49.761205 27035 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:25:49.762617 27483 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:49.794588 27513 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:49.761940 27511 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27.
I20260709 12:25:49.810320 18021 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:49.827764 27322 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.822886 27482 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:49.834892 26986 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:49.849512 27420 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:49.804661 27473 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:49.811533 27504 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.809365 27558 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:49.847097 18172 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:25:49.793452 27475 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27.
I20260709 12:25:49.750844 27317 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.866546 27513 consensus_queue.cc:1243] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:25:49.870373 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 320957us
I20260709 12:25:49.812392 27109 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
W20260709 12:25:49.870852 27515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.989s	user 0.014s	sys 0.028s
I20260709 12:25:49.897549 27435 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:49.897989 27317 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.898195 27435 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:49.869798 27473 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:49.876713 27482 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.899334 27435 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.899868 27473 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.900749 27435 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:49.901067 27473 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:49.883780 27515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.014s	sys 0.028s
I20260709 12:25:49.906189 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:49.907931 17987 leader_election.cc:400] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26.
I20260709 12:25:49.908339 17987 leader_election.cc:403] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.909973 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:49.911187 18810 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 29 because replica has last-logged OpId of term: 27 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 21.
I20260709 12:25:49.913291 27317 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:49.914031 27379 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:49.913952 27317 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.914911 27379 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:49.917173 27317 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.918917 27113 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:49.919441 27113 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:49.920478 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.920445 27113 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.929785 27503 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:49.871706 27420 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:49.936581 27420 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:49.940548 27540 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.872769 27558 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:49.942941 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:49.943686 27483 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:49.872900 27504 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:49.947306 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:49.979242 18810 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:49.957026 18023 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:49.960357 18015 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
W20260709 12:25:50.000777 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.993523 27113 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:50.017436 27420 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:50.019896 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:50.018397 27420 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.051445 18172 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:50.062054 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 94659.4us
I20260709 12:25:50.032096 27500 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.063606 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:49.987174 27529 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:50.073313 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 314074us
I20260709 12:25:50.045701 27482 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.074891 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:50.091174 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:50.062304 27497 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:50.095646 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 167815us
W20260709 12:25:50.087803 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.535s (SENT) [suppressed 21 similar messages]
I20260709 12:25:50.109560 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.073038 27500 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.114755 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:50.116560 18098 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:50.116817 27500 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:50.116997 18098 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:50.130263 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:50.134512 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 414234us
I20260709 12:25:50.113759 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.134748 18091 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:50.072860 27035 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:50.142793 18780 leader_election.cc:400] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:50.148649 18091 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:50.085243 27566 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.085665 27514 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:50.152189 18780 leader_election.cc:403] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:25:50.139161 18064 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.096963 27482 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:50.180792 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 86370.6us
I20260709 12:25:50.102272 27475 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:50.141283 18098 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.102712 27400 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.100660 27419 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:50.140554 27564 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.204535 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:50.204950 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:50.210927 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71947.2us
I20260709 12:25:50.212894 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:50.205334 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.249801 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:50.250695 18097 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:50.231875 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94920.1us
I20260709 12:25:50.252409 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.198522 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4458) took 1022 ms. Trace:
I20260709 12:25:50.231293 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:50.140600 27568 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.256026 18092 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:50.256675 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.259446 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:50.261430 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:50.253232 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.271674 18810 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:50.190600 27035 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.279366 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:50.280429 18810 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:50.261660 18094 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:50.253232 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.235423 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.283685 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.253896 18023 rpcz_store.cc:276] 0709 12:25:49.176531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.188621 (+ 12090us) service_pool.cc:224] Handling call
0709 12:25:50.198199 (+1009578us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b5400268754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:50.198418 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1711243520}
I20260709 12:25:50.200814 27511 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.237677 18098 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:50.109989 27497 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.285153 18090 raft_consensus.cc:2368] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27.
I20260709 12:25:50.311223 18094 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:25:50.278172 27513 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:25:50.312327 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 224545us
I20260709 12:25:50.284111 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:50.231345 27348 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.325690 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:50.225440 27317 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:50.224562 27503 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:50.329468 18138 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.577s (SENT)
I20260709 12:25:50.327014 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:50.330412 18138 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.328080 18172 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:50.210276 27501 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.342541 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:50.344871 18169 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:50.239238 27529 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.263001 27562 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:50.259963 27113 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24.
I20260709 12:25:50.364177 18098 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:25:50.395884 18098 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 12
I20260709 12:25:50.396874 18098 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260709 12:25:50.389087 26986 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:25:50.398118 27475 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:50.398528 27475 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:50.399420 27475 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.400552 18098 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:50.388267 18092 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
W20260709 12:25:50.402683 27567 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:25:50.403229 18810 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:25:50.332504 27473 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:25:50.347828 27540 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:50.348264 27348 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:50.421131 27435 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.005s
I20260709 12:25:50.421969 27435 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:50.421876 27562 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.422542 27435 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:50.422943 27435 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:50.424466 27562 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:50.425105 27562 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.306612 27568 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.437515 27322 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:50.437891 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4459) took 1210 ms. Trace:
I20260709 12:25:50.258311 27400 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.210095 27514 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.438277 18024 rpcz_store.cc:276] 0709 12:25:49.227344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.628807 (+401463us) service_pool.cc:224] Handling call
0709 12:25:49.629291 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:49.646108 (+ 16817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:49.646171 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:49.659398 (+ 13227us) log.cc:844] Serialized 26 byte log entry
0709 12:25:49.952631 (+293233us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:50.095273 (+142642us) log.cc:844] Serialized 14 byte log entry
0709 12:25:50.097587 (+  2314us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.097716 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:50.437584 (+339868us) raft_consensus.cc:1685] finished
0709 12:25:50.437625 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:50.437800 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":190362,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":19509,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":17316,"thread_start_us":400762,"threads_started":2,"wal-append.queue_time_us":429099}
I20260709 12:25:50.510649 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:50.518956 18166 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:50.454221 27509 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:50.256832 27566 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.425598 27562 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.306109 27564 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:50.544325 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.328032 27511 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:50.549319 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 289956us
I20260709 12:25:50.535972 18172 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:25:50.551482 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.558820 18169 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:50.552773 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:50.572631 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 317916us
I20260709 12:25:50.580430 27322 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:50.581344 27322 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:50.581758 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.582654 27322 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:50.547610 27379 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.632s	user 0.019s	sys 0.000s
I20260709 12:25:50.382740 27503 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.588214 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:50.511037 27501 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.605445 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.606848 27290 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.607478 27290 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:50.608409 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 271049us
I20260709 12:25:50.613128 18166 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
W20260709 12:25:50.615970 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 286635us
W20260709 12:25:50.616745 18138 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.597s (SENT)
I20260709 12:25:50.591950 27392 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:50.621973 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.616833 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:50.602492 27379 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26.
I20260709 12:25:50.615736 27473 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:50.615104 27035 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:50.614638 27529 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:50.640525 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.642225 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.647907 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.618896 27589 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:50.656690 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51335us
I20260709 12:25:50.659304 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.617401 27289 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:50.663935 27530 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:50.665270 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:50.668969 27530 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:50.669561 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82671.3us
I20260709 12:25:50.633122 27419 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:50.680567 27497 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.681175 27420 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:50.681525 27564 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:50.655272 27504 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.664s	user 0.021s	sys 0.003s
W20260709 12:25:50.655520 27558 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.665s	user 0.007s	sys 0.007s
I20260709 12:25:50.650161 27272 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:50.702366 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.703604 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:50.708303 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:50.714074 18021 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:25:50.727145 18024 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:25:50.776762 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.728232 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.779906 18091 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:25:50.670380 27392 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:25:50.653970 27529 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.794092 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:50.802107 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 548921us
I20260709 12:25:50.803661 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.778649 27588 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:25:50.812366 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 195644us
W20260709 12:25:50.813733 18138 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.627s (SENT)
I20260709 12:25:50.814355 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.682315 27592 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.142s
I20260709 12:25:50.694556 27589 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.111s
W20260709 12:25:50.821623 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.826395 27582 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
W20260709 12:25:50.828644 27578 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260709 12:25:50.829365 27348 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.830271 27566 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.831629 27503 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.832687 27587 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:50.832984 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:50.833993 27562 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:50.678009 27530 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:50.848773 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 175916us
I20260709 12:25:50.807025 27590 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:50.677765 27569 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:50.817589 27540 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.822935 27400 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.791545 27272 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.856309 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.685847 27593 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.822506 27392 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:50.823832 27588 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:50.862990 27529 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.865000 27587 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:50.872087 27569 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:50.859550 27562 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:50.859637 27540 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.852759 27530 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:50.853017 27589 consensus_queue.cc:1243] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:25:50.687747 27419 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.851527 27497 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:50.897413 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77363.8us
I20260709 12:25:50.901049 18023 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:50.908254 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:50.728605 27504 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.950156 27504 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:50.920825 27435 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.952831 27435 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:50.922277 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89434.9us
I20260709 12:25:50.900584 27595 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:50.954293 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.954555 27595 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.956555 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.902983 27593 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.913389 27562 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:50.958067 27562 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:50.932631 27540 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:50.931420 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:50.959161 18095 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:50.917512 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 260868us
I20260709 12:25:50.959154 18090 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:50.943784 18808 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 33.
I20260709 12:25:50.943980 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:50.728406 27558 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:50.963284 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:50.960297 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:50.972323 18811 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
W20260709 12:25:50.905201 27513 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.003s	sys 0.006s
I20260709 12:25:50.898164 27392 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:50.975773 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:50.978921 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:50.979840 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:50.980583 26986 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:50.980921 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:50.981264 27435 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:50.982262 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:50.961180 27497 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:50.983258 27594 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:50.990988 27609 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:50.989990 27509 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:50.986172 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:50.995664 18171 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:25:50.900915 27568 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.000186 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:50.960899 27511 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:51.003625 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.056700 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:50.984871 27419 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.004338 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:50.987303 27540 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:50.916545 27473 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:51.061456 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57274.9us
I20260709 12:25:50.967156 27113 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.064594 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:50.994933 27513 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.003s	sys 0.007s
W20260709 12:25:51.045783 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 190333us
W20260709 12:25:51.067153 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 51346.8us
I20260709 12:25:51.048346 18167 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:50.973718 27348 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:50.960574 27501 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:51.071951 27513 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:51.072374 18064 leader_election.cc:400] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:25:51.059510 27595 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.960691 27529 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 12:25:51.011698 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:51.057493 18166 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:51.047994 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91482.6us
I20260709 12:25:51.051988 27611 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:51.012831 27435 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:51.118337 27435 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:51.121171 27616 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:51.068307 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82242.3us
I20260709 12:25:51.069026 18811 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:51.123203 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:51.125507 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:51.002601 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 188899us
I20260709 12:25:51.072758 18064 leader_election.cc:403] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:51.129070 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56852.1us
I20260709 12:25:51.057216 27604 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.065584 18171 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
W20260709 12:25:51.136812 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 125258us
I20260709 12:25:50.285619 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:51.140666 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:51.141726 18024 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:51.070737 27113 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.115129 27392 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.083690 26986 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:51.126955 27497 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:51.129643 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:50.933916 27587 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.130630 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.094347 27611 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.170193 27611 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:51.171687 27317 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.834s	user 0.017s	sys 0.001s
I20260709 12:25:51.158582 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:51.180617 18015 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:51.160776 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:51.146839 27392 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:51.162983 27419 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:51.165258 27564 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:51.166357 27503 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:51.188108 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:51.167356 26986 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:51.148102 27540 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:51.172034 27587 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.069979 27596 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:51.176079 18022 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:51.194728 18810 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:51.179371 18166 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:51.181453 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:51.184289 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100410us
W20260709 12:25:51.185447 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.200277 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.115193 27558 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:51.142200 27514 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:50.985478 27588 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.203037 18812 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:25:51.191862 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:51.145684 27501 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:51.213155 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.213564 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:51.214053 18806 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:25:51.207077 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77549.7us
I20260709 12:25:51.209478 18812 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:25:51.201366 18808 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:25:51.213970 18807 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:25:51.216506 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:25:51.147428 27589 consensus_queue.cc:1175] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ae9641d5c3184572a102d51a75f670ee" responder_term: 26 status { last_received { term: 26 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 26 index: 15 } }
I20260709 12:25:51.073112 27513 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.206719 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.188716 27317 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:51.190402 27609 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:51.190096 27540 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.190191 27593 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.233002 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.194388 27596 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.142462 27616 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.235356 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:51.236677 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:51.229524 27514 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:51.211265 27497 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:51.252215 18807 raft_consensus.cc:2376] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24. Re-sending same reply.
I20260709 12:25:51.147692 27604 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.228868 27419 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:51.229645 27618 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23.
I20260709 12:25:51.237572 27540 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:51.190982 27564 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:51.282409 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.280782 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 91773.5us
W20260709 12:25:51.285442 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.254029 27514 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:51.250146 27566 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:51.271726 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.276572 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.293073 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:51.288203 18021 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:51.254231 27596 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:51.268734 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:51.218174 27473 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.266744 27289 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:51.304492 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:51.312129 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43648.8us
I20260709 12:25:51.317790 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.319216 18810 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:51.320953 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:51.293601 27514 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:51.294459 27597 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:51.279261 27419 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:51.332541 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 140755us
I20260709 12:25:51.305331 27615 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.338152 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:51.338768 27620 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:25:51.339885 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.329260 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:51.349462 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:51.322634 27034 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:51.360775 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 216038us
I20260709 12:25:51.322695 27272 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:51.362819 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:51.338183 27585 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:51.399674 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63815.3us
I20260709 12:25:51.401188 17987 leader_election.cc:400] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27.
I20260709 12:25:51.401806 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:51.355883 27595 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.406065 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.365902 27500 raft_consensus.cc:2764] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election decision vote started in defunct term 29: won
I20260709 12:25:51.414217 27610 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:51.428737 27597 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.429565 27597 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.431145 27272 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.431777 27272 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.432489 27588 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:51.433108 27588 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.435175 27501 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:51.435648 27586 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:51.436220 27586 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:51.439038 27564 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:51.439563 27624 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:51.441985 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.442559 27604 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:51.450855 27623 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:25:51.451820 27475 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:51.452409 27419 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.413244 27558 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:51.454203 27558 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:51.454658 27558 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:51.414134 27587 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:25:51.415166 27595 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.464282 27595 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:51.471484 27500 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:51.472609 27500 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:51.353504 27615 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.473034 27500 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:51.475154 27615 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:51.486565 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48772.6us
W20260709 12:25:51.428270 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78915.4us
W20260709 12:25:51.498297 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135567us
I20260709 12:25:51.504818 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:51.505769 18807 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:25:51.505985 18064 leader_election.cc:400] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:51.506567 18064 leader_election.cc:403] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:51.492060 27419 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.515256 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.519862 18096 raft_consensus.cc:2410] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:25:51.503389 27420 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.528548 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:51.530648 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:51.531916 18023 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:51.536321 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:51.539914 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:51.546450 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:51.550813 18021 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:51.551910 27591 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:25:51.518169 27290 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:51.554550 27290 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:51.555356 27290 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:51.556725 27290 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.563493 27607 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:51.574296 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.568419 26986 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.547623 27595 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:51.530103 27593 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.437533 27586 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.511890 27610 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:51.591987 18171 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:51.592233 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.593125 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.627936 18096 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 19.
I20260709 12:25:51.530766 27419 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.596724 18098 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:25:51.596513 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.632076 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.599436 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.633914 18090 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:25:51.541898 27420 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.600736 27629 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:51.510227 27500 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.609495 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:51.610931 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.609305 18166 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
W20260709 12:25:51.612948 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.526563 27035 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 12:25:51.615008 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:51.615278 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.621359 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.650918 18172 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:51.623632 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.612555 27607 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:51.627808 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:51.629161 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:51.625020 26986 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:51.525697 27530 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.572s	user 0.016s	sys 0.000s
I20260709 12:25:51.543574 27034 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260709 12:25:51.543484 27615 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
I20260709 12:25:51.601745 27113 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.603384 27588 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.541705 27630 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.095s
I20260709 12:25:51.605370 27597 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:51.638914 27596 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.624598 18021 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:51.585790 27435 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.594331 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:51.654520 27562 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.002s
I20260709 12:25:51.656641 18168 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:51.645103 27616 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:51.663933 27585 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:51.665508 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.667779 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.627548 27624 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:25:51.671667 18164 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:51.672245 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.672698 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:51.624331 27595 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:51.685878 18169 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:51.687408 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:51.656648 27587 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:51.671419 27611 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.501s	user 0.010s	sys 0.005s
W20260709 12:25:51.731593 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59034.7us
I20260709 12:25:51.734231 18095 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:51.735411 18165 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 29. Current term is 30. Ops: [29.24-29.24]
W20260709 12:25:51.723379 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 183899us
I20260709 12:25:51.723538 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.724143 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.736208 18170 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33.
W20260709 12:25:51.737892 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50621.4us
W20260709 12:25:51.740810 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.733528 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:51.628224 27593 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.762290 18093 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:25:51.762835 18093 pending_rounds.cc:85] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 12
I20260709 12:25:51.763092 18093 pending_rounds.cc:107] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:25:51.763341 18093 raft_consensus.cc:2889] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:51.676463 27530 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.732817 27587 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.785904 18098 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:51.757719 27615 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:51.732136 27613 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:51.752444 27475 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:51.757629 27616 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:51.759963 27529 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.763715 27609 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:51.717569 27562 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.819640 18092 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:25:51.834401 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 285930us
I20260709 12:25:51.798926 27587 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:51.837782 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:51.751981 27611 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.806784 27615 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:51.763904 27633 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.078s
W20260709 12:25:51.857892 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 235895us
W20260709 12:25:51.857275 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.861238 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:51.864995 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 328730us
I20260709 12:25:51.866830 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:51.867784 18813 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:51.916306 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 55876.9us
I20260709 12:25:51.842594 27483 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.088s
I20260709 12:25:51.932931 27596 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:51.934005 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:51.934842 27638 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:25:51.844811 27529 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.936226 27435 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:51.936733 27634 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:25:51.940905 27420 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:51.942384 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:51.943270 27630 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:51.943760 27630 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.944139 27630 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:51.849996 27475 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:51.968887 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115485us
I20260709 12:25:51.937803 27435 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:51.764362 27610 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:51.934887 18093 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:51.946651 18023 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 34: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:51.971968 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105272us
I20260709 12:25:51.970858 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:51.976137 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 113230us
I20260709 12:25:51.955071 27035 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:51.866076 27589 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:51.866760 27635 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:25:51.998570 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58659.4us
I20260709 12:25:51.814523 27607 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:51.858537 27595 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.032758 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:52.013664 27420 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.984618 27475 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:52.028438 27514 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.682s	user 0.013s	sys 0.002s
I20260709 12:25:52.016870 27635 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:52.025782 27035 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:52.005368 27596 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:25:52.048920 27420 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.043676 27034 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:25:52.082418 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:52.083545 18165 raft_consensus.cc:606] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:25:52.084110 18165 raft_consensus.cc:614] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:52.049316 27589 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:52.032825 27645 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:52.095006 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.098026 18015 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:52.097316 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 126550us
I20260709 12:25:52.094578 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:52.102892 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 124278us
I20260709 12:25:52.105221 27113 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.116336 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:52.054719 27514 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.032613 27615 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:52.104225 27392 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.916s	user 0.019s	sys 0.000s
I20260709 12:25:52.139843 18015 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:52.061326 27585 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.056s
I20260709 12:25:52.103190 27530 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.116981 27290 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:52.097327 27633 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.172853 18090 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
W20260709 12:25:52.182518 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 149912us
W20260709 12:25:52.194095 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.115396 27645 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:52.187681 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58931.5us
I20260709 12:25:52.172801 18170 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:52.107514 27589 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:52.108469 27647 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.077s
W20260709 12:25:52.183180 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.117139 27616 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:52.195299 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:52.220037 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.223094 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.224690 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:52.224915 27615 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:52.225737 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.195470 27595 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:52.189908 27562 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:52.194212 27290 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:52.254001 18165 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:52.119370 27641 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.116s
I20260709 12:25:52.248761 18170 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:52.118534 27113 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.213331 27611 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.263327 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:52.273998 18095 raft_consensus.cc:2376] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22. Re-sending same reply.
I20260709 12:25:52.271227 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:52.278939 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 174562us
W20260709 12:25:52.279392 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84269.4us
I20260709 12:25:52.280061 18164 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:52.283318 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.220824 27633 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.258558 27605 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:52.289083 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.307365 18165 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 29 because replica has last-logged OpId of term: 27 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 21.
W20260709 12:25:52.277587 27540 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.984s	user 0.014s	sys 0.002s
I20260709 12:25:52.267714 27290 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:52.267066 27656 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:52.181558 27035 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.276412 27530 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.308776 18093 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:25:52.310873 18169 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47bac54b53844de9814420f86103a6f8"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:52.314563 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:52.323734 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:52.324977 18093 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:25:52.326637 18098 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:52.327936 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 44756.2us
I20260709 12:25:52.328362 18170 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:52.330075 18169 raft_consensus.cc:606] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Received request to transfer leadership to TS cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:52.330761 18169 raft_consensus.cc:614] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:52.329742 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:52.127313 27635 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:52.334247 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 217990us
I20260709 12:25:52.339016 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.339929 18168 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
W20260709 12:25:52.340178 27655 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:25:52.341094 27420 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:52.342806 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:52.311349 27647 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:52.309966 27475 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.346513 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:52.352149 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:52.346726 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:52.310235 27654 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.359772 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.361461 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:52.362394 18808 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:25:52.363283 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.363566 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.364239 18166 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:52.366348 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.368060 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.367177 18173 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:52.369345 18166 raft_consensus.cc:2410] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:52.370191 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.370471 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.336309 27585 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:52.371428 18166 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:52.372638 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:52.372061 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:52.370915 18173 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:52.374908 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:52.375581 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:52.375988 18094 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:52.376405 18810 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:52.319103 27540 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.344110 27605 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:52.376883 27664 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:52.381994 27652 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:25:52.365188 27654 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.383018 27661 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.369022 27562 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.015s
I20260709 12:25:52.350159 27420 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:52.385324 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:52.411346 18167 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:52.411986 27562 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:52.386305 18168 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:25:52.412983 27514 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:52.414294 27540 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.414933 18168 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [23.11-26.12]
I20260709 12:25:52.415113 27661 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.417783 27661 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.387324 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:52.330543 27530 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.438092 27639 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 12:25:52.438784 27605 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.439850 27639 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:52.440326 27639 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:52.440718 27530 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:52.442499 27649 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
I20260709 12:25:52.443310 27654 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:52.389752 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:52.372941 27585 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.319522 27589 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:52.386597 27650 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.280s	user 0.001s	sys 0.001s
I20260709 12:25:52.330224 27656 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:52.407725 27629 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.739s	user 0.012s	sys 0.002s
I20260709 12:25:52.333097 27645 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:52.409283 27663 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:52.419559 18166 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:52.412863 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:52.418236 27513 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:52.422377 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:52.445220 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73276.5us
I20260709 12:25:52.510195 27633 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.335426 27616 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:52.466122 27558 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.011s	user 0.019s	sys 0.000s
I20260709 12:25:52.513911 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.441386 27639 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:52.459429 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.418383 27619 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.127s
W20260709 12:25:52.510712 27497 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.250s	user 0.023s	sys 0.012s
I20260709 12:25:52.530596 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:52.569135 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.446247 27585 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:52.575181 18094 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:25:52.576673 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 163935us
I20260709 12:25:52.580078 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:52.581537 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.581738 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:25:52.529956 27643 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260709 12:25:52.585389 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 163084us
W20260709 12:25:52.536700 27646 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.278s	user 0.000s	sys 0.002s
I20260709 12:25:52.535112 27113 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:52.599992 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.587679 27497 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:52.587908 27665 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:52.587833 27654 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:52.579373 27657 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:52.544842 27540 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.587754 27667 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:52.576110 27558 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:52.522236 27630 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.578s	user 0.011s	sys 0.007s
I20260709 12:25:52.590236 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.584992 27530 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:52.555154 27656 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:52.580011 27650 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:25:52.597704 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.494244 27629 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:52.601555 27633 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:52.585217 27641 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:52.615341 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:52.615599 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.615697 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.616470 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.588649 27639 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:52.619993 18170 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:52.624792 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.647573 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:52.648108 18019 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27. Re-sending same reply.
I20260709 12:25:52.627766 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:52.649282 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.650118 18019 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:52.650354 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.651041 18015 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:52.651178 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.631734 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.618270 27611 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:52.655460 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.633708 18021 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:52.656141 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.637609 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.627708 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:52.655546 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:52.654709 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.604671 27475 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:52.626950 27667 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:52.614353 27665 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:52.623543 27661 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:52.623831 27420 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.658113 18166 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:25:52.671499 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.658727 18016 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27.
I20260709 12:25:52.658933 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:52.652288 18017 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:52.626717 27671 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:52.652961 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.660226 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1383) took 1131 ms. Trace:
I20260709 12:25:52.676602 18022 rpcz_store.cc:276] 0709 12:25:51.528395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.528543 (+   148us) service_pool.cc:224] Handling call
0709 12:25:51.529842 (+  1299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:51.541488 (+ 11646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:51.541528 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:51.542225 (+   697us) log.cc:844] Serialized 26 byte log entry
0709 12:25:51.654277 (+112052us) spinlock_profiling.cc:243] Waited 30.3 ms on lock 0x7b1800007de0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:25:51.848428 (+194151us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:51.848474 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.848564 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:52.657648 (+809084us) raft_consensus.cc:1685] finished
0709 12:25:52.657676 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.660142 (+  2466us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4956,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":14976,"spinlock_wait_cycles":69670656,"thread_start_us":193897,"threads_started":1,"wal-append.queue_time_us":529211}
I20260709 12:25:52.618626 27540 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.677449 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:52.673346 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.684473 18166 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:25:52.654079 27641 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:52.685704 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.687371 27641 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:52.687600 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.648007 27633 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.688935 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.690770 27633 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.690965 27558 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.690510 27667 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.675601 27671 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.693068 27671 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:52.675974 27661 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.694186 27661 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:52.665509 27668 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:52.662191 27611 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.659101 27514 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:52.698827 27514 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:52.699357 27514 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:52.700086 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:52.700467 27611 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:52.702019 27678 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:25:52.662935 27475 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.627106 27113 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.033s
I20260709 12:25:52.668071 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:52.635612 27497 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:52.649557 26986 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.974s	user 0.011s	sys 0.006s
I20260709 12:25:52.679839 27420 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:52.717026 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:52.698271 27668 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:52.704228 18020 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:25:52.626154 27630 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.722604 18815 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:52.723819 27420 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:52.724756 27630 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.724536 27290 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.725664 18095 raft_consensus.cc:2368] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 29. Current term is 30.
I20260709 12:25:52.727427 27272 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.727792 27290 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:52.712198 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.713428 18015 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:52.675038 27656 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:52.718623 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:52.720551 27113 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:52.721315 27475 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.709702 27497 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.722092 27562 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:52.730481 27668 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:52.734807 27035 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:52.736992 27665 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 27.23, Last appended by leader: 23, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.740139 27514 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.743760 18020 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:25:52.744850 27540 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:52.748028 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1787) took 1617 ms. Trace:
I20260709 12:25:52.748224 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:52.755509 27419 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.660034 18018 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:25:52.788257 27656 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:52.788669 27587 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.931s	user 0.016s	sys 0.000s
I20260709 12:25:52.683269 27645 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:52.798776 18020 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:52.801005 18018 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:52.772746 27562 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:52.771364 27113 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.772236 27497 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:52.813006 18018 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:52.820343 18018 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:52.813756 18015 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:25:52.817067 18020 raft_consensus.cc:1217] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Deduplicated request from leader. Original: 22.13->[23.14-23.14]   Dedup: 23.14->[]
I20260709 12:25:52.822376 18020 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:52.792217 18813 rpcz_store.cc:276] 0709 12:25:51.128631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.130207 (+  1576us) service_pool.cc:224] Handling call
0709 12:25:51.158707 (+ 28500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:51.868389 (+709682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.745987 (+877598us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54006f2cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:52.746196 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1630031232}
I20260709 12:25:52.826579 18020 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
W20260709 12:25:52.826503 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94996.9us
I20260709 12:25:52.828375 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:52.832067 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:52.833287 18172 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:52.834059 18172 raft_consensus.cc:606] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:25:52.835207 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:52.821836 27113 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:52.805577 27595 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:25:52.813858 27587 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.839278 27629 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:52.844799 27615 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.004s
I20260709 12:25:52.851467 18094 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260709 12:25:52.852036 18094 pending_rounds.cc:85] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:25:52.852344 18094 pending_rounds.cc:107] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260709 12:25:52.852684 18094 raft_consensus.cc:2889] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:52.839833 27630 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:52.841096 27619 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 21.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:25:52.896241 27514 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.883253 27630 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:52.897789 27630 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:52.899287 27676 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:52.900059 27676 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:52.944321 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:52.849198 27562 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.24, Last appended by leader: 24, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.853962 27656 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.937098 27627 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:52.967923 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.968775 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:52.901224 27676 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.976089 18813 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:52.970271 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1626) took 1766 ms. Trace:
I20260709 12:25:52.971297 18807 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:52.977087 18809 rpcz_store.cc:276] 0709 12:25:51.203452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.203585 (+   133us) service_pool.cc:224] Handling call
0709 12:25:51.810597 (+607012us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:51.810894 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:51.810935 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:51.811004 (+    69us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:51.811060 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.811156 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.970041 (+1158885us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54005f5850. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:52.970190 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1395102976}
I20260709 12:25:52.948830 27585 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.991271 27585 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:52.992040 27629 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:52.992535 27629 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:52.994634 27658 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:52.995185 27658 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:52.996332 27658 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.000916 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:53.002023 18809 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:52.978937 27627 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.012838 27627 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:53.012645 27671 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.013048 27598 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.014802 27654 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:53.015180 27598 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:53.031607 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3729) took 2324 ms (client timeout 2023 ms). Trace:
W20260709 12:25:53.032184 18092 rpcz_store.cc:269] 0709 12:25:50.707383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.707971 (+   588us) service_pool.cc:224] Handling call
0709 12:25:53.031311 (+2323340us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002968d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:53.031479 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2553326464}
I20260709 12:25:52.988024 27562 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:53.034034 27504 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.034682 27504 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.037101 27504 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:53.038343 18023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1384) took 1426 ms (client timeout 1572 ms). Trace:
W20260709 12:25:53.038856 18023 rpcz_store.cc:269] 0709 12:25:51.614524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.614704 (+   180us) service_pool.cc:224] Handling call
0709 12:25:53.038055 (+1423351us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:53.038240 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510313728}
I20260709 12:25:53.039527 27630 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.041146 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:53.042557 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.043098 27630 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:53.043272 18813 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:25:53.044728 27661 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:52.993512 27629 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 26.12, Last appended by leader: 12, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.047134 27629 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:53.049933 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.051405 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4610) took 1787 ms. Trace:
I20260709 12:25:53.053215 27658 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:53.051896 18099 rpcz_store.cc:276] 0709 12:25:51.263642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.263967 (+   325us) service_pool.cc:224] Handling call
0709 12:25:51.264421 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:51.396387 (+131966us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:51.466354 (+ 69967us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.726614 (+260260us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:51.726683 (+    69us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:51.726735 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.051074 (+1324339us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:53.051145 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.051319 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94124,"prepare.queue_time_us":81885,"prepare.run_cpu_time_us":4846,"prepare.run_wall_time_us":6638,"raft.queue_time_us":29134,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":24621,"spinlock_wait_cycles":302850304,"thread_start_us":235049,"threads_started":1,"wal-append.queue_time_us":377725,"wal-append.run_cpu_time_us":18841,"wal-append.run_wall_time_us":1078818}
W20260709 12:25:53.056565 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.056744 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:53.057642 18099 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:53.038424 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.058166 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.059372 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.060230 18167 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:53.060576 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80961us
I20260709 12:25:53.060698 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.033694 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.061928 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.062059 18092 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:53.062943 18169 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:53.064349 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.064808 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.065299 18172 raft_consensus.cc:2410] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:53.065582 18169 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:53.067076 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.067592 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.067996 18023 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:53.068250 18169 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:53.069768 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.071074 18015 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:53.071329 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.070262 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.074746 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.075789 18018 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:53.076064 18020 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:53.077064 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.077557 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.078635 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.078739 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.080005 18015 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:53.080083 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.083261 18168 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:25:53.085057 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.087594 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.088500 18096 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:53.081948 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.084300 27475 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.091104 27682 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:53.083551 27109 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.082129 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.099006 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.099320 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.109694 18172 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:53.111181 27611 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:53.111936 27611 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.112353 27611 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:53.099867 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.101043 18808 raft_consensus.cc:2368] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29.
W20260709 12:25:53.102691 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.104346 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.114943 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.115483 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.116595 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.117431 18092 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:53.118815 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.106912 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.107429 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.120397 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.083175 27661 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.107991 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.124534 18015 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:53.098950 27680 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.102581 27639 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:53.120198 27109 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.127261 27604 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:53.128786 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43842.7us
I20260709 12:25:53.137308 18064 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27.
I20260709 12:25:53.133649 18809 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:25:53.134275 18167 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 34.
I20260709 12:25:53.122812 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:53.122107 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.109732 27630 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:53.141987 18807 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:53.114169 27598 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:53.146335 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.144163 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.152680 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.154809 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.155606 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.149503 18020 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:53.152551 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.105403 27113 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.162729 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.171113 27497 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.152382 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:53.106123 27475 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.138126 27639 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:53.178365 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.194121 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:53.195588 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:53.203223 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 47710.5us
I20260709 12:25:53.174239 18021 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:53.204883 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.179445 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:53.190991 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.191571 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:53.208185 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.192034 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:53.150321 26986 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:53.209233 18809 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 20: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:53.207573 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.172710 27501 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:53.172385 27651 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.216019 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.178285 27630 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.222038 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.200683 27627 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:53.200384 27639 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:53.143568 27668 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.145478 27604 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.161339 27676 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:25:53.202641 27597 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:53.159406 27657 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:25:53.228277 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.218061 27272 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:53.218103 27501 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.222332 27667 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:53.225751 27619 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:53.251768 18815 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:53.251827 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.256245 18020 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:53.238849 27627 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:53.253427 18021 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:53.260913 27700 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:53.263314 27693 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:25:53.253602 27501 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:53.263996 27619 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.248986 27633 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.250722 27272 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.254817 26986 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:53.285442 18021 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:53.265657 27657 consensus_queue.cc:1243] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:25:53.288533 18021 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:53.290174 18021 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:53.267794 27700 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:53.299189 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 91778.5us
I20260709 12:25:53.276168 27667 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:53.282289 27272 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:53.282044 27671 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.295600 27615 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.299139 18020 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:25:53.318357 26986 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.12, Last appended by leader: 12, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.323472 18099 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:53.294361 27564 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.303428 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.308694 27703 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
W20260709 12:25:53.309453 27672 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.788s	user 0.000s	sys 0.005s
W20260709 12:25:53.309664 27702 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:25:53.298547 27680 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.310277 27640 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:25:53.310845 27668 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.300657 27700 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:53.286469 27676 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.022s
I20260709 12:25:53.327946 27562 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:53.281311 27630 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.023s
I20260709 12:25:53.361227 27113 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.298833 27657 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:25:53.363046 27680 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.363619 27654 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:53.363464 27667 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.364235 27633 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:53.365835 27109 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.297744 27613 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:25:53.337000 27379 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:53.393157 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:53.393738 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 186222us
I20260709 12:25:53.410172 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:53.396487 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 172948us
I20260709 12:25:53.329553 27615 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.408978 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.360410 27564 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.392376 27497 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:53.434573 27113 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:53.417969 27604 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.446286 27113 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:53.322830 27627 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.391283 27562 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.429564 27705 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:53.413673 27475 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:53.398094 27654 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.414650 27613 consensus_queue.cc:1243] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:25:53.458990 27709 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:53.493723 27709 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:53.494344 27709 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:53.496026 27587 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.459723 27113 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.450013 27705 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:53.497726 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3494) took 1142 ms. Trace:
I20260709 12:25:53.498232 18170 rpcz_store.cc:276] 0709 12:25:52.355168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.355449 (+   281us) service_pool.cc:224] Handling call
0709 12:25:52.355831 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.647611 (+291780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.497425 (+849814us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:53.497647 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606986624}
I20260709 12:25:53.498275 27705 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.466042 27706 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:53.507265 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.508153 18167 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:53.508392 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.508718 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.509404 18168 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:53.509405 18172 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:53.508316 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.510984 18166 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:53.511173 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.511999 18018 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 34: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:53.514750 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.515062 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.468292 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.468777 27529 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.521136 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.522099 18170 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 34: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:53.470779 27109 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:53.471870 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:53.524931 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53161.4us
W20260709 12:25:53.520382 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52183.8us
I20260709 12:25:53.474220 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3647) took 1111 ms. Trace:
I20260709 12:25:53.475540 27619 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.482051 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3495) took 1111 ms. Trace:
I20260709 12:25:53.483618 27593 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:53.485518 27700 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:53.490093 27688 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.594s	user 0.000s	sys 0.002s
I20260709 12:25:53.515695 18019 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
W20260709 12:25:53.543069 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.526352 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.526824 18171 rpcz_store.cc:276] 0709 12:25:52.363202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.364235 (+  1033us) service_pool.cc:224] Handling call
0709 12:25:52.379242 (+ 15007us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:52.390145 (+ 10903us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:52.418057 (+ 27912us) log.cc:844] Serialized 14 byte log entry
0709 12:25:53.045669 (+627612us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b18001d53c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:25:53.055433 (+  9764us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:53.074738 (+ 19305us) log.cc:844] Serialized 26 byte log entry
0709 12:25:53.074936 (+   198us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:53.074974 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.075061 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:53.287125 (+212064us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:53.473958 (+186833us) raft_consensus.cc:1685] finished
0709 12:25:53.473987 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.474144 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6301,"prepare.queue_time_us":28365,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":16536,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":24167,"spinlock_wait_cycles":168841728,"thread_start_us":34520,"threads_started":1,"wal-append.queue_time_us":102803}
I20260709 12:25:53.534267 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.529362 18173 rpcz_store.cc:276] 0709 12:25:52.370265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.375064 (+  4799us) service_pool.cc:224] Handling call
0709 12:25:52.729835 (+354771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.730093 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:52.730128 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:52.730188 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:52.730236 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.730317 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.481749 (+751432us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54000438d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:53.481953 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":814996352}
I20260709 12:25:53.526897 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:53.526708 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.518275 27706 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:53.595935 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.471791 27564 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:53.573460 18167 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:25:53.587265 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.588068 27605 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:53.470835 27680 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:53.501435 27671 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.544484 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.531632 27475 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:53.607789 27671 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.616354 27562 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:53.618366 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91577.2us
I20260709 12:25:53.619457 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.621055 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:53.533453 27529 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:53.542089 27705 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:53.669806 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48845.6us
I20260709 12:25:53.680845 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:53.630723 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 104417us
I20260709 12:25:53.670584 27564 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:53.670806 27648 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.703588 27648 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.706364 27317 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:53.707100 27317 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:53.709024 27501 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:53.710624 27379 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:53.711802 27615 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.713068 27633 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:53.715111 27671 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:53.677750 27529 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.670974 27587 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:53.716056 27529 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:53.716334 27587 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.716930 27587 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:53.719010 27643 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:25:53.692328 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:53.688657 18166 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:53.678102 27109 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:53.677731 27593 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:53.671088 27603 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:53.697938 27654 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:53.671949 27658 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.618s	user 0.016s	sys 0.000s
W20260709 12:25:53.603224 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59172.5us
I20260709 12:25:53.693703 27707 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:53.753551 27593 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:53.750816 27564 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:53.603477 27706 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:53.754561 27603 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:53.668359 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77890.3us
I20260709 12:25:53.750397 27109 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.789491 27109 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:53.794610 27317 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:53.771816 18098 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:53.811734 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:53.815960 18172 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:25:53.820998 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.857230 18171 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:53.857690 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:53.858669 18172 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:25:53.823114 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.824661 18020 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:53.826061 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.826157 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:53.828084 18169 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: [26.13-26.13]
W20260709 12:25:53.767621 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.843585 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3408) took 1211 ms. Trace:
I20260709 12:25:53.843838 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:53.864310 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.863531 18024 rpcz_store.cc:276] 0709 12:25:52.631714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.702008 (+ 70294us) service_pool.cc:224] Handling call
0709 12:25:52.702374 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:52.702792 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:52.702825 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:52.706290 (+  3465us) log.cc:844] Serialized 26 byte log entry
0709 12:25:52.839921 (+133631us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:52.940419 (+100498us) log.cc:844] Serialized 14 byte log entry
0709 12:25:52.940761 (+   342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.940837 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:53.489140 (+548303us) spinlock_profiling.cc:243] Waited 7.52 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:53.825344 (+336204us) raft_consensus.cc:1685] finished
0709 12:25:53.843233 (+ 17889us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.843491 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":112609,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":5930,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":29298,"spinlock_wait_cycles":17826304,"thread_start_us":186815,"threads_started":2,"wal-append.queue_time_us":183387}
I20260709 12:25:53.874393 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.874753 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.875458 18019 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:53.806824 27648 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:53.851507 27629 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.804s	user 0.014s	sys 0.003s
I20260709 12:25:53.806502 27633 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.807441 27564 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:53.881685 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:53.896881 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.014s	sys 0.025s
W20260709 12:25:53.897316 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.014s	sys 0.025s
W20260709 12:25:53.897753 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 609331us
I20260709 12:25:53.882639 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:53.825278 27501 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:53.892304 18172 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:53.900542 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:53.903465 18778 connection.cc:256] RPC call timeout handler was delayed by -2.2127208960000644s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21850000000000003s for a response
W20260709 12:25:53.908833 18778 connection.cc:256] RPC call timeout handler was delayed by -1.836953726000047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:25:53.909510 18778 connection.cc:256] RPC call timeout handler was delayed by -1.4581367739997404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:25:53.909955 18778 connection.cc:256] RPC call timeout handler was delayed by -1.1347664650002116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260709 12:25:53.911583 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.905110 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:53.893921 27676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.012s	sys 0.025s
I20260709 12:25:53.914852 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:53.887842 27668 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:53.886001 27603 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:53.920905 27611 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.808s	user 0.016s	sys 0.000s
I20260709 12:25:53.972043 27680 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:25:53.917346 27676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.012s	sys 0.026s
I20260709 12:25:53.913502 27654 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:53.914235 27629 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:53.994050 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:53.995020 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:53.884922 27504 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
W20260709 12:25:54.008921 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.914772 27671 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:53.990281 27688 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:25:53.933182 27667 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:54.000453 27717 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.246s	user 0.002s	sys 0.000s
I20260709 12:25:53.972546 27562 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:54.013624 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:54.003562 27113 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.006s	sys 0.019s
I20260709 12:25:53.904460 27633 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:54.009678 27639 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.764s	user 0.006s	sys 0.009s
I20260709 12:25:54.006937 27668 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:54.023947 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:54.021998 27272 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.698s	user 0.009s	sys 0.005s
I20260709 12:25:54.029811 18806 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
I20260709 12:25:54.007970 27665 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:54.035009 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:54.011613 27648 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.042654 18806 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:54.024281 27113 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.006s	sys 0.019s
W20260709 12:25:54.097446 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.128033 18019 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e"
dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:51724
I20260709 12:25:54.020408 27710 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:54.128778 18019 raft_consensus.cc:606] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:54.119572 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.129256 18019 raft_consensus.cc:614] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:54.013067 27680 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:54.010125 27611 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.029289 27700 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:54.176826 27709 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.682s	user 0.000s	sys 0.015s
I20260709 12:25:54.177553 27709 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.041203 27504 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:54.017796 27629 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.146531 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.184109 27504 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:53.995649 27593 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.022262 27667 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.200320 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.033589 27639 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.018585 27671 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.205087 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 304628us
I20260709 12:25:54.148908 27034 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:54.206817 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 292155us
I20260709 12:25:54.148172 27632 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.072s
I20260709 12:25:54.127043 27272 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.188395 26986 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:54.132834 27648 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.128512 27665 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.227069 18810 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:54.231721 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.028096 27564 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.214684 27586 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:54.221381 27668 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:54.260480 18016 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:54.225805 27603 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.253245 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:54.265380 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.268663 18809 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:54.270356 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.271742 18809 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:54.229337 26986 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:54.220197 27034 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.281962 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.287055 18809 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:54.277725 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:54.237491 27562 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:54.292073 27639 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.292523 27671 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.292850 27731 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.261298 27725 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:54.236697 27504 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.273561 27700 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.299314 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:54.304312 18809 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 34: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:54.284001 26986 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:54.309276 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.314193 18806 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:25:54.310303 18809 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:54.315761 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.316111 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.272553 27710 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:54.279335 27603 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.372455 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.336319 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:54.256243 27680 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.362447 27727 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:25:54.399328 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
W20260709 12:25:54.397248 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 262054us
I20260709 12:25:54.362653 27742 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:54.432448 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:54.404789 27735 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.411154 27272 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:54.498366 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3491) took 2189 ms. Trace:
I20260709 12:25:54.308504 27700 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.317024 27671 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:54.423395 27034 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:54.420760 27665 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:54.415328 27722 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
W20260709 12:25:54.470340 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.017s (SENT)
I20260709 12:25:54.518193 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:54.418311 27710 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.416601 27562 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.409649 27724 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:25:54.373015 27504 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:54.309551 27654 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:54.408725 27109 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.001s
I20260709 12:25:54.496227 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:54.399923 27633 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:54.498837 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66569.1us
I20260709 12:25:54.485615 27737 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.503924 18165 rpcz_store.cc:276] 0709 12:25:52.309200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.309926 (+   726us) service_pool.cc:224] Handling call
0709 12:25:52.310381 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.392194 (+ 81813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:52.392231 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:52.392275 (+    44us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:52.392322 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.498067 (+2105745us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:54.498126 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:54.498296 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187550592}
W20260709 12:25:54.514240 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.500408 27629 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:54.534363 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64114.5us
I20260709 12:25:54.308523 27725 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.502032 27611 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.537503 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3749) took 1505 ms. Trace:
I20260709 12:25:54.379129 27668 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.311393 27731 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.538563 27705 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.817s	user 0.014s	sys 0.002s
I20260709 12:25:54.573699 18165 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:25:54.549139 18166 raft_consensus.cc:606] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:25:54.575450 18166 raft_consensus.cc:614] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:54.552476 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:54.506325 27735 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.556164 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.536727 27113 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:54.541134 27741 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.564615 18172 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:54.591105 18172 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:25:54.592213 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.592315 18172 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.549721 27529 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.833s	user 0.013s	sys 0.004s
I20260709 12:25:54.560217 18099 rpcz_store.cc:276] 0709 12:25:53.031770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.334628 (+302858us) service_pool.cc:224] Handling call
0709 12:25:53.335068 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:53.435444 (+100376us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:53.802110 (+366666us) log.cc:844] Serialized 14 byte log entry
0709 12:25:54.230844 (+428734us) log.cc:844] Serialized 14 byte log entry
0709 12:25:54.232087 (+  1243us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:54.232817 (+   730us) log.cc:844] Serialized 26 byte log entry
0709 12:25:54.233023 (+   206us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:54.233070 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.233189 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:54.537175 (+303986us) raft_consensus.cc:1685] finished
0709 12:25:54.537199 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:54.537406 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27158,"prepare.queue_time_us":438076,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":3175,"raft.queue_time_us":163232,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":19757,"spinlock_wait_cycles":49449472,"thread_start_us":736387,"threads_started":3,"wal-append.queue_time_us":454371}
I20260709 12:25:54.569370 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.570616 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:54.570976 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.580279 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:54.593777 18172 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:54.549949 27667 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:54.554808 27109 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.583662 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.602182 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.609126 18166 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:54.611739 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.612500 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
W20260709 12:25:54.602998 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.682s (SENT)
I20260709 12:25:54.596397 18173 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:54.618183 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:54.619079 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.599300 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:54.624281 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.625237 18099 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:54.625720 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.626556 18169 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:54.573802 27648 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:54.626905 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.586243 27633 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:54.637755 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.643585 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:54.637490 18172 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:54.646983 18806 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:54.648459 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.649281 18172 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:54.611140 27611 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:54.559965 27737 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.664054 27732 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:54.653695 27611 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.627983 27734 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.629978 27113 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:54.670289 27603 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.629384 27705 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:54.640396 27654 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.649894 27633 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.671378 27597 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:54.609850 27529 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.672518 27735 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:54.673285 27587 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.956s	user 0.010s	sys 0.007s
I20260709 12:25:54.664053 27731 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:54.674949 27733 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.364s	user 0.000s	sys 0.002s
I20260709 12:25:54.675753 27725 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:54.679895 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:25:54.635247 18099 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 20: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:54.685408 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.744467 18167 raft_consensus.cc:606] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:25:54.744900 18167 raft_consensus.cc:614] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:54.745254 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:51710 (request call id 0) took 1608 ms. Trace:
I20260709 12:25:54.693976 18173 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
W20260709 12:25:54.746050 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 110815us
I20260709 12:25:54.693755 18166 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:54.704721 18806 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:54.628052 27272 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:54.706112 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138315us
W20260709 12:25:54.706388 18090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3740) took 3051 ms (client timeout 2726 ms). Trace:
I20260709 12:25:54.750898 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.650765 27648 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.652925 27629 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.695391 27732 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.712235 27611 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:54.741400 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:54.762727 18173 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:54.744750 18093 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:54.763940 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.764526 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3763) took 1026 ms. Trace:
I20260709 12:25:54.764732 18173 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:54.610445 27741 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.687099 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1456) took 2490 ms (client timeout 2017 ms). Trace:
I20260709 12:25:54.745801 18023 rpcz_store.cc:276] 0709 12:25:53.129342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.255246 (+125904us) service_pool.cc:224] Handling call
0709 12:25:54.737171 (+1481925us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b54001090d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bd735 00007f6c166b59fa 00007f6c1b67d4c9 00007f6c14bf9c54 00007f6c14bf9be2 00007f6c14bf9b11 00007f6c14bf9a8c 00007f6c14bf8d42 00007f6c149f248c 00007f6c149f18c6
0709 12:25:54.737418 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103548288}
I20260709 12:25:54.726117 27633 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:54.764974 18093 rpcz_store.cc:276] 0709 12:25:53.738162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.685001 (+946839us) service_pool.cc:224] Handling call
0709 12:25:54.764454 (+ 79453us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:54.768049 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:54.769385 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.755424 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:25:54.774749 27746 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:25:54.758574 27272 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.780133 27272 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.781332 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.782235 18093 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:54.782814 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3762) took 1046 ms. Trace:
I20260709 12:25:54.783484 27603 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:54.783870 18093 rpcz_store.cc:276] 0709 12:25:53.736943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.781004 (+1044061us) service_pool.cc:224] Handling call
0709 12:25:54.782743 (+  1739us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:54.785010 27603 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:54.787262 27529 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.766737 27629 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.787569 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.792368 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:54.793089 18093 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:54.716388 27734 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.751323 18090 rpcz_store.cc:269] 0709 12:25:51.655413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.656455 (+  1042us) service_pool.cc:224] Handling call
0709 12:25:54.706172 (+3049717us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b5400052654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:54.706306 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1042711808}
I20260709 12:25:54.757274 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:54.763641 27732 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:54.765938 18095 rpcz_store.cc:269] 0709 12:25:52.196361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.287566 (+ 91205us) service_pool.cc:224] Handling call
0709 12:25:54.686805 (+2399239us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54002977d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:54.687000 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":928351616}
I20260709 12:25:54.661621 18172 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:54.798185 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.799753 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.751729 18167 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:54.801424 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.756136 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.796586 18099 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:54.764032 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:54.804795 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.805341 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4623) took 2295 ms. Trace:
I20260709 12:25:54.805847 18094 rpcz_store.cc:276] 0709 12:25:52.510091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.578177 (+ 68086us) service_pool.cc:224] Handling call
0709 12:25:52.578684 (+   507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.853358 (+274674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.805033 (+1951675us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400214654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:54.805245 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":625853824}
I20260709 12:25:54.807874 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:54.810391 27113 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:54.818686 27741 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:54.821512 27737 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.785502 27603 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:54.761229 27648 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.828476 27648 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:54.831341 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.832594 27734 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:54.832969 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.833638 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.833752 18806 raft_consensus.cc:2376] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22. Re-sending same reply.
I20260709 12:25:54.834388 18090 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 20.
I20260709 12:25:54.835376 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:54.835872 26986 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.518s	user 0.012s	sys 0.002s
I20260709 12:25:54.836110 18806 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 21.
I20260709 12:25:54.836364 26986 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.837186 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:54.743981 27587 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.839900 27587 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:54.806213 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.841933 18167 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:54.842362 27586 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.583s	user 0.014s	sys 0.000s
I20260709 12:25:54.791385 27731 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:54.843032 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.843510 27731 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:54.843838 18167 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:54.843973 27731 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:54.849077 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:54.855688 18168 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:54.858842 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:54.859503 18814 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:54.844939 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.871443 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:54.872049 18167 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:54.884210 18173 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:25:54.887378 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 29 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.878531 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:54.896090 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.897161 18173 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:54.899256 18099 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:25:54.903914 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.904834 18095 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:25:54.905455 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3764) took 1166 ms. Trace:
I20260709 12:25:54.905807 18095 rpcz_store.cc:276] 0709 12:25:53.739214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.900704 (+1161490us) service_pool.cc:224] Handling call
0709 12:25:54.905377 (+  4673us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:54.907325 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.908562 18099 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 24 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 18.
I20260709 12:25:54.860194 27569 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.918937 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:54.900980 27751 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:25:54.890998 27586 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:54.936952 27586 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:54.937542 27586 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:54.926292 18809 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:54.936125 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.941115 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 29 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.942003 18095 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:25:54.901111 27034 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:54.942674 27034 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.943116 27034 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:54.943193 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.945508 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:54.900919 27704 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:54.946596 27704 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:54.946976 27597 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.947587 27597 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.947643 27704 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.937071 27755 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:25:54.900780 27752 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.950081 27569 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:54.950538 27752 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:54.951169 26986 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:54.951566 18091 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:25:54.951953 18091 pending_rounds.cc:85] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:25:54.952198 18091 pending_rounds.cc:107] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:25:54.952448 18091 raft_consensus.cc:2889] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:54.949086 27704 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:54.957657 26986 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.958810 26986 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:54.972203 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4626) took 2236 ms (client timeout 2147 ms). Trace:
W20260709 12:25:54.973021 18098 rpcz_store.cc:269] 0709 12:25:52.735523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.042280 (+306757us) service_pool.cc:224] Handling call
0709 12:25:54.971812 (+1929532us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b54002ca954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:54.972023 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1675315584}
I20260709 12:25:54.974540 27501 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:54.975466 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.975278 27501 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:54.981570 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.926666 18166 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:54.934724 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:54.989151 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:54.995827 18099 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29. Re-sending same reply.
I20260709 12:25:55.003631 18166 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:55.003758 18173 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:25:55.004297 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.023635 27501 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.022609 18098 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:55.026026 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3752) took 1993 ms. Trace:
I20260709 12:25:55.026680 18099 rpcz_store.cc:276] 0709 12:25:53.032755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.025037 (+1992282us) service_pool.cc:224] Handling call
0709 12:25:55.025438 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.025638 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.025671 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:55.025722 (+    51us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:55.025759 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.025843 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.025955 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.021368 18093 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:55.032891 18173 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:55.031257 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3766) took 1284 ms. Trace:
I20260709 12:25:55.033787 18093 rpcz_store.cc:276] 0709 12:25:53.746874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.792078 (+1045204us) service_pool.cc:224] Handling call
0709 12:25:55.031135 (+239057us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.025132 18166 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:25:55.038568 18806 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 24. Preceding OpId from leader: term: 30 index: 24. (term mismatch)
I20260709 12:25:55.039090 18806 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 23
I20260709 12:25:55.039350 18806 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 29.24
I20260709 12:25:55.039621 18806 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:55.034727 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:55.048199 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3765) took 1308 ms. Trace:
I20260709 12:25:55.048867 18173 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:55.049489 18093 rpcz_store.cc:276] 0709 12:25:53.740029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.047244 (+1307215us) service_pool.cc:224] Handling call
0709 12:25:55.047619 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.047844 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.047873 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:55.047922 (+    49us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:55.047956 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.048013 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.048123 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:55.053596 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78869.5us
I20260709 12:25:55.011267 27653 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:25:55.096086 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.099327 18812 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:55.060258 18098 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:25:55.056152 27618 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:25:55.060173 27435 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:25:55.070621 27719 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:25:55.072345 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3490) took 2775 ms (client timeout 1715 ms). Trace:
I20260709 12:25:55.060171 27527 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.016s
W20260709 12:25:55.077445 27712 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.482s	user 0.012s	sys 0.006s
I20260709 12:25:55.048517 27725 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.034s
I20260709 12:25:55.083591 27732 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.086699 27633 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.087849 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3733) took 3493 ms. Trace:
I20260709 12:25:55.087941 26986 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.092195 27109 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:55.046162 27589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.495s	user 0.013s	sys 0.036s
I20260709 12:25:55.059468 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1463) took 1126 ms. Trace:
I20260709 12:25:55.099061 27604 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:25:55.103941 18164 rpcz_store.cc:269] 0709 12:25:52.297143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.072066 (+2774923us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:55.072241 (+   175us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:55.104756 18098 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:55.117146 18098 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:55.094434 27569 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:55.105559 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48014.8us
I20260709 12:25:55.090579 27611 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:55.122291 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.121680 18098 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.109643 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:55.109488 18091 rpcz_store.cc:276] 0709 12:25:51.594805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.594966 (+   161us) service_pool.cc:224] Handling call
0709 12:25:51.600320 (+  5354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:54.953043 (+3352723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.087550 (+134507us) spinlock_profiling.cc:243] Waited 3.35 s on lock 0x7b5400051ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:55.087774 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7706112256}
I20260709 12:25:55.123976 18166 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:55.113431 18093 rpcz_store.cc:276] 0709 12:25:53.932956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.053715 (+1120759us) service_pool.cc:224] Handling call
0709 12:25:55.058916 (+  5201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.059120 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.059144 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:55.059191 (+    47us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:55.059230 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.059282 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.059401 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.114653 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:55.115326 27589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.562s	user 0.013s	sys 0.036s
I20260709 12:25:55.106176 27435 consensus_queue.cc:1243] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:55.106184 27710 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:55.112319 27706 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.306s	user 0.026s	sys 0.006s
I20260709 12:25:55.112509 27752 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.127682 18098 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:55.128717 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.130040 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:55.116413 27653 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:55.127180 27597 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.133414 27732 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.120771 27604 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.170928 27653 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:55.178324 27653 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.134789 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1865) took 1109 ms. Trace:
I20260709 12:25:55.180037 18814 rpcz_store.cc:276] 0709 12:25:54.025831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.219049 (+193218us) service_pool.cc:224] Handling call
0709 12:25:55.134537 (+915488us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54006153d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.134711 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1436908800}
I20260709 12:25:55.135648 27633 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.181975 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.145509 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4488) took 2034 ms. Trace:
I20260709 12:25:55.184082 18015 rpcz_store.cc:276] 0709 12:25:53.110678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.125477 (+ 14799us) service_pool.cc:224] Handling call
0709 12:25:53.125924 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:53.132675 (+  6751us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:53.132731 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:53.137412 (+  4681us) log.cc:844] Serialized 26 byte log entry
0709 12:25:53.387929 (+250517us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:53.387982 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.388076 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.076819 (+1688743us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b240011fee8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:25:55.136522 (+ 59703us) raft_consensus.cc:1685] finished
0709 12:25:55.136546 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.145418 (+  8872us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6897,"raft.queue_time_us":6290,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7110,"spinlock_wait_cycles":127792896,"thread_start_us":170758,"threads_started":1,"wal-append.queue_time_us":262859}
I20260709 12:25:55.175940 27680 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.010s
I20260709 12:25:55.147099 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1804) took 1672 ms. Trace:
I20260709 12:25:55.102777 18167 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:55.149494 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:55.186738 18812 rpcz_store.cc:276] 0709 12:25:53.474456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.474649 (+   193us) service_pool.cc:224] Handling call
0709 12:25:53.475006 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.099854 (+1624848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.146806 (+ 46952us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54006f2cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:55.147023 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3734941056}
I20260709 12:25:55.188015 27709 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.188160 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:55.195048 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.130152 27710 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.155335 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:55.151823 27611 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.176402 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:55.160195 27629 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:55.163652 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.164692 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1858) took 1247 ms. Trace:
W20260709 12:25:55.133237 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.175973 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3753) took 2143 ms. Trace:
I20260709 12:25:55.171255 27589 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.179355 27604 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.182839 18814 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:55.148769 18808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1647) took 2249 ms (client timeout 2115 ms). Trace:
I20260709 12:25:55.188812 18169 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:55.207995 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.129470 27109 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:55.208024 18808 rpcz_store.cc:269] 0709 12:25:52.899000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.915174 (+ 16174us) service_pool.cc:224] Handling call
0709 12:25:55.148447 (+2233273us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54005f1c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.148649 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358091136}
I20260709 12:25:55.206420 18099 rpcz_store.cc:276] 0709 12:25:53.033037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.028050 (+1995013us) service_pool.cc:224] Handling call
0709 12:25:55.028416 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:55.032545 (+  4129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.032574 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:55.033188 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:25:55.118180 (+ 84992us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:55.118232 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.118320 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.175403 (+ 57083us) raft_consensus.cc:1685] finished
0709 12:25:55.175669 (+   266us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.175886 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4570,"raft.queue_time_us":1260,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":4700,"thread_start_us":61946,"threads_started":1,"wal-append.queue_time_us":105232}
I20260709 12:25:55.155700 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:55.197345 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:55.158394 27742 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:55.214867 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.188262 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.202018 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.202201 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:55.203531 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.204772 18810 rpcz_store.cc:276] 0709 12:25:53.917025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.917230 (+   205us) service_pool.cc:224] Handling call
0709 12:25:55.164419 (+1247189us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b5400781754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.164574 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":497452288}
I20260709 12:25:55.208650 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.218375 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.219467 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.221375 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.180841 27653 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:55.213876 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:55.213965 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58490.6us
I20260709 12:25:55.204984 27588 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:55.222231 18093 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:55.223603 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.205413 27719 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.224372 27562 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:55.216347 27589 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.225423 27604 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:55.220258 27629 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:55.220333 27654 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:55.220487 27742 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:55.244936 27569 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:55.204432 27680 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:55.210594 27109 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.246322 27109 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:55.247720 27589 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.245036 27654 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:55.250674 27654 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:55.251138 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:55.253347 27651 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 12:25:55.266788 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.268249 18099 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 21: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:55.229666 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.200776 27710 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:55.231879 18814 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:55.206804 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:55.234282 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.234283 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.235026 18167 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:25:55.238514 18137 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:55.271942 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65256.9us
I20260709 12:25:55.272966 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:55.243155 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:55.253258 18815 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1646) took 2505 ms (client timeout 2032 ms). Trace:
I20260709 12:25:55.217980 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:55.254271 18811 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:55.257491 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.269847 18812 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:25:55.165586 27034 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.278052 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.279327 18093 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:25:55.275246 18815 rpcz_store.cc:269] 0709 12:25:52.747650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.747839 (+   189us) service_pool.cc:224] Handling call
0709 12:25:55.252948 (+2505109us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54005ad554. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.253164 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1050410240}
I20260709 12:25:55.280761 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:55.280933 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63059.5us
I20260709 12:25:55.283643 18814 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:55.286939 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:55.288708 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54523.9us
I20260709 12:25:55.294368 18098 raft_consensus.cc:2376] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28. Re-sending same reply.
I20260709 12:25:55.295730 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.289966 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:55.254200 27588 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:55.308423 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:55.308135 27588 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.296422 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:55.299912 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.166690 27706 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.264529 27604 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:55.298977 18093 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 12. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:25:55.281059 27647 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:55.283829 27709 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:55.317284 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:55.296473 27597 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:55.239332 27731 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.298441 27034 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.254153 27719 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.324223 27752 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.325348 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:55.313700 27569 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:55.330791 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.325354 27734 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:55.301108 18098 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:25:55.301564 27527 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:25:55.324662 27709 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.297583 27586 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.354657 18806 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:55.347520 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:55.326743 27611 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:55.357419 27558 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.365887 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.361645 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.368269 18094 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:55.337862 27569 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.333200 27597 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:55.376888 18099 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:55.374039 17987 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:55.327100 27633 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:25:55.384704 27633 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:55.366061 27611 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.385349 27633 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:55.388928 27700 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.367923 27322 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:55.401790 27597 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.408846 27633 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:55.409149 27762 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:25:55.385468 27611 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:55.387984 27719 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.346254 27732 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:55.346973 27752 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.380007 27569 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:55.391695 27529 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:55.390511 27731 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.363822 27709 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:55.363210 27734 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:25:55.423606 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127355us
W20260709 12:25:55.487154 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3737) took 3876 ms (client timeout 1924 ms). Trace:
I20260709 12:25:55.379731 27558 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.434635 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:55.421857 27475 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:55.423197 27322 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:25:55.440270 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:55.496563 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:55.501260 18138 connection.cc:256] RPC call timeout handler was delayed by -1.7838405039997269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20260709 12:25:55.501739 18138 connection.cc:256] RPC call timeout handler was delayed by -1.7802405039997211s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
I20260709 12:25:55.496104 27322 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:55.502601 27725 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.503358 27725 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.510614 27734 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.513970 27725 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:55.517704 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.520146 27752 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:55.524009 27586 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:55.453222 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.378080 27728 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.068s
I20260709 12:25:55.441840 27753 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:55.454778 27642 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:25:55.462085 27776 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:55.468715 18170 pending_rounds.cc:85] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 11
I20260709 12:25:55.581223 18170 pending_rounds.cc:107] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:25:55.581745 18170 raft_consensus.cc:2889] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:55.470639 27706 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.432370 27700 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:25:55.421834 27604 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.490330 18093 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:55.592458 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.593418 18093 raft_consensus.cc:2368] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29.
I20260709 12:25:55.490494 18094 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
W20260709 12:25:55.487634 18097 rpcz_store.cc:269] 0709 12:25:51.610470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.610629 (+   159us) service_pool.cc:224] Handling call
0709 12:25:55.486900 (+3876271us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54002972d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.487055 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2309922048}
I20260709 12:25:55.492609 18169 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:55.600211 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:55.526726 27713 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 2.048s	user 0.006s	sys 0.008s
W20260709 12:25:55.527782 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4624) took 2925 ms (client timeout 1645 ms). Trace:
W20260709 12:25:55.601953 18096 rpcz_store.cc:269] 0709 12:25:52.602911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.627476 (+ 24565us) service_pool.cc:224] Handling call
0709 12:25:55.527550 (+2900074us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54002986d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.527690 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":781962496}
W20260709 12:25:55.533752 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica cab4a2f5d3ca48a2a803ee8ec3906e1d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:25:55.604545 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70874.1us
I20260709 12:25:55.535763 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:55.536437 27721 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.330s	user 0.000s	sys 0.001s
I20260709 12:25:55.537340 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:55.538802 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1801) took 2570 ms. Trace:
I20260709 12:25:55.538697 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:55.540246 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:55.540771 27635 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:55.541249 18020 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:55.541157 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.618445 18099 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:55.544360 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:55.548287 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3669) took 1726 ms. Trace:
I20260709 12:25:55.546098 27752 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:55.550872 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:55.547480 18167 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:55.626511 18024 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:25:55.588037 27587 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:55.643277 27716 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:55.643851 27716 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:55.643668 27587 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.644939 27716 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.603457 27562 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:55.647576 27109 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.647826 27562 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.650733 27654 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.595973 27639 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.652073 27639 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:55.438239 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73194.1us
I20260709 12:25:55.655238 27639 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:55.553036 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1392) took 2050 ms (client timeout 1969 ms). Trace:
I20260709 12:25:55.529695 27710 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:55.556480 27704 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.603s	user 0.010s	sys 0.009s
W20260709 12:25:55.556589 18136 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.062s (SENT) [suppressed 26 similar messages]
I20260709 12:25:55.567607 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:55.848112 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 280619us
W20260709 12:25:55.502257 18138 connection.cc:256] RPC call timeout handler was delayed by -1.7532405039996775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260709 12:25:55.605718 18097 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 21. Preceding OpId from leader: term: 29 index: 23. (index mismatch)
W20260709 12:25:55.852314 18138 connection.cc:256] RPC call timeout handler was delayed by -1.6722405039995465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260709 12:25:55.852788 18138 connection.cc:256] RPC call timeout handler was delayed by -1.6353405039994868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20260709 12:25:55.853315 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.608592 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.855252 18093 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:25:55.614230 18169 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:55.856205 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.610862 18811 rpcz_store.cc:276] 0709 12:25:52.968006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.969350 (+  1344us) service_pool.cc:224] Handling call
0709 12:25:55.254041 (+2284691us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.276909 (+ 22868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.538517 (+261608us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54005ad550. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.538692 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5253399936}
I20260709 12:25:55.620982 18171 rpcz_store.cc:276] 0709 12:25:53.822277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.822448 (+   171us) service_pool.cc:224] Handling call
0709 12:25:53.857000 (+ 34552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.869264 (+ 12264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.548079 (+1678815us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5400042ed0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.548226 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16414592}
I20260709 12:25:55.857055 18810 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
W20260709 12:25:55.853400 18138 connection.cc:256] RPC call timeout handler was delayed by -1.5894405039998674s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20260709 12:25:55.857996 18138 connection.cc:256] RPC call timeout handler was delayed by -1.4877405039997029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
W20260709 12:25:55.858258 18138 connection.cc:256] RPC call timeout handler was delayed by -1.3860405039995385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
W20260709 12:25:55.858796 18138 connection.cc:256] RPC call timeout handler was delayed by -1.3509405039994817s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20120000000000002s for a response
W20260709 12:25:55.859339 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2528405039997779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
I20260709 12:25:55.488773 27529 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.591140 27700 consensus_queue.cc:1243] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 26.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:55.635838 18167 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 29 index: 23. (index mismatch)
I20260709 12:25:55.547720 27603 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.626235 27752 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.641960 27558 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.656893 18015 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:55.656561 27322 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.486368 27732 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:55.876088 27732 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.663966 27660 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.664381 27611 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.882359 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:55.882500 27660 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.838088 27767 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.665616 18099 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:55.883692 27767 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.670676 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.867260 27348 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:55.884555 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.885041 27348 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:55.672482 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1440) took 1108 ms. Trace:
I20260709 12:25:55.885869 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.886669 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:25:55.886143 18168 rpcz_store.cc:276] 0709 12:25:54.564316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.568908 (+  4592us) service_pool.cc:224] Handling call
0709 12:25:55.672243 (+1103335us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54000438d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.672375 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":597183744}
I20260709 12:25:55.887846 27348 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:55.888900 27603 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:55.674988 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.681438 18166 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:25:55.890580 27742 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.005s
I20260709 12:25:55.890780 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.891109 27742 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.892074 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.681027 27587 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:55.681770 27500 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:55.683282 18813 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:55.695415 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.898681 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1650) took 2779 ms. Trace:
I20260709 12:25:55.899242 18813 rpcz_store.cc:276] 0709 12:25:53.119074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.682648 (+2563574us) service_pool.cc:224] Handling call
0709 12:25:55.683044 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.898278 (+215234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.898556 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.686436 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1658) took 1034 ms. Trace:
I20260709 12:25:55.688861 27109 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:55.701504 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.702004 27716 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:55.705964 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1654) took 1458 ms. Trace:
I20260709 12:25:55.709273 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.695416 27562 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:55.918219 18815 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:55.718571 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:55.764904 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 190795us
W20260709 12:25:55.809055 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:55.733511 18016 rpcz_store.cc:269] 0709 12:25:53.503147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.516335 (+ 13188us) service_pool.cc:224] Handling call
0709 12:25:55.552765 (+2036430us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b54001d0454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:55.552945 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1529909504}
I20260709 12:25:55.620379 18018 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:25:55.859751 18138 connection.cc:256] RPC call timeout handler was delayed by -1.2231405039997298s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2154s for a response
W20260709 12:25:55.923147 18138 connection.cc:256] RPC call timeout handler was delayed by -1.004825563999475s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
I20260709 12:25:55.878211 27706 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.046s
I20260709 12:25:55.900238 27558 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:55.926477 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:55.837328 27772 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.243s	user 0.001s	sys 0.001s
I20260709 12:25:55.853629 27665 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:55.874037 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.876875 27322 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.662400 27773 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:55.878089 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.877938 27641 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:55.884235 27611 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:55.887804 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.892191 18166 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:55.869920 27656 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.074s
I20260709 12:25:55.699317 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.700261 27630 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:55.846933 27708 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.885905 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.900597 18806 rpcz_store.cc:276] 0709 12:25:54.652372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.040855 (+388483us) service_pool.cc:224] Handling call
0709 12:25:55.041212 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.363634 (+322422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.686099 (+322465us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54004a60d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:55.686347 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":720010112}
I20260709 12:25:55.989421 18806 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:25:55.989969 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 405042us
I20260709 12:25:55.916733 18812 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:55.897130 27709 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.924767 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:55.927116 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.862061 27776 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.009796 27776 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.864646 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:56.011667 18169 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:56.013265 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:55.907897 27768 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.570s	user 0.000s	sys 0.008s
I20260709 12:25:55.944357 18093 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24. Re-sending same reply.
I20260709 12:25:56.022192 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.946678 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:55.971495 18097 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:25:56.022992 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 466393us
I20260709 12:25:55.979296 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:56.024145 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:55.893648 18165 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:55.933511 18807 rpcz_store.cc:276] 0709 12:25:54.247401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.852041 (+604640us) service_pool.cc:224] Handling call
0709 12:25:54.852592 (+   551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.853171 (+   579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:54.853217 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:54.870394 (+ 17177us) log.cc:844] Serialized 26 byte log entry
0709 12:25:55.442949 (+572555us) spinlock_profiling.cc:243] Waited 9.04 ms on lock 0x7b1800198d00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:25:55.486480 (+ 43531us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:55.491178 (+  4698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.491324 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.705487 (+214163us) raft_consensus.cc:1685] finished
0709 12:25:55.705531 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.705883 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15384,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":14712,"spinlock_wait_cycles":20781312,"thread_start_us":43145,"threads_started":1,"wal-append.queue_time_us":832810}
I20260709 12:25:55.942440 27564 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.946080 27501 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:55.956707 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:55.884136 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:55.945909 27639 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.038081 18808 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:56.038219 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:55.992530 18806 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:55.996145 18811 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:56.002692 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:56.040091 18099 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:25:56.006078 27707 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
W20260709 12:25:56.006716 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82021.8us
I20260709 12:25:56.007256 27647 raft_consensus.cc:2764] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:25:55.981668 27641 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.013883 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:25:56.072916 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59176.4us
I20260709 12:25:55.975103 27728 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.925500 27558 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:55.946810 27665 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:55.905602 27654 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.023712 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:55.986235 27630 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:56.078032 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:56.083086 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:56.092818 18806 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:25:56.076164 27633 consensus_meta.cc:287] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.624s	user 0.009s	sys 0.005s
I20260709 12:25:55.995383 27708 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.074324 27742 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:56.041714 27671 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.056s
I20260709 12:25:56.107949 27633 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.070024 27639 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.084380 27665 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:56.085244 27680 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:25:56.115315 27630 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.025568 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:55.893637 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.299341 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.044221 27501 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.035725 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.307061 18093 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:25:56.307891 27630 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:56.309114 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:56.030318 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.247637 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.314347 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3525) took 1048 ms. Trace:
W20260709 12:25:56.315572 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.017984 18095 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 19.
I20260709 12:25:56.249387 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3691) took 1417 ms. Trace:
I20260709 12:25:56.319312 18164 rpcz_store.cc:276] 0709 12:25:54.831238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.135413 (+304175us) service_pool.cc:224] Handling call
0709 12:25:55.135890 (+   477us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:55.136461 (+   571us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.136498 (+    37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:55.144010 (+  7512us) log.cc:844] Serialized 78 byte log entry
0709 12:25:55.200822 (+ 56812us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:55.216975 (+ 16153us) log.cc:844] Serialized 14 byte log entry
0709 12:25:55.232672 (+ 15697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.232773 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.706333 (+473560us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:56.248124 (+541791us) raft_consensus.cc:1685] finished
0709 12:25:56.248379 (+   255us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:56.248983 (+   604us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15921,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":54,"raft.queue_time_us":15238,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":10402,"spinlock_wait_cycles":58250112,"thread_start_us":56334,"threads_started":1,"wal-append.queue_time_us":153106}
I20260709 12:25:56.045090 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.353754 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:56.354529 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 309428us
I20260709 12:25:56.356151 27654 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:56.356760 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.030879 18169 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:56.357200 27654 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.359858 27647 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:56.259212 27732 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.085883 27562 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:56.367358 27562 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:56.367936 27562 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:56.143103 27710 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.183290 27709 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.086692 27728 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.135037 27752 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.172792 27767 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.171768 27776 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.271759 27607 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:56.273002 27569 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.784s	user 0.014s	sys 0.001s
I20260709 12:25:56.274772 27529 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.275146 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:56.040498 27564 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:56.383741 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 109054us
I20260709 12:25:56.255403 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:56.325408 27633 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.379236 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:56.378926 27569 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.383885 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3683) took 1766 ms. Trace:
W20260709 12:25:56.326839 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 301260us
I20260709 12:25:56.387799 27776 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:56.393618 18096 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.398138 18098 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:56.398650 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.399628 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.390039 18165 rpcz_store.cc:276] 0709 12:25:54.617752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.618766 (+  1014us) service_pool.cc:224] Handling call
0709 12:25:56.383252 (+1764486us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003b8154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:56.383628 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2928737152}
I20260709 12:25:56.404649 27753 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:56.406242 27639 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.407372 27641 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:56.408452 27558 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.408727 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:56.409996 27564 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.377168 27728 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.314934 18170 rpcz_store.cc:276] 0709 12:25:55.265808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.467856 (+202048us) service_pool.cc:224] Handling call
0709 12:25:55.468361 (+   505us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:55.592019 (+123658us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.592056 (+    37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:55.592836 (+   780us) log.cc:844] Serialized 78 byte log entry
0709 12:25:55.945148 (+352312us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:56.253149 (+308001us) log.cc:844] Serialized 14 byte log entry
0709 12:25:56.312994 (+ 59845us) log.cc:844] Serialized 14 byte log entry
0709 12:25:56.313443 (+   449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.313618 (+   175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:56.313734 (+   116us) raft_consensus.cc:1685] finished
0709 12:25:56.313747 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:56.314100 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":240079,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":162,"raft.queue_time_us":26458,"raft.run_cpu_time_us":868,"raft.run_wall_time_us":9788,"thread_start_us":583638,"threads_started":2,"wal-append.queue_time_us":483578}
I20260709 12:25:56.419253 27641 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:56.405575 27753 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:56.423363 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.424728 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1867) took 2236 ms. Trace:
I20260709 12:25:56.424899 18093 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:56.432617 27740 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.433794 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:56.435477 27624 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:56.434069 27589 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:25:56.437662 27589 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:56.436163 27624 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.425298 18809 rpcz_store.cc:276] 0709 12:25:54.187761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.413959 (+226198us) service_pool.cc:224] Handling call
0709 12:25:54.414494 (+   535us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:54.939403 (+524909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.423995 (+1484592us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b54004b41d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:56.424428 (+   433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1176316800}
I20260709 12:25:56.438512 27624 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:56.453071 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.452376 27647 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:56.456216 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.462348 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:56.463956 18093 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:56.425459 27732 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.470417 27501 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.471099 27633 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:56.473769 27569 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.443022 27589 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:56.457765 27664 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.485935 27664 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.488075 27708 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.491189 18814 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:56.455636 27647 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:56.493218 27647 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:56.494257 27800 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:25:56.477986 18090 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:56.482937 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 76977.9us
I20260709 12:25:56.505100 27732 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.507336 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3786) took 1160 ms. Trace:
I20260709 12:25:56.508023 18098 rpcz_store.cc:276] 0709 12:25:55.349070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.355798 (+  6728us) service_pool.cc:224] Handling call
0709 12:25:55.356315 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.398970 (+1042655us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.506901 (+107931us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002968d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:56.507234 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2378444032}
I20260709 12:25:56.511267 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.512550 18098 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
W20260709 12:25:56.521468 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87938.7us
I20260709 12:25:56.522951 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:56.524552 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.434687 27740 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.525182 27732 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:56.533121 27435 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 12:25:56.537606 27435 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:56.538501 27435 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:56.578622 27737 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:56.547982 27654 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.535261 18096 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:56.540498 27435 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.583312 18097 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:25:56.584081 27529 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.585050 27529 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:56.578140 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55457.6us
I20260709 12:25:56.558679 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:56.559827 27739 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.593183 18808 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:25:56.569765 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:56.560665 27588 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:56.596805 27588 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:56.599610 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.602948 18091 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:25:56.603578 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:56.601290 18095 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1478) took 1788 ms (client timeout 2225 ms). Trace:
W20260709 12:25:56.605026 18095 rpcz_store.cc:269] 0709 12:25:54.812994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.945156 (+132162us) service_pool.cc:224] Handling call
0709 12:25:56.600734 (+1655578us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b5400259754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:56.601010 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2004259712}
I20260709 12:25:56.608150 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4631) took 1016 ms. Trace:
I20260709 12:25:56.608639 18095 rpcz_store.cc:276] 0709 12:25:55.592144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.606603 (+1014459us) service_pool.cc:224] Handling call
0709 12:25:56.607244 (+   641us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.607642 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.607679 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:56.607757 (+    78us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:56.607832 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.607892 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:56.608056 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:56.593802 27739 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.613767 18097 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 26 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 10.
I20260709 12:25:56.615069 27776 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:56.615201 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 258676us
I20260709 12:25:56.616039 27776 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.619405 18095 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:56.620481 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1475) took 1881 ms. Trace:
I20260709 12:25:56.570688 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.621222 18091 rpcz_store.cc:276] 0709 12:25:54.739213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.126856 (+387643us) service_pool.cc:224] Handling call
0709 12:25:56.602038 (+1475182us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.603596 (+  1558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.620098 (+ 16502us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400259750. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:56.620390 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3376350080}
I20260709 12:25:56.621958 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.623298 18091 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:56.624084 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4634) took 1013 ms. Trace:
I20260709 12:25:56.624512 18091 rpcz_store.cc:276] 0709 12:25:55.610399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.622515 (+1012116us) service_pool.cc:224] Handling call
0709 12:25:56.622970 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.623765 (+   795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.624022 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:56.626310 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:56.627694 27739 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.598333 27588 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.575439 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:56.553308 27660 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:56.626214 27529 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.571067 27708 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:25:56.629593 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 146244us
I20260709 12:25:56.630915 27776 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.624286 18094 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 27 index: 23. (index mismatch)
I20260709 12:25:56.620584 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4633) took 1017 ms. Trace:
W20260709 12:25:56.626127 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 116352us
W20260709 12:25:56.634488 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105003us
I20260709 12:25:56.566196 27740 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.657430 18095 rpcz_store.cc:276] 0709 12:25:55.603288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.618312 (+1015024us) service_pool.cc:224] Handling call
0709 12:25:56.619007 (+   695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.620234 (+  1227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.620489 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:56.659240 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:56.645457 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.661931 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:56.587947 27664 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.665779 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4638) took 1005 ms. Trace:
I20260709 12:25:56.668615 18094 rpcz_store.cc:276] 0709 12:25:55.655235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.623396 (+968161us) service_pool.cc:224] Handling call
0709 12:25:56.623842 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.656862 (+ 33020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.660517 (+  3655us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:56.671339 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45185.7us
I20260709 12:25:56.678531 27588 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:56.687546 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:56.674088 27569 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:56.696727 27569 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.697304 27569 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:56.684978 17916 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 14 to 15, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 15 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:56.695863 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.703732 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.704901 18098 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.708747 18096 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:56.711833 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1486) took 1015 ms. Trace:
I20260709 12:25:56.712270 18096 rpcz_store.cc:276] 0709 12:25:55.696563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.672124 (+975561us) service_pool.cc:224] Handling call
0709 12:25:56.711750 (+ 39626us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:56.708712 27740 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:56.726791 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.713444 27435 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:56.733893 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:56.741240 27776 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:56.742300 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75063.9us
I20260709 12:25:56.722934 27664 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:56.741443 27651 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:56.733085 27660 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:56.744616 27776 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.746579 27660 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:56.746857 27776 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:56.747418 27651 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.747952 27660 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:56.750303 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 175210us
W20260709 12:25:56.744470 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 128025us
W20260709 12:25:56.755376 18815 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.808s	user 0.013s	sys 0.009s
I20260709 12:25:56.736370 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:56.724025 27708 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.760609 27651 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.761920 27530 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:56.762418 18099 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.682s	user 0.021s	sys 0.002s
I20260709 12:25:56.763870 27753 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.769853 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.786617 27530 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.789234 18099 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:56.792698 18092 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260709 12:25:56.793182 18092 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 16
I20260709 12:25:56.793448 18092 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 12:25:56.794171 18091 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:56.791730 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.793751 18092 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:56.786962 18815 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:56.795071 27739 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:56.800366 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104608us
I20260709 12:25:56.813282 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4650) took 1123 ms. Trace:
I20260709 12:25:56.813863 18091 rpcz_store.cc:276] 0709 12:25:55.687239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.644838 (+957599us) service_pool.cc:224] Handling call
0709 12:25:56.793946 (+149108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.794640 (+   694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.810044 (+ 15404us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400235d50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:56.810255 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":310359424}
I20260709 12:25:56.805061 17916 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 16 to 18, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 18 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:25:56.736630 27589 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:56.817852 27603 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.914s	user 0.009s	sys 0.015s
I20260709 12:25:56.820072 18099 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:56.821451 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:56.822319 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52637.4us
I20260709 12:25:56.822916 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.823606 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.823757 18099 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:56.825600 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:56.827275 27289 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:56.828250 27289 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:56.829044 27289 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.832592 27289 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:56.834007 18810 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: [22.16-24.17]
I20260709 12:25:56.816131 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1886) took 1187 ms. Trace:
I20260709 12:25:56.837004 18815 rpcz_store.cc:276] 0709 12:25:55.620301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.708836 (+ 88535us) service_pool.cc:224] Handling call
0709 12:25:56.803172 (+1094336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:56.805609 27664 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:56.814024 27529 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:56.799857 27740 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
I20260709 12:25:56.848804 27747 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:56.854758 27589 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:56.846174 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:56.864457 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 160622us
I20260709 12:25:56.853615 18778 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:56.849532 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:56.807447 27739 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.875026 27753 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:56.878896 27708 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:56.882341 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 56853us
I20260709 12:25:56.883425 18090 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:56.883664 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:56.886929 18097 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:56.889153 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74632.6us
W20260709 12:25:56.893934 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44541us
I20260709 12:25:56.897909 18091 pending_rounds.cc:85] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 11
I20260709 12:25:56.898398 18091 pending_rounds.cc:107] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 26.12
I20260709 12:25:56.898706 18091 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:56.900043 27109 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:56.901053 27109 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:56.901450 27660 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:56.902098 27660 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:56.902612 27660 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:56.903443 27753 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:56.904107 27753 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:56.903940 27109 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.905907 27664 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.882346 27739 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:25:56.908339 27624 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:56.906843 27664 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.909855 18093 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:56.910128 27289 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:56.910627 27289 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:56.911556 27664 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:56.911725 27289 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.913417 27289 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:56.913489 27624 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:56.914157 27716 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.987s	user 0.011s	sys 0.019s
I20260709 12:25:56.916294 27664 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:56.917383 18138 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:56.917771 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:56.918481 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65152.7us
I20260709 12:25:56.905335 27753 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:56.922504 27732 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:56.927337 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:56.929982 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4628) took 4081 ms. Trace:
I20260709 12:25:56.934859 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4632) took 1331 ms. Trace:
W20260709 12:25:56.942013 27665 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.008s
W20260709 12:25:56.943567 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.976883 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.978052 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.978190 18098 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:56.945425 18096 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:56.950749 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:56.953293 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:25:56.941453 27500 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.047s	user 0.011s	sys 0.015s
I20260709 12:25:56.973991 18092 rpcz_store.cc:276] 0709 12:25:52.848885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.196543 (+347658us) service_pool.cc:224] Handling call
0709 12:25:53.196920 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.841416 (+3644496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.929599 (+ 88183us) spinlock_profiling.cc:243] Waited 3.59 s on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:56.929892 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8267813248}
W20260709 12:25:56.983247 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.978572 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.974488 18097 rpcz_store.cc:276] 0709 12:25:55.592445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.615423 (+1022978us) service_pool.cc:224] Handling call
0709 12:25:56.886554 (+271131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.887509 (+   955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.923675 (+ 36166us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b540040c4d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:56.923860 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622339328}
I20260709 12:25:56.953516 18093 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:56.976188 27732 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:57.000876 27648 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:56.990402 27665 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.005236 27529 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:57.005448 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54892us
I20260709 12:25:57.006580 27501 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:57.006726 27610 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.007118 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:57.007568 27501 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:57.007642 27610 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.008656 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:57.006021 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88379.1us
I20260709 12:25:57.010051 27708 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.010655 27708 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.011354 27419 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:57.011875 27419 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.011991 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:57.013268 27716 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.013563 27564 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:57.014125 27564 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.014613 27564 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:57.006737 17916 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 8 to 13, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 13 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:57.011361 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84193.2us
I20260709 12:25:57.018098 27624 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:57.018967 27624 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.019381 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:57.019881 27569 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.020498 27648 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:57.021422 18094 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:57.021801 27648 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:57.012379 27419 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:57.009002 27501 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.026090 27716 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.027495 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:57.028276 18815 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:57.001384 27562 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.633s	user 0.018s	sys 0.001s
I20260709 12:25:56.928766 27647 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.028899 27562 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.005448 27665 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:57.031280 27807 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:25:57.031519 27665 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:57.032048 27665 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:57.033499 27665 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.037366 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:57.039327 27529 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:25:57.039913 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 286) took 30860 ms (client timeout 15000 ms). Trace:
I20260709 12:25:57.041801 27569 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:57.057305 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52731.8us
W20260709 12:25:57.058545 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.001155 27664 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:57.070187 27624 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.071880 27737 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:57.071254 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50651.1us
I20260709 12:25:57.074783 18096 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:25:57.072755 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:57.076164 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 158962us
W20260709 12:25:57.078929 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.079098 27562 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.065979 27716 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:57.096110 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3788) took 1175 ms. Trace:
I20260709 12:25:57.068460 27741 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:57.090350 27737 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:57.072467 27708 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.091135 18811 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:57.090451 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:57.096766 18094 rpcz_store.cc:276] 0709 12:25:55.920753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.675519 (+754766us) service_pool.cc:224] Handling call
0709 12:25:56.740474 (+ 64955us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.021923 (+281449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.095738 (+ 73815us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b540017ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:57.096022 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645521792}
I20260709 12:25:57.084126 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:25:57.070446 27739 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:25:57.090586 27530 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.080880 27664 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:57.117426 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:57.111778 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:57.123484 18098 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:57.126188 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:57.104060 27647 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:57.099889 27741 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.093703 27610 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:57.123152 27664 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.135603 27741 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:57.138124 27529 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:57.123844 27708 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:25:57.181325 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91043.8us
I20260709 12:25:57.148572 27665 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:57.160847 27610 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:57.146513 27776 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.175289 27562 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:57.119354 27732 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.201905 18172 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:57.200152 18815 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:25:57.188511 27529 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:57.201238 27588 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:57.245353 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 205285us
I20260709 12:25:57.246601 27588 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:57.206141 27562 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.203411 27708 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.207453 27660 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.193264 27737 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:57.270320 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 254030us
I20260709 12:25:57.267086 27588 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:57.284694 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 155097us
W20260709 12:25:57.066032 17920 rpcz_store.cc:269] 0709 12:25:26.179135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.080336 (+6901201us) service_pool.cc:224] Handling call
0709 12:25:57.039457 (+23959121us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:45.449541 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:45.451215 (+  1674us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:45.451272 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:25:45.451297 (+    25us) tablet.cc:662] Decoding operations
0709 12:25:45.452990 (+  1693us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:45.453113 (+   123us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:45.453134 (+    21us) tablet.cc:685] Acquiring locks for 28 operations
0709 12:25:45.480242 (+ 27108us) tablet.cc:701] Row locks acquired
0709 12:25:45.480328 (+    86us) write_op.cc:260] PREPARE: finished
0709 12:25:45.508760 (+ 28432us) write_op.cc:270] Start()
0709 12:25:45.523420 (+ 14660us) write_op.cc:276] Timestamp: P: 1783599945505500 usec, L: 0
0709 12:25:45.525517 (+  2097us) op_driver.cc:348] REPLICATION: starting
0709 12:25:45.543215 (+ 17698us) log.cc:844] Serialized 10998 byte log entry
0709 12:25:46.450188 (+906973us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:47.649714 (+1199526us) op_driver.cc:464] REPLICATION: finished
0709 12:25:47.863759 (+214045us) write_op.cc:301] APPLY: starting
0709 12:25:47.863897 (+   138us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:48.303253 (+439356us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:48.303301 (+    48us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:48.768279 (+464978us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:48.768873 (+   594us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=56,key_file_lookups=56,delta_file_lookups=0,mrs_lookups=0
0709 12:25:48.768948 (+    75us) write_op.cc:312] APPLY: finished
0709 12:25:48.769693 (+   745us) log.cc:844] Serialized 244 byte log entry
0709 12:25:49.604349 (+834656us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b
0709 12:25:49.903310 (+298961us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:49.903802 (+   492us) write_op.cc:454] Released schema lock
0709 12:25:49.903916 (+   114us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:49.904542 (+   626us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:49.904580 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4799428,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":213528,"apply.run_cpu_time_us":91888,"apply.run_wall_time_us":2040794,"cfile_cache_hit":114,"cfile_cache_hit_bytes":121335,"num_ops":28,"prepare.queue_time_us":97978,"prepare.run_cpu_time_us":33967,"prepare.run_wall_time_us":215240,"raft.queue_time_us":788047,"raft.run_cpu_time_us":33212,"raft.run_wall_time_us":241118,"replication_time_us":2123889,"spinlock_wait_cycles":57753344,"thread_start_us":2378403,"threads_started":5,"wal-append.queue_time_us":1606514,"wal-append.run_cpu_time_us":36764,"wal-append.run_wall_time_us":3745229}]]}
W20260709 12:25:57.289769 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 279262us
I20260709 12:25:57.299878 27776 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.276221 27530 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.286952 27732 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:57.321745 18064 leader_election.cc:336] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.715s (SENT)
I20260709 12:25:57.326011 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1820) took 1029 ms. Trace:
I20260709 12:25:57.326722 18811 rpcz_store.cc:276] 0709 12:25:56.350156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.350365 (+   209us) service_pool.cc:224] Handling call
0709 12:25:56.377767 (+ 27402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.114172 (+736405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.312925 (+198753us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b5400781754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:57.325885 (+ 12960us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589517056}
I20260709 12:25:57.276280 27784 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.128s
I20260709 12:25:57.306413 27530 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.340541 27530 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:57.292954 27501 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:25:57.295403 27610 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
W20260709 12:25:57.360630 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.360373 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:57.362119 18098 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:57.295250 27529 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:57.380524 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 271239us
I20260709 12:25:57.387130 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:57.382907 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.391137 27822 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:57.364362 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:57.389396 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:57.395646 27822 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:57.401257 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:57.406142 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:57.414386 27808 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:25:57.423130 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3787) took 1503 ms. Trace:
I20260709 12:25:57.423712 18095 rpcz_store.cc:276] 0709 12:25:55.920328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.692768 (+772440us) service_pool.cc:224] Handling call
0709 12:25:56.753227 (+ 60459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.841225 (+ 87998us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:56.847183 (+  5958us) log.cc:844] Serialized 14 byte log entry
0709 12:25:56.963298 (+116115us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.964175 (+   877us) log.cc:844] Serialized 26 byte log entry
0709 12:25:56.970350 (+  6175us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:56.970910 (+   560us) log.cc:844] Serialized 14 byte log entry
0709 12:25:56.971349 (+   439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.971473 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.150215 (+178742us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b18002dad80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca522 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:25:57.422719 (+272504us) raft_consensus.cc:1685] finished
0709 12:25:57.422762 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.423023 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11297,"prepare.queue_time_us":5086,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":103,"raft.queue_time_us":31063,"raft.run_cpu_time_us":511,"raft.run_wall_time_us":25491,"spinlock_wait_cycles":17086592,"thread_start_us":114229,"threads_started":1,"wal-append.queue_time_us":316650}
W20260709 12:25:57.428728 27810 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260709 12:25:57.434191 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3518) took 2657 ms (client timeout 1715 ms). Trace:
I20260709 12:25:57.435940 18173 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:57.436136 27708 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:57.436604 27739 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:57.437160 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.441845 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 131547us
I20260709 12:25:57.460614 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4651) took 1143 ms. Trace:
I20260709 12:25:57.309363 27109 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:25:57.440215 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:57.458086 18172 rpcz_store.cc:269] 0709 12:25:54.776747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.830974 (+ 54227us) service_pool.cc:224] Handling call
0709 12:25:57.433772 (+2602798us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b5400030a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:57.434023 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5006449920}
I20260709 12:25:57.341387 27776 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
W20260709 12:25:57.454280 27589 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.567s	user 0.014s	sys 0.001s
I20260709 12:25:57.478587 18098 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:57.467612 18090 rpcz_store.cc:276] 0709 12:25:56.317595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.885441 (+567846us) service_pool.cc:224] Handling call
0709 12:25:56.885977 (+   536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.887099 (+  1122us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.887134 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.887787 (+   653us) log.cc:844] Serialized 26 byte log entry
0709 12:25:57.274139 (+386352us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:57.274240 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.274380 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.460180 (+185800us) raft_consensus.cc:1685] finished
0709 12:25:57.460226 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.460504 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":152,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":148,"thread_start_us":267243,"threads_started":1,"wal-append.queue_time_us":429569}
I20260709 12:25:57.487414 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:57.474393 27822 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:57.486306 27530 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:57.486889 27562 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.520958 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:25:57.523154 27813 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:25:57.527235 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 126629us
W20260709 12:25:57.536046 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.536357 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3796) took 1604 ms. Trace:
I20260709 12:25:57.474377 27739 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:57.537333 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.550132 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.515431 27109 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:57.523207 27530 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:25:57.562731 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 283) took 15950 ms (client timeout 15000 ms). Trace:
W20260709 12:25:57.563264 17920 rpcz_store.cc:269] 0709 12:25:41.606125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.535985 (+15929860us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:57.556305 (+ 20320us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:57.518000 27589 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:25:57.536862 18091 rpcz_store.cc:276] 0709 12:25:55.931638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.841073 (+909435us) service_pool.cc:224] Handling call
0709 12:25:56.875909 (+ 34836us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:56.954722 (+ 78813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.954779 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:56.955764 (+   985us) log.cc:844] Serialized 52 byte log entry
0709 12:25:57.055916 (+100152us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b48001e0258. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1b5e0aa1 00007f6c1b5d7b37 00007f6c1b5d79ca 00007f6c1b6cf26f 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508 00007f6c1b6e3488 00007f6c1b6e3443 00007f6c1b6e26ec 00007f6c1a60ca96
0709 12:25:57.251705 (+195789us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:57.251760 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.251850 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.524755 (+272905us) raft_consensus.cc:1685] finished
0709 12:25:57.524782 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.536272 (+ 11490us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8462,"raft.run_cpu_time_us":2793,"raft.run_wall_time_us":90995,"spinlock_wait_cycles":26771712,"thread_start_us":295403,"threads_started":1,"wal-append.queue_time_us":355875}
I20260709 12:25:57.520339 27419 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.574343 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:57.575508 27821 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:57.560124 27501 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:57.582028 27645 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:25:57.582629 27501 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:57.584303 27501 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.551502 27660 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:57.586843 27660 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:57.587479 27660 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:57.553387 27109 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:57.591221 27564 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.576s	user 0.015s	sys 0.006s
I20260709 12:25:57.592016 27564 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:57.594484 27289 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.644s	user 0.014s	sys 0.005s
I20260709 12:25:57.531513 27753 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:25:57.596325 27589 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:57.596804 27753 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:57.596999 27589 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:57.597414 27753 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:57.598690 27753 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.599282 27603 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.599768 27747 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:25:57.603663 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 260182us
I20260709 12:25:57.601286 27660 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.608388 27603 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.572307 27610 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:57.613631 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 265129us
W20260709 12:25:57.615340 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 159545us
W20260709 12:25:57.616874 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98012.8us
I20260709 12:25:57.602900 27739 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.605161 27708 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.615900 27821 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:57.625196 27753 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:57.637851 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.638137 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 65914.2us
I20260709 12:25:57.648893 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:57.624403 27589 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.652746 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:57.653437 27589 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.626605 27660 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.642354 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:57.643009 18172 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:57.646389 27603 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:25:57.647934 27812 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.312s	user 0.004s	sys 0.000s
I20260709 12:25:57.656275 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:57.660588 18093 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:57.671595 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:57.671845 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:57.672595 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3516) took 2896 ms. Trace:
I20260709 12:25:57.672868 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3793) took 1742 ms. Trace:
I20260709 12:25:57.673046 18173 rpcz_store.cc:276] 0709 12:25:54.776097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.076333 (+300236us) service_pool.cc:224] Handling call
0709 12:25:57.435266 (+2358933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.458321 (+ 23055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.664947 (+206626us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b5400030a50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:57.672490 (+  7543us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5423449344}
I20260709 12:25:57.673388 18099 rpcz_store.cc:276] 0709 12:25:55.930569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.830277 (+899708us) service_pool.cc:224] Handling call
0709 12:25:56.831837 (+  1560us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.863190 (+ 31353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.863218 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.876636 (+ 13418us) log.cc:844] Serialized 26 byte log entry
0709 12:25:57.184632 (+307996us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:57.341876 (+157244us) log.cc:844] Serialized 14 byte log entry
0709 12:25:57.342743 (+   867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.342933 (+   190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.647147 (+304214us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:57.672537 (+ 25390us) raft_consensus.cc:1685] finished
0709 12:25:57.672657 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.672804 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21796,"prepare.queue_time_us":253172,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":33278,"raft.run_cpu_time_us":444,"raft.run_wall_time_us":16146,"spinlock_wait_cycles":13504256,"thread_start_us":372221,"threads_started":2,"wal-append.queue_time_us":325153}
I20260709 12:25:57.676167 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:57.661451 18098 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:25:57.633119 27530 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.632443 27109 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.682861 27109 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:57.684463 18170 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:57.685748 27836 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:57.666121 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:57.686331 27836 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.686592 18096 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:57.686776 27836 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:57.630852 27815 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.171s
I20260709 12:25:57.695111 27648 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:57.695729 27648 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:57.695941 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:57.696934 27648 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.699214 18172 raft_consensus.cc:2368] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:57.653983 27821 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:57.626138 27708 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:57.717553 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93227.9us
I20260709 12:25:57.724421 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:57.725319 18815 raft_consensus.cc:2393] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 31: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:57.725087 18098 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:25:57.723858 27419 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:57.699199 27784 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:57.731369 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:57.738761 18170 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:57.739751 18099 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:57.741974 18094 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:57.748335 18097 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:25:57.746138 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:57.722751 27739 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:25:57.758103 27648 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.706521 27741 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.761057 27741 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:57.762058 18167 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 31. Ops: []
W20260709 12:25:57.764657 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.771024 27564 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:57.773408 27664 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.006s
I20260709 12:25:57.774027 27664 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.776914 27647 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:57.777427 27647 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:57.778437 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:57.778596 27647 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:57.780151 27647 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:57.783560 27708 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:57.784013 27500 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:57.784544 18138 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:57.784430 27645 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:57.752204 27784 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:57.786890 27784 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.789666 27529 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:57.782990 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 55649.2us
I20260709 12:25:57.791990 18096 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:25:57.807278 18092 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:57.699764 27737 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.087s
I20260709 12:25:57.776671 27664 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:57.818773 27529 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:57.822633 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:57.823580 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.844167 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:57.845508 18171 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 23 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:25:57.846647 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:57.847316 18171 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.850006 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:57.830415 27741 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:57.853735 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:57.856519 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:57.802045 27588 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.850489 27529 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.806547 27654 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:25:57.863641 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:25:57.864362 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.864792 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.860323 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:57.859933 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:57.861723 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92638.8us
I20260709 12:25:57.868350 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:57.874408 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:25:57.875664 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91250.1us
I20260709 12:25:57.875432 18811 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:57.877453 18169 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 35 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:25:57.869768 27753 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:57.867538 27562 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:57.888767 27753 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:57.895994 18166 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:57.898134 18166 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:57.903149 18063 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:57.879074 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:57.908182 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:57.909339 18164 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:25:57.877238 27656 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:57.869063 27741 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:57.877640 27776 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:57.928218 18091 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:25:57.935748 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.935582 18166 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:25:57.940400 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.915761 27603 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:57.960391 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:57.966131 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.972254 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.973019 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44005.1us
I20260709 12:25:57.994833 18169 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:25:57.990509 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:57.999063 27849 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:58.007159 27562 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:57.963413 27603 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.010157 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:58.019400 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141121us
I20260709 12:25:58.020686 18173 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: [30.20-30.20]
I20260709 12:25:58.022097 18166 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:58.035142 18171 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:57.930874 27656 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:58.042971 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73071.8us
I20260709 12:25:58.021818 27603 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:58.045970 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:25:58.049574 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.051438 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.049674 27854 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:58.052676 27854 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.053169 27854 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:58.065485 18166 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:58.067795 18171 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
W20260709 12:25:58.068095 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 191542us
I20260709 12:25:58.078820 18018 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:58.075239 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.083079 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.075636 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:58.062593 27821 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:25:58.098218 18173 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:58.098526 27753 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:58.101079 27847 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.099s	user 0.004s	sys 0.000s
I20260709 12:25:58.077672 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:58.108855 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.109612 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.110677 18093 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:58.110374 18166 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:58.048678 27660 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:25:58.120566 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:58.082705 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 142232us
I20260709 12:25:58.124207 18171 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:25:58.084095 27588 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.098256 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:58.145208 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:58.068634 27776 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:25:58.161481 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.166105 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90539.4us
I20260709 12:25:58.170351 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.128391 27500 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.171676 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.172941 18172 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:25:58.174135 18023 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.170387 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.136690 27664 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:58.183514 27855 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.114s	user 0.000s	sys 0.002s
I20260709 12:25:58.069422 27562 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.24, Last appended by leader: 24, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.152441 27564 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:58.214524 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.178676 27739 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:58.215787 18016 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 12:25:58.260303 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 74881.5us
I20260709 12:25:58.208598 18170 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
W20260709 12:25:58.214699 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137187us
I20260709 12:25:58.221453 18019 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:25:58.222038 27836 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.005s
I20260709 12:25:58.222394 18018 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:58.223704 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.224171 18015 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
I20260709 12:25:58.226380 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.226598 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.190213 27500 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.226643 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:58.227504 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:58.227674 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.229212 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.230548 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.232772 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.233628 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:58.168541 27753 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
I20260709 12:25:58.233946 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:58.104969 27741 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.234673 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.234740 18172 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:58.237283 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:58.238499 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.240113 27851 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:58.294092 18095 raft_consensus.cc:2368] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:58.294632 18017 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.283478 18164 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:58.250368 27776 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.251431 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:58.252326 27853 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:58.252503 27656 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:58.256080 18169 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:25:58.212198 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.255796 27822 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 12:25:58.268790 18173 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:58.268929 18015 pending_rounds.cc:85] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 19
I20260709 12:25:58.299803 18015 pending_rounds.cc:107] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:25:58.270727 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.271301 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.273628 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.278885 18165 raft_consensus.cc:2368] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:58.280082 18096 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:58.282614 18022 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.284716 18094 raft_consensus.cc:2368] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23.
I20260709 12:25:58.294945 18815 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:25:58.248538 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.301297 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:58.300168 18015 raft_consensus.cc:2889] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:58.303902 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:58.304297 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.306942 18021 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.311142 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.307176 18023 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:25:58.307842 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:58.313022 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:58.324994 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.325800 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.326498 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.326952 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.314827 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.317319 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.319970 18019 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.329682 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:58.329790 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 79108.7us
I20260709 12:25:58.330910 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:58.328323 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.332158 18169 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:25:58.296633 27589 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:25:58.328805 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.316123 18167 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:25:58.326124 18170 raft_consensus.cc:2376] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23. Re-sending same reply.
I20260709 12:25:58.336786 18097 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: [27.23-30.24]
I20260709 12:25:58.337992 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.338675 18173 raft_consensus.cc:2376] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27. Re-sending same reply.
I20260709 12:25:58.336485 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.339980 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.340780 18018 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.345614 18023 raft_consensus.cc:2368] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:58.340093 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.347123 18016 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:58.348975 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47750.3us
I20260709 12:25:58.341126 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.352610 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.353767 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.354764 18169 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:58.358389 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.324553 27664 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:58.279141 27836 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.366317 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.367126 18093 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:58.367228 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 147466us
I20260709 12:25:58.364267 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:58.339859 27647 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.559s	user 0.016s	sys 0.002s
I20260709 12:25:58.375849 18017 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:25:58.362993 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44897.5us
I20260709 12:25:58.347427 27708 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:25:58.379248 27860 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.379465 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.379549 27708 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:58.380055 27860 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.328418 27651 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:25:58.383252 18092 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:58.384080 27651 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:58.384514 27651 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:58.386229 18023 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:58.387028 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:58.375492 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:58.393682 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:58.394196 27822 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:58.394630 27588 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:58.395617 27647 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:58.389317 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.390087 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.398538 18020 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.382005 18095 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:58.399499 18018 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:58.399425 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.400478 18173 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:58.400931 18018 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.390303 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.397699 18170 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:58.402472 18167 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.352358 27419 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:25:58.403833 18167 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:58.311434 27853 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.374089 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.407328 18173 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.384433 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.381551 18022 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:58.418843 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:58.369031 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:58.441612 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:58.437536 18165 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:25:58.312127 27862 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:25:58.469558 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:58.370976 27863 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:58.523728 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.525036 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 156139us
W20260709 12:25:58.530687 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89212.7us
I20260709 12:25:58.536990 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:25:58.540045 27859 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.540836 27859 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.544477 27859 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:58.546057 27501 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.546885 27501 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.404312 27419 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:58.552232 27419 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:25:58.553836 27419 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.557498 27501 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:58.385677 27651 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.561489 27651 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:58.562395 18168 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
W20260709 12:25:58.563313 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.564668 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:25:58.566751 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:25:58.567687 27664 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:58.361569 27821 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.113s
I20260709 12:25:58.359299 27589 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:58.599419 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.599161 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 174842us
I20260709 12:25:58.588086 27529 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:58.601851 27530 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.602562 18136 leader_election.cc:400] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:58.629957 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:58.630476 18136 leader_election.cc:403] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:25:58.535214 27822 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.590284 27753 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:58.624166 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:58.595242 27660 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:58.621747 18016 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:25:58.623027 27647 raft_consensus.cc:2764] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:25:58.637672 27854 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.581s	user 0.013s	sys 0.004s
W20260709 12:25:58.673425 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 352052us
I20260709 12:25:58.670405 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:58.630941 27529 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:58.624672 27836 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.684170 18171 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:25:58.641860 27603 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.592s	user 0.014s	sys 0.004s
W20260709 12:25:58.684945 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82634.9us
I20260709 12:25:58.708850 18136 leader_election.cc:400] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23.
I20260709 12:25:58.709503 18136 leader_election.cc:403] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:58.636878 27530 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:58.689608 27848 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.738s	user 0.013s	sys 0.011s
I20260709 12:25:58.691931 27860 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.695902 18015 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:58.661401 27875 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260709 12:25:58.698026 27564 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:58.699668 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.703657 18172 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:25:58.660507 27648 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:58.714102 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:58.733614 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 260256us
I20260709 12:25:58.669610 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:58.604642 27863 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.745743 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:58.758693 18094 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:58.755216 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 85746.2us
I20260709 12:25:58.749305 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:58.676880 27753 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:58.714239 27529 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:58.768153 18164 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.767342 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:58.769055 18172 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:58.779433 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:58.779844 18172 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260709 12:25:58.773360 18808 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:58.777305 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:58.642509 27822 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:58.768157 27530 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.757851 27647 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 22. Current term is 23.
I20260709 12:25:58.780249 18172 pending_rounds.cc:85] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 15
I20260709 12:25:58.801329 18172 pending_rounds.cc:107] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:25:58.765236 27656 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:58.800397 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:58.803870 18171 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:58.812835 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.723745 27603 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.828980 18164 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:58.809620 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.701145 27854 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.801713 18172 raft_consensus.cc:2889] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:58.815750 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.797189 27435 raft_consensus.cc:2764] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
W20260709 12:25:58.810123 27867 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260709 12:25:58.859759 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:58.765688 27664 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.048s
I20260709 12:25:58.871311 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.873077 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:58.874384 18017 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:58.880986 27863 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:58.881916 27843 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:25:58.883520 27589 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.886281 27853 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:58.847599 27660 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:58.887285 27660 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.887691 27660 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:58.890214 27753 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.896407 27822 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:25:58.831677 27784 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.044s	user 0.006s	sys 0.011s
I20260709 12:25:58.898187 27784 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.900964 27784 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.906215 27664 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:58.906807 27664 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:58.907819 27664 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:58.914413 18024 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:58.914489 18091 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:25:58.921625 18815 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:58.909538 27664 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:58.924854 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.925786 27501 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:58.927729 27603 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:58.947155 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:25:58.927806 27878 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 12:25:58.955839 18094 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:25:58.956822 18019 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
W20260709 12:25:58.958518 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.960695 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 288831us
I20260709 12:25:58.961045 18015 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:58.947968 18093 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.574s	user 0.020s	sys 0.000s
I20260709 12:25:58.962294 18093 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
W20260709 12:25:58.967732 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 179407us
W20260709 12:25:58.971448 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.972698 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 296878us
I20260709 12:25:58.975026 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:58.996213 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:58.997094 18814 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:58.998276 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.001142 27854 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.003531 18015 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:59.003844 18024 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:59.005401 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.005959 18017 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:59.006378 18016 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:59.006706 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.007125 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.007611 18024 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:25:59.008601 18018 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:59.008715 27647 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:59.009258 27647 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:25:59.010265 27647 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:59.013664 27588 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.619s	user 0.012s	sys 0.001s
I20260709 12:25:59.013778 18015 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:59.014321 27588 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:59.015029 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:59.011299 27500 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:59.015785 18093 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:25:58.961571 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:59.018517 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:59.039145 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.973940 27849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.973s	user 0.014s	sys 0.011s
I20260709 12:25:59.017665 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:25:59.021029 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:59.018586 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:59.039865 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78392.3us
I20260709 12:25:59.040890 18094 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:25:59.068893 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.041944 27849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.042s	user 0.015s	sys 0.011s
I20260709 12:25:59.071615 18020 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:59.076416 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:59.069037 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:25:59.056463 27589 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:59.102118 27589 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:59.102447 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.102985 27589 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:59.104637 18093 raft_consensus.cc:2410] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 27 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:59.055676 27876 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:59.095633 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.105737 27876 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.106258 27876 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:59.110242 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.111418 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.112524 18018 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:59.113889 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.114183 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.079226 27849 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:25:59.116566 27849 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:59.099949 27651 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.538s	user 0.016s	sys 0.001s
I20260709 12:25:59.100878 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.122448 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.121049 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52113.8us
I20260709 12:25:59.120606 27895 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:59.139261 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.127969 18023 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.006s	sys 0.009s
W20260709 12:25:59.142438 18023 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.006s	sys 0.009s
I20260709 12:25:59.144300 18097 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:25:59.145776 27419 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:59.148134 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.149034 18022 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:25:59.127429 27784 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:59.152230 27895 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.140625 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.163312 18016 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:59.181083 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.176087 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 349993us
I20260709 12:25:59.184638 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3729) took 1108 ms. Trace:
W20260709 12:25:59.184015 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 107722us
I20260709 12:25:59.185230 18016 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.175952 27648 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:25:59.188832 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.189989 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.189885 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.193315 18064 leader_election.cc:400] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:59.193971 18064 leader_election.cc:403] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:25:59.198134 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:59.173599 27784 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.199813 18020 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.185506 18173 rpcz_store.cc:276] 0709 12:25:58.074125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.660334 (+586209us) service_pool.cc:224] Handling call
0709 12:25:58.691820 (+ 31486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.692154 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:58.692563 (+   409us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.175015 (+482452us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.175345 (+   330us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.175411 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:59.175468 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.182681 (+  7213us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.182839 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1067,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":52527,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":156,"spinlock_wait_cycles":10231040,"thread_start_us":463742,"threads_started":2,"wal-append.queue_time_us":180062}
I20260709 12:25:59.203445 18092 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:25:59.205494 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:59.204550 18780 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.969s (SENT)
I20260709 12:25:59.222718 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.207312 18090 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.020s	sys 0.017s
W20260709 12:25:59.231216 18090 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.020s	sys 0.017s
W20260709 12:25:59.234880 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.205464 27784 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:59.180339 27501 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:25:59.204692 27890 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:25:59.249986 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.250968 18016 raft_consensus.cc:2368] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:25:59.253460 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.260761 27881 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:59.261940 27892 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:59.262699 27892 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:59.263872 27892 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.241185 27822 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.265952 27858 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:59.266249 27822 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.266947 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.268196 18024 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:25:59.268817 27109 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.260809 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.269430 27109 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.279875 27589 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.281472 18097 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:59.284755 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.285318 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.298416 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 109598us
I20260709 12:25:59.314309 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.256580 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:25:59.261494 27881 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:59.247579 27435 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.253290 27846 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:59.332774 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.376569 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 120338us
I20260709 12:25:59.379729 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.335737 27881 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:59.391065 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.361552 27656 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:59.440286 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 68834.3us
I20260709 12:25:59.443517 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.444339 27708 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.444963 27708 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.446312 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.447064 18016 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25. Re-sending same reply.
I20260709 12:25:59.450723 27708 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.451627 27708 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:59.455605 27863 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.456622 27863 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.363471 27846 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:59.478798 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.485983 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.497260 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.426978 27890 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:59.475703 27846 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:59.475473 27875 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:25:59.502389 27875 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.367033 27435 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:25:59.476970 27529 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.659s	user 0.009s	sys 0.007s
W20260709 12:25:59.515305 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.476452 27501 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.470715 27894 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:59.429793 27660 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.535311 18016 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:25:59.540649 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.500296 27822 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.536058 18016 pending_rounds.cc:85] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 14
I20260709 12:25:59.554674 18016 pending_rounds.cc:107] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:25:59.555157 18016 raft_consensus.cc:2889] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:59.555804 27664 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.005s
I20260709 12:25:59.550029 27500 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.484566 27656 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.531481 27894 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:25:59.538149 27501 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:25:59.576239 27903 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:25:59.564599 27876 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.552582 27895 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.549584 27645 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.523818 27529 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.562170 27897 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:59.603443 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 160210us
I20260709 12:25:59.609078 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.619935 27894 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.629985 18021 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:59.637292 18015 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
W20260709 12:25:59.639158 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.639803 27435 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.642277 27529 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.656443 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.593954 27500 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.616827 27909 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.663277 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.621769 27894 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:59.623528 27501 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.666739 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.606614 27645 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.722172 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.723464 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:59.724097 18813 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:59.724251 27645 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.670282 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4660) took 1434 ms. Trace:
I20260709 12:25:59.724965 18091 rpcz_store.cc:276] 0709 12:25:58.236123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.312738 (+ 76615us) service_pool.cc:224] Handling call
0709 12:25:59.670065 (+1357327us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b54001afc54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:59.670202 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1254184192}
I20260709 12:25:59.671110 27500 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.671494 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:59.674306 27895 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:59.728116 18807 raft_consensus.cc:2410] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:25:59.727974 27895 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.728775 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:59.729316 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.675721 27876 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.675248 27784 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.731545 27895 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.678725 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.732645 27784 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.733007 18022 raft_consensus.cc:2368] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:25:59.729802 18806 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:25:59.737499 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:59.737717 27589 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.684834 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:59.750466 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:25:59.741520 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.755394 27900 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.287s	user 0.002s	sys 0.000s
I20260709 12:25:59.755753 18812 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:59.684701 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.757519 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:25:59.758981 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.763456 27109 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.624065 27897 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.770697 27897 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:25:59.691552 27908 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:25:59.693099 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4668) took 1333 ms. Trace:
I20260709 12:25:59.771852 18092 rpcz_store.cc:276] 0709 12:25:58.360236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.384127 (+ 23891us) service_pool.cc:224] Handling call
0709 12:25:59.692821 (+1308694us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54004119d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:25:59.693008 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1591555840}
I20260709 12:25:59.700285 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3836) took 1025 ms. Trace:
I20260709 12:25:59.772930 18096 rpcz_store.cc:276] 0709 12:25:58.701556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.719820 (+ 18264us) service_pool.cc:224] Handling call
0709 12:25:58.720299 (+   479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:59.154568 (+434269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:59.154605 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:59.187960 (+ 33355us) log.cc:844] Serialized 26 byte log entry
0709 12:25:59.475273 (+287313us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:59.475318 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.476851 (+  1533us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:59.699900 (+223049us) raft_consensus.cc:1685] finished
0709 12:25:59.699937 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.700173 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7725,"raft.queue_time_us":173411,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":191,"thread_start_us":644337,"threads_started":2,"wal-append.queue_time_us":392136}
I20260709 12:25:59.705016 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:25:59.775552 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70712.1us
I20260709 12:25:59.776530 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:25:59.779093 27895 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:25:59.781428 27858 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.515s	user 0.012s	sys 0.003s
I20260709 12:25:59.708954 27660 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:25:59.714718 27875 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:25:59.718130 27863 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:25:59.718667 27909 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.816674 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:25:59.682617 27906 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
W20260709 12:25:59.724577 27896 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:25:59.731405 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.678210 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.742887 18813 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:25:59.777653 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3435) took 1243 ms. Trace:
I20260709 12:25:59.748755 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:25:59.783102 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3431) took 1552 ms. Trace:
I20260709 12:25:59.783601 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.857630 18021 rpcz_store.cc:276] 0709 12:25:58.534657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.534909 (+   252us) service_pool.cc:224] Handling call
0709 12:25:58.566328 (+ 31419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.630595 (+1064267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.777295 (+146700us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:25:59.777545 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443471232}
I20260709 12:25:59.784461 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:59.731336 18807 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:59.646567 27893 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:59.665782 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:25:59.792141 18806 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:59.814460 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.869002 18018 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.815629 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.869415 18166 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 22: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:25:59.841560 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.867408 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.841620 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:59.817481 18019 raft_consensus.cc:2376] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25. Re-sending same reply.
I20260709 12:25:59.645519 27881 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:25:59.875527 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:59.877641 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.873054 18164 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:59.880956 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.805476 27849 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:59.860160 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:25:59.864727 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.886381 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.887301 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:25:59.867019 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 201384us
I20260709 12:25:59.888797 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.720021 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:25:59.865401 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.817219 27858 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:59.865522 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:25:59.880426 18809 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:59.858306 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:25:59.898778 18016 raft_consensus.cc:2393] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.884841 18017 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.898665 18813 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:25:59.853310 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.901897 18015 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.902144 18020 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:59.910765 18022 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.891870 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.911805 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:59.912706 18172 raft_consensus.cc:2376] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25. Re-sending same reply.
I20260709 12:25:59.741698 27846 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:25:59.917757 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:59.922827 18173 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:59.858255 27863 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.918601 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:25:59.919245 18024 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 21: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:25:59.919783 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1676) took 1157 ms. Trace:
I20260709 12:25:59.921202 18016 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:59.882606 27651 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:59.858318 18023 rpcz_store.cc:276] 0709 12:25:58.230634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.387264 (+156630us) service_pool.cc:224] Handling call
0709 12:25:58.387695 (+   431us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:58.403469 (+ 15774us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:58.403843 (+   374us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.157751 (+753908us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:59.183645 (+ 25894us) log.cc:844] Serialized 52 byte log entry
0709 12:25:59.183912 (+   267us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:59.183967 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.184076 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:59.754512 (+570436us) spinlock_profiling.cc:243] Waited 7.56 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:25:59.782579 (+ 28067us) raft_consensus.cc:1685] finished
0709 12:25:59.782761 (+   182us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.782995 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15183,"prepare.queue_time_us":31553,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":22529,"raft.queue_time_us":16028,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":6384,"spinlock_wait_cycles":17470208,"thread_start_us":738892,"threads_started":1,"wal-append.queue_time_us":923751}
I20260709 12:25:59.932418 18091 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:59.956036 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:25:59.933235 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.923133 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.886565 27784 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.921753 27501 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:59.913590 27875 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:25:59.902050 27419 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 12:25:59.902638 27564 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:25:59.911137 27822 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
W20260709 12:25:59.960116 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6267714939999678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20260709 12:25:59.991925 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:25:59.999922 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:00.029196 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6249714939999649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
W20260709 12:26:00.007084 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.035068 18166 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:25:59.983629 27895 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.884543 27893 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.074543 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:00.076282 18064 connection.cc:256] RPC call timeout handler was delayed by -1.34687149399997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260709 12:26:00.076996 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3288087270002507s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260709 12:26:00.077564 18064 connection.cc:256] RPC call timeout handler was delayed by -1.255971493999823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20260709 12:26:00.078187 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1623714940001264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20990000000000003s for a response
I20260709 12:26:00.075639 18167 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:25:59.919279 27849 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:00.078812 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.040055 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:00.083782 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.005015 27901 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.428s	user 0.002s	sys 0.000s
I20260709 12:26:00.107028 18096 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:00.002359 27918 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.145s	user 0.002s	sys 0.000s
I20260709 12:26:00.000856 27648 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:25:59.986299 27651 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:00.094012 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54130.1us
I20260709 12:26:00.010839 27419 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:00.028453 27822 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:59.925374 18810 rpcz_store.cc:276] 0709 12:25:58.762044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.762212 (+   168us) service_pool.cc:224] Handling call
0709 12:25:58.767694 (+  5482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.767962 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:59.230970 (+463008us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.673868 (+442898us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b18004bca60. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:25:59.919266 (+245398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.919355 (+    89us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:59.919428 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.919533 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.919700 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":465423,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":2631,"raft.queue_time_us":3333,"spinlock_wait_cycles":101415680,"thread_start_us":705485,"threads_started":2}
I20260709 12:26:00.009976 27845 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:00.027135 27881 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:00.151645 18095 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:00.016978 27656 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.150296 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.176047 18098 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:26:00.177194 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 289974us
I20260709 12:26:00.117432 18093 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:00.124562 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.149765 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:00.180083 18092 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:00.161665 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:00.181005 18166 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:00.001010 27647 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:00.194247 27647 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.074280 27664 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:00.081502 27603 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:00.167538 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.025372 27846 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.012513 27909 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.028646 27784 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.160727 27419 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:00.094274 27876 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.179960 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:00.247749 18096 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 27
W20260709 12:26:00.078588 18064 connection.cc:256] RPC call timeout handler was delayed by -1.0309910820001278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260709 12:25:59.994978 27645 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.157572 18173 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:00.163599 27648 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.191614 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:00.192147 27849 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:00.174998 27883 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.349s	user 0.008s	sys 0.004s
I20260709 12:26:00.284545 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:00.287367 18019 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:00.199506 18097 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:00.000226 27914 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.164s
I20260709 12:26:00.197595 27892 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:00.294167 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 399791us
W20260709 12:26:00.200837 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51223.8us
I20260709 12:25:59.986761 27863 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:00.211269 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.297859 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.055833 27875 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.159549 27651 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:00.251839 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.160521 27822 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:00.197255 27501 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.271355 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:00.278971 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.292784 27647 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.092478 27895 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.304487 18173 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:00.305013 18167 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:00.242826 27897 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:00.309248 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.310153 18095 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:26:00.311743 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4666) took 1999 ms. Trace:
I20260709 12:26:00.312263 18090 rpcz_store.cc:276] 0709 12:25:58.312151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.320508 (+  8357us) service_pool.cc:224] Handling call
0709 12:25:58.337923 (+ 17415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.346772 (+  8849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.346830 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:58.360093 (+ 13263us) log.cc:844] Serialized 26 byte log entry
0709 12:25:59.231694 (+871601us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:59.231743 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.231848 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:00.004224 (+772376us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:00.296883 (+292659us) raft_consensus.cc:1685] finished
0709 12:26:00.296913 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.311635 (+ 14722us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":45385,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":5876608,"thread_start_us":644141,"threads_started":1,"wal-append.queue_time_us":1119424}
I20260709 12:26:00.320930 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:00.326818 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 378614us
I20260709 12:26:00.322064 18167 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:00.333897 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.336248 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:00.337086 18172 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:00.275579 27845 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:00.344355 18170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.006s	sys 0.012s
W20260709 12:26:00.344946 18170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.006s	sys 0.012s
I20260709 12:26:00.242385 27664 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:00.348271 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 156786us
I20260709 12:26:00.336421 17920 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 14 to 24, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 24 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:00.349009 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.258069 27909 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:00.361610 27648 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.362466 27648 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.361589 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:00.363126 27648 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.364806 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:00.362336 27909 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.311118 27564 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:00.257377 27603 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.293500 27645 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.365495 27648 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.353605 27845 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.450337 27603 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:00.414408 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.319830 27647 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:00.309707 27501 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.273399 27876 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.337453 27897 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:00.367089 18172 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:26:00.476307 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.477560 18172 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:26:00.431129 27589 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20260709 12:26:00.450314 27648 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:00.410633 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 477769us
I20260709 12:26:00.527165 27589 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:00.527803 27589 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:00.528177 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.469475 18166 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:00.532984 18166 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:00.423414 18096 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
W20260709 12:26:00.473237 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44419.4us
W20260709 12:26:00.453661 27915 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
W20260709 12:26:00.443292 27917 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260709 12:26:00.546468 27603 consensus_queue.cc:1059] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:00.444373 27919 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:26:00.471781 27647 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.330130 27892 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:00.483209 27894 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.773s	user 0.016s	sys 0.000s
I20260709 12:26:00.485278 27664 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.486752 27863 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:00.435882 27822 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.488327 27909 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.477946 27893 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.472885 27529 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.017s
I20260709 12:26:00.492004 27876 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.492413 27895 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.494550 27501 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:00.496934 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.499137 27875 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.503501 27784 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.511202 27645 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.514645 27897 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:00.468930 18806 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.589s	user 0.014s	sys 0.000s
I20260709 12:26:00.597191 18806 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:00.528899 27739 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:00.528769 27589 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.529670 27926 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:00.530396 27849 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:00.531154 27927 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:26:00.540021 18164 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.659s	user 0.015s	sys 0.000s
I20260709 12:26:00.605360 18164 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:26:00.543331 27881 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:00.563376 18166 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:26:00.577977 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.002s	sys 0.038s
I20260709 12:26:00.585359 18167 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
W20260709 12:26:00.587430 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 240941us
W20260709 12:26:00.611775 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.004s	sys 0.040s
I20260709 12:26:00.549104 27564 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:00.594722 18165 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
W20260709 12:26:00.595340 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.599609 18172 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
W20260709 12:26:00.603578 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 238908us
I20260709 12:26:00.525336 27648 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.597782 27863 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:26:00.608017 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 94202.6us
W20260709 12:26:00.612658 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 718370us
I20260709 12:26:00.615226 18172 pending_rounds.cc:85] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 16
I20260709 12:26:00.559412 27846 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:26:00.618022 18172 pending_rounds.cc:107] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:26:00.619786 18172 raft_consensus.cc:2889] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:00.554050 27925 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
W20260709 12:26:00.627279 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.603236 27664 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:00.628340 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100286us
I20260709 12:26:00.608846 27741 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:00.601495 27893 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 12:26:00.635375 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.611186 27909 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.633406 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:00.639166 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:00.642773 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.644093 18172 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:26:00.647928 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.630858 27656 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:00.651991 27656 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:26:00.653127 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92282.7us
I20260709 12:26:00.655755 27854 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:00.659511 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.645623 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.665746 18172 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:00.670894 17920 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 21, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 21 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:00.597419 27913 consensus_queue.cc:1059] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:26:00.680985 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.683148 27822 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:00.683321 27603 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:00.616377 27645 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:26:00.684655 27648 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.684456 27935 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:26:00.709517 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.506s (SENT) [suppressed 35 similar messages]
I20260709 12:26:00.686553 27109 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:00.727900 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47003.9us
I20260709 12:26:00.687317 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.626143 27895 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:00.678647 27500 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.042s
W20260709 12:26:00.695883 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56830.1us
W20260709 12:26:00.679693 27919 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:26:00.705168 27934 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:00.706346 27931 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:26:00.711736 27892 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:00.712570 27894 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:00.640033 27664 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.715574 18169 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:00.745314 27895 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.698566 27913 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:00.729290 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.746162 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:00.693933 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:00.681298 27909 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:00.755435 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 120153us
W20260709 12:26:00.806715 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 173454us
I20260709 12:26:00.786124 27946 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:26:00.784801 27929 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.212s	user 0.001s	sys 0.001s
W20260709 12:26:00.784231 27937 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:26:00.683140 27942 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.123s
I20260709 12:26:00.830202 27925 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.715850 27943 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:26:00.831276 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.832017 27894 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:00.829152 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 82922.2us
I20260709 12:26:00.841503 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1899) took 1163 ms. Trace:
I20260709 12:26:00.857797 27500 consensus_queue.cc:1243] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:26:00.846251 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 100203us
I20260709 12:26:00.847986 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.846354 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:00.851241 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 122125us
I20260709 12:26:00.887535 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.824677 27648 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:00.825188 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.831959 27660 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:00.827468 27876 raft_consensus.cc:3055] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:00.857000 27945 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:00.839159 27913 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:00.788856 27934 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.857430 27603 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:00.850530 27946 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:00.857025 27647 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:00.855841 27947 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:26:00.828395 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 180617us
I20260709 12:26:00.881274 27645 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.884011 18812 rpcz_store.cc:276] 0709 12:25:59.678092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.756686 (+ 78594us) service_pool.cc:224] Handling call
0709 12:25:59.757045 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:59.757468 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:59.757499 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:59.764093 (+  6594us) log.cc:844] Serialized 26 byte log entry
0709 12:26:00.085707 (+321614us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:00.094245 (+  8538us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.094677 (+   432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.094785 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:00.841171 (+746386us) raft_consensus.cc:1685] finished
0709 12:26:00.841204 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.841398 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23564,"prepare.queue_time_us":422,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":91354,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":13509,"thread_start_us":321096,"threads_started":1,"wal-append.queue_time_us":602523}
I20260709 12:26:00.915115 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.874683 27925 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:00.896250 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48321.3us
W20260709 12:26:00.897229 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72180.2us
I20260709 12:26:00.901510 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.905334 27946 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.907838 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:00.917721 18019 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:00.917989 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:00.911641 18090 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:00.925998 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 79748.8us
I20260709 12:26:00.927009 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:00.888711 27648 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:00.928164 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.929595 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1468) took 1152 ms. Trace:
W20260709 12:26:00.929996 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 220523us
I20260709 12:26:00.893770 27660 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.928949 18097 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:00.930058 18170 rpcz_store.cc:276] 0709 12:25:59.777173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.785026 (+  7853us) service_pool.cc:224] Handling call
0709 12:25:59.785397 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:59.785921 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:59.785950 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:59.786523 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:26:00.345445 (+558922us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:00.345515 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.345630 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:00.929297 (+583667us) raft_consensus.cc:1685] finished
0709 12:26:00.929322 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.929522 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54547,"raft.queue_time_us":3167,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":16579,"thread_start_us":558400,"threads_started":1,"wal-append.queue_time_us":745922}
I20260709 12:26:00.898811 27876 raft_consensus.cc:740] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:00.899441 27863 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:00.939388 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.941110 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:00.903822 27892 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.903286 27501 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:00.905462 27846 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:00.952216 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.954883 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3561) took 1003 ms. Trace:
I20260709 12:26:00.959692 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:00.960811 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:00.915920 27603 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:00.969219 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:00.957394 18136 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:00.958707 27647 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:00.940128 27660 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:00.985208 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 83782.5us
I20260709 12:26:00.972069 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:00.986267 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.938620 27881 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:00.905628 27938 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
W20260709 12:26:00.980952 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66369.8us
I20260709 12:26:00.974201 18016 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:00.939507 27947 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:00.986356 18020 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:00.998607 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 59380us
I20260709 12:26:00.999557 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:01.001217 27109 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:00.961076 27956 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:00.935349 27645 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:00.974606 27529 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:01.014132 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.053071 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:01.014663 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54014.1us
I20260709 12:26:01.027789 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.028860 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:01.028860 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:01.030128 27863 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.062529 18019 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:01.031021 18090 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:01.013996 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:00.984598 27935 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:00.903620 27945 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:00.980362 27647 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:01.078061 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91905.6us
I20260709 12:26:01.078325 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.056154 18016 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:01.056586 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:01.058066 18024 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:01.060334 27645 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:01.064169 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.081713 18017 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:26:00.966536 27588 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.090s
I20260709 12:26:01.031296 27938 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:01.009888 27846 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:00.959497 27892 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.070366 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.013619 27564 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.015069 27924 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:01.081749 18812 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:01.065068 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:01.079068 18810 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:26:01.090375 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.035074 27956 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:01.112068 27952 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.117189 27951 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:01.117401 27952 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.118050 27951 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:01.119823 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.121104 18813 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:01.122262 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.124783 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.127175 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.091341 27943 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:01.138095 18814 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:01.100771 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:00.980145 27894 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.082592 27647 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:01.113637 18017 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:01.075214 27529 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:01.154476 27529 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.158795 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.160878 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.163302 18017 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:26:01.165010 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.166277 27895 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:01.161761 18018 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:01.167354 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.168184 18095 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:01.169282 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.169880 27941 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.141s
I20260709 12:26:01.145726 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.151093 27952 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:01.152094 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.170181 18017 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:01.098860 27938 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:01.033353 27876 consensus_queue.cc:260] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.168058 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:01.172420 27964 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:26:01.176039 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:01.230623 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54695.7us
I20260709 12:26:01.230947 27876 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.233260 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:01.176967 27954 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:26:01.235015 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.169557 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.236097 18095 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:26:01.237775 27934 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:01.121471 27956 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:01.247073 27940 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.409s	user 0.004s	sys 0.000s
I20260709 12:26:01.182729 18015 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:01.231477 27936 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:01.183085 27939 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:01.254796 27936 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.255005 27939 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:01.255375 27936 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:01.256428 27959 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.174s	user 0.002s	sys 0.000s
I20260709 12:26:01.188103 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.257319 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:01.189093 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.190054 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.119450 27603 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.190991 27935 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:01.191471 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.192288 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.272377 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:00.989502 27913 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.173748 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.192611 18023 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:01.060880 27648 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.204188 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.278987 18809 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:01.279250 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.279381 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.280598 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:01.281448 18780 connection.cc:256] RPC call timeout handler was delayed by -1.012882245000128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1584s for a response
I20260709 12:26:01.207932 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:01.209220 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 116711us
I20260709 12:26:00.955441 18164 rpcz_store.cc:276] 0709 12:25:59.951174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.606458 (+655284us) service_pool.cc:224] Handling call
0709 12:26:00.606875 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:00.607132 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:26:00.607448 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.718482 (+111034us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b180043d860. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:26:00.954407 (+235925us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:00.954478 (+    71us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:00.954568 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.954653 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.954808 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75448,"prepare.queue_time_us":405,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"raft.queue_time_us":75766,"spinlock_wait_cycles":18349440,"thread_start_us":160127,"threads_started":1}
I20260709 12:26:01.215973 27589 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:01.218187 27945 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:01.219585 27664 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:01.219393 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.221505 27892 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:01.314668 18017 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:01.173758 27846 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.141388 27947 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.221441 27845 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:01.225666 18024 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:01.277594 27934 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:01.226352 27881 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.181318 27955 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:26:01.240178 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.219424 27875 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:01.219414 27925 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:01.243891 27951 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:01.244189 27651 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.924s	user 0.010s	sys 0.006s
I20260709 12:26:01.239622 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3751) took 1440 ms. Trace:
I20260709 12:26:01.182605 18016 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:01.330554 18096 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:01.253921 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.187846 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.206306 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4672) took 1122 ms. Trace:
I20260709 12:26:01.173358 27924 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.283532 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.285790 18098 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:01.291555 27109 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.333222 18091 rpcz_store.cc:276] 0709 12:26:00.084404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.085364 (+   960us) service_pool.cc:224] Handling call
0709 12:26:00.085873 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:00.450637 (+364764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.450687 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:00.450748 (+    61us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:00.450797 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.205964 (+755167us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:01.206037 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.206219 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":838157184}
I20260709 12:26:01.311798 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:01.256376 27939 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.315565 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:01.320897 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.142849 27419 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.332607 18807 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:01.342254 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.351512 18016 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:01.352643 18097 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:01.328346 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.303066 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 219921us
I20260709 12:26:01.330463 18168 rpcz_store.cc:276] 0709 12:25:59.798767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.577278 (+778511us) service_pool.cc:224] Handling call
0709 12:26:00.599758 (+ 22480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:00.614375 (+ 14617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.614420 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:00.624286 (+  9866us) log.cc:844] Serialized 52 byte log entry
0709 12:26:00.703179 (+ 78893us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:00.703238 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.703324 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.239386 (+536062us) raft_consensus.cc:1685] finished
0709 12:26:01.239411 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.239555 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14072,"raft.queue_time_us":21285,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":5387,"thread_start_us":19650,"threads_started":1,"wal-append.queue_time_us":258361}
I20260709 12:26:01.322297 27934 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:01.321079 27959 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:26:01.337543 18023 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:01.322219 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.355014 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:01.326654 27875 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:01.330682 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44099.8us
I20260709 12:26:01.361807 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:01.372030 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.337376 27784 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:01.369262 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:01.337235 27924 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:01.362732 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 348846us
W20260709 12:26:01.399665 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84222.2us
W20260709 12:26:01.402145 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.406059 18094 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:26:01.374241 27952 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:26:01.365073 27845 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:26:01.407092 27875 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.408839 27875 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:01.361317 27925 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:01.364007 27934 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:01.362766 18137 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:01.361559 27881 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:01.420946 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58265.4us
I20260709 12:26:01.373391 27501 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:01.340088 27651 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.377274 27939 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:01.425503 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:01.414868 18780 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:01.450812 27753 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:01.452725 18094 raft_consensus.cc:1217] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Deduplicated request from leader. Original: 23.14->[24.15-28.17]   Dedup: 26.16->[28.17-28.17]
I20260709 12:26:01.361562 27949 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:26:01.417490 27925 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.413439 27846 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:01.464937 27925 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:01.465713 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93824.1us
I20260709 12:26:01.467972 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.469123 27893 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:01.416123 27892 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.470237 27892 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.473161 27892 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:01.476857 27966 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:26:01.479063 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3559) took 1528 ms. Trace:
I20260709 12:26:01.479595 18171 rpcz_store.cc:276] 0709 12:25:59.950418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.604015 (+653597us) service_pool.cc:224] Handling call
0709 12:26:00.604531 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:00.831630 (+227099us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.831750 (+   120us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:00.860686 (+ 28936us) log.cc:844] Serialized 26 byte log entry
0709 12:26:01.074209 (+213523us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:01.074255 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.074337 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.478718 (+404381us) raft_consensus.cc:1685] finished
0709 12:26:01.478753 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.478974 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74942,"raft.queue_time_us":174816,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":168,"spinlock_wait_cycles":140032,"thread_start_us":258803,"threads_started":2,"wal-append.queue_time_us":431141}
I20260709 12:26:01.453384 18094 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [24.15-28.17]
I20260709 12:26:01.455833 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3563) took 1504 ms. Trace:
I20260709 12:26:01.486545 18018 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:01.469652 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.378674 27946 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.486745 18166 rpcz_store.cc:276] 0709 12:25:59.951769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.611475 (+659706us) service_pool.cc:224] Handling call
0709 12:26:00.626403 (+ 14928us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:00.687331 (+ 60928us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.687372 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:00.688159 (+   787us) log.cc:844] Serialized 26 byte log entry
0709 12:26:00.894932 (+206773us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:00.920763 (+ 25831us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.921705 (+   942us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.921847 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.255816 (+333969us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b24001cff58. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:26:01.439940 (+184124us) raft_consensus.cc:1685] finished
0709 12:26:01.455541 (+ 15601us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.455752 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60383,"prepare.queue_time_us":25078,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":135063,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":27637,"spinlock_wait_cycles":21144448,"thread_start_us":172020,"threads_started":1,"wal-append.queue_time_us":257694}
W20260709 12:26:01.503413 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 77971.2us
I20260709 12:26:01.438107 27501 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.543596 27708 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.437664 27664 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:26:01.559813 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.561635 18096 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:01.567184 18092 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:01.540710 27645 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:01.577014 27946 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:01.448220 27881 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.542989 27501 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:01.375277 27935 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:01.592521 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 190110us
W20260709 12:26:01.560577 27822 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.838s	user 0.014s	sys 0.004s
W20260709 12:26:01.590906 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.598289 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 128729us
I20260709 12:26:01.602077 18064 leader_election.cc:400] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:26:01.602645 18064 leader_election.cc:403] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:26:01.606051 27645 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:01.606662 27708 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21.
I20260709 12:26:01.606796 27645 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.605361 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.613327 27708 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:01.615857 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.617915 27708 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:01.620323 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.562125 27664 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:01.635015 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.639099 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:01.686488 18168 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 26 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:26:01.640646 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:01.644953 18807 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 17. (term mismatch)
I20260709 12:26:01.687122 18807 pending_rounds.cc:85] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 16
I20260709 12:26:01.687407 18807 pending_rounds.cc:107] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260709 12:26:01.687659 18807 raft_consensus.cc:2889] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:01.647976 18166 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:26:01.650274 27972 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:26:01.589843 27753 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:01.672502 27971 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:26:01.698153 27753 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:01.703090 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82937.7us
I20260709 12:26:01.673267 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:01.678725 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 227729us
I20260709 12:26:01.682806 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.708941 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.592903 27909 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:01.720180 18815 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:26:01.738139 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:01.676415 27975 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:01.739065 18808 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:01.681073 27895 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:01.688548 27664 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:01.682152 27564 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 12:26:01.729637 27957 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
W20260709 12:26:01.771299 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 62484.8us
I20260709 12:26:01.781904 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.783628 27603 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:01.783697 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:01.784795 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.784704 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105762us
I20260709 12:26:01.786172 27946 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:01.790046 18064 leader_election.cc:400] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:01.769172 27893 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:01.801534 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:01.805060 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.805999 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.808674 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.812507 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:01.844810 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.740536 27956 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.849671 18097 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:01.772195 27909 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:26:01.767007 27975 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:01.769249 27895 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:01.866235 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.780761 27741 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:26:01.871093 27530 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.814663 27651 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:01.869365 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 147790us
W20260709 12:26:01.881577 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 91719.7us
I20260709 12:26:01.887066 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:01.888638 18780 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.726s (SENT)
I20260709 12:26:01.890121 18097 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:01.889369 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:01.897912 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96447.5us
I20260709 12:26:01.902046 18093 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:26:01.902104 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:01.918345 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 142944us
I20260709 12:26:01.930264 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:01.915863 27651 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.959265 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:01.970489 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84631.7us
I20260709 12:26:01.971477 27893 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:01.972819 27708 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:01.973410 27708 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:01.974052 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:01.975044 18096 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:01.974717 27708 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:01.976351 27708 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.978500 18091 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 31 index: 25. (index mismatch)
I20260709 12:26:01.874677 27925 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:02.002669 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72496us
I20260709 12:26:02.004841 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.011345 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.018110 27753 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.018457 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.019321 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:26:02.021108 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.024061 27753 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.024912 27975 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.031745 27894 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:02.035264 27895 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:02.036453 27875 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.616s	user 0.010s	sys 0.004s
W20260709 12:26:02.041908 27876 consensus_meta.cc:287] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.810s	user 0.016s	sys 0.000s
I20260709 12:26:02.043985 27956 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:02.044579 27660 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.034s	user 0.014s	sys 0.000s
W20260709 12:26:02.044776 27647 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.872s	user 0.013s	sys 0.003s
I20260709 12:26:01.982649 27979 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:02.051220 27983 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 12:26:01.998342 27589 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.709s	user 0.015s	sys 0.000s
I20260709 12:26:02.055022 18099 raft_consensus.cc:2410] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:26:02.045195 27647 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.045104 27660 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.041316 27645 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.066414 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:02.069526 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48724.3us
I20260709 12:26:02.077857 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.079042 18173 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:02.084264 18092 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:02.100715 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:02.101066 27976 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.626s	user 0.002s	sys 0.000s
I20260709 12:26:02.101598 18097 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:26:02.112783 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.116389 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:02.124398 27925 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:02.125314 27603 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:02.127136 27660 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:02.129516 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.139271 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.140264 18096 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:26:02.122627 27979 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:02.146708 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4690) took 1020 ms. Trace:
I20260709 12:26:02.147301 18095 rpcz_store.cc:276] 0709 12:26:01.125931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.237053 (+111122us) service_pool.cc:224] Handling call
0709 12:26:01.237525 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:01.238006 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:01.238041 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:01.238665 (+   624us) log.cc:844] Serialized 26 byte log entry
0709 12:26:01.409795 (+171130us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:01.409883 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.409990 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:02.100435 (+690445us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:02.140621 (+ 40186us) raft_consensus.cc:1685] finished
0709 12:26:02.140646 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:02.146588 (+  5942us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1440,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":113,"spinlock_wait_cycles":40375040,"thread_start_us":170634,"threads_started":1,"wal-append.queue_time_us":235580}
W20260709 12:26:02.149242 27934 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.702s	user 0.015s	sys 0.000s
W20260709 12:26:02.155355 27936 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.899s	user 0.014s	sys 0.002s
I20260709 12:26:02.150677 27664 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.129246 27951 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:26:02.163810 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.136746 27892 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:02.174557 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.175789 18097 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:02.122386 27980 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
W20260709 12:26:02.155372 27501 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.512s	user 0.006s	sys 0.007s
I20260709 12:26:02.174762 27892 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.178494 27753 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:02.187825 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.190089 27647 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:02.191126 27990 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:26:02.192924 27664 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:02.197666 27998 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.164s	user 0.002s	sys 0.000s
I20260709 12:26:02.185634 27892 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:02.199244 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:02.200809 27924 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.788s	user 0.011s	sys 0.004s
I20260709 12:26:02.162184 27934 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.223798 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:02.231716 27708 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:02.231973 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:02.232218 27708 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:02.232721 27708 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:02.234372 27994 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.202s	user 0.000s	sys 0.003s
I20260709 12:26:02.225324 27924 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.229974 27603 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:02.166538 27936 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.236539 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:02.243779 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:02.245254 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:02.246255 27603 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:02.246799 27603 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:02.247731 27603 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:02.250870 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.236435 27589 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:02.184803 27501 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.266084 27501 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:02.269721 27938 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.271266 27938 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.261514 27936 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:02.275367 27958 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.196s	user 0.006s	sys 0.000s
I20260709 12:26:02.282943 27934 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:02.283541 27924 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.290719 27648 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:02.299115 27979 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:02.299726 18092 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:02.301873 27939 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.843s	user 0.012s	sys 0.004s
I20260709 12:26:02.254223 27894 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:02.302829 27894 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:02.302994 18099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 3862) took 1401 ms (client timeout 1575 ms). Trace:
I20260709 12:26:02.303318 27894 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:02.303481 18099 rpcz_store.cc:269] 0709 12:26:00.901596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.902532 (+   936us) service_pool.cc:224] Handling call
0709 12:26:02.302775 (+1400243us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540032b9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:02.302929 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2619023616}
W20260709 12:26:02.314978 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.318394 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.319211 18091 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.330902 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:02.331211 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.331881 18099 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.332353 18096 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:26:02.308625 27564 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.308698 27501 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.333842 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.297605 27881 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.331637 27863 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:02.339831 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.331682 27845 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.341230 27845 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.343581 27845 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:02.344370 27708 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:02.355260 27989 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.315s	user 0.003s	sys 0.000s
I20260709 12:26:02.355743 18099 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:02.356909 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 397725us
I20260709 12:26:02.358263 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.361420 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.362617 18018 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:02.363852 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.370178 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.371232 18016 raft_consensus.cc:2376] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27. Re-sending same reply.
I20260709 12:26:02.373087 18017 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:26:02.373703 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 368964us
I20260709 12:26:02.377923 27845 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.385969 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.386756 18016 raft_consensus.cc:2376] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27. Re-sending same reply.
I20260709 12:26:02.388275 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.389182 18016 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:02.350032 27501 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:02.392213 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 201552us
I20260709 12:26:02.392500 27845 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.390570 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.396592 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.396657 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.394586 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.398121 18019 raft_consensus.cc:2376] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31. Re-sending same reply.
I20260709 12:26:02.356942 27741 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:02.404373 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.337976 27863 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.395597 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.360347 27924 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.408843 18017 raft_consensus.cc:2368] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:02.383358 27942 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.420568 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.421188 27949 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:02.423069 27955 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:02.423663 27955 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:02.409216 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.426787 18016 raft_consensus.cc:2368] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:26:02.428822 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.429556 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:02.430630 27928 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:02.437009 27928 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:02.439854 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:02.424919 27955 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.440085 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:02.441210 18167 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.441354 27955 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.443372 27928 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:02.444106 27928 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:02.445446 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66172.3us
I20260709 12:26:02.446498 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.336771 27564 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.446553 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.447693 18019 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:02.448978 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:02.452318 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77145.5us
I20260709 12:26:02.408829 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.458683 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.407330 27741 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:02.461357 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:02.469033 18171 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.446775 27895 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:02.467051 18018 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:02.468246 18173 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:02.445122 27928 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.482256 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:02.462564 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53843.2us
I20260709 12:26:02.472481 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.487193 18169 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:02.487357 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:02.471735 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.453681 27996 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:02.494733 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:02.495664 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:02.496474 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.498572 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.503110 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.507699 18164 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:02.508975 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:02.509270 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:02.474023 27952 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:02.521000 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:02.446545 27924 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.524756 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:02.469914 28000 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:02.479874 27895 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:02.495189 27947 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:02.513867 27708 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:02.454442 27942 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.504511 27996 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:02.569933 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.562574 18167 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:02.580600 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:02.589789 18164 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.556635 18015 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:02.503504 27999 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:02.578018 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.502938 27957 consensus_queue.cc:1059] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:26:02.505363 27980 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.062s
I20260709 12:26:02.591759 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.586447 18168 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
W20260709 12:26:02.592998 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 117313us
I20260709 12:26:02.643121 17920 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 24, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 24 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:02.570261 18171 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:02.545717 28002 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.545023 27858 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:26:02.646799 18021 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 13.
I20260709 12:26:02.520267 27945 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 12:26:02.580410 27952 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:02.592842 28015 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:26:02.660749 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.668648 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:02.666479 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.598392 27999 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:02.696266 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 241784us
I20260709 12:26:02.696988 18809 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:26:02.697237 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.698670 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.698724 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.574657 27947 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.718465 18164 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:02.719748 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.722818 18015 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 19.
I20260709 12:26:02.722856 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:02.726509 28029 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:02.727258 28029 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:02.727753 27968 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.356s	user 0.005s	sys 0.004s
I20260709 12:26:02.728349 27753 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:02.728588 28020 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:02.729965 28016 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260709 12:26:02.729508 28020 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:02.730371 27974 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.251s	user 0.000s	sys 0.005s
W20260709 12:26:02.731815 28013 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:26:02.731714 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.733032 28000 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:02.733948 28022 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:26:02.734622 27936 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.734833 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.736076 27530 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:02.737078 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:02.737464 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.581377 28003 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.123s
I20260709 12:26:02.738238 27956 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:02.738667 27956 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:02.739070 27956 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:02.739426 28019 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:02.704128 27999 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.741564 27647 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:02.739740 27845 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.728852 27753 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:02.743856 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:02.738265 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.755434 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.754916 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:02.751362 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.750275 18018 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
W20260709 12:26:02.761590 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:02.765578 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 278306us
I20260709 12:26:02.724445 27953 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.717489 27952 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:02.768993 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.704774 27975 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:02.773942 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.776784 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.778798 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.741999 27647 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:02.792436 18099 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:02.793676 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 320365us
I20260709 12:26:02.794243 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.746007 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.794934 18018 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:02.797451 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:02.796370 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:02.798626 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:02.796583 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50713.4us
I20260709 12:26:02.739094 27897 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:26:02.806042 27992 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:02.815359 27992 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:02.815812 27992 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:02.816769 27648 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.004s
W20260709 12:26:02.817299 27894 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.003s
I20260709 12:26:02.817916 18165 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 12. (term mismatch)
I20260709 12:26:02.808928 27564 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:02.818850 27892 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.004s
I20260709 12:26:02.819442 27892 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.803663 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.822876 18098 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:02.824309 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.795874 27952 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:02.826589 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.818363 18165 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 11
I20260709 12:26:02.807631 18780 leader_election.cc:400] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:02.808125 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.825135 18098 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:02.808246 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:02.778589 27957 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.799448 27645 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.827886 18165 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20260709 12:26:02.834179 18165 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:02.829098 18095 raft_consensus.cc:2368] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29.
I20260709 12:26:02.803035 18091 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:02.810114 18138 leader_election.cc:400] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:26:02.702267 27895 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.836501 18138 leader_election.cc:403] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:26:02.799511 27863 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.828099 18780 leader_election.cc:403] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:02.843248 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.843492 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.845321 18018 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:02.846102 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:02.824653 27942 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.847668 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:02.844115 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 371567us
I20260709 12:26:02.808164 27822 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:02.837711 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:02.818065 27894 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.864862 18164 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:26:02.861032 18169 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:02.868109 18169 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
W20260709 12:26:02.861028 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 138250us
I20260709 12:26:02.863881 18170 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:02.789917 27953 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.871392 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:02.872802 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70007.8us
I20260709 12:26:02.818187 28031 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:02.874366 18091 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:02.835515 27934 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.884809 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:02.886000 18169 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:02.764878 27996 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.878994 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:02.903324 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:02.848453 27664 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:02.865180 27500 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.849696 27648 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.886435 18091 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:02.856936 27924 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.854260 27975 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:02.854282 28019 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.840066 27951 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:26:02.906875 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:02.907297 18092 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:02.880076 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.940922 18018 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:02.855813 28011 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:02.879870 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48618us
I20260709 12:26:02.873965 18099 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:02.876345 28003 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.856379 28027 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:02.868722 27892 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.938912 18098 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:02.947129 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:02.956753 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:02.934543 28031 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.938247 27975 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:02.967641 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:02.963824 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:02.968457 18018 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:02.906384 27934 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.937861 27938 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:02.848778 27753 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.25, Last appended by leader: 25, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.918213 27936 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:26:02.944815 28011 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:02.948894 28028 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.971664 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:02.943431 28020 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.954056 28027 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:02.961403 27951 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:02.986822 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:03.021885 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 50372.2us
I20260709 12:26:03.022305 18018 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:02.987152 27952 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:02.988808 27953 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:02.989348 27753 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:02.943334 28036 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
W20260709 12:26:03.006162 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 127319us
I20260709 12:26:02.946595 28019 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:02.970333 27897 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:02.909070 27647 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:03.057739 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.934072 27500 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:02.972433 27975 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:03.021387 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3769) took 1413 ms. Trace:
I20260709 12:26:03.066015 18172 rpcz_store.cc:276] 0709 12:26:01.608518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.608706 (+   188us) service_pool.cc:224] Handling call
0709 12:26:01.609246 (+   540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.609575 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:01.777480 (+167905us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.353858 (+576378us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b1800109960. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:26:02.995447 (+641589us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b1800109960. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287
0709 12:26:03.020920 (+ 25473us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:03.021003 (+    83us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:03.021079 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.021164 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.021319 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":233138,"prepare.queue_time_us":286121,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":29241,"raft.queue_time_us":516450,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":8359,"spinlock_wait_cycles":58422272,"thread_start_us":457787,"threads_started":3,"wal-append.queue_time_us":300139}
I20260709 12:26:02.977079 27664 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:03.023949 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:03.024035 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.072937 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:02.974546 27935 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:02.984879 28011 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:03.052587 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.053896 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.091570 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:03.026961 28020 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:03.024310 27913 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:03.079475 18018 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:03.079435 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.027812 27909 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:03.022037 27951 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:03.013672 28027 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:03.110625 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:03.086210 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.106592 18172 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:03.106513 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.115404 18171 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:03.098204 27664 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:03.124153 27419 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:03.128549 27894 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:03.171010 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.172644 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:03.148583 28021 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 12:26:03.175756 18806 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:03.175925 18808 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:26:03.176586 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.180034 28033 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:26:03.180940 28004 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:26:03.181671 28026 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:26:03.202979 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.214476 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:03.219007 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.220316 27897 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:03.220826 27897 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.221223 27897 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:03.221618 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.224417 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.232682 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:03.241060 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.243763 28029 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.246140 28029 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:03.251606 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1522) took 1104 ms. Trace:
I20260709 12:26:03.252190 18095 rpcz_store.cc:276] 0709 12:26:02.163339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.163492 (+   153us) service_pool.cc:224] Handling call
0709 12:26:03.229767 (+1066275us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400410fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:03.251505 (+ 21738us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1518883712}
I20260709 12:26:03.254808 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.257351 27980 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:03.258617 18165 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:26:03.258211 27980 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.262279 18015 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:26:03.248250 27909 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:03.264247 27909 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:03.266702 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.271313 27997 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.228s	user 0.000s	sys 0.003s
I20260709 12:26:03.275056 27996 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:03.248987 27939 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:03.279827 27939 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:03.279940 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 373144us
I20260709 12:26:03.269572 27501 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:03.283131 27501 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:03.221168 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.287002 18136 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:03.288517 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:03.289619 27953 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:03.289935 27936 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:03.290113 27784 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.290519 27936 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.290984 27936 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:03.290743 27784 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.293349 27784 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.293466 27934 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:03.298698 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.300407 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.306059 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.306169 27938 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.307296 27938 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.309561 28036 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.310094 27924 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.310607 27822 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.310292 28036 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.310737 27924 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.311295 27822 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.313438 27924 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.314738 28036 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.315599 27822 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.280407 27939 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:03.317873 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.321769 27564 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:03.321568 28036 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:03.323195 27928 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:26:03.315821 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.331220 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.323642 27928 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:03.349799 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.351048 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:03.378291 27863 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:03.387444 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.396718 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:03.397810 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.399151 18091 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:03.407300 18091 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:03.409478 28047 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 12:26:03.263907 27980 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:26:03.404836 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:03.397243 18092 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:26:03.424173 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54178.9us
I20260709 12:26:03.408352 18091 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.366741 27935 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:03.437071 18170 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.006s	sys 0.027s
W20260709 12:26:03.437579 18170 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.006s	sys 0.027s
I20260709 12:26:03.392673 27942 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.379086 27863 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:03.448844 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52234.4us
I20260709 12:26:03.471844 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:03.473416 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.468016 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3872) took 1169 ms. Trace:
I20260709 12:26:03.475729 18097 rpcz_store.cc:276] 0709 12:26:02.302189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.303365 (+  1176us) service_pool.cc:224] Handling call
0709 12:26:02.303782 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:02.304187 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:02.304229 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:02.304841 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:26:02.444958 (+140117us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:02.445012 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.445244 (+   232us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:03.467647 (+1022403us) raft_consensus.cc:1685] finished
0709 12:26:03.467692 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.467913 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4745,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":19437,"thread_start_us":139619,"threads_started":1,"wal-append.queue_time_us":415533}
I20260709 12:26:03.432591 28036 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:03.482349 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.483572 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:03.462760 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.200115 27951 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.478823 18093 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:03.487290 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.488893 18095 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:03.484406 18092 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:03.489647 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:03.492405 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.429949 18091 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:03.503482 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.504693 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:03.505564 18093 raft_consensus.cc:2410] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:03.515496 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:03.487032 27645 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.000s
I20260709 12:26:03.466709 28051 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:03.478627 27990 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:03.433107 27909 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.399102 27664 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:03.487974 28026 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:26:03.535917 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.536876 18807 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:03.537158 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:03.541160 18778 leader_election.cc:400] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29.
I20260709 12:26:03.526778 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.459543 27822 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 12:26:03.528780 27645 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:03.562959 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1438) took 1187 ms. Trace:
I20260709 12:26:03.548704 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:03.549049 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1439) took 1173 ms. Trace:
I20260709 12:26:03.541706 18778 leader_election.cc:403] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:26:03.542330 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.542403 18811 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:26:03.491932 27928 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.469678 27942 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.547863 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:03.544559 28043 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.419s	user 0.002s	sys 0.000s
I20260709 12:26:03.564924 18022 rpcz_store.cc:276] 0709 12:26:02.375370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.447198 (+ 71828us) service_pool.cc:224] Handling call
0709 12:26:02.467783 (+ 20585us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:02.479842 (+ 12059us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:02.479878 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:02.494619 (+ 14741us) log.cc:844] Serialized 26 byte log entry
0709 12:26:02.762013 (+267394us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:02.779271 (+ 17258us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.800156 (+ 20885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.800255 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:03.562611 (+762356us) raft_consensus.cc:1685] finished
0709 12:26:03.562639 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.562860 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31714,"prepare.queue_time_us":27391,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":53927,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":8478,"spinlock_wait_cycles":894336,"thread_start_us":241783,"threads_started":1,"wal-append.queue_time_us":357675}
I20260709 12:26:03.576716 18091 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:26:03.558320 28044 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
W20260709 12:26:03.561237 28049 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.387s	user 0.001s	sys 0.000s
W20260709 12:26:03.560472 28048 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
W20260709 12:26:03.559446 28045 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:26:03.502710 28028 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.072s
I20260709 12:26:03.619168 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.592935 18812 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:03.544301 28019 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:03.595772 27603 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:03.565660 18019 rpcz_store.cc:276] 0709 12:26:02.375702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.450651 (+ 74949us) service_pool.cc:224] Handling call
0709 12:26:02.468497 (+ 17846us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:02.641078 (+172581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:02.641121 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:02.641763 (+   642us) log.cc:844] Serialized 26 byte log entry
0709 12:26:02.851250 (+209487us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:02.901926 (+ 50676us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.906263 (+  4337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.906385 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:03.279015 (+372630us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b1800335560. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c1a58b58f 00007f6c135ca571 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd
0709 12:26:03.538804 (+259789us) raft_consensus.cc:1685] finished
0709 12:26:03.548785 (+  9981us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.548967 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32701,"prepare.queue_time_us":50300,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":80,"raft.queue_time_us":32157,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":23275,"spinlock_wait_cycles":30308096,"thread_start_us":241739,"threads_started":2,"wal-append.queue_time_us":265806}
I20260709 12:26:03.558534 28046 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.532799 28036 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 12:26:03.617095 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.557206 28053 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.117s
W20260709 12:26:03.606417 27648 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.689s	user 0.007s	sys 0.008s
I20260709 12:26:03.533197 27990 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.598060 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1446) took 1151 ms. Trace:
I20260709 12:26:03.632898 18170 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:26:03.557498 28038 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.147s
I20260709 12:26:03.646150 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.616247 27934 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:03.569999 27936 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.639264 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:03.687220 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 168277us
W20260709 12:26:03.651248 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.624516 28054 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:26:03.645565 27938 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:03.664376 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.694160 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.694883 18167 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:03.696147 18806 raft_consensus.cc:2410] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:26:03.666355 27784 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:03.676542 27909 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:03.649622 27500 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.615666 27892 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:03.545389 27935 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.641317 18016 rpcz_store.cc:276] 0709 12:26:02.446906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.265432 (+818526us) service_pool.cc:224] Handling call
0709 12:26:03.265766 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.296248 (+ 30482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.296281 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:03.296843 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 12:26:03.593709 (+296866us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:03.597420 (+  3711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.597619 (+   199us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:03.597694 (+    75us) raft_consensus.cc:1685] finished
0709 12:26:03.597715 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.597958 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30024,"raft.queue_time_us":32323,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"thread_start_us":296266,"threads_started":1,"wal-append.queue_time_us":284359}
W20260709 12:26:03.693267 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 144626us
I20260709 12:26:03.657511 28030 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:03.646593 27648 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28.
I20260709 12:26:03.592002 27863 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.701892 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:03.735248 18167 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:03.737565 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.713608 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:03.719271 28056 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:03.710444 27999 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.970s	user 0.011s	sys 0.005s
W20260709 12:26:03.698940 27992 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.883s	user 0.009s	sys 0.009s
W20260709 12:26:03.731494 28062 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.083s	user 0.001s	sys 0.001s
W20260709 12:26:03.717410 27975 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.620s	user 0.014s	sys 0.002s
I20260709 12:26:03.742166 17988 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:03.741427 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:03.762243 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.638056 27603 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:03.716750 27955 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.275s	user 0.013s	sys 0.005s
I20260709 12:26:03.766381 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:03.767371 18016 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:03.644382 28046 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.780210 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:03.759660 27741 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:03.815508 27753 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.764s	user 0.009s	sys 0.005s
I20260709 12:26:03.817250 27938 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:03.818171 27938 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:03.819087 17988 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:03.819674 27938 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:03.820109 27938 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.820480 27938 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:03.822330 28071 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:03.822566 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1442) took 1446 ms. Trace:
I20260709 12:26:03.823042 18023 rpcz_store.cc:276] 0709 12:26:02.376499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.252407 (+875908us) service_pool.cc:224] Handling call
0709 12:26:03.252832 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.294726 (+ 41894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.294764 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:03.295340 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 12:26:03.638768 (+343428us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:03.638844 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.638969 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:03.822257 (+183288us) raft_consensus.cc:1685] finished
0709 12:26:03.822293 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.822487 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42843,"raft.queue_time_us":44853,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":189,"thread_start_us":334113,"threads_started":1,"wal-append.queue_time_us":496547}
I20260709 12:26:03.822475 27942 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:03.826534 27939 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.003s
I20260709 12:26:03.761610 28056 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.827129 27939 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.829327 28056 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:03.830075 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.832192 27956 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.093s	user 0.011s	sys 0.005s
I20260709 12:26:03.832778 27956 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:03.835489 27419 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.681s	user 0.011s	sys 0.001s
I20260709 12:26:03.836342 27529 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.836953 27529 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.762161 27999 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:03.840619 28031 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.866s	user 0.012s	sys 0.004s
I20260709 12:26:03.840729 27603 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.841353 28031 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:03.842057 18169 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:03.783259 27897 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.561s	user 0.013s	sys 0.002s
W20260709 12:26:03.845037 27913 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.718s	user 0.015s	sys 0.000s
I20260709 12:26:03.761729 27992 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.845855 27990 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:03.735157 27892 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.822950 28071 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:03.847664 27953 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.557s	user 0.014s	sys 0.000s
I20260709 12:26:03.838798 27936 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:03.836355 28052 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.358s	user 0.000s	sys 0.004s
I20260709 12:26:03.853261 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.853960 18171 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:03.843098 27897 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.854506 18164 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:03.855763 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 136883us
I20260709 12:26:03.790391 27975 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:03.856168 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1440) took 1480 ms. Trace:
I20260709 12:26:03.862775 18020 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:03.862789 18090 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:03.857686 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:03.864768 18024 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:03.871028 18016 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:03.801118 18167 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:03.872294 18023 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:03.885746 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:03.887007 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:03.887921 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.888471 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.874662 18814 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 28. Ops: []
I20260709 12:26:03.894122 18166 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:03.897258 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:03.898357 18168 raft_consensus.cc:2393] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:03.899148 27956 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:03.899361 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.855137 27894 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:03.900089 18168 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:03.900190 27894 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.901278 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.902959 27992 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:03.903877 28003 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.932s	user 0.013s	sys 0.002s
I20260709 12:26:03.904551 28003 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:03.889214 18023 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:03.906595 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:03.908092 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.879213 27953 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:03.910403 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:03.912220 18023 raft_consensus.cc:2376] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22. Re-sending same reply.
I20260709 12:26:03.881120 18018 rpcz_store.cc:276] 0709 12:26:02.376006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.123680 (+747674us) service_pool.cc:224] Handling call
0709 12:26:03.148167 (+ 24487us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.301279 (+153112us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.301318 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:03.302244 (+   926us) log.cc:844] Serialized 26 byte log entry
0709 12:26:03.511917 (+209673us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:03.709496 (+197579us) log.cc:844] Serialized 14 byte log entry
0709 12:26:03.852285 (+142789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.852384 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:03.852435 (+    51us) raft_consensus.cc:1685] finished
0709 12:26:03.852450 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:03.855311 (+  2861us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":188844,"prepare.queue_time_us":297744,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":232899,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":10086,"thread_start_us":467583,"threads_started":3,"wal-append.queue_time_us":291427}
W20260709 12:26:03.912583 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.912815 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.915642 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.917006 18165 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 31. Current term is 32. Ops: [31.25-31.25]
I20260709 12:26:03.917303 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:03.918375 18167 raft_consensus.cc:2393] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:03.921938 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:03.922915 18172 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:03.923280 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.924919 18016 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 22.
I20260709 12:26:03.923672 18167 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:26:03.886605 27530 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.871434 28071 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:03.876493 28070 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:26:03.945295 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:03.950878 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:03.914160 27990 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:03.946403 18023 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 21.
I20260709 12:26:03.957754 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:03.958006 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:03.959856 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:03.963791 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.969587 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59021.3us
I20260709 12:26:03.972886 18018 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:03.992555 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3497) took 1275 ms. Trace:
I20260709 12:26:03.993121 18018 rpcz_store.cc:276] 0709 12:26:02.717238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.972220 (+1254982us) service_pool.cc:224] Handling call
0709 12:26:03.972632 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.992079 (+ 19447us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.992460 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:03.994973 18018 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:03.965076 28000 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.231s	user 0.015s	sys 0.003s
I20260709 12:26:03.947888 27530 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:03.959527 28067 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.962159 27952 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:03.978181 27955 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 12:26:03.987763 27935 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.989452 27784 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:04.012091 27996 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.736s	user 0.011s	sys 0.007s
W20260709 12:26:04.031484 28011 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.927s	user 0.009s	sys 0.007s
I20260709 12:26:04.034289 28011 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.039541 27897 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.040499 28046 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.007946 28000 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:04.042002 27784 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.044409 28054 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29.
I20260709 12:26:04.044618 27784 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.045646 18815 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:04.052124 18173 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:04.011418 27952 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:04.077101 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.288013 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:04.102954 17920 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 21, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 21 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:04.021971 27935 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:04.122177 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.127465 28004 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:04.128149 28004 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.133917 18813 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:04.139521 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3782) took 1278 ms. Trace:
I20260709 12:26:04.150636 18169 rpcz_store.cc:276] 0709 12:26:02.861493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.906236 (+ 44743us) service_pool.cc:224] Handling call
0709 12:26:02.906608 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.842603 (+935995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.139044 (+296441us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b54002026d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.139397 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2150769024}
I20260709 12:26:04.153323 27892 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.149113 27947 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.155383 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:04.159689 27975 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.161592 18096 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:04.165014 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:04.166249 28011 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.166939 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:04.167821 18093 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:04.168402 28071 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.016544 28067 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.170358 27881 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:04.171370 28058 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.508s	user 0.003s	sys 0.000s
I20260709 12:26:04.171208 27881 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.172392 28067 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.173377 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1445) took 1769 ms. Trace:
I20260709 12:26:04.174603 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:04.173787 18015 rpcz_store.cc:276] 0709 12:26:02.404028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.263313 (+859285us) service_pool.cc:224] Handling call
0709 12:26:03.263708 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.296129 (+ 32421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.296170 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:03.296774 (+   604us) log.cc:844] Serialized 26 byte log entry
0709 12:26:03.502673 (+205899us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:03.502741 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.502862 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:04.170644 (+667782us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:04.172995 (+  2351us) raft_consensus.cc:1685] finished
0709 12:26:04.173023 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:04.173298 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31914,"raft.queue_time_us":34145,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":58406656,"thread_start_us":205369,"threads_started":1,"wal-append.queue_time_us":322169}
I20260709 12:26:04.175236 27999 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:04.176038 28063 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:04.176344 18015 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 28. Ops: []
I20260709 12:26:04.176724 28063 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.178097 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1487) took 1041 ms. Trace:
I20260709 12:26:04.178719 18171 rpcz_store.cc:276] 0709 12:26:03.136274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.242613 (+106339us) service_pool.cc:224] Handling call
0709 12:26:03.243245 (+   632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.872638 (+629393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.177693 (+305055us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.178004 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1403927936}
I20260709 12:26:04.179972 28064 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.180606 28064 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:04.182519 18090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4683) took 3060 ms (client timeout 1966 ms). Trace:
I20260709 12:26:04.182696 27935 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:04.183193 28064 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:04.183043 18090 rpcz_store.cc:269] 0709 12:26:01.121699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.121907 (+   208us) service_pool.cc:224] Handling call
0709 12:26:04.182277 (+3060370us) spinlock_profiling.cc:243] Waited 2.74 s on lock 0x7b5400214154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:04.182388 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6300342528}
I20260709 12:26:04.183892 18023 raft_consensus.cc:1240] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 27. Ops: []
I20260709 12:26:04.102875 27952 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:26:04.108489 28029 consensus_meta.cc:287] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.855s	user 0.012s	sys 0.004s
I20260709 12:26:04.183951 28064 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:04.194617 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.196278 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:04.198906 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:04.201540 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:04.203387 18095 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.205672 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:04.206164 18019 raft_consensus.cc:2410] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:04.207093 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:04.207846 18096 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:04.210430 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:04.211817 18098 raft_consensus.cc:2410] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:26:04.211953 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:04.214471 18015 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:04.216174 18015 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
I20260709 12:26:04.219579 18173 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:04.200944 28027 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.066s	user 0.016s	sys 0.000s
I20260709 12:26:04.226155 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:04.227321 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:04.229640 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:04.179538 28063 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.234242 27947 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:26:04.234886 27947 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:04.235320 27947 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:04.238036 27881 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.201938 28064 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.245066 28064 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.250540 28071 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.251025 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1489) took 1109 ms. Trace:
I20260709 12:26:04.251605 18165 rpcz_store.cc:276] 0709 12:26:03.141382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.259531 (+118149us) service_pool.cc:224] Handling call
0709 12:26:03.259890 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.921128 (+661238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.250616 (+329488us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x7b54000424d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.250935 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1510381312}
W20260709 12:26:04.255189 28066 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.580s	user 0.001s	sys 0.000s
I20260709 12:26:04.258227 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:04.262878 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:04.265635 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.266335 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:04.265861 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.271070 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.275923 27501 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:04.276491 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1699) took 1125 ms. Trace:
I20260709 12:26:04.276677 27501 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:04.279227 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:04.280109 18169 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
W20260709 12:26:04.281772 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52216.3us
I20260709 12:26:04.284266 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:04.286609 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:04.287150 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:04.277042 18814 rpcz_store.cc:276] 0709 12:26:03.150969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.151190 (+   221us) service_pool.cc:224] Handling call
0709 12:26:03.171020 (+ 19830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.894224 (+723204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.276146 (+381922us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54005f5854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.276405 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612063488}
I20260709 12:26:04.303071 18096 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:04.282950 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:04.314460 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 304825us
I20260709 12:26:04.249656 27928 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:04.319813 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:04.324234 18814 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:04.320441 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:04.324874 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:04.321648 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:04.325470 18171 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:04.325716 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.327003 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:04.413443 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.229972 28027 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.323962 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:04.443020 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 119198us
W20260709 12:26:04.444034 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.375061 27881 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:04.444181 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:04.472280 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.478225 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.477958 27979 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 29.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.005s
I20260709 12:26:04.483806 27979 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:04.484282 27979 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:04.471678 27881 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:04.486200 28064 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:04.488035 27996 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:04.488662 27996 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.491254 27996 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:04.467602 18095 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:04.492076 27603 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:04.493669 27603 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:04.495345 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.494975 27603 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.486328 27881 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:04.479275 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:04.497309 18097 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:04.489230 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3491) took 1788 ms. Trace:
I20260709 12:26:04.501300 18023 rpcz_store.cc:276] 0709 12:26:02.715411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.968010 (+1252599us) service_pool.cc:224] Handling call
0709 12:26:03.979343 (+ 11333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.184511 (+205168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.488923 (+304412us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54000fe954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.489147 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":469854592}
W20260709 12:26:04.506368 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.497915 18809 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:04.577411 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98231.2us
I20260709 12:26:04.499581 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:04.477408 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:04.500211 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:04.579275 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101961us
I20260709 12:26:04.492719 18814 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:04.450951 28067 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:04.606885 18095 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 32 index: 25. (index mismatch)
W20260709 12:26:04.603657 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 104189us
I20260709 12:26:04.597136 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:04.612233 27660 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:26:04.621197 27660 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:04.614547 28050 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:04.622092 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:04.622560 28050 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:04.623883 28050 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.576417 28063 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:04.605129 27645 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.626338 27645 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.579944 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:04.582317 28016 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:26:04.633739 27645 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:04.638255 27996 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:04.638898 27996 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.604029 28011 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:04.634479 28039 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:04.642279 28039 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:04.643575 28039 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.635928 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:04.648113 18095 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:04.319473 27501 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.651785 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:04.618152 18814 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 31 index: 25. Preceding OpId from leader: term: 32 index: 25. (term mismatch)
I20260709 12:26:04.641360 27996 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.653599 18814 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 24
I20260709 12:26:04.654037 18814 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 31.25
I20260709 12:26:04.654381 18814 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:04.657099 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:04.660825 28088 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.661278 27897 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:04.661854 27897 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.662442 27897 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:04.661623 28088 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.485450 27979 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.666815 27979 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:04.670606 28089 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:26:04.573892 28056 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:04.610615 27938 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.783s	user 0.016s	sys 0.000s
W20260709 12:26:04.661528 28087 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:26:04.695143 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.678308 28028 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:04.670708 27892 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:04.659821 28011 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:04.720594 27952 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.536s	user 0.010s	sys 0.004s
I20260709 12:26:04.721238 27952 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.709303 28056 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.722564 18807 raft_consensus.cc:1240] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:04.722399 28056 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.724858 27952 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:04.725509 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.724858 28056 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:04.736958 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.800966 18173 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3787) took 1867 ms (client timeout 1813 ms). Trace:
I20260709 12:26:04.813735 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1484) took 1706 ms. Trace:
I20260709 12:26:04.823437 18172 rpcz_store.cc:276] 0709 12:26:03.107062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.137319 (+ 30257us) service_pool.cc:224] Handling call
0709 12:26:03.146431 (+  9112us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.923334 (+776903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.813290 (+889956us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b540002a3d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.813623 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1785244928}
W20260709 12:26:04.815081 18164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3783) took 1953 ms (client timeout 1626 ms). Trace:
I20260709 12:26:04.815413 18169 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:26:04.825726 18164 rpcz_store.cc:269] 0709 12:26:02.861768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.866439 (+  4671us) service_pool.cc:224] Handling call
0709 12:26:04.814780 (+1948341us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:04.814971 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2008760704}
W20260709 12:26:04.801730 18173 rpcz_store.cc:269] 0709 12:26:02.938090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.018693 (+ 80603us) service_pool.cc:224] Handling call
0709 12:26:04.800615 (+1781922us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54003025d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:04.800834 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2255239552}
I20260709 12:26:04.847848 28084 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:04.849082 28084 consensus_queue.cc:1243] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:26:04.853715 28078 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:04.854749 28078 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.859884 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:04.862248 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1486) took 1734 ms. Trace:
I20260709 12:26:04.862840 18166 rpcz_store.cc:276] 0709 12:26:03.127773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.135852 (+  8079us) service_pool.cc:224] Handling call
0709 12:26:03.136282 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.894833 (+758551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.861856 (+967023us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b5400042ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:04.862130 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1716332928}
W20260709 12:26:04.875262 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.750882 28095 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:26:04.741626 27947 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:26:04.679214 27996 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.143s
W20260709 12:26:04.890506 28060 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.204s	user 0.004s	sys 0.002s
I20260709 12:26:04.892892 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:04.894021 28030 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:04.894225 28078 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:04.894821 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.894980 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3490) took 2194 ms. Trace:
I20260709 12:26:04.752398 28011 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:04.884841 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3495) took 2144 ms (client timeout 1818 ms). Trace:
I20260709 12:26:04.896554 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:04.887228 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:04.890097 28071 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:26:04.890465 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4564) took 1578 ms (client timeout 1909 ms). Trace:
W20260709 12:26:04.901059 18019 rpcz_store.cc:269] 0709 12:26:03.312145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.707221 (+395076us) service_pool.cc:224] Handling call
0709 12:26:04.890155 (+1182934us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54001fbb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:04.890348 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1001984384}
I20260709 12:26:04.854200 28083 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:04.793740 27645 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:04.897074 18024 rpcz_store.cc:269] 0709 12:26:02.716679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.752258 (+ 35579us) service_pool.cc:224] Handling call
0709 12:26:04.860960 (+2108702us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b540010a9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:04.861106 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2276777728}
W20260709 12:26:04.909466 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.895584 18017 rpcz_store.cc:276] 0709 12:26:02.700462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.265517 (+565055us) service_pool.cc:224] Handling call
0709 12:26:03.265858 (+   341us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:03.298038 (+ 32180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.298088 (+    50us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:03.299027 (+   939us) log.cc:844] Serialized 78 byte log entry
0709 12:26:03.491885 (+192858us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:03.700919 (+209034us) log.cc:844] Serialized 14 byte log entry
0709 12:26:03.701388 (+   469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.701476 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:04.889543 (+1188067us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:04.894552 (+  5009us) raft_consensus.cc:1685] finished
0709 12:26:04.894707 (+   155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:04.894897 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31598,"prepare.queue_time_us":254427,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":13758,"raft.queue_time_us":56580,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":212,"spinlock_wait_cycles":13360000,"thread_start_us":346095,"threads_started":2,"wal-append.queue_time_us":339268}
W20260709 12:26:04.899672 28004 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.771s	user 0.013s	sys 0.003s
I20260709 12:26:04.714516 27892 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:04.680078 27938 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:04.924631 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.922811 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:04.754019 28028 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:04.929778 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.905404 28071 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.930547 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:04.932895 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.904645 28056 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:04.928682 27858 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:04.937312 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:04.885558 28091 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.045s
I20260709 12:26:04.926964 27892 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.922102 27645 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.974713 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:04.975590 18019 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:26:04.934119 28071 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:04.980225 27530 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.974843 27913 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:04.926631 28004 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.983075 27951 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.981256 27913 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.983959 28004 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:04.931381 27564 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:26:04.986763 27913 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.987380 27913 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:04.987902 27913 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.913324 27947 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:04.991797 27947 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:04.992660 27529 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.997720 27647 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:04.955608 27845 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:26:05.006676 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:04.964306 28036 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.960122 28056 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:04.921221 28083 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.021091 18166 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:05.014552 27913 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:04.981256 27530 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:05.030977 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 451092us
I20260709 12:26:04.980319 27645 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:04.974221 27941 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:04.992625 27947 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:05.035183 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:05.010819 28004 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.019250 27753 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:05.053448 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3791) took 1739 ms. Trace:
I20260709 12:26:05.032269 28067 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:05.047839 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:05.041479 27941 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.056183 18168 rpcz_store.cc:276] 0709 12:26:03.315134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.914428 (+599294us) service_pool.cc:224] Handling call
0709 12:26:03.914795 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.915066 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:03.950123 (+ 35057us) log.cc:844] Serialized 14 byte log entry
0709 12:26:04.822044 (+871921us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b180002bd20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:26:05.052893 (+230849us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:05.052955 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:05.053021 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.053100 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.053377 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21790,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":3529,"raft.queue_time_us":4326,"spinlock_wait_cycles":13119744,"thread_start_us":227848,"threads_started":1}
I20260709 12:26:05.048668 27913 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:05.026602 28036 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.068750 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:05.075512 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:05.038398 27530 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.045404 27947 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.061031 27941 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:05.048089 28004 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:05.090801 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.0926e+06us
I20260709 12:26:05.053805 27753 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.094746 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.096256 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.099525 18098 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:05.099802 18095 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260709 12:26:05.099929 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:05.101222 27938 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.101697 27913 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:05.103971 28056 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.106244 28028 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.074288 28099 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:05.110453 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:05.076670 28104 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:05.116412 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1441) took 2740 ms. Trace:
I20260709 12:26:05.118534 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.121538 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:05.122432 18093 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:05.125649 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.126628 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:26:05.130357 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:05.131390 18099 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.112963 27895 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:05.138849 27845 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.139518 27845 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.143180 27947 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.144157 27897 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:05.145987 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.146487 28083 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.114667 28068 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:05.147578 28004 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.148066 28100 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:05.148672 28100 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:05.149907 28100 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.152364 27753 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:05.153148 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1698) took 2002 ms. Trace:
I20260709 12:26:05.155280 27530 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.153616 18809 rpcz_store.cc:276] 0709 12:26:03.150560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.151653 (+  1093us) service_pool.cc:224] Handling call
0709 12:26:03.180745 (+ 29092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.577423 (+1396678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.152823 (+575400us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001d9f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:05.153070 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3028647808}
I20260709 12:26:05.111366 28099 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:05.109066 27913 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:05.170492 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.193490 18092 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:26:05.111277 27603 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:26:05.162626 27881 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.587s	user 0.006s	sys 0.007s
W20260709 12:26:05.219442 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51450.1us
I20260709 12:26:05.203719 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:05.149672 27858 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:05.129863 27938 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.121305 18021 rpcz_store.cc:276] 0709 12:26:02.376241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.250124 (+873883us) service_pool.cc:224] Handling call
0709 12:26:03.250631 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:03.299550 (+ 48919us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.299590 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:03.300290 (+   700us) log.cc:844] Serialized 52 byte log entry
0709 12:26:03.597137 (+296847us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:03.712695 (+115558us) log.cc:844] Serialized 14 byte log entry
0709 12:26:03.991408 (+278713us) log.cc:844] Serialized 14 byte log entry
0709 12:26:04.819551 (+828143us) spinlock_profiling.cc:243] Waited 7.22 ms on lock 0x7b18004c5460. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:26:05.114278 (+294727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.114382 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:05.116142 (+  1760us) raft_consensus.cc:1685] finished
0709 12:26:05.116166 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.116342 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79943,"prepare.queue_time_us":240949,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":3592,"raft.queue_time_us":1005749,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":388,"spinlock_wait_cycles":16603904,"thread_start_us":952713,"threads_started":4,"wal-append.queue_time_us":277655,"wal-append.run_cpu_time_us":3998,"wal-append.run_wall_time_us":1452324}
I20260709 12:26:05.236333 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.166285 28099 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:05.238538 18093 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:05.163894 28104 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:05.207607 18099 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:05.242256 18094 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:26:05.238039 27928 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.861s	user 0.007s	sys 0.008s
I20260709 12:26:05.238299 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:05.192127 28079 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:05.229279 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:05.200008 27845 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.266722 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:05.261282 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 294343us
I20260709 12:26:05.227768 27660 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:05.251720 18098 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:05.265849 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:05.147624 28068 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.236953 27881 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.309598 17987 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:05.292210 27980 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:05.317195 18091 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:05.262401 27938 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.295228 27784 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:26:05.308167 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 195344us
I20260709 12:26:05.281160 28079 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.306413 27957 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:05.327443 28104 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.291821 27660 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.323644 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:05.295439 28071 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.331303 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:05.317571 28102 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:26:05.344053 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.344568 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.345544 18090 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:05.322145 27784 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.345769 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.349330 18170 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:05.350349 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.350965 18170 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:05.340806 27603 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:05.324196 28108 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:05.355854 28079 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.353577 27603 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:05.358033 27603 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.358944 18166 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:26:05.347875 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:05.364964 18167 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:26:05.299348 27913 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.365986 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.366235 27881 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.366773 18170 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:05.368335 27938 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.369001 27784 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.369386 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.370091 18170 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:05.372241 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.372871 18170 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:05.375500 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.375730 28071 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.375344 27941 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.376367 18170 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:05.378335 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.379580 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.380534 18170 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:05.330493 27957 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.382197 27957 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.384325 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.385248 18170 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:05.388622 28114 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:26:05.330794 28036 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.390900 27941 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.348297 28111 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:26:05.402527 27999 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.430846 27999 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.403308 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:05.432626 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.365736 27953 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:05.407078 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.410195 28063 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:05.434090 28063 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:05.434293 18064 connection.cc:256] RPC call timeout handler was delayed by -1.5373096630000873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20260709 12:26:05.434839 28091 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:05.435194 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:05.435566 28091 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.438074 28111 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:05.438273 28117 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:26:05.438584 28011 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.007s
I20260709 12:26:05.438827 28111 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.439327 28011 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.339164 27858 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:05.414108 28077 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:05.416224 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1704) took 1954 ms. Trace:
I20260709 12:26:05.440709 28077 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:05.441547 28111 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.441982 28077 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.420768 28064 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:05.443251 28064 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:05.446309 28077 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:05.450098 17916 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 183) took 78558 ms (client timeout 15000 ms). Trace:
W20260709 12:26:05.450963 17916 rpcz_store.cc:269] 0709 12:24:46.936292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.952055 (+ 15763us) service_pool.cc:224] Handling call
0709 12:26:05.449945 (+78497890us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:57.406303 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:57.407262 (+   959us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:57.407340 (+    78us) write_op.cc:435] Acquired schema lock
0709 12:25:57.407358 (+    18us) tablet.cc:662] Decoding operations
0709 12:25:57.408326 (+   968us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:57.408466 (+   140us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:57.408519 (+    53us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:25:57.409209 (+   690us) tablet.cc:701] Row locks acquired
0709 12:25:57.409276 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:25:57.409528 (+   252us) write_op.cc:270] Start()
0709 12:25:57.409744 (+   216us) write_op.cc:276] Timestamp: P: 1783599957409418 usec, L: 0
0709 12:25:57.409792 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 12:25:57.422223 (+ 12431us) log.cc:844] Serialized 1289 byte log entry
0709 12:25:57.900124 (+477901us) op_driver.cc:464] REPLICATION: finished
0709 12:25:58.170677 (+270553us) write_op.cc:301] APPLY: starting
0709 12:25:58.184252 (+ 13575us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:58.217098 (+ 32846us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b432a 00007f6c16639f14 00007f6c16639ec3 00007f6c16639e38 00007f6c16639df3 00007f6c166390dc 00007f6c16662e16 00007f6c1664acbb 00007f6c1663539d 00007f6c16638d75
0709 12:25:58.238398 (+ 21300us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:58.238444 (+    46us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:58.240650 (+  2206us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:58.241012 (+   362us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 12:25:58.241083 (+    71us) write_op.cc:312] APPLY: finished
0709 12:25:58.241436 (+   353us) log.cc:844] Serialized 41 byte log entry
0709 12:25:58.241666 (+   230us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:58.241786 (+   120us) write_op.cc:454] Released schema lock
0709 12:25:58.254428 (+ 12642us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:58.254947 (+   519us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:58.254991 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4795561,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":190395,"apply.run_cpu_time_us":8215,"apply.run_wall_time_us":116852,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16685,"num_ops":3,"prepare.queue_time_us":312817,"prepare.run_cpu_time_us":15466,"prepare.run_wall_time_us":194356,"raft.queue_time_us":176544,"raft.run_cpu_time_us":52924,"raft.run_wall_time_us":344947,"replication_time_us":477932,"spinlock_wait_cycles":7284608,"thread_start_us":552728,"threads_started":4,"wal-append.queue_time_us":245959,"wal-append.run_cpu_time_us":22005,"wal-append.run_wall_time_us":1552797}]]}
I20260709 12:26:05.424561 27892 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.429670 18022 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:26:05.404575 27894 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.433924 27953 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.410436 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.500808 18169 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:05.419260 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4568) took 2103 ms. Trace:
I20260709 12:26:05.323378 27980 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.440833 18815 rpcz_store.cc:276] 0709 12:26:03.461514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.461718 (+   204us) service_pool.cc:224] Handling call
0709 12:26:03.462148 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.046173 (+584025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.415814 (+1369641us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54004e5f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:05.416117 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1341337344}
I20260709 12:26:05.458400 28091 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.413300 27660 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.459937 27501 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:05.460939 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.507316 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:05.423892 27939 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.462462 27999 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:05.463522 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:05.473173 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.473593 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:05.498467 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:05.500664 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:05.446753 17920 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 22 to 23, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.513953 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.513530 17920 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 14 to 24, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 24 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:05.515126 18097 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:05.516077 17920 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 21, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 21 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:05.502251 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.501699 18018 rpcz_store.cc:276] 0709 12:26:03.316388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.996799 (+680411us) service_pool.cc:224] Handling call
0709 12:26:03.997272 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.054979 (+1057707us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.055016 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.114677 (+ 59661us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.282041 (+167364us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:05.282085 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.282164 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:05.416755 (+134591us) raft_consensus.cc:1685] finished
0709 12:26:05.416785 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.419150 (+  2365us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":822158,"raft.queue_time_us":849931,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":297663,"threads_started":2,"wal-append.queue_time_us":219347}
W20260709 12:26:05.522130 17916 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 287) took 15997 ms (client timeout 15000 ms). Trace:
I20260709 12:26:05.522801 18170 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:05.522938 17916 rpcz_store.cc:269] 0709 12:25:49.519730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.506282 (+15986552us) service_pool.cc:211] Skipping call since client already timed out
0709 12:26:05.516641 (+ 10359us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:26:05.519980 17920 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:05.470675 28011 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.496230 28078 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.445703 28111 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 12:26:05.530854 17920 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 26, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 26 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.538949 17920 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 26, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 26 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.554493 17920 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 24, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 24 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.590476 17920 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 23. New cstate: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.521453 27753 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:05.563617 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.523875 27979 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:05.594336 17920 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 24, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 24 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.604446 17920 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:05.592926 28024 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.106s	user 0.004s	sys 0.033s
I20260709 12:26:05.606797 28099 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.608314 28064 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.609229 27957 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.538538 28078 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:05.643612 28121 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:26:05.644598 18166 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:26:05.643726 27957 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:05.646999 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:05.651742 28099 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:05.633420 28024 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.152s	user 0.005s	sys 0.033s
I20260709 12:26:05.658365 28064 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.660274 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:05.664760 28064 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:05.665876 28064 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.667086 27980 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.673656 27892 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.676237 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:05.680823 28111 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.681577 28111 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.602480 27753 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.683597 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:05.684783 18811 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:05.687486 28027 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.693383 28064 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:05.625980 28120 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:26:05.696079 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.635032 17920 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 10 to 25, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.537942 27784 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:05.699659 27784 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.697535 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:05.705734 18164 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:05.707100 27953 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:05.706847 17920 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 29, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 29 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.626313 27979 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:05.714411 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:05.715103 18164 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.716858 27979 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.722002 18809 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.722189 27753 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.724434 28111 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.727254 28088 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.729400 18173 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:05.731092 18173 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:05.742815 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:05.743650 18172 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
W20260709 12:26:05.748917 18138 connection.cc:256] RPC call timeout handler was delayed by -1.4517010650001794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20260709 12:26:05.751571 18138 connection.cc:256] RPC call timeout handler was delayed by -1.4283010650001415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
W20260709 12:26:05.753095 18138 connection.cc:256] RPC call timeout handler was delayed by -1.1475010650001423s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260709 12:26:05.753520 18138 connection.cc:256] RPC call timeout handler was delayed by -1.0692010650000157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
I20260709 12:26:05.755681 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:05.709264 17920 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 16 to 24, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 24 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:05.747283 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:05.765758 17920 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 25, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.767257 18809 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:05.770947 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:05.768679 17920 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 21, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 21 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.773981 18815 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:05.776062 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:05.843663 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 308109us
I20260709 12:26:05.843724 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:05.775449 17920 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 25. New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.770467 28050 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:05.849080 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73102us
I20260709 12:26:05.841441 18165 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:26:05.983989 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1494) took 1711 ms. Trace:
I20260709 12:26:05.984644 18165 rpcz_store.cc:276] 0709 12:26:04.272718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.272970 (+   252us) service_pool.cc:224] Handling call
0709 12:26:05.777969 (+1504999us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592eee 000000000036eb39 00007f6c1b6061e0 00007f6c1b6cf0c4 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:26:05.835741 (+ 57772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.983482 (+147741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.983890 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11261184}
I20260709 12:26:05.772821 27753 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:05.846791 17920 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 25, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.970577 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:05.962141 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:05.988008 17920 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 28, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 28 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.989691 18811 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:05.967586 27664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.295s	user 0.020s	sys 0.003s
I20260709 12:26:05.909003 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1498) took 1623 ms. Trace:
I20260709 12:26:05.906601 18169 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 12:26:05.913671 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.933650 28031 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:05.894361 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.997664 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.938757 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4736) took 1301 ms. Trace:
I20260709 12:26:05.998615 18091 rpcz_store.cc:276] 0709 12:26:04.637854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.638034 (+   180us) service_pool.cc:224] Handling call
0709 12:26:04.638413 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.318025 (+679612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.938453 (+620428us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:05.938672 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1560268032}
I20260709 12:26:05.835167 27975 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:05.900305 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4513) took 4889 ms (client timeout 1642 ms). Trace:
I20260709 12:26:05.901043 27741 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:05.899469 18813 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1873) took 2014 ms (client timeout 2121 ms). Trace:
I20260709 12:26:05.905848 27529 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:05.930104 18022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4562) took 2619 ms (client timeout 1597 ms). Trace:
I20260709 12:26:05.867635 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:05.855532 18164 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:05.967901 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:05.929388 28134 consensus_queue.cc:1059] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.017s
I20260709 12:26:05.842044 28063 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:05.850745 28133 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:26:05.848062 28064 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:05.937402 28068 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:05.910663 28126 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:26:05.967438 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:05.842378 27979 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:05.936410 28078 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:05.990373 17920 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 24. New cstate: current_term: 24 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.984019 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:05.994513 18167 rpcz_store.cc:276] 0709 12:26:04.286105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.286270 (+   165us) service_pool.cc:224] Handling call
0709 12:26:05.908834 (+1622564us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54003b9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:05.908941 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2477859072}
W20260709 12:26:06.005595 18063 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.121s (SENT) [suppressed 24 similar messages]
I20260709 12:26:06.154361 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1458) took 1217 ms. Trace:
I20260709 12:26:06.193701 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3908) took 1096 ms. Trace:
I20260709 12:26:05.990895 27939 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:05.991848 27753 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.986304 28046 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.062s
W20260709 12:26:06.001471 18020 rpcz_store.cc:269] 0709 12:26:01.010665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.011784 (+  1119us) service_pool.cc:224] Handling call
0709 12:26:05.900072 (+4888288us) spinlock_profiling.cc:243] Waited 2.78 s on lock 0x7b540024d954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:05.900199 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6395981056}
W20260709 12:26:06.146733 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144002us
I20260709 12:26:06.012914 27529 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:06.148101 27897 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:06.149647 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 282157us
I20260709 12:26:06.206306 18096 rpcz_store.cc:276] 0709 12:26:05.097244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.099029 (+  1785us) service_pool.cc:224] Handling call
0709 12:26:05.101639 (+  2610us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.108534 (+  6895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.108582 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.129522 (+ 20940us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.927172 (+797650us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b18003b60e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:26:06.138274 (+211102us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:06.138328 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.138418 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:06.193246 (+ 54828us) raft_consensus.cc:1685] finished
0709 12:26:06.193286 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.193608 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11081,"raft.queue_time_us":9371,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":2533,"spinlock_wait_cycles":80329088,"thread_start_us":152738,"threads_started":1,"wal-append.queue_time_us":1061846}
I20260709 12:26:06.205651 18024 rpcz_store.cc:276] 0709 12:26:04.936670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.944391 (+  7721us) service_pool.cc:224] Handling call
0709 12:26:04.963214 (+ 18823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.963518 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:05.230560 (+267042us) log.cc:844] Serialized 14 byte log entry
0709 12:26:06.153616 (+923056us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b1800398ce0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:26:06.153879 (+   263us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:06.153967 (+    88us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:06.154037 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.154138 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.154283 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15506,"prepare.queue_time_us":360986,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":806154,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":123,"spinlock_wait_cycles":263475584,"thread_start_us":331587,"threads_started":2,"wal-append.queue_time_us":301775}
I20260709 12:26:06.211690 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:06.006868 18813 rpcz_store.cc:269] 0709 12:26:03.884635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.885477 (+   842us) service_pool.cc:224] Handling call
0709 12:26:05.899228 (+2013751us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54005f64d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:05.899384 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365246336}
W20260709 12:26:06.212148 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 228221us
W20260709 12:26:06.016690 18022 rpcz_store.cc:269] 0709 12:26:03.310984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.620432 (+309448us) service_pool.cc:224] Handling call
0709 12:26:05.929850 (+2309418us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:05.930016 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3634432768}
I20260709 12:26:06.191762 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:05.997090 27664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.322s	user 0.020s	sys 0.003s
W20260709 12:26:06.155501 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 807649us
I20260709 12:26:06.159076 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3496) took 3442 ms. Trace:
W20260709 12:26:06.162178 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.206586 18812 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.007671 28031 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:06.192243 28050 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:06.205497 17920 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 24, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 24 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:06.157744 28136 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.309s	user 0.000s	sys 0.004s
I20260709 12:26:06.272730 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:06.272147 17920 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 27, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 27 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:06.209071 27753 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:06.273720 18096 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:06.208817 28039 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:06.213587 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:06.148830 28054 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:06.215360 18807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1872) took 2312 ms. Trace:
W20260709 12:26:06.168267 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.257434 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:06.258497 18098 pending_rounds.cc:85] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:06.286356 18098 pending_rounds.cc:107] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:26:06.286854 18098 raft_consensus.cc:2889] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:06.259042 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:06.288843 18097 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.255578 18016 rpcz_store.cc:276] 0709 12:26:02.716957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.970074 (+1253117us) service_pool.cc:224] Handling call
0709 12:26:03.990035 (+ 19961us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.076150 (+1086115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.076194 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.115456 (+ 39262us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.313655 (+198199us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:05.524163 (+210508us) log.cc:844] Serialized 14 byte log entry
0709 12:26:06.158574 (+634411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.158693 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:06.158753 (+    60us) raft_consensus.cc:1685] finished
0709 12:26:06.158769 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.158963 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1474993,"prepare.queue_time_us":76383,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":1495076,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":249,"thread_start_us":512852,"threads_started":3,"wal-append.queue_time_us":193710}
I20260709 12:26:06.259469 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:06.255523 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43894.5us
I20260709 12:26:06.290685 18099 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:26:06.291126 18778 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:06.277544 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:06.292371 18094 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 20: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:06.209445 27939 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:06.155683 28107 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:06.285133 18807 rpcz_store.cc:276] 0709 12:26:03.884297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.885036 (+   739us) service_pool.cc:224] Handling call
0709 12:26:04.041805 (+156769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.723050 (+681245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.073606 (+1350556us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b540059fe54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:06.196466 (+122860us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540059fe50. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:06.196694 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1924555392}
I20260709 12:26:06.296015 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:06.278138 18095 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:06.169981 27951 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.163827 27858 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:06.276084 17920 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 19, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 19 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:06.303444 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.192504 27957 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.311064 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:06.154994 28108 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:06.321360 17920 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 25, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 25 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:06.323949 17920 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:06.284686 27897 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:06.254613 28064 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 30 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:06.327840 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:06.195659 27975 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.110s
I20260709 12:26:06.338254 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.340134 18813 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:06.339953 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:06.341133 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.341965 18015 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.342605 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:06.343060 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:06.344842 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:06.345646 18808 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:06.320299 27664 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:06.347527 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:06.348295 18808 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:06.348865 18018 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:26:06.351991 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.352869 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.283077 28067 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:26:06.341162 18809 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.354756 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.359184 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.340253 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.295248 28107 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.385038 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.218024 27953 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.111s
I20260709 12:26:06.300498 27939 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.374135 18023 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:06.378760 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.397023 18018 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.383491 18809 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:06.373359 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.385583 18097 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:06.397571 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.389400 18015 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:06.390533 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.329768 28042 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:06.228075 27979 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.300065 28038 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:06.400766 18016 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:06.372725 27664 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.421649 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:06.391790 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.426599 18017 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.399119 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.411218 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:06.413257 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54704.7us
I20260709 12:26:06.428170 18022 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:06.412096 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:06.283900 28050 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.427985 18023 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:26:06.424067 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:06.390981 27501 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.451623 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:06.398516 27897 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:06.412940 28082 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:06.321774 27951 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.396802 28107 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.459995 27897 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:06.482192 27897 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.415449 27975 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:06.328523 27957 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.459046 27949 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:26:06.469094 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45129.4us
I20260709 12:26:06.423971 28063 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:06.471616 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:06.472899 28039 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:06.502745 18023 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:06.474264 27858 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.474334 28108 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.479094 27564 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:06.436039 27822 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:26:06.468197 27501 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.480356 27979 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:06.522480 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.523777 18022 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:06.484588 18018 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:06.398015 27938 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.469782 28129 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:06.406973 28030 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:06.432240 27664 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.496155 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.496857 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.530690 18024 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:06.532286 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.487746 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:06.412563 28110 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:06.504542 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.534171 18016 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:26:06.483922 27975 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.507905 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.401543 28031 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.518428 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.466765 28082 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:06.505728 27949 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:06.502462 27952 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.526589 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.487282 27953 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:06.535984 28078 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:06.536665 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.537990 27938 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.539700 28129 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.540436 28110 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:06.542129 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54508.4us
I20260709 12:26:06.421416 28064 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.543241 18023 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:06.420032 28077 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.090s
W20260709 12:26:06.544514 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 538910us
I20260709 12:26:06.430151 28038 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.347271 27529 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.607733 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:06.503096 28039 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:06.552062 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.621245 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.622668 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.551990 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.623986 27529 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.624751 27957 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:06.625347 27784 consensus_queue.cc:1175] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "768723648a9444fda53659e6a8cc3091" responder_term: 27 status { last_received { term: 27 index: 13 } last_committed_idx: 13 last_received_current_leader { term: 27 index: 13 } }
I20260709 12:26:06.607319 27419 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260709 12:26:06.618496 28080 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:06.628286 28080 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.551975 18019 raft_consensus.cc:2368] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
W20260709 12:26:06.560972 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.573056 18164 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:06.487072 28068 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:06.632894 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.633574 28068 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.634284 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.635293 27951 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:06.635548 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.636071 28068 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:06.575369 18167 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:26:06.576584 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.636508 28080 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:06.559422 27953 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:06.642202 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.643018 18022 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:06.642762 27953 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.643947 27953 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:06.645319 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.578019 28031 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.625128 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:06.550936 28030 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.648043 18016 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.648811 28110 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:06.650036 28030 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:06.654661 28063 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.655341 28063 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.576185 28129 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:06.659164 18815 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.032s	sys 0.078s
W20260709 12:26:06.659631 18815 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.032s	sys 0.078s
I20260709 12:26:06.595810 18809 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
W20260709 12:26:06.552922 27741 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.004s
I20260709 12:26:06.549080 27822 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:06.661592 27822 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.545023 27975 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:06.551460 27949 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:06.659490 28038 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:06.575155 27952 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.663700 18171 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260709 12:26:06.664224 18811 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.637758 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.673616 27957 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.681949 28145 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:06.682041 18018 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:06.682570 27957 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.683068 28145 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.684101 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.677938 27949 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.672551 28011 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:06.666592 18019 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:06.672904 18017 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:06.759531 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:06.760593 18017 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:06.673444 18812 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:06.616659 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 425017us
I20260709 12:26:06.766146 18813 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:06.675921 18016 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:06.768954 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:06.770539 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:06.676252 18023 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 20.
I20260709 12:26:06.773711 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:06.657701 28063 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:06.662081 27822 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:06.795375 18811 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:06.796535 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:06.797802 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:06.799677 18168 raft_consensus.cc:2393] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:06.661971 28082 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.811810 18165 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:06.812709 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.000s	sys 0.049s
W20260709 12:26:06.813936 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.001s	sys 0.049s
I20260709 12:26:06.814854 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:06.816202 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 602604us
I20260709 12:26:06.817783 18166 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:06.817508 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:06.820819 28082 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:06.815768 27501 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:06.856137 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.832217 28050 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:06.814468 27939 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.828812 28063 consensus_queue.cc:1059] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:26:06.890847 28146 raft_consensus.cc:3055] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:06.891601 28146 raft_consensus.cc:740] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:06.893038 28146 consensus_queue.cc:260] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.882579 27939 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:06.898759 28145 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:06.898379 27975 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.833946 28039 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:06.923084 28129 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.925235 28129 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:06.664651 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:06.972062 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 307271us
W20260709 12:26:06.990307 17989 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.005s	sys 0.006s
W20260709 12:26:06.991173 17989 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.006s	sys 0.006s
W20260709 12:26:06.991519 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:06.991868 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 561534us
W20260709 12:26:06.992389 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.000797 17989 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.626s (SENT)
I20260709 12:26:06.753499 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:07.016927 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.018277 18167 raft_consensus.cc:606] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:26:07.019052 18167 raft_consensus.cc:614] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:07.029027 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1713) took 1068 ms. Trace:
I20260709 12:26:07.030650 27501 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:07.031919 27953 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:06.913116 27975 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:07.035596 28030 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:07.037043 28030 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.039085 27938 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.030247 18815 rpcz_store.cc:276] 0709 12:26:05.960491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.960995 (+   504us) service_pool.cc:224] Handling call
0709 12:26:05.961388 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.969093 (+  7705us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:26:05.992862 (+ 23769us) log.cc:844] Serialized 14 byte log entry
0709 12:26:06.085127 (+ 92265us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b1800424b40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:06.666957 (+581830us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:06.667746 (+   789us) log.cc:844] Serialized 26 byte log entry
0709 12:26:06.667948 (+   202us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:26:06.668006 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.668104 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.027873 (+359769us) raft_consensus.cc:1685] finished
0709 12:26:07.028133 (+   260us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.028584 (+   451us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8186,"prepare.queue_time_us":20134,"prepare.run_cpu_time_us":4427,"prepare.run_wall_time_us":54845,"raft.queue_time_us":29198,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":10955,"spinlock_wait_cycles":32225536,"thread_start_us":666350,"threads_started":1,"wal-append.queue_time_us":919312}
I20260709 12:26:07.002964 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:07.043538 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.046277 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:07.048009 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.003206 18022 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
I20260709 12:26:07.049413 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.051528 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:07.005522 18024 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:26:07.007093 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 399709us
I20260709 12:26:07.035619 27979 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.058240 27979 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.009992 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.060163 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 34.
I20260709 12:26:07.060563 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.062391 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.062868 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.063658 18807 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:26:07.064230 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.011751 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.066820 27564 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:07.067471 27564 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:07.068084 18093 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:07.049186 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:07.078629 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.090422 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:07.070468 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 252940us
I20260709 12:26:07.070245 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.041481 27858 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:07.071976 18814 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:07.099388 18814 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:07.031972 27501 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:07.070734 28030 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.074652 27939 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.091634 27979 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:26:07.114943 28129 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:07.115615 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.116053 28129 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.119585 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:07.127074 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.069775 27564 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.132378 27564 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:07.128539 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.144322 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.112849 27892 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:07.143338 18814 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.146572 27858 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:06.918874 28107 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.153324 18021 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:07.149370 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.158071 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.158258 18807 raft_consensus.cc:2376] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29. Re-sending same reply.
I20260709 12:26:07.155545 27501 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.159461 27939 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.161933 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.166567 18091 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:26:07.168210 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.169170 18091 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:07.164616 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.083992 28064 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:26:07.174583 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:07.180120 18090 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:07.182258 18090 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:07.181680 27897 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.186647 18092 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:07.188521 27858 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.189646 18815 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:26:07.191217 18019 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:07.193073 18019 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:07.173497 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:07.195118 28129 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.195428 18016 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:07.179921 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.198920 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:07.199187 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:07.200138 18017 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:07.200775 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:07.200961 27949 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.511s	user 0.016s	sys 0.013s
I20260709 12:26:07.201921 18099 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:07.202402 18096 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 32: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:07.203634 18096 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:07.203778 18093 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:26:07.204918 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.212811 18095 raft_consensus.cc:2393] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:07.211508 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.215463 27957 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:07.219724 28031 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.011s
I20260709 12:26:07.220818 18806 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:07.228947 28146 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:26:07.272657 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98310.4us
W20260709 12:26:07.271112 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60295.4us
I20260709 12:26:07.273752 18814 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:07.277854 27957 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:07.279183 28159 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.280172 28159 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.285035 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:07.287230 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.201756 27949 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.286806 27892 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.316725 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.320278 18096 raft_consensus.cc:2368] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26.
I20260709 12:26:07.321384 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.308770 27939 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:07.318152 27957 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:07.318732 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:07.309162 27938 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.321719 27892 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.324651 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.327556 27938 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.328406 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.331096 27938 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.332538 18096 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:26:07.312043 28064 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:07.334295 28064 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:07.334830 18096 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
W20260709 12:26:07.319327 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.338200 18094 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 26: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:07.342748 28134 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:07.343442 28134 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:07.334932 28064 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:07.335755 27957 consensus_queue.cc:1243] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:26:07.317905 18091 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:07.323118 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:07.365797 28107 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.359153 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:07.361581 17989 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.813s (SENT)
I20260709 12:26:07.379571 18019 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:07.375159 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:07.367290 27741 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:07.402762 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.402907 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:07.374737 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51850.2us
I20260709 12:26:07.406024 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:07.407078 28063 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:07.407105 27741 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:07.407829 28145 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:26:07.410110 27753 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.103s	user 0.008s	sys 0.018s
I20260709 12:26:07.391631 27858 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:07.411481 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.392838 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3926) took 1135 ms. Trace:
W20260709 12:26:07.394348 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3798) took 3067 ms (client timeout 2253 ms). Trace:
W20260709 12:26:07.412675 18171 rpcz_store.cc:269] 0709 12:26:04.326587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.326735 (+   148us) service_pool.cc:224] Handling call
0709 12:26:07.393702 (+3066967us) spinlock_profiling.cc:243] Waited 2.34 s on lock 0x7b54001b5b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.394032 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5371507968}
I20260709 12:26:07.378098 27892 raft_consensus.cc:2764] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:26:07.351332 28134 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.415961 28134 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:07.384220 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70852.7us
I20260709 12:26:07.418664 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:07.411403 27858 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:07.421702 27858 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.422331 27741 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:07.423808 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1934) took 1079 ms. Trace:
I20260709 12:26:07.424316 18806 rpcz_store.cc:276] 0709 12:26:06.344350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.344538 (+   188us) service_pool.cc:224] Handling call
0709 12:26:07.423447 (+1078909us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.423677 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2012006272}
I20260709 12:26:07.412073 18098 rpcz_store.cc:276] 0709 12:26:06.257674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.257807 (+   133us) service_pool.cc:224] Handling call
0709 12:26:06.258239 (+   432us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:06.661131 (+402892us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.661168 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:06.661963 (+   795us) log.cc:844] Serialized 52 byte log entry
0709 12:26:07.291713 (+629750us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:07.294279 (+  2566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.294397 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.391170 (+ 96773us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b54003596a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662cdbe 00007f6c16637930 00007f6c166378d8 00007f6c16637858 00007f6c16637813 00007f6c16636a9c 00007f6c1c5222f6 00007f6c1c51caa8 00007f6c166737cb 00007f6c166762c0
0709 12:26:07.392201 (+  1031us) raft_consensus.cc:1685] finished
0709 12:26:07.392238 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.392605 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":366396,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"spinlock_wait_cycles":4764544,"thread_start_us":859176,"threads_started":2,"wal-append.queue_time_us":568731}
I20260709 12:26:07.386998 28159 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.429955 28145 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:07.430634 28145 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.431068 28145 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:07.431085 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 1933) took 1088 ms. Trace:
I20260709 12:26:07.434024 18811 rpcz_store.cc:276] 0709 12:26:06.342572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.342733 (+   161us) service_pool.cc:224] Handling call
0709 12:26:07.430625 (+1087892us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54006f0ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.430872 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":736743680}
I20260709 12:26:07.434762 18166 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:07.436329 18165 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:07.437381 18165 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:26:07.438377 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.439396 18167 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
W20260709 12:26:07.440944 28082 consensus_meta.cc:287] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.619s	user 0.011s	sys 0.011s
I20260709 12:26:07.442338 27924 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:07.442361 18018 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:26:07.443876 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:07.444285 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 90846.1us
I20260709 12:26:07.444519 27952 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.446323 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.446749 18099 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:07.443185 27924 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:07.447472 27924 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:07.447777 27897 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:07.448812 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:07.450347 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:07.451911 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.393340 28039 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:07.412343 27753 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.462477 27942 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:07.463140 27942 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:07.463867 18810 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:07.464347 27942 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.465723 27753 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.467201 18812 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 29 index: 23. Preceding OpId from leader: term: 30 index: 24. (index mismatch)
I20260709 12:26:07.469230 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.474249 27941 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:07.474633 28158 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.476078 28157 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:07.479128 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1718) took 1509 ms. Trace:
I20260709 12:26:07.481137 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.482184 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.504688 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.482372 18166 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:07.371848 27949 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:07.412369 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.484021 18170 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:07.486559 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65077.9us
W20260709 12:26:07.490051 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43577.7us
I20260709 12:26:07.491611 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.493007 18811 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 29 index: 23. (index mismatch)
I20260709 12:26:07.407724 28063 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:07.493647 18165 raft_consensus.cc:614] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:07.494704 18020 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:07.503157 27741 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:07.481124 27664 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.929s	user 0.011s	sys 0.016s
I20260709 12:26:07.509995 18099 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:07.506269 18810 rpcz_store.cc:276] 0709 12:26:05.969934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.970156 (+   222us) service_pool.cc:224] Handling call
0709 12:26:07.477535 (+1507379us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400557154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.477801 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3389451520}
I20260709 12:26:07.515681 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.515395 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.524585 18091 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:26:07.525772 18098 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:07.518510 18171 raft_consensus.cc:2376] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28. Re-sending same reply.
I20260709 12:26:07.515365 18167 raft_consensus.cc:2376] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28. Re-sending same reply.
I20260709 12:26:07.527921 18813 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
W20260709 12:26:07.537724 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.542815 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4578) took 1205 ms. Trace:
I20260709 12:26:07.543385 18020 rpcz_store.cc:276] 0709 12:26:06.337709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.337893 (+   184us) service_pool.cc:224] Handling call
0709 12:26:07.542490 (+1204597us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b5400250b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.542701 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1529697664}
I20260709 12:26:07.545029 18168 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 32: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.546262 27947 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.017s
I20260709 12:26:07.550599 27892 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.551419 27892 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.554947 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:07.558344 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.562017 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:07.563875 27975 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:07.564844 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:07.567642 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:07.568382 27858 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.572785 27741 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:07.555068 27822 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.738s	user 0.003s	sys 0.017s
I20260709 12:26:07.574081 18814 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:26:07.574523 18814 pending_rounds.cc:85] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 18
I20260709 12:26:07.574756 18814 pending_rounds.cc:107] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:26:07.575037 18814 raft_consensus.cc:2889] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:07.577011 27529 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:07.579021 27529 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:07.579876 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:07.582903 28030 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:07.583230 28112 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:26:07.586155 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1715) took 1623 ms. Trace:
I20260709 12:26:07.585954 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:07.590719 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:07.596043 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.596300 28111 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.068s
I20260709 12:26:07.597498 18091 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:07.531131 27664 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.571547 27924 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.607947 28063 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.557669 27897 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:07.573773 27822 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.528479 28158 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:07.606840 28165 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:26:07.629978 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.622509 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.600356 18814 rpcz_store.cc:276] 0709 12:26:05.965905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.966100 (+   195us) service_pool.cc:224] Handling call
0709 12:26:05.966574 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:07.071553 (+1104979us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54005add20. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c1662edc0 00007f6c166b91c0 00007f6c166ba231 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c
0709 12:26:07.582837 (+511284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.586069 (+  3232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710677888}
I20260709 12:26:07.614180 18168 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:07.641127 18168 raft_consensus.cc:606] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:07.641635 18168 raft_consensus.cc:614] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:07.619786 18015 raft_consensus.cc:2368] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24.
I20260709 12:26:07.620211 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.627243 18095 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:07.644789 18023 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
W20260709 12:26:07.645975 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60190.1us
I20260709 12:26:07.633574 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:07.644789 18099 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:07.650228 18170 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:07.622712 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.659451 18171 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:07.660422 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56020.1us
I20260709 12:26:07.630292 27924 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.609771 28063 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.664862 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.666172 18091 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:07.670887 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.665696 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.666433 18165 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.678472 18166 raft_consensus.cc:2410] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:26:07.680429 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.616309 27664 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.688796 18169 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:07.686084 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:07.623433 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.634274 27897 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:07.623523 27822 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.697367 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:07.628973 28158 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.699509 18170 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:07.709529 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.619629 28064 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.713789 18171 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.679327 27975 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.713683 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.666728 28110 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:07.657272 27957 consensus_queue.cc:1059] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.061s
W20260709 12:26:07.723647 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.736580 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.718534 18091 raft_consensus.cc:1217] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Deduplicated request from leader. Original: 23.14->[24.15-30.18]   Dedup: 26.16->[28.17-30.18]
I20260709 12:26:07.737560 18098 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:07.673453 27892 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.743609 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.744560 18168 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:07.745523 18169 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: [23.14-23.14]
W20260709 12:26:07.744589 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.746766 27858 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.747682 18167 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.721140 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3528) took 1173 ms. Trace:
I20260709 12:26:07.697494 27741 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:07.723399 28154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.026s	sys 0.009s
I20260709 12:26:07.723704 18172 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.730594 27897 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.730394 27975 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:07.720350 28110 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:07.731859 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 141331us
W20260709 12:26:07.739710 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 58126.5us
I20260709 12:26:07.737699 18091 pending_rounds.cc:85] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 16
I20260709 12:26:07.755895 18091 pending_rounds.cc:107] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20260709 12:26:07.756183 18091 raft_consensus.cc:2889] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:07.723304 18814 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:07.752590 18094 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:07.749347 18023 rpcz_store.cc:276] 0709 12:26:06.546323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.772407 (+226084us) service_pool.cc:224] Handling call
0709 12:26:07.448410 (+676003us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:26:07.719848 (+271438us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.719979 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":872616576}
I20260709 12:26:07.751197 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:07.715090 27529 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
W20260709 12:26:07.752202 28154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.027s	sys 0.009s
I20260709 12:26:07.764636 18167 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:07.775638 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.771445 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3516) took 1272 ms. Trace:
I20260709 12:26:07.793221 18166 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:07.795588 18098 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:07.776854 18015 rpcz_store.cc:276] 0709 12:26:06.498708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.536294 (+ 37586us) service_pool.cc:224] Handling call
0709 12:26:07.759888 (+1223594us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400320854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.771347 (+ 11459us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2435291904}
I20260709 12:26:07.798187 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.799732 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:07.803350 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:07.806391 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.807928 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.808743 18168 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 32: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.809980 27741 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.812668 27741 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:07.813328 27741 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:07.772753 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:07.762799 28110 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.809675 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.824347 18170 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:07.734028 28145 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.774462 27858 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:07.827512 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.834878 18098 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.835978 18168 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:07.829639 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57030.6us
I20260709 12:26:07.842532 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.843283 18165 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:07.844895 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.846140 18170 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:07.851162 18173 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:07.844115 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:07.858554 18169 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:26:07.838450 27949 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:07.863348 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:07.864897 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.867622 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.845600 27939 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:07.888875 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.889657 18169 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:07.882530 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 476628us
W20260709 12:26:07.883173 18093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.013s	sys 0.019s
W20260709 12:26:07.892073 18093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.013s	sys 0.019s
I20260709 12:26:07.874218 27949 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.854339 27858 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:07.893741 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.897881 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.896562 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.842648 27822 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:26:07.874058 27924 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:07.892463 28146 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:07.914410 18165 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:07.912910 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.908874 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:07.915474 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.898763 27949 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:07.925810 18171 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260709 12:26:07.928931 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:07.931460 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 103302us
I20260709 12:26:07.920631 27741 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:07.974050 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:07.975751 18168 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:07.950047 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 109420us
W20260709 12:26:07.952483 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.954010 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.958137 27975 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:07.942962 27892 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:07.991346 27892 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:07.963611 27664 raft_consensus.cc:3055] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:07.965376 27564 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:07.966686 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:07.971086 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1509) took 1192 ms. Trace:
I20260709 12:26:07.972453 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:07.946267 27858 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.004555 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:07.944600 28171 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:26:08.006204 18167 raft_consensus.cc:606] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:26:07.998677 18172 rpcz_store.cc:276] 0709 12:26:06.778709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.780562 (+  1853us) service_pool.cc:224] Handling call
0709 12:26:07.970733 (+1190171us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54001b4c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:07.970955 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1901835008}
I20260709 12:26:08.006906 18167 raft_consensus.cc:614] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:08.007355 18097 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:07.978790 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:07.998028 27564 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:08.014882 27564 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.016252 27564 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:07.933185 27924 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:08.019198 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:26:08.022317 27924 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:07.996167 27664 raft_consensus.cc:740] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:07.983991 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.984292 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 376676us
I20260709 12:26:07.985901 28064 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:07.988945 27741 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:07.991896 27892 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:07.998723 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.000980 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:08.024067 28146 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:08.025533 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:07.983016 28145 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:08.028714 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.004659 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:08.033430 18169 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:08.013486 28174 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.095s	user 0.000s	sys 0.002s
I20260709 12:26:08.033679 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:08.052086 18098 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:26:08.049531 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77918.7us
I20260709 12:26:08.061792 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:08.062867 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69886.2us
I20260709 12:26:08.036087 28146 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:08.068163 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:08.072307 18814 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:08.071146 18812 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:08.077805 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:08.060110 28063 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:08.098104 18806 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:08.075935 27975 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:08.105952 27741 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.117626 18015 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:26:08.119570 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:08.072829 27501 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.099157 28063 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.130389 18023 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:08.127401 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:08.137131 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.140729 18812 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:26:08.141428 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.106360 27975 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.123646 27501 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.153139 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:08.135725 27949 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.136497 28063 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:08.158000 28145 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.158668 28145 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.163712 28011 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:08.164258 27975 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:08.173970 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.088074 27529 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:26:08.195307 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:08.213910 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 118378us
I20260709 12:26:08.216413 28111 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:08.217031 28111 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:08.217496 28111 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:08.083194 27664 consensus_queue.cc:260] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:08.213225 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60250.5us
I20260709 12:26:08.199090 28112 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:08.253541 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 157866us
W20260709 12:26:08.264263 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51095.1us
W20260709 12:26:08.271973 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.275722 18814 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:08.280300 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.304582 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:08.277122 27501 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:08.308404 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 221330us
I20260709 12:26:08.313966 18812 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:08.260355 28112 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.334702 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:08.331223 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:08.336949 18091 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:08.344750 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:08.316644 28145 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:08.354478 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50069.9us
I20260709 12:26:08.356666 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:08.357563 18091 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:08.326414 27957 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:08.361104 18814 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 22.
I20260709 12:26:08.336809 27501 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.317173 28110 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.382599 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:08.362538 27822 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:08.386576 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:08.356901 27741 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:08.396385 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49061.5us
W20260709 12:26:08.425110 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64562.9us
I20260709 12:26:08.397020 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:08.426272 18020 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:26:08.413478 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.422788 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:08.428421 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:08.387197 28182 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.260s	user 0.001s	sys 0.001s
I20260709 12:26:08.375108 27942 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:26:08.502723 28145 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:08.505134 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 76829us
I20260709 12:26:08.505874 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:08.506170 27941 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:08.380209 27501 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.442734 28159 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:08.506558 27897 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:08.398627 27949 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:08.548141 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.486202 27938 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:26:08.506669 27941 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:08.553843 28039 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:08.554553 28039 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:08.554450 27941 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.555861 28039 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.556460 27941 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:08.558734 28194 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260709 12:26:08.561395 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 164541us
I20260709 12:26:08.540109 27897 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.570374 27897 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:08.570650 27939 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:08.571401 27939 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:08.572400 27939 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.574323 27664 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:08.540318 28159 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.576311 28159 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:08.577085 28201 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.079s	user 0.003s	sys 0.000s
W20260709 12:26:08.558876 28200 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
W20260709 12:26:08.595753 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.497869 27858 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.428800 27822 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:08.612049 28080 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:08.612711 28080 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:08.613998 27858 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.621899 28011 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:08.632373 28011 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.633028 28011 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:08.634652 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.613741 28080 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.511659 27741 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.658030 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:08.676241 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 266411us
W20260709 12:26:08.677250 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 249034us
I20260709 12:26:08.649389 28202 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.669368 28080 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:08.684478 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 301324us
W20260709 12:26:08.719430 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 61527.8us
W20260709 12:26:08.723961 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 355915us
I20260709 12:26:08.687886 28110 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:08.730849 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:08.740906 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:08.732342 27501 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:08.746124 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.747030 18097 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:08.729205 28030 raft_consensus.cc:2764] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:08.731621 28172 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:08.765107 28172 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.752861 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:08.753898 27938 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:08.758155 28211 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:26:08.759450 28063 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.602s	user 0.013s	sys 0.013s
I20260709 12:26:08.759754 28039 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:08.760150 28111 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.005s
I20260709 12:26:08.761096 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:08.755800 28207 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:26:08.715943 27822 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:08.764011 28192 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:26:08.814417 28203 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.826598 28203 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.689286 28202 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.747285 28209 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.010s
I20260709 12:26:08.750833 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:08.830677 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80002.3us
I20260709 12:26:08.831115 28222 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.831782 28222 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.833009 27938 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:08.833521 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.774927 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.846093 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.779098 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:08.800925 28063 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.810370 28220 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:26:08.820197 28208 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:26:08.829841 28202 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:08.767419 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.882846 28222 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:08.833678 27938 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:08.885200 18094 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:08.886152 28220 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:08.887458 28220 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.888059 18095 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:08.833858 28110 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:08.822548 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:08.889545 28110 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.890302 18164 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:26:08.891592 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:08.892107 28110 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:08.836093 28203 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:08.892611 18813 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:08.836186 28111 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.804134 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:08.894105 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 147804us
I20260709 12:26:08.894882 28111 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:08.837323 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.845005 18168 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.797s	user 0.012s	sys 0.010s
I20260709 12:26:08.896546 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:08.896973 18168 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
W20260709 12:26:08.825543 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 143571us
I20260709 12:26:08.898176 27939 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:08.898866 27939 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:08.900080 27892 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.850s	user 0.028s	sys 0.002s
I20260709 12:26:08.900537 27892 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:08.901340 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:08.901780 18099 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:08.848209 18091 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:08.903272 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:08.781379 28112 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:08.766917 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 113607us
I20260709 12:26:08.897677 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3831) took 1095 ms. Trace:
I20260709 12:26:08.854429 18170 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:26:08.850347 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.898882 18138 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:08.879395 28158 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.906625 18168 rpcz_store.cc:276] 0709 12:26:07.802351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.922023 (+119672us) service_pool.cc:224] Handling call
0709 12:26:08.897566 (+975543us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:08.885712 27938 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:08.901387 27939 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:08.906076 27924 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:08.824992 28172 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:08.804615 28039 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.001590 28213 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:26:08.912305 18090 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:09.108383 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 280111us
I20260709 12:26:08.885340 28063 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.150220 27858 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.150871 28214 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:09.151889 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:09.152786 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 244781us
I20260709 12:26:09.154016 28209 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:09.154829 28211 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:09.158980 28207 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:09.156090 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 257332us
W20260709 12:26:09.157290 28146 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.078s	user 0.015s	sys 0.019s
W20260709 12:26:09.142381 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 213972us
W20260709 12:26:09.142879 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 231134us
W20260709 12:26:09.161911 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.163755 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.182722 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.155359 27564 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.139s	user 0.026s	sys 0.014s
W20260709 12:26:09.189989 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.180652 27941 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.617s	user 0.023s	sys 0.001s
I20260709 12:26:09.158351 28191 raft_consensus.cc:2764] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:26:09.193455 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:09.165330 28112 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.185700 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:09.181265 28080 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.163794 28158 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.195211 28146 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.219064 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:09.220052 28215 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.220486 28212 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:09.221153 28146 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:09.198024 27858 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:26:09.222785 18091 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:09.224874 28225 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:09.220774 28215 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.229506 18092 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:09.203197 27924 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:09.255698 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:09.256760 18815 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:09.199795 28224 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
W20260709 12:26:09.262233 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45301.6us
I20260709 12:26:09.211839 28080 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.232120 27741 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:09.213722 27941 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:26:09.208276 28112 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:09.269594 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.006s	sys 0.009s
W20260709 12:26:09.282780 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.007s	sys 0.011s
W20260709 12:26:09.283298 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 921487us
W20260709 12:26:09.275059 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.284809 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.248095 27924 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:09.235288 28064 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:09.277294 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95844.9us
I20260709 12:26:09.246320 27892 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:09.261457 28145 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.756s	user 0.020s	sys 0.011s
I20260709 12:26:09.221187 28212 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:09.285794 27897 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.715s	user 0.027s	sys 0.003s
W20260709 12:26:09.286118 28159 consensus_meta.cc:287] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.709s	user 0.013s	sys 0.020s
W20260709 12:26:09.309441 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.316494 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.317297 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 114635us
W20260709 12:26:09.370427 28011 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.737s	user 0.017s	sys 0.003s
W20260709 12:26:09.370890 27975 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.206s	user 0.023s	sys 0.012s
I20260709 12:26:09.371028 28011 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.371487 27975 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.372359 28215 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:09.373659 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 63992.9us
I20260709 12:26:09.374639 28039 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:09.380849 28158 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:09.295269 28064 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:09.385547 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:09.390743 18814 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:09.383064 18091 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:09.392004 18091 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:09.395865 18064 leader_election.cc:400] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:26:09.396433 18064 leader_election.cc:403] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:09.309762 28145 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:09.401836 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:09.402582 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.404224 18807 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:26:09.410106 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105399us
I20260709 12:26:09.409276 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:09.394827 28064 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:09.348910 27957 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:26:09.424448 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:09.424965 18096 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:09.426345 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:09.436736 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 173548us
I20260709 12:26:09.439095 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:09.358752 28159 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.440934 18095 raft_consensus.cc:2393] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:09.423245 18806 raft_consensus.cc:2393] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:09.373659 28011 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.446802 18814 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:26:09.358178 27897 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.455870 28112 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.464110 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:09.477620 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:09.467456 28063 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:09.469189 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:09.436754 18091 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.467698 27741 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:09.423460 28158 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:26:09.457671 28080 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.485999 18093 raft_consensus.cc:2368] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23.
I20260709 12:26:09.442175 28224 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:09.488931 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:09.488095 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:09.488207 18097 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:09.514144 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 137910us
I20260709 12:26:09.515617 18091 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:09.476970 28110 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:09.518306 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:09.479394 28039 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:09.494094 27938 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:09.504176 27564 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:09.497226 27892 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:09.513635 28224 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.539884 28011 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:09.540365 28112 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:26:09.554013 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.541960 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:09.520473 28080 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:09.555539 18020 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:09.557109 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:09.493708 27741 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.573370 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.572691 18168 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:09.582505 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.594456 18170 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:09.584187 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:09.587414 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:09.543402 27938 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:09.582386 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.471117 27924 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.530277 28234 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.545178 27939 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.595749 18167 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:26:09.601933 18815 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 33.
I20260709 12:26:09.546867 28213 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:09.546583 28224 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:09.595690 18165 raft_consensus.cc:2376] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29. Re-sending same reply.
I20260709 12:26:09.678653 18091 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260709 12:26:09.680743 18091 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:26:09.681136 18091 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260709 12:26:09.681509 18091 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:09.529371 28039 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:09.686299 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.690912 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97794.5us
I20260709 12:26:09.694334 27939 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.693068 18020 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:26:09.575265 27892 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:09.697403 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.502s (SENT)
I20260709 12:26:09.691828 18063 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:09.701737 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:09.685107 28235 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:26:09.695953 28234 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.716888 28234 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:09.719146 27741 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:09.719434 28064 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.706530 27892 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.709434 28039 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:09.710144 28063 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:26:09.711130 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 192952us
I20260709 12:26:09.695408 28202 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:26:09.738132 28202 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:09.739317 28202 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.740698 28202 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:09.711615 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:09.744379 18091 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:26:09.744772 28211 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.546s	user 0.008s	sys 0.005s
I20260709 12:26:09.745281 28211 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:09.746302 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44835.5us
I20260709 12:26:09.745483 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:09.730723 27957 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.733758 27938 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.677049 28220 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:09.751313 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53978.8us
I20260709 12:26:09.730036 28064 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.776312 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:09.781427 18095 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:09.783079 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.610s (SENT)
I20260709 12:26:09.752801 28234 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:09.730330 28212 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.751279 28145 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.769691 27741 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:09.796912 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.778096 28203 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:09.816438 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.774063 28039 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:09.807101 28234 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.835945 28234 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:09.778432 28225 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 12:26:09.838279 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:09.839215 18815 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:09.837393 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:09.842320 28246 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.843266 28246 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.844334 28223 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:09.844974 28223 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:09.816421 28145 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.850118 28238 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:09.850804 28238 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:09.815826 27957 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.852031 28238 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.854182 28241 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.858074 28225 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:09.858791 28225 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.819444 27741 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.787207 28220 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.792402 28240 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:09.862397 28240 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:09.863469 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.863903 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4769) took 1186 ms. Trace:
I20260709 12:26:09.864363 18092 rpcz_store.cc:276] 0709 12:26:08.677929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.678143 (+   214us) service_pool.cc:224] Handling call
0709 12:26:08.688357 (+ 10214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:09.231696 (+543339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.863540 (+631844us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b54002959d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:09.863808 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1238462208}
I20260709 12:26:09.868893 28223 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:09.871029 28223 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:09.872077 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:09.857458 28236 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.183s	user 0.000s	sys 0.002s
W20260709 12:26:09.842718 28228 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.616s	user 0.006s	sys 0.000s
I20260709 12:26:09.853489 28238 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:09.825800 28063 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:09.886328 18167 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:09.909760 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.911612 27924 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.912303 27924 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.913988 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:09.914198 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:09.914839 27924 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:09.915481 28145 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:09.915616 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:09.915834 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:09.916580 18022 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:09.916596 18020 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:09.914937 18015 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:09.915627 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.925909 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3975) took 1235 ms. Trace:
I20260709 12:26:09.926656 18098 rpcz_store.cc:276] 0709 12:26:08.690806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.712814 (+ 22008us) service_pool.cc:224] Handling call
0709 12:26:08.713213 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:08.852701 (+139488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:08.852749 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:08.928816 (+ 76067us) log.cc:844] Serialized 26 byte log entry
0709 12:26:09.402891 (+474075us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:09.402949 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.403040 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:09.856296 (+453256us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:09.925351 (+ 69055us) raft_consensus.cc:1685] finished
0709 12:26:09.925392 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.925766 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7360,"raft.queue_time_us":33388,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":159017344,"thread_start_us":360793,"threads_started":2,"wal-append.queue_time_us":370788}
I20260709 12:26:09.929702 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.828747 28159 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:09.983223 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:09.984159 18164 raft_consensus.cc:2393] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:09.989609 18815 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:09.990872 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:09.991662 18017 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:09.854918 28241 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:09.992193 18015 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:09.993791 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:09.990285 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:09.984047 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:09.994602 18020 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:10.003710 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 210931us
I20260709 12:26:09.992087 18063 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:09.993963 28232 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.009s
I20260709 12:26:10.007844 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.003142 28240 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.010272 18164 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:10.009262 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.011050 18168 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 29 index: 23. (index mismatch)
I20260709 12:26:09.969369 27564 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:10.014611 18171 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:26:09.982579 27897 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:10.008375 18064 connection.cc:256] RPC call timeout handler was delayed by -3.017056268000033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1627s for a response
W20260709 12:26:10.027488 18064 connection.cc:256] RPC call timeout handler was delayed by -2.9423562679999122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20260709 12:26:10.027505 18015 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:10.027508 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.036233 18169 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.036404 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.037231 18168 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:10.037981 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.038754 18020 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:10.038400 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.040205 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.040444 18164 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
W20260709 12:26:10.043471 18064 connection.cc:256] RPC call timeout handler was delayed by -2.9261562680003408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260709 12:26:10.044076 18064 connection.cc:256] RPC call timeout handler was delayed by -2.8982562680002957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20260709 12:26:10.044412 18064 connection.cc:256] RPC call timeout handler was delayed by -2.8649562680002418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260709 12:26:10.044723 18064 connection.cc:256] RPC call timeout handler was delayed by -2.8469562680002127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
W20260709 12:26:10.045001 18064 connection.cc:256] RPC call timeout handler was delayed by -2.8253562680001778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
W20260709 12:26:10.045274 18064 connection.cc:256] RPC call timeout handler was delayed by -2.7641562680000789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
W20260709 12:26:10.045701 18064 connection.cc:256] RPC call timeout handler was delayed by -2.7461562680000497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
W20260709 12:26:10.046718 18064 connection.cc:256] RPC call timeout handler was delayed by -2.7073438419997728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
W20260709 12:26:10.047094 18064 connection.cc:256] RPC call timeout handler was delayed by -2.6642562679999173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20190000000000002s for a response
W20260709 12:26:10.047447 18064 connection.cc:256] RPC call timeout handler was delayed by -2.5858343499999137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260709 12:26:10.047811 18064 connection.cc:256] RPC call timeout handler was delayed by -2.5175442550003027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20260709 12:26:10.048170 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4986562680001043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
W20260709 12:26:10.058218 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4932562680000956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2209s for a response
W20260709 12:26:10.058681 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4823343500002011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
W20260709 12:26:10.059146 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4716562680000607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
W20260709 12:26:10.059504 18064 connection.cc:256] RPC call timeout handler was delayed by -2.4598343500001647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
W20260709 12:26:10.059866 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2862562680002156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2439s for a response
I20260709 12:26:10.061364 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:10.074141 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1575748420000309s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2209s for a response
I20260709 12:26:10.074812 18166 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:10.064960 18018 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 23: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:10.025661 28249 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:10.022723 28214 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.809s	user 0.025s	sys 0.013s
I20260709 12:26:10.074646 18016 raft_consensus.cc:1240] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:10.061194 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.018646 28145 raft_consensus.cc:2804] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:09.948585 28039 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.077854 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.106616 18017 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:10.078274 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.107904 18166 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:10.084506 18022 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:10.080261 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.028380 28240 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:10.074717 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8619410100000096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
I20260709 12:26:10.079973 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.042627 28234 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.109848 18172 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:26:10.138976 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7219562680002127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3066s for a response
W20260709 12:26:10.139643 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4906562680002935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33230000000000004s for a response
I20260709 12:26:10.085148 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:10.144610 18024 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:10.146073 18021 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:26:10.150174 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.151425 18168 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:10.097788 28214 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.026s	sys 0.013s
I20260709 12:26:10.073467 27741 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.172010 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.172863 18023 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:10.076867 28225 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:10.176545 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.177696 18165 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:10.186221 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.187091 18172 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.168843 28234 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.192415 28234 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.149166 28240 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.078392 28249 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.196223 28240 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.211678 18166 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:10.118690 27957 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:10.229168 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 144150us
I20260709 12:26:10.225291 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.226634 18172 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:10.181141 18017 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:10.231886 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:10.233655 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.231410 18166 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:10.235798 18172 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:10.143985 27564 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:10.252405 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.197474 28249 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.239426 18168 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:10.244256 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:10.254683 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.243920 27892 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:10.256222 18169 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:10.256776 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.257485 18172 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.244740 28172 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:10.245388 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:10.261307 28172 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:10.261629 18806 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:26:10.172425 28214 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:10.236521 27941 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.262578 28172 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.263074 27941 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.263489 28214 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.228727 28030 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.274554 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:10.274904 28212 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.275352 27892 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:10.275869 27892 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:10.149628 28145 raft_consensus.cc:697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:10.255097 18023 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:10.277390 28191 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:26:10.277173 28145 consensus_queue.cc:237] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.249987 27822 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:10.260011 18169 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 29 index: 23. Preceding OpId from leader: term: 30 index: 24. (index mismatch)
I20260709 12:26:10.256793 28253 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:10.237771 27938 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.266917 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.261010 27564 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.268989 28039 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.270476 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:10.271116 27957 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:10.272037 27941 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:10.274314 28030 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.298480 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4773) took 1148 ms. Trace:
I20260709 12:26:10.275436 28212 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:10.261529 28237 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.410s	user 0.007s	sys 0.000s
I20260709 12:26:10.249301 27975 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.255667 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.276820 27892 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.309558 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.310106 27892 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:10.310446 18093 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:10.236754 28209 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:10.311476 28209 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.281747 18806 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:10.281955 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:10.314337 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.284181 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.315769 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.316334 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.316541 18092 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:10.316942 18169 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:26:10.318065 28249 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:10.318539 28249 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.318950 28249 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.319337 18169 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:10.285498 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.320417 18173 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:26:10.321153 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.322474 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.291810 27938 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.323166 18023 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:26:10.324076 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.326256 27564 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.302378 28250 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:10.329416 28250 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:10.330425 28242 log.cc:927] Time spent T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.643s	user 0.003s	sys 0.003s
I20260709 12:26:10.290331 18172 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:10.324102 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.144838 28063 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 32.25, Last appended by leader: 25, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.295471 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:10.297338 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.343793 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48402.2us
I20260709 12:26:10.344791 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:10.279325 28191 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.301503 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.299002 18096 rpcz_store.cc:276] 0709 12:26:09.143846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.143984 (+   138us) service_pool.cc:224] Handling call
0709 12:26:09.185506 (+ 41522us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800037e98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:26:10.292112 (+1106606us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400291b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:10.292327 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527870976}
I20260709 12:26:10.282306 27822 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:10.354763 18098 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:10.355899 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:10.286535 28172 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:10.307153 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 3973) took 1617 ms. Trace:
I20260709 12:26:10.308380 18168 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.294481 28211 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.366367 28211 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.367085 18173 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: [27.14-28.15]
I20260709 12:26:10.368892 28211 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:10.292747 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.334771 18023 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:10.371140 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:10.372982 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.374035 28209 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.375150 27938 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:10.375809 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.766s (SENT)
I20260709 12:26:10.377977 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:10.378118 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:10.379217 18807 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:10.347666 28252 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.382941 28252 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.362967 18090 rpcz_store.cc:276] 0709 12:26:08.690059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.691015 (+   956us) service_pool.cc:224] Handling call
0709 12:26:08.712445 (+ 21430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:09.188824 (+476379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.306764 (+1117940us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b540035d9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:10.307062 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447205120}
I20260709 12:26:10.317560 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:10.342036 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75372.7us
I20260709 12:26:10.348632 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1955) took 1001 ms. Trace:
I20260709 12:26:10.342608 28250 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.357796 28191 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:10.391368 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:10.405782 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 137826us
I20260709 12:26:10.392757 28030 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.394469 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:10.412206 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67575.2us
I20260709 12:26:10.413120 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:10.406657 17989 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.661s (SENT)
I20260709 12:26:10.395776 18017 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.398885 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.414131 27957 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:10.414371 18090 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:10.414626 27957 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.413589 17987 leader_election.cc:400] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23.
I20260709 12:26:10.415091 27957 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.415323 17987 leader_election.cc:403] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:10.417326 28209 raft_consensus.cc:3060] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:26:10.420113 28030 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:10.420991 18095 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:10.422024 28211 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:10.422551 28211 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:10.423154 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:10.423600 28211 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.424108 28039 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:10.425143 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:10.425410 28211 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.426025 28039 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:10.426476 28039 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:10.426625 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:10.426909 28039 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.428510 18093 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:10.431214 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.432765 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.433379 18023 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:10.435642 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.435993 28255 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:10.436591 28255 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:10.436983 28255 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.437274 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.438712 28209 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 22. Current term is 23.
W20260709 12:26:10.439926 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.446889 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.447609 18173 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:10.447854 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.448616 28220 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.449805 18166 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:10.449992 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.451331 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.451853 28246 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.452212 18166 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:10.453240 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.440990 18168 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:10.441699 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.451351 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:10.440891 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.454272 18018 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:10.448781 18165 raft_consensus.cc:2393] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:10.455492 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.454838 18097 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 20.
I20260709 12:26:10.456832 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.444595 27939 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.457708 18165 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:26:10.458817 18166 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:10.460120 18166 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:10.462960 18018 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:10.421990 17989 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:10.472616 18017 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
W20260709 12:26:10.472772 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46234.1us
I20260709 12:26:10.483443 18173 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:26:10.460394 18168 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:10.390542 28212 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:10.407117 27975 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:26:10.488313 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 79168.6us
I20260709 12:26:10.488502 28243 raft_consensus.cc:2764] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:26:10.489410 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:10.489627 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.490018 28241 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.360395 28207 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.114s
I20260709 12:26:10.491137 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.491289 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:10.492379 28159 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:10.492568 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.492893 28159 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:10.493386 28159 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:10.494156 28064 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:10.494654 28064 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:10.495061 28064 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:10.498263 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.482442 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:10.396073 18812 rpcz_store.cc:276] 0709 12:26:09.347203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.368508 (+ 21305us) service_pool.cc:224] Handling call
0709 12:26:09.369020 (+   512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.382730 (+ 13710us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:09.382764 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:09.406804 (+ 24040us) log.cc:844] Serialized 26 byte log entry
0709 12:26:09.535512 (+128708us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:09.680379 (+144867us) log.cc:844] Serialized 14 byte log entry
0709 12:26:09.703396 (+ 23017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.703484 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:10.329586 (+626102us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:10.332459 (+  2873us) raft_consensus.cc:1685] finished
0709 12:26:10.348343 (+ 15884us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.348560 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56678,"prepare.queue_time_us":291583,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":36957,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":20420,"spinlock_wait_cycles":82288384,"thread_start_us":218840,"threads_started":2,"wal-append.queue_time_us":154450}
I20260709 12:26:10.489410 18095 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:10.485117 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.474424 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:10.465525 27938 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:10.498394 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:10.483942 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.498793 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.502662 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.487179 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.385742 28252 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.498306 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:10.560125 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.640605 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 142476us
I20260709 12:26:10.435730 27957 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:10.688467 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46772.8us
W20260709 12:26:10.706086 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.667865 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:10.648869 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.696274 18017 raft_consensus.cc:2468] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:10.679176 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.667795 18169 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:10.666606 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:10.657685 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 141252us
I20260709 12:26:10.751565 18016 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:10.657261 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.684988 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:10.677026 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.753326 18022 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:10.675846 18015 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:10.749626 18024 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:10.672497 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 265888us
I20260709 12:26:10.750733 18020 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:10.675199 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:10.646509 18023 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:10.763394 18017 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:10.748625 18018 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:26:10.757982 18097 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:10.770648 18024 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260709 12:26:10.776410 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.777333 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:10.778415 18806 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:10.779488 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.780483 18095 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:10.504243 27975 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:10.755304 28224 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.060s	user 0.015s	sys 0.005s
W20260709 12:26:10.757299 17989 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.737s (SENT)
I20260709 12:26:10.678619 27664 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.514843 27942 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:10.796864 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:10.797760 18021 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:26:10.842577 18812 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:10.785071 28110 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.843664 28110 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.752179 28202 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:10.846215 18020 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:10.772876 28213 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
W20260709 12:26:10.800969 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 116157us
I20260709 12:26:10.846823 28202 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:10.848320 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:10.850353 28220 raft_consensus.cc:2764] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:10.851106 28110 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:10.850113 27939 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
I20260709 12:26:10.852859 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:10.829522 18024 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:10.854485 27939 consensus_queue.cc:1243] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 28.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:26:10.856110 27939 raft_consensus.cc:2764] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:10.745460 27938 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.857537 27938 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.515360 28243 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:10.851248 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93972.3us
I20260709 12:26:10.860198 28243 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:10.861117 18019 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.008s	sys 0.033s
W20260709 12:26:10.863951 18019 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.008s	sys 0.035s
I20260709 12:26:10.783782 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:10.866801 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 83133.9us
I20260709 12:26:10.864696 28243 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:10.867470 27939 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:10.870450 27939 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.871134 27939 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:10.873402 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 375074us
I20260709 12:26:10.876875 28243 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 32.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:26:10.823362 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.887485 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.890444 18095 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:26:10.891541 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.892212 18095 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:10.893265 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.505087 28209 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.291s
W20260709 12:26:10.896780 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.863521 17989 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.842s (SENT)
I20260709 12:26:10.901633 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:10.902403 18095 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:26:10.876230 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:10.802990 28241 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:10.785974 28063 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:10.839007 28214 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.009s	sys 0.037s
W20260709 12:26:10.838560 28215 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.024s	sys 0.027s
W20260709 12:26:10.910395 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 457297us
I20260709 12:26:10.922964 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:10.904799 28145 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.011s	sys 0.011s
I20260709 12:26:10.943408 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:10.955616 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:10.763456 28253 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.154s
I20260709 12:26:10.774029 28252 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.140s
W20260709 12:26:10.929544 28215 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.025s	sys 0.027s
I20260709 12:26:10.791337 28234 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.138s
W20260709 12:26:10.929970 28214 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.009s	sys 0.037s
W20260709 12:26:10.948004 27892 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.002s
I20260709 12:26:10.886675 28224 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:10.963127 18165 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.009s	sys 0.022s
I20260709 12:26:10.917222 27664 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:11.036921 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 184154us
W20260709 12:26:11.031888 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 170243us
I20260709 12:26:11.069630 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:11.032639 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.1729e+06us
W20260709 12:26:11.071509 18780 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.180s (SENT) [suppressed 29 similar messages]
W20260709 12:26:11.033962 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 258128us
I20260709 12:26:11.072623 28238 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.073099 27664 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:11.073365 28238 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.077689 18091 raft_consensus.cc:2368] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
W20260709 12:26:10.961174 28145 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.011s	sys 0.011s
W20260709 12:26:11.013281 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 144326us
I20260709 12:26:11.090157 18016 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:26:10.962554 18092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.008s	sys 0.010s
I20260709 12:26:11.092867 18136 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:11.054929 28223 consensus_meta.cc:287] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.168s	user 0.013s	sys 0.003s
W20260709 12:26:11.068331 18165 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.011s	sys 0.023s
I20260709 12:26:10.974665 28202 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:11.105593 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:11.012123 27975 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:10.978372 28220 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.132s
I20260709 12:26:10.960711 28241 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:26:11.128706 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.132540 18092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.008s	sys 0.011s
W20260709 12:26:11.017786 28039 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.590s	user 0.012s	sys 0.001s
I20260709 12:26:11.133178 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:11.135215 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.110397 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.107933 18812 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:26:11.017896 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.018s	sys 0.012s
W20260709 12:26:11.138059 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 63399.5us
I20260709 12:26:11.165648 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:11.124965 28255 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.687s	user 0.006s	sys 0.009s
I20260709 12:26:10.964183 28080 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:11.180984 28039 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.144695 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:11.161516 18166 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:26:11.188067 18166 pending_rounds.cc:85] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 13
I20260709 12:26:11.188426 18166 pending_rounds.cc:107] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:26:11.188737 18166 raft_consensus.cc:2889] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:11.162796 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.019s	sys 0.013s
I20260709 12:26:11.108048 28224 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:11.133250 28234 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:11.108598 27664 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:11.107028 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica cab4a2f5d3ca48a2a803ee8ec3906e1d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 30 committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:11.197623 17908 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 90805.2us
W20260709 12:26:11.171224 28267 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:26:11.172654 27938 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.173978 28255 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.173617 28207 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:11.200675 28030 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:11.182677 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:11.186998 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:11.195686 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62585.9us
W20260709 12:26:11.233570 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 64085.3us
I20260709 12:26:11.235065 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.236340 28207 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:11.237061 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:11.237434 28207 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:11.238066 28207 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.238718 28207 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:11.234092 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47231.5us
I20260709 12:26:11.134308 28241 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:11.203205 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.162009 27924 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:11.223671 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.264981 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.265543 28241 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 32.25, Last appended by leader: 25, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.238755 28274 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:11.266929 28064 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.771s	user 0.014s	sys 0.000s
I20260709 12:26:11.267320 28241 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:11.267429 28064 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.269937 28275 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:26:11.233412 28030 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.271063 28030 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:11.184458 28277 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:11.193596 27564 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:11.227206 18015 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:11.123060 28261 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:26:11.160441 18811 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.200349 28220 consensus_queue.cc:1243] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer cab4a2f5d3ca48a2a803ee8ec3906e1d log is divergent from this leader: its last log entry 29.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:26:11.234436 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.202129 27664 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.161705 27941 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:26:11.093061 28063 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:11.207984 28214 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.211126 28223 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:26:11.262061 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.270328 18094 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 22: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:11.224375 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:11.211360 28234 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:11.286128 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.287253 27957 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:11.287565 28159 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.793s	user 0.013s	sys 0.000s
I20260709 12:26:11.317354 18807 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:11.317898 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:11.318859 18172 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:26:11.310999 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:11.321908 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.287856 28276 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:11.299345 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:11.302616 18806 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:11.305119 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.305240 18095 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:26:11.233644 28255 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:11.310410 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.313783 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:11.314620 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:11.322383 18090 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:11.215207 27822 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.325363 18173 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:11.279563 28277 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.330179 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:11.333907 18090 raft_consensus.cc:2410] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:26:11.284026 27664 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:11.338551 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:11.341351 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:11.303853 28223 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:11.347138 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.351994 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:11.281041 27924 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.356287 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.359691 18815 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:11.360282 28238 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:11.360873 18806 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:11.361707 18813 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:26:11.361896 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.362351 28209 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:11.362768 28209 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:11.363312 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.363721 27938 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:11.364840 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:11.365103 28265 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:26:11.365715 18809 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:11.366986 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.368984 18092 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:11.362668 18810 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:11.369637 18096 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:11.370703 28211 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.908s	user 0.012s	sys 0.003s
I20260709 12:26:11.339797 28274 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:26:11.372177 28064 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:11.327946 28276 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:11.374977 28276 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:11.374686 28234 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:11.379387 28268 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:26:11.379763 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4624) took 1142 ms. Trace:
I20260709 12:26:11.380219 18015 rpcz_store.cc:276] 0709 12:26:10.237358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.798648 (+561290us) service_pool.cc:224] Handling call
0709 12:26:10.799017 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.281132 (+482115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:11.379444 (+ 98312us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54001d0954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:11.379676 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984182016}
W20260709 12:26:11.383790 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 240676us
I20260709 12:26:11.384807 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:11.386629 18815 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:11.388381 18806 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:11.369697 18091 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4783) took 1636 ms (client timeout 1801 ms). Trace:
W20260709 12:26:11.389854 18091 rpcz_store.cc:269] 0709 12:26:09.733567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.745146 (+ 11579us) service_pool.cc:224] Handling call
0709 12:26:11.369483 (+1624337us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540011d854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:11.369630 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3058729344}
I20260709 12:26:11.300941 27939 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.404464 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.382033 28039 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:11.389338 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:11.408983 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.343575 28277 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:11.389961 28253 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:11.324481 28275 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:26:11.430732 18166 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:26:11.443281 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:11.428987 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:11.327951 28159 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.307116 28214 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.441035 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.452255 18810 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:11.452402 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:11.353518 28223 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:11.411657 28172 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.015s	user 0.009s	sys 0.005s
W20260709 12:26:11.447625 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 571731us
W20260709 12:26:11.457176 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72478.7us
I20260709 12:26:11.464919 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:11.458978 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:11.469980 18815 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:11.441533 28039 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:11.469524 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.438226 28253 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:11.490516 18093 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:11.487448 18099 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:26:11.489620 18806 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 32 index: 25. Preceding OpId from leader: term: 33 index: 26. (index mismatch)
I20260709 12:26:11.485543 18165 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:11.516515 18165 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:11.517828 18811 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:11.463497 27938 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:11.461097 28280 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:11.363878 28209 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.534487 27822 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:11.540159 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 445621us
I20260709 12:26:11.485450 28039 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.493983 18136 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:11.521500 28250 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.044s	user 0.014s	sys 0.003s
I20260709 12:26:11.546787 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.564147 18815 raft_consensus.cc:2368] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
W20260709 12:26:11.557265 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67084.4us
I20260709 12:26:11.554476 18138 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.554241 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:11.567361 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 103937us
I20260709 12:26:11.558288 18165 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.519158 28211 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:11.484794 27942 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:26:11.572096 18136 leader_election.cc:400] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:26:11.555706 27941 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:11.546384 27938 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:11.578189 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 108823us
W20260709 12:26:11.587338 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.546384 28214 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:11.563447 28039 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.589907 18136 leader_election.cc:403] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:11.573709 18138 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.589057 18165 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:11.598940 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.587826 28249 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.268s	user 0.015s	sys 0.004s
I20260709 12:26:11.566748 28279 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.034s
I20260709 12:26:11.600364 28249 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.603796 28249 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:11.584836 28211 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.610374 28211 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:11.612238 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.576987 28293 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:11.605677 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:11.617861 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.553205 28080 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:26:11.563004 28172 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:11.551064 28280 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:11.635164 28280 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:11.567010 28246 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:11.637200 28246 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.638118 18064 leader_election.cc:400] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26.
I20260709 12:26:11.638845 28159 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:11.639374 18137 leader_election.cc:336] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.064s (SENT)
I20260709 12:26:11.580572 28224 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:11.619699 28281 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.624119 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.654577 18812 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:26:11.641402 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 67870.4us
I20260709 12:26:11.594383 27938 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:11.593287 27941 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.637691 28246 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:11.660562 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:11.645088 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 78910.5us
I20260709 12:26:11.569154 28250 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:11.656811 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 164325us
W20260709 12:26:11.590430 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 99155.2us
I20260709 12:26:11.663450 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.658367 27941 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:11.666294 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:11.669055 28291 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:26:11.679141 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4791) took 1327 ms. Trace:
I20260709 12:26:11.677697 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 4000) took 1205 ms. Trace:
I20260709 12:26:11.684113 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.686806 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.682535 18099 rpcz_store.cc:276] 0709 12:26:10.352154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.352358 (+   204us) service_pool.cc:224] Handling call
0709 12:26:10.391723 (+ 39365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.521480 (+1129757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:11.678762 (+157282us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002fbcd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:11.679036 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2518649216}
I20260709 12:26:11.692509 27924 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:11.659188 28281 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.662879 28294 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.694155 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:11.684019 18093 rpcz_store.cc:276] 0709 12:26:10.472698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.478047 (+  5349us) service_pool.cc:224] Handling call
0709 12:26:11.677400 (+1199353us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54002cae54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:11.677599 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1683869440}
W20260709 12:26:11.650112 28191 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.238s	user 0.015s	sys 0.012s
I20260709 12:26:11.706626 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:11.708429 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46324.9us
I20260709 12:26:11.709535 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:11.710176 18171 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:11.659495 28224 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:11.589501 28253 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.680433 28255 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:11.738055 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.742031 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.731331 28215 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:11.744673 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:11.745461 27942 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:11.746106 27942 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.746534 27942 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.746698 18064 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:11.748587 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:11.756796 28276 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.757934 27924 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:11.758688 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.777211 18165 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:11.778189 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:11.704587 27975 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.726497 28243 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.725960 28220 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:11.759795 28224 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.763980 28250 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:11.764217 28063 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:11.729701 28279 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.822748 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:11.769027 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.180s (SENT)
I20260709 12:26:11.785104 28064 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:26:11.774255 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.787283 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 82001.1us
I20260709 12:26:11.698426 28241 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:26:11.791750 18810 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:11.834574 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.835505 18809 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:11.836113 28030 consensus_meta.cc:287] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.000s
I20260709 12:26:11.836817 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.820062 28255 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:26:11.705853 28191 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.800944 28243 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.839059 28223 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.840382 28191 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:11.820865 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.793764 28232 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260709 12:26:11.841307 28243 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:11.744484 28296 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:11.823784 18812 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:26:11.788157 28249 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:11.856923 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.788573 28275 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:26:11.828887 28272 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:26:11.828593 28279 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.833458 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:11.836678 28030 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.841415 28223 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:11.824512 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55539.2us
W20260709 12:26:11.842307 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93844.1us
I20260709 12:26:11.893106 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.857722 18812 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:11.894745 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111911us
I20260709 12:26:11.884166 18807 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:11.731920 27939 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:11.896828 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 192493us
I20260709 12:26:11.783139 27942 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.826368 28064 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:11.919754 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.367s (SENT)
I20260709 12:26:11.926337 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:11.921619 17988 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:11.806501 27975 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:11.864423 28281 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:11.885349 28214 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:11.954087 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:11.954864 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 212705us
I20260709 12:26:11.857764 28241 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.090s
I20260709 12:26:11.939049 28064 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:11.988060 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:11.991530 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71827us
I20260709 12:26:11.991819 18807 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:11.996174 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:11.996464 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75025.8us
W20260709 12:26:11.997658 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 151967us
I20260709 12:26:11.886157 27941 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.999751 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:11.949522 28290 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:12.000604 18813 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:11.886186 28255 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.081s
I20260709 12:26:11.943782 28272 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:11.918871 28249 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:12.008678 18808 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:26:12.026353 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.020771 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:12.031234 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86874.8us
I20260709 12:26:11.994796 28250 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.999943 28298 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.030061 18807 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.042586 18169 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:11.959064 27939 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 12:26:12.046461 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:12.024811 28211 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:11.988956 27938 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:12.032425 28277 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.581s	user 0.004s	sys 0.010s
I20260709 12:26:12.031657 28275 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:12.027365 27822 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 12:26:12.102443 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:12.086748 28110 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:12.094553 28272 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.060019 28298 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:12.110553 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:12.182677 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.103936 28249 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:12.119263 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.008471 28315 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.200923 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:12.201650 18810 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.131088 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:12.139106 28294 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
W20260709 12:26:12.139956 28309 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:26:12.141005 27938 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:12.141275 28303 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:26:12.141364 28030 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.144315 28276 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:12.146100 27975 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:12.148103 28308 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:26:12.155085 27942 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:12.156279 28172 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.523s	user 0.011s	sys 0.002s
I20260709 12:26:12.186784 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:12.253286 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 223954us
W20260709 12:26:12.188675 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.253870 18093 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:26:12.190598 18097 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:12.192694 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:12.194836 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.212890 18807 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:12.214629 27941 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:12.184003 28039 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.555s	user 0.012s	sys 0.002s
I20260709 12:26:12.217617 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:12.184966 27664 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.850s	user 0.008s	sys 0.008s
I20260709 12:26:12.221019 18813 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 21.
W20260709 12:26:12.284286 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.225914 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.063169 28250 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.258596 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:12.285202 18096 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.232877 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:12.285593 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:12.263479 28172 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:12.283253 28039 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.292641 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:12.182097 28281 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:12.294504 28281 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:12.240793 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.295814 28281 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.297449 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:12.241881 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 301994us
I20260709 12:26:12.243491 18173 raft_consensus.cc:606] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:26:12.301163 18173 raft_consensus.cc:614] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:12.256143 18093 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:12.130786 28280 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.258895 18810 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
W20260709 12:26:12.262090 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1522) took 2248 ms (client timeout 2478 ms). Trace:
W20260709 12:26:12.302232 18171 rpcz_store.cc:269] 0709 12:26:10.013488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.166577 (+153089us) service_pool.cc:224] Handling call
0709 12:26:10.482205 (+315628us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b5800032d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b67d0a0 00007f6c1b67c984 00007f6c14bf76d4 00007f6c14bf7662
0709 12:26:12.261795 (+1779590us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003025d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:12.262012 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2791030400}
W20260709 12:26:12.302821 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44345us
I20260709 12:26:12.254879 18097 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:12.170964 28277 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.263963 18095 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:12.231772 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 185449us
W20260709 12:26:12.305289 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.331090 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:12.301891 18093 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:12.257287 28275 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:12.352167 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.337694 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:12.366074 18168 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:12.366888 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.350656 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:12.373342 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:12.374385 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.228274 28315 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:12.384217 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45627.2us
I20260709 12:26:12.330809 28293 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:26:12.365433 28207 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.108s	user 0.020s	sys 0.000s
I20260709 12:26:12.345538 18097 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.335418 28290 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:12.337062 27941 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:12.333982 27938 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:12.307950 27664 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.376371 27564 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:12.426086 18097 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:12.426518 18167 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3861) took 1978 ms (client timeout 1766 ms). Trace:
W20260709 12:26:12.427055 18167 rpcz_store.cc:269] 0709 12:26:10.448245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.451014 (+  2769us) service_pool.cc:224] Handling call
0709 12:26:12.424899 (+1973885us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:12.426420 (+  1521us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443570048}
W20260709 12:26:12.428584 28323 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:26:12.429728 28313 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:12.432286 27822 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:12.432804 27822 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:12.434101 28030 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.456534 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:12.392442 28279 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:12.396870 28296 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:12.459820 18096 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:12.461344 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:12.462070 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72315.3us
I20260709 12:26:12.471851 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:12.394066 28290 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:12.473261 28275 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.474161 28290 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:12.391525 28207 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.468263 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:12.478091 28314 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:26:12.480329 28207 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.455549 27941 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:12.482465 27941 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.483007 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4799) took 1206 ms. Trace:
I20260709 12:26:12.483443 18091 rpcz_store.cc:276] 0709 12:26:11.276865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.391139 (+114274us) service_pool.cc:224] Handling call
0709 12:26:11.391483 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.843287 (+451804us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:11.843333 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:11.843390 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:11.843427 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.482734 (+639307us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400411ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:12.482799 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:12.482940 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36082,"raft.queue_time_us":3166,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":14181,"spinlock_wait_cycles":1037934848}
W20260709 12:26:12.485666 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.486634 28298 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:12.492242 28280 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.492050 28290 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.495746 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:12.496948 18173 raft_consensus.cc:606] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Received request to transfer leadership to TS 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:12.498199 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 105875us
I20260709 12:26:12.499259 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:12.499742 18812 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 34. Current term is 35. Ops: []
I20260709 12:26:12.500397 28279 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:12.500891 28279 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:12.501315 28279 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:12.504019 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.504412 28315 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:12.505074 28211 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:12.517755 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56519.5us
I20260709 12:26:12.519490 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:12.458413 28039 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.525388 18097 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:26:12.529848 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:12.530289 18097 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 17
I20260709 12:26:12.531036 18097 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:26:12.531440 18097 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:12.538491 18173 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a29e00b535474906948e66659110db3b"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:12.539404 18173 raft_consensus.cc:606] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:26:12.539836 18173 raft_consensus.cc:614] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:12.455302 28030 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.528267 28207 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.523916 28322 raft_consensus.cc:2764] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:12.543852 28207 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:12.550838 28327 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:26:12.555315 28290 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:12.465241 27938 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:12.577661 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 105900us
I20260709 12:26:12.464787 27822 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.545296 27949 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:12.544003 28326 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:12.611810 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:12.612922 18166 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0"
dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
mode: GRACEFUL
new_leader_uuid: "768723648a9444fda53659e6a8cc3091"
 from {username='slave'} at 127.0.0.1:45622
I20260709 12:26:12.614158 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:12.616910 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.541433 28172 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.070s
I20260709 12:26:12.617619 18166 raft_consensus.cc:606] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Received request to transfer leadership to TS 768723648a9444fda53659e6a8cc3091
I20260709 12:26:12.628012 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.628373 18166 raft_consensus.cc:614] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:12.633046 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102522us
I20260709 12:26:12.607887 27564 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:12.638164 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.615052 28321 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:12.641547 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:12.666183 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 145292us
I20260709 12:26:12.667449 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:12.670557 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:26:12.642966 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:12.646929 28277 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:12.648417 27938 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.649175 27564 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.651163 18064 leader_election.cc:400] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24.
I20260709 12:26:12.683667 18064 leader_election.cc:403] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:12.651463 28063 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:12.621690 28293 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:12.651620 28039 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:26:12.652895 28246 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.942s	user 0.013s	sys 0.001s
I20260709 12:26:12.653388 28209 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:12.616011 28326 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:12.628098 28159 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:12.658174 28343 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:26:12.664740 28064 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.665s	user 0.012s	sys 0.002s
I20260709 12:26:12.674125 18812 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:12.689927 28344 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:26:12.678658 28232 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:12.687162 18778 leader_election.cc:336] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.848s (SENT)
I20260709 12:26:12.669536 28250 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:12.709033 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:12.685590 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.680727 28321 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:12.713696 18090 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:12.690140 28063 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:12.750921 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99954.1us
I20260709 12:26:12.751972 18096 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:12.691349 28207 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.701324 28293 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:12.765045 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77946.4us
I20260709 12:26:12.712607 28159 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:12.791797 28159 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:12.792024 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:12.793223 18815 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:12.766635 18812 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:12.766875 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:12.786839 18095 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:12.796803 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:12.759373 27664 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:12.805526 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:12.807411 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.806656 18096 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:12.813436 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:12.713374 28326 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:12.819149 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:12.814756 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:12.820614 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 151864us
I20260709 12:26:12.750707 28321 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.805022 28030 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:12.827767 18023 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 25. Ops: []
I20260709 12:26:12.806722 27938 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:12.815936 28249 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:12.710496 28246 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.760857 28232 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.833550 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:12.726166 28064 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.842628 18095 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:12.732071 28279 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.845932 18097 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:12.847431 28279 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:12.851890 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.864116 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:12.865272 18095 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:12.870906 18172 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 33: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:12.832221 28351 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:12.830258 28348 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:12.842366 28321 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:26:12.877584 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:12.880610 18780 connection.cc:256] RPC call timeout handler was delayed by -1.5383848579999722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
W20260709 12:26:12.881072 18780 connection.cc:256] RPC call timeout handler was delayed by -1.3502848580001228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
I20260709 12:26:12.877506 18097 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:12.866917 27938 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:12.876092 28279 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:12.895401 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80710.1us
I20260709 12:26:12.897367 27975 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.904239 27975 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.905408 28253 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:12.881392 18780 connection.cc:256] RPC call timeout handler was delayed by -1.0541848580000988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24780000000000002s for a response
I20260709 12:26:12.911042 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:12.911360 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:12.902494 28039 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.006s
I20260709 12:26:12.883401 28348 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:12.833719 28063 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.920590 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:12.934525 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:12.923902 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:12.953636 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.965585 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:12.907096 27975 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:12.935580 28347 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:26:12.975488 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:12.936147 27664 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:12.937188 28215 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24.
I20260709 12:26:12.918705 28279 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:12.903736 28232 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:12.984490 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:12.908839 28250 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.067s
I20260709 12:26:12.935967 28280 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.065s
I20260709 12:26:12.992807 28312 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:26:12.993219 28352 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:12.993484 28312 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:12.993992 28352 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:12.994607 28312 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:12.996423 28312 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:12.961217 28348 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:12.977717 28249 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:12.971449 28039 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:12.988399 27975 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.002817 28249 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.003468 28279 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.011119 27949 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:13.015275 18020 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:13.026301 28280 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:13.027135 28280 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:13.028383 28280 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:13.032192 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:13.032785 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.035166 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:12.988199 28063 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:13.040462 18164 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:26:13.040828 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.041226 18164 pending_rounds.cc:85] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 16
I20260709 12:26:12.996433 28352 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.041682 18164 pending_rounds.cc:107] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:26:13.041808 18166 raft_consensus.cc:2393] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.042121 18164 raft_consensus.cc:2889] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:13.044528 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.048810 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:13.050689 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.051970 18167 raft_consensus.cc:2368] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27.
I20260709 12:26:13.054924 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.054179 28246 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.056582 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.056815 28064 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.057278 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.058156 18098 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.029657 28326 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.060289 28348 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.061769 28326 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.035122 27938 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.062781 18812 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:13.064038 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.064743 18167 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:26:13.066298 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.038360 28290 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:13.056095 18173 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:12.930369 28351 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:13.071244 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.076673 18166 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:26:13.076790 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.077951 18095 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:26:13.075922 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.082304 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.104903 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.106056 18167 raft_consensus.cc:2368] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25.
I20260709 12:26:13.107275 18169 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:13.055222 28211 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:13.123577 18016 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 28. Ops: []
I20260709 12:26:13.114401 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.116931 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.111729 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.126156 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:13.074334 28296 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.004s
I20260709 12:26:13.127025 18099 raft_consensus.cc:2368] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26.
I20260709 12:26:13.125618 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.125640 18095 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:26:13.132423 18166 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:13.133078 18090 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:13.063467 28348 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:13.134280 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.136197 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.137933 18094 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.138079 28360 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.131031 27975 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.138628 28360 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.139082 27975 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:13.140177 28330 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.652s	user 0.000s	sys 0.010s
I20260709 12:26:13.141279 27975 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.141498 28039 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.141808 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3883) took 1105 ms. Trace:
I20260709 12:26:13.142050 28039 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.144801 28039 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.145077 18167 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:13.141316 28360 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.148948 27975 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.150827 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.151517 18173 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:13.153942 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.155525 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.156323 18167 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:13.157331 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.158277 18167 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.149511 27975 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.132148 28290 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:13.161360 28290 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.163272 28293 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.135179 28296 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:13.166779 18022 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:13.168515 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.168377 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.168386 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.170154 18165 rpcz_store.cc:276] 0709 12:26:12.036136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.038325 (+  2189us) service_pool.cc:224] Handling call
0709 12:26:12.044315 (+  5990us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:12.106107 (+ 61792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.106161 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:12.292252 (+186091us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b2c001268f8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:26:12.292957 (+   705us) log.cc:844] Serialized 52 byte log entry
0709 12:26:12.390331 (+ 97374us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:12.390404 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.390496 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.139434 (+748938us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:13.141524 (+  2090us) raft_consensus.cc:1685] finished
0709 12:26:13.141555 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.141710 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61182,"raft.queue_time_us":61905,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"spinlock_wait_cycles":106998528,"thread_start_us":96624,"threads_started":1,"wal-append.queue_time_us":194050}
I20260709 12:26:13.169773 18169 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:13.181097 28159 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.182662 18096 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:13.183969 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.184710 28159 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.185045 18167 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.191058 18090 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:13.191136 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.193397 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.193495 18096 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:13.195360 28351 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.198568 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.199875 18098 raft_consensus.cc:2410] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:26:13.202466 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.209300 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.216197 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.217768 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.218586 18024 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:13.219823 18173 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:13.213372 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:13.226052 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.227883 18017 raft_consensus.cc:1275] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:26:13.229466 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.230362 18017 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:13.232862 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.222765 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:13.235136 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:13.261615 18017 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:13.260849 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.260912 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.279503 18019 raft_consensus.cc:2368] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:13.279793 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.223146 28320 raft_consensus.cc:2764] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:26:13.282043 18018 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:13.279435 18015 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:13.285079 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:13.285809 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.823s (SENT)
I20260709 12:26:13.205283 28346 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:13.221648 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 310703us
I20260709 12:26:13.262007 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:13.288419 18165 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:13.295083 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:13.282135 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.304212 18017 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:13.304486 18173 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:13.307786 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.308166 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.308667 18015 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.308738 18019 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:13.232460 28296 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:13.223177 28360 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.313279 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.260762 28232 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.316457 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:13.313458 18166 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 35.
I20260709 12:26:13.275629 28209 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:13.276114 27938 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:13.326620 27975 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:13.315464 28340 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:13.331431 18015 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:13.314950 28296 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.332828 28296 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:13.305908 27949 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:13.338443 27949 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.339125 27949 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:13.342346 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.343564 18173 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.344729 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.345749 28358 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:13.346841 28209 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.347371 28209 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:13.305224 28346 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.350514 28064 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:13.350795 28246 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.351231 28064 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.346052 28360 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.353490 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 1973) took 1460 ms. Trace:
I20260709 12:26:13.337996 28232 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.353948 18809 rpcz_store.cc:276] 0709 12:26:11.893336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.894380 (+  1044us) service_pool.cc:224] Handling call
0709 12:26:11.943349 (+ 48969us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:12.023558 (+ 80209us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:12.131153 (+107595us) log.cc:844] Serialized 14 byte log entry
0709 12:26:13.025637 (+894484us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b1800499a80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:26:13.136973 (+111336us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:13.180681 (+ 43708us) log.cc:844] Serialized 26 byte log entry
0709 12:26:13.210992 (+ 30311us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:13.211041 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.211286 (+   245us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.353132 (+141846us) raft_consensus.cc:1685] finished
0709 12:26:13.353255 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.353409 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9619,"prepare.queue_time_us":74186,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":5406,"raft.queue_time_us":103597,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":18430,"spinlock_wait_cycles":58386304,"thread_start_us":174353,"threads_started":2,"wal-append.queue_time_us":973902}
I20260709 12:26:13.356514 18165 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:13.351325 28246 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.361282 18778 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:13.362345 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.364256 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:13.371745 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55418.5us
I20260709 12:26:13.376488 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:13.378255 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 341841us
I20260709 12:26:13.365759 27975 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:13.381933 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.363765 28340 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:13.383203 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.384018 18167 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 32 index: 25. Preceding OpId from leader: term: 33 index: 26. (index mismatch)
I20260709 12:26:13.373176 28343 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:13.390448 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:13.395319 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9787750390000838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 12:26:13.396104 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9427750390000256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
I20260709 12:26:13.396987 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.398339 18166 raft_consensus.cc:2368] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
W20260709 12:26:13.380434 28275 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.016s	sys 0.004s
I20260709 12:26:13.372735 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:13.400389 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:13.411391 18809 raft_consensus.cc:2368] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:26:13.411381 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:13.422431 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:13.401453 18064 connection.cc:256] RPC call timeout handler was delayed by -1.9346750390000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
I20260709 12:26:13.405866 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:13.423906 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:13.424423 18808 raft_consensus.cc:2410] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 34 because replica has last-logged OpId of term: 33 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 25.
I20260709 12:26:13.410491 18813 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [28.20-28.20]
W20260709 12:26:13.426192 18064 connection.cc:256] RPC call timeout handler was delayed by -1.8239750390002882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
I20260709 12:26:13.427794 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:13.429443 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7996750390002489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1748s for a response
W20260709 12:26:13.486442 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7960750390002431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
I20260709 12:26:13.456247 18809 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:26:13.401974 28358 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:13.400738 28159 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:13.470909 18019 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:13.401017 28063 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:13.418643 28275 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s	user 0.017s	sys 0.005s
I20260709 12:26:13.488178 18137 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:13.490334 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 106466us
I20260709 12:26:13.530746 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:13.531474 28329 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.067s	user 0.000s	sys 0.002s
W20260709 12:26:13.532881 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.940s (SENT)
I20260709 12:26:13.533398 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.496033 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:13.500998 18167 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:13.407200 28320 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
W20260709 12:26:13.550576 28363 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:26:13.533144 28063 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.552713 28376 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:13.498636 28357 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:26:13.506235 18173 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:13.505797 28321 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.618s	user 0.014s	sys 0.005s
I20260709 12:26:13.554962 28321 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.557485 28321 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.513267 18023 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 35.
I20260709 12:26:13.429782 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:13.422134 28211 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:13.561504 28211 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:13.528120 28159 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.562690 28211 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.450575 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3885) took 1413 ms. Trace:
W20260709 12:26:13.519527 18064 connection.cc:256] RPC call timeout handler was delayed by -1.78797503900023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
W20260709 12:26:13.564608 18064 connection.cc:256] RPC call timeout handler was delayed by -1.7789750390002155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
W20260709 12:26:13.565254 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6565750390000176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260709 12:26:13.565908 18064 connection.cc:256] RPC call timeout handler was delayed by -1.6214750389999608s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260709 12:26:13.506598 28346 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:13.546164 28272 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.071s	user 0.021s	sys 0.012s
I20260709 12:26:13.546597 28359 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.548919 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:13.566269 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4387750390001202s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
I20260709 12:26:13.567227 18167 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:13.567413 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3946750390000489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
I20260709 12:26:13.521466 28358 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:13.567960 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3604750389999936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
I20260709 12:26:13.568346 28358 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:13.568387 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3262750389999383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22740000000000002s for a response
W20260709 12:26:13.568961 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2173750390002169s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23950000000000002s for a response
I20260709 12:26:13.500499 28232 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:13.550367 28320 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:26:13.567406 28272 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.092s	user 0.021s	sys 0.013s
I20260709 12:26:13.569804 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.387109 27975 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.570133 28320 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:13.564548 18170 rpcz_store.cc:276] 0709 12:26:12.036878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.037004 (+   126us) service_pool.cc:224] Handling call
0709 12:26:12.037384 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:12.366829 (+329445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.366882 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:12.390614 (+ 23732us) log.cc:844] Serialized 26 byte log entry
0709 12:26:12.517708 (+127094us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:12.517760 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.450067 (+932307us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b5400202bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:13.450135 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.450191 (+    56us) raft_consensus.cc:1685] finished
0709 12:26:13.450207 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.450483 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30210,"raft.queue_time_us":18851,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":7843,"spinlock_wait_cycles":713298816,"thread_start_us":89274,"threads_started":1,"wal-append.queue_time_us":131992}
I20260709 12:26:13.533500 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4811) took 1341 ms. Trace:
I20260709 12:26:13.575361 18024 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:13.519491 28360 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.575934 18092 rpcz_store.cc:276] 0709 12:26:12.191736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.191897 (+   161us) service_pool.cc:224] Handling call
0709 12:26:12.200837 (+  8940us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:12.201283 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.201315 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:12.214386 (+ 13071us) log.cc:844] Serialized 26 byte log entry
0709 12:26:12.293838 (+ 79452us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:12.293896 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.293986 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.530765 (+1236779us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:13.533162 (+  2397us) raft_consensus.cc:1685] finished
0709 12:26:13.533185 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.533428 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":682,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":17949,"spinlock_wait_cycles":32946944,"thread_start_us":70800,"threads_started":1,"wal-append.queue_time_us":154435}
I20260709 12:26:13.588367 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:13.590195 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.609602 27975 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.566366 28064 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.578439 28272 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:13.579424 28232 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:13.637888 18019 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:13.639747 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:13.642603 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3870) took 2330 ms (client timeout 1833 ms). Trace:
W20260709 12:26:13.643102 18168 rpcz_store.cc:269] 0709 12:26:11.312614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.312779 (+   165us) service_pool.cc:224] Handling call
0709 12:26:13.642372 (+2329593us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:13.642518 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2353921920}
I20260709 12:26:13.630053 27938 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:13.631457 28320 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.615527 28367 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.646662 28250 raft_consensus.cc:993] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: : Instructing follower 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:26:13.634724 28159 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:13.516109 28340 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.647337 28250 raft_consensus.cc:1081] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Signalling peer 910f8c8d787e4abbb092413c1f90ebf3 to start an election
I20260709 12:26:13.650552 18093 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.651917 18093 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:13.655388 28322 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:13.656194 28322 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.656235 28209 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.660061 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 4021) took 1022 ms. Trace:
I20260709 12:26:13.660529 28209 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.660862 18097 rpcz_store.cc:276] 0709 12:26:12.637616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.637783 (+   167us) service_pool.cc:224] Handling call
0709 12:26:13.659793 (+1022010us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:13.659973 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475753216}
I20260709 12:26:13.665413 18094 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 25. Ops: []
I20260709 12:26:13.668176 28293 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.660221 18167 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
W20260709 12:26:13.669607 18172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3884) took 1633 ms (client timeout 1776 ms). Trace:
W20260709 12:26:13.670257 18172 rpcz_store.cc:269] 0709 12:26:12.036520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.038546 (+  2026us) service_pool.cc:224] Handling call
0709 12:26:13.669380 (+1630834us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b5400050fd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:13.669540 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1608475392}
I20260709 12:26:13.676697 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:13.678021 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:13.679025 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.680401 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.681643 18015 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:13.681795 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.682973 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.683872 18015 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:13.684654 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.686124 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.651674 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:13.693969 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 103877us
W20260709 12:26:13.704324 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 291999us
I20260709 12:26:13.707015 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:13.709227 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.709285 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.710284 18024 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
W20260709 12:26:13.716526 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 65082.8us
W20260709 12:26:13.728299 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.738610 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.748028 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.749564 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:13.754721 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.704562 28359 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:13.617401 28281 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.646718 27938 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.757404 28279 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:26:13.758142 27938 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.759279 18018 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:13.760842 27938 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.737424 28209 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.762280 28209 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.762745 28377 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:13.771493 28377 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:13.775665 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.772114 28377 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.741951 28367 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:13.785045 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:13.738339 28241 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:13.823405 28241 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.795056 28305 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:13.826408 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.827201 28281 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:13.797917 28272 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:13.801052 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.827936 28281 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:13.828022 28272 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:13.828678 28272 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:13.829098 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:13.801733 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.795043 28355 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:13.783404 28290 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:13.802846 28367 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:13.802976 28326 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:13.780853 28320 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:26:13.806679 28368 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.298s	user 0.003s	sys 0.000s
I20260709 12:26:13.807323 28322 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.808439 28277 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:13.811208 28209 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:13.784713 28279 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:26:13.841732 28355 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:13.834599 28367 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:13.847337 28279 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:13.847819 28279 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:13.847858 28355 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.848863 28279 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.818199 28246 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:13.829129 28281 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.818876 28359 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.850550 28241 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:13.855801 28358 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:13.860061 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1484) took 1033 ms. Trace:
I20260709 12:26:13.860615 18020 rpcz_store.cc:276] 0709 12:26:12.826754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.826944 (+   190us) service_pool.cc:224] Handling call
0709 12:26:12.828402 (+  1458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.015796 (+187394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.859775 (+843979us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54000ffd54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:13.859985 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":429174016}
I20260709 12:26:13.862124 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.863533 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.865636 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3888) took 1529 ms. Trace:
I20260709 12:26:13.865420 28358 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:13.866196 18171 rpcz_store.cc:276] 0709 12:26:12.336592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.345943 (+  9351us) service_pool.cc:224] Handling call
0709 12:26:12.347907 (+  1964us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:12.390819 (+ 42912us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.390856 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:12.391359 (+   503us) log.cc:844] Serialized 26 byte log entry
0709 12:26:12.621377 (+230018us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:12.621445 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.865066 (+1243621us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b540029dbd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:13.865146 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.865221 (+    75us) raft_consensus.cc:1685] finished
0709 12:26:13.865240 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.865555 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43250,"raft.queue_time_us":5562,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":7488,"spinlock_wait_cycles":87162240,"thread_start_us":229604,"threads_started":1,"wal-append.queue_time_us":187715,"wal-append.run_cpu_time_us":1821,"wal-append.run_wall_time_us":1003208}
I20260709 12:26:13.868758 18810 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:13.871178 18094 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:13.871881 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.871996 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.872952 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.874181 18095 raft_consensus.cc:2410] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:13.875638 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.820672 27949 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.821305 18018 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:13.778925 28296 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.827445 18021 raft_consensus.cc:2393] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:13.886364 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.797201 28343 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.740619 28159 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.054s
I20260709 12:26:13.843276 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.844732 28277 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:13.882202 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 97252.5us
I20260709 12:26:13.898010 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.883113 18090 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:13.899019 18098 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 30: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:13.885399 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.835069 28326 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:13.891883 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.899873 18092 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:13.898398 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.901693 18093 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:26:13.903120 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4814) took 1647 ms. Trace:
I20260709 12:26:13.903517 18093 rpcz_store.cc:276] 0709 12:26:12.255515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.255651 (+   136us) service_pool.cc:224] Handling call
0709 12:26:12.255994 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.650132 (+1394138us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b4c00314ec0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14983a20 00007f6c14994a29 00007f6c1495a644 00007f6c14981f2d 00007f6c14981b93 00007f6c166b218b 00007f6c166b2cb5 00007f6c166adf30 00007f6c166af7de 00007f6c166b9b49 00007f6c166ba416
0709 12:26:13.902829 (+252697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.903061 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83241344}
W20260709 12:26:13.909395 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 336392us
I20260709 12:26:13.906993 18022 raft_consensus.cc:2393] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 30: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:13.906332 18020 raft_consensus.cc:2393] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 30: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:26:13.913883 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 425814us
I20260709 12:26:13.918429 18019 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:13.919862 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.911199 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:13.922173 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.929421 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.920298 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.931535 18021 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 30: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:13.931726 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.933054 18064 leader_election.cc:400] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:13.933482 18064 leader_election.cc:403] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:26:13.934522 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.935491 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.936224 18017 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 23.
I20260709 12:26:13.937312 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.940056 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.941229 18017 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:13.943537 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:13.951581 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:13.952721 18171 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:13.955034 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.955950 18137 leader_election.cc:400] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 24. Current term is 25.
I20260709 12:26:13.955179 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.956787 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:13.960532 18169 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:13.961400 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.962466 18093 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 13.
I20260709 12:26:13.969339 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.972675 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.981012 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:13.974278 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:13.981807 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:13.983819 18172 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:26:13.983484 18138 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:13.986522 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:13.987735 18166 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:13.994232 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:13.995730 18021 raft_consensus.cc:2393] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:13.919615 28064 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:14.000277 18090 raft_consensus.cc:2393] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:14.001624 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:14.009326 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:14.012387 18095 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:26:14.012954 18095 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:13.980122 27957 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.015367 27957 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:13.972118 28277 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.020466 18099 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:14.021157 18099 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:14.021489 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:14.022477 28246 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:14.022374 18099 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.024165 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:14.021497 28232 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:14.025435 28232 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:14.026171 28232 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:14.027217 28275 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:14.027725 28275 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.028115 28275 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:14.028316 18095 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.029475 18095 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:14.032815 28306 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:14.033556 28306 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.034972 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:14.036164 28306 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:14.037756 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54412.4us
I20260709 12:26:13.947866 28215 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:14.038741 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.027413 28232 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:14.024256 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:14.042680 18099 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.960887 28353 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:14.052585 28353 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.055325 28353 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:14.055485 28360 consensus_queue.cc:1059] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:14.056722 28360 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:14.057157 28360 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:14.058132 27664 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:14.052551 28241 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:14.061385 28360 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.071616 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:13.948459 28348 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.100s
I20260709 12:26:14.075261 18023 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:13.937683 28351 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.103s
I20260709 12:26:14.077126 18166 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:26:13.966491 28358 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.091s
W20260709 12:26:14.051898 28330 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:26:14.066390 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:14.052961 28306 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:14.106621 18021 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:14.079298 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.025630 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:14.045455 18017 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 33.
I20260709 12:26:14.008953 28250 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.075s
I20260709 12:26:14.038177 28234 consensus_queue.cc:1059] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260709 12:26:14.113430 18017 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: [28.17-30.18]
I20260709 12:26:14.123751 18021 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:26:14.124290 18017 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [28.20-28.20]
I20260709 12:26:14.094332 18171 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:14.050787 28232 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:14.143400 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 117851us
I20260709 12:26:14.144208 18021 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:14.117211 18138 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.118891 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:14.084834 28377 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.162951 28274 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:14.163568 28274 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.163934 28377 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.164599 28377 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.164750 28232 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.166005 28315 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:14.165156 28377 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.166503 28232 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:14.169665 28360 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:14.126983 28353 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.046288 28326 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.170840 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:14.171214 28326 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:26:14.171584 18170 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:14.172344 27822 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.172453 28353 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.116156 28243 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:14.110016 27957 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:14.106137 28324 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.138666 28385 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:14.183259 18095 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 16. (term mismatch)
I20260709 12:26:14.195845 18095 pending_rounds.cc:85] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 15
I20260709 12:26:14.197237 18095 pending_rounds.cc:107] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260709 12:26:14.198523 18095 raft_consensus.cc:2889] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:14.201853 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84692.5us
I20260709 12:26:14.203274 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:14.146523 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 428987us
I20260709 12:26:14.157655 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:14.161216 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:14.108007 28343 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.173548 18092 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:26:14.208276 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49666.7us
W20260709 12:26:14.208554 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 47460.1us
I20260709 12:26:14.205320 18172 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:14.209307 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:14.216539 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.210420 18170 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:14.163962 28274 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:14.231292 28211 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:14.232004 28211 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:14.236008 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.217804 18099 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 18. (term mismatch)
I20260709 12:26:14.218348 28110 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:14.218513 27942 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 12:26:14.233014 28211 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.174568 28377 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.243920 18099 pending_rounds.cc:85] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 17
I20260709 12:26:14.266736 18099 pending_rounds.cc:107] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 27.18
I20260709 12:26:14.206434 18780 leader_election.cc:400] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27.
I20260709 12:26:14.256283 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:14.249578 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46436.7us
I20260709 12:26:14.207839 28280 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:14.271821 18170 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:14.277586 18099 raft_consensus.cc:2889] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:14.273407 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:14.219951 28385 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:14.290390 18138 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:14.292021 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48928.4us
I20260709 12:26:14.293704 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.254433 28281 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:14.258229 28215 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:26:14.308482 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.254499 27957 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:14.306399 18172 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:14.311828 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.368229 28234 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.210430 28324 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.271117 28110 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:14.369886 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:14.386840 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 180607us
I20260709 12:26:14.390705 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:14.393467 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:14.394680 18099 raft_consensus.cc:2376] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27. Re-sending same reply.
I20260709 12:26:14.219131 28343 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:14.450271 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.315538 28243 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:14.463842 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.315230 28064 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:14.469523 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 213382us
W20260709 12:26:14.470484 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80308.2us
W20260709 12:26:14.473160 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 161466us
W20260709 12:26:14.474229 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.483148 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.478610 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:14.487403 18099 raft_consensus.cc:2376] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28. Re-sending same reply.
I20260709 12:26:14.478838 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:14.481837 18064 leader_election.cc:400] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:14.490825 18064 leader_election.cc:403] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
W20260709 12:26:14.494622 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.502767 28320 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:14.397543 28234 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.503454 28320 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:14.503952 28320 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:14.507501 28396 consensus_queue.cc:1048] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:14.510309 28064 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.516268 28320 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.516108 28275 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.476774 18136 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:14.522097 28243 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:14.522768 28243 raft_consensus.cc:697] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:14.523254 28275 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:14.523682 28063 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:14.523854 28243 consensus_queue.cc:237] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.462864 28352 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:26:14.475365 28401 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.528632 28401 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.529662 28320 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.531833 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3874) took 3215 ms. Trace:
I20260709 12:26:14.533012 18164 rpcz_store.cc:276] 0709 12:26:11.316388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.422676 (+106288us) service_pool.cc:224] Handling call
0709 12:26:11.422936 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.043236 (+1620300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.531427 (+1488191us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b5400032ad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:14.531714 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3718535936}
I20260709 12:26:14.518061 28110 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.541998 27949 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.520238 28316 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:14.543347 28316 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.519412 18780 leader_election.cc:400] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25.
I20260709 12:26:14.544358 28343 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.544442 18780 leader_election.cc:403] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:26:14.531530 28401 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.520292 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:14.547338 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70775.6us
W20260709 12:26:14.562183 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.477403 28340 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:26:14.555970 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 255858us
I20260709 12:26:14.570883 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:14.575980 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1752) took 1186 ms. Trace:
I20260709 12:26:14.576728 18810 rpcz_store.cc:276] 0709 12:26:13.389660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.392440 (+  2780us) service_pool.cc:224] Handling call
0709 12:26:14.575684 (+1183244us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x7b54004e5f54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:14.575872 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1021020544}
I20260709 12:26:14.578130 28401 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:14.579010 28401 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:14.580375 28401 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:14.582216 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100571us
I20260709 12:26:14.583483 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:14.590531 28377 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:14.591872 28377 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:14.592382 28377 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:14.592898 28377 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.596184 18164 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: [29.23-29.23]
W20260709 12:26:14.599084 28390 log.cc:927] Time spent T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:26:14.559264 28316 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:14.602089 28402 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:14.602864 28402 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:14.606705 28388 log.cc:927] Time spent T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.209s	user 0.000s	sys 0.000s
W20260709 12:26:14.173635 28290 consensus_queue.cc:1175] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "768723648a9444fda53659e6a8cc3091" responder_term: 25 status { last_received { term: 23 index: 17 } last_committed_idx: 17 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 25. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:26:14.480093 28351 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:14.617164 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:14.547319 27942 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:26:14.637315 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141990us
I20260709 12:26:14.605546 28402 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:14.645766 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:14.624998 18099 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:14.649036 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:14.640825 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:14.650614 18172 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:14.653492 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:14.570264 28063 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.570052 28340 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:14.661974 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:14.623270 28320 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 24. Current term is 25.
I20260709 12:26:14.629355 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:14.657421 28272 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.828s	user 0.015s	sys 0.001s
I20260709 12:26:14.696924 18167 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:14.674746 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:14.735949 18172 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:26:14.669039 28315 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:14.750169 18166 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:14.750712 18169 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:14.677040 28340 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:14.755822 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 126585us
I20260709 12:26:14.678221 28358 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:14.661834 28316 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:26:14.687125 27942 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:14.492497 28385 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.755577 28280 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:14.676797 28063 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.754503 28351 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:14.757949 28397 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:14.809736 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1482) took 1974 ms. Trace:
I20260709 12:26:14.758878 28404 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.759045 28320 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:26:14.691881 28274 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:14.816610 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:14.693179 28393 consensus_queue.cc:1059] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.129s
I20260709 12:26:14.765085 28224 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:26:14.832018 18016 rpcz_store.cc:276] 0709 12:26:12.826083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.826226 (+   143us) service_pool.cc:224] Handling call
0709 12:26:12.826700 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.124148 (+297448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.799821 (+1675673us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54001095d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:14.800095 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682168064}
I20260709 12:26:14.839520 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:14.827939 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:14.852804 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:14.839247 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:14.856608 18023 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 30 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 13.
I20260709 12:26:14.749877 28272 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:14.859629 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:14.860607 18015 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:14.861680 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:14.770301 28159 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.063s
I20260709 12:26:14.831525 28280 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:14.837924 28367 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.986s	user 0.018s	sys 0.011s
I20260709 12:26:14.857416 18022 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:26:14.912484 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.856761 18021 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 35.
I20260709 12:26:14.834679 28315 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:14.909258 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 141127us
I20260709 12:26:14.858412 28320 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:14.813145 28358 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:14.863601 27822 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.691s	user 0.017s	sys 0.006s
W20260709 12:26:15.014716 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.010877 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:15.016839 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:14.862815 28385 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:15.019280 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:15.020598 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.021189 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.022464 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.025995 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 364097us
I20260709 12:26:14.860984 28324 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:15.027752 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 72685.9us
W20260709 12:26:15.033790 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.040223 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.041412 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:14.842770 27949 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.169s
I20260709 12:26:14.836372 28414 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.183s
I20260709 12:26:15.052618 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:14.912464 28280 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.840510 28351 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.048317 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:14.911147 28316 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.053018 27822 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:15.053156 28392 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:26:15.053472 27975 raft_consensus.cc:2764] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
W20260709 12:26:15.054118 28408 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.361s	user 0.001s	sys 0.000s
W20260709 12:26:15.042596 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 270448us
W20260709 12:26:15.270253 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.270988 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.849491 28404 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.064198 18812 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:15.075213 18811 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 12:26:15.168908 18096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 4025) took 2350 ms (client timeout 2020 ms). Trace:
W20260709 12:26:15.279626 18096 rpcz_store.cc:269] 0709 12:26:12.818690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.818851 (+   161us) service_pool.cc:224] Handling call
0709 12:26:15.168619 (+2349768us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b540005fad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:15.168776 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826459392}
I20260709 12:26:15.013518 28367 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.194824 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:15.197608 28413 log.cc:927] Time spent T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:26:15.198321 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4835) took 1179 ms. Trace:
I20260709 12:26:15.200487 28234 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:15.010566 27942 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.228348 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:15.232223 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4037) took 1266 ms. Trace:
I20260709 12:26:15.035248 28355 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:15.324959 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 272426us
I20260709 12:26:15.086686 18810 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:26:15.272783 18024 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:15.183218 28422 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:15.190304 27975 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:15.151048 27822 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.083149 28316 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.286036 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:15.294903 18097 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:15.294256 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:15.290628 18099 rpcz_store.cc:276] 0709 12:26:14.018984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.661411 (+642427us) service_pool.cc:224] Handling call
0709 12:26:14.662167 (+   756us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:14.663804 (+  1637us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:14.664153 (+   349us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.909684 (+245531us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:14.910333 (+   649us) log.cc:844] Serialized 26 byte log entry
0709 12:26:14.910593 (+   260us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:14.910640 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.910726 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:15.197949 (+287223us) raft_consensus.cc:1685] finished
0709 12:26:15.197984 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.198217 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16935,"prepare.queue_time_us":1198,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":25249,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":14894,"thread_start_us":228995,"threads_started":1,"wal-append.queue_time_us":504031}
I20260709 12:26:15.303313 18093 rpcz_store.cc:276] 0709 12:26:13.966064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.971220 (+  5156us) service_pool.cc:224] Handling call
0709 12:26:13.971580 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.971859 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:14.462041 (+490182us) log.cc:844] Serialized 14 byte log entry
0709 12:26:15.231790 (+769749us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:15.231863 (+    73us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:15.231917 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.231998 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.232148 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2485,"prepare.queue_time_us":79470,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":23672,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":13012,"thread_start_us":1197644,"threads_started":3,"wal-append.queue_time_us":548890}
I20260709 12:26:15.326073 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:15.283829 28424 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:15.182600 28315 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.217108 28320 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.361510 18097 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:15.350869 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:15.336942 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4032) took 1411 ms. Trace:
I20260709 12:26:15.370719 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:15.372207 18099 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:26:15.345081 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:15.316004 28396 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.327661 27938 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:15.380735 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86596.5us
I20260709 12:26:15.388399 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:15.369692 18094 rpcz_store.cc:276] 0709 12:26:13.925684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.925909 (+   225us) service_pool.cc:224] Handling call
0709 12:26:13.926483 (+   574us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.926840 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:13.962332 (+ 35492us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.912948 (+950616us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b18002f7ee0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a 00007f6c166c1a2a
0709 12:26:15.335425 (+422477us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:15.335508 (+    83us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:15.336651 (+  1143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.336736 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.336871 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":35228,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":23738,"raft.queue_time_us":131169,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":6571,"spinlock_wait_cycles":6636416,"thread_start_us":421228,"threads_started":2,"wal-append.queue_time_us":1334497}
I20260709 12:26:15.395933 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:15.343019 28063 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:15.398133 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.398523 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:15.401561 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:15.364549 27957 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:15.343001 28274 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:15.440933 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 95974.5us
W20260709 12:26:15.423784 28064 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.005s	sys 0.008s
I20260709 12:26:15.333770 28355 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.336866 28422 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.433672 28358 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:15.453514 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 174783us
I20260709 12:26:15.393419 28159 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
W20260709 12:26:15.467346 18020 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4654) took 1669 ms (client timeout 1940 ms). Trace:
I20260709 12:26:15.399192 27938 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:15.473518 18020 rpcz_store.cc:269] 0709 12:26:13.797661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.863211 (+ 65550us) service_pool.cc:224] Handling call
0709 12:26:15.467034 (+1603823us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54002ffb54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:15.467255 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96530688}
I20260709 12:26:15.446534 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:15.367529 28424 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.482580 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:15.453228 28063 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:15.484766 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:15.485985 18810 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:15.393622 28396 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:15.455528 28064 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.005s	sys 0.008s
I20260709 12:26:15.468540 28281 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:15.516286 18810 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:15.485013 28351 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.482664 28159 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:15.521988 18099 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
W20260709 12:26:15.524739 28418 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 12:26:15.502082 28367 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:15.525740 28419 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
W20260709 12:26:15.525910 28428 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:26:15.524567 18808 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:15.526962 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:15.529235 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:15.545444 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.503738 27975 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:15.492252 27957 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:15.524835 28433 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:15.598512 27957 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.603493 28433 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.603814 27957 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:26:15.603925 28416 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.308s	user 0.002s	sys 0.000s
I20260709 12:26:15.604789 28440 consensus_queue.cc:1059] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:26:15.606119 28443 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.607048 28444 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.502856 28063 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:15.584707 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 194802us
I20260709 12:26:15.587545 18815 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:15.519958 28272 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:15.530306 28281 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.607620 28431 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:15.608917 28447 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.628504 28447 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.509346 28415 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:15.524801 28435 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.583308 28430 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.509577 28234 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.082s
W20260709 12:26:15.588811 28421 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.296s	user 0.001s	sys 0.001s
I20260709 12:26:15.606771 28443 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.583961 18806 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:26:15.610790 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4833) took 1603 ms. Trace:
I20260709 12:26:15.596199 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:15.612010 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:15.589671 28404 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.631606 28447 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:15.611255 28427 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.286s	user 0.002s	sys 0.000s
I20260709 12:26:15.603993 28433 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:15.621767 28281 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:15.619138 28326 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.447s	user 0.019s	sys 0.000s
I20260709 12:26:15.609725 28296 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.612483 28432 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.713286 28432 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.669057 18090 rpcz_store.cc:276] 0709 12:26:14.007975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.040776 (+ 32801us) service_pool.cc:224] Handling call
0709 12:26:14.041176 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:14.041459 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:14.041486 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:14.605244 (+563758us) spinlock_profiling.cc:243] Waited 6.59 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:26:14.606108 (+   864us) log.cc:844] Serialized 26 byte log entry
0709 12:26:14.836513 (+230405us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:14.836550 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.836634 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:15.610426 (+773792us) raft_consensus.cc:1685] finished
0709 12:26:15.610450 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.610717 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15160064,"thread_start_us":229850,"threads_started":1,"wal-append.queue_time_us":545556}
I20260709 12:26:15.621980 27957 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.617772 28436 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:15.533143 28425 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.263s
I20260709 12:26:15.703833 27975 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:15.707171 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4632) took 2028 ms (client timeout 1549 ms). Trace:
I20260709 12:26:15.695050 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:15.726330 18815 raft_consensus.cc:2368] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:15.663866 28435 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:15.710062 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 206866us
I20260709 12:26:15.583431 28159 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.664577 28313 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:15.686830 28316 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:15.688660 28446 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.707333 28404 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:26:15.698177 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86290.3us
I20260709 12:26:15.734275 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:15.610949 28063 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.728758 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:15.737965 18138 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:15.712000 28447 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:15.658566 28415 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:15.723850 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.011s	sys 0.014s
I20260709 12:26:15.718983 28326 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.609016 28444 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:15.725591 18019 rpcz_store.cc:269] 0709 12:26:13.678521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.678675 (+   154us) service_pool.cc:224] Handling call
0709 12:26:15.706931 (+2028256us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:15.707077 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":523862912}
W20260709 12:26:15.747742 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.011s	sys 0.014s
I20260709 12:26:15.747552 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:15.743234 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4838) took 1353 ms. Trace:
I20260709 12:26:15.765976 18091 rpcz_store.cc:276] 0709 12:26:14.389788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.805901 (+416113us) service_pool.cc:224] Handling call
0709 12:26:14.808031 (+  2130us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:14.832284 (+ 24253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:14.832324 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:14.839612 (+  7288us) log.cc:844] Serialized 26 byte log entry
0709 12:26:15.080672 (+241060us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:15.080725 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.080824 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:15.742932 (+662108us) raft_consensus.cc:1685] finished
0709 12:26:15.742968 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.743156 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19152,"raft.queue_time_us":38826,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":11582,"thread_start_us":240655,"threads_started":1,"wal-append.queue_time_us":392905}
I20260709 12:26:15.769944 28429 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.770766 28429 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.772156 18810 raft_consensus.cc:2368] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
W20260709 12:26:15.765686 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 738295us
I20260709 12:26:15.696043 28430 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:15.741894 28385 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.709s	user 0.014s	sys 0.003s
I20260709 12:26:15.733115 28404 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:15.778906 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:15.779753 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:15.783702 18090 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:26:15.780666 28454 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:15.782243 28422 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.785930 28454 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:15.782020 28431 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.786482 27975 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.787122 28431 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.787595 28422 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:15.788439 28422 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.789919 28431 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.790649 28454 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:15.791106 28454 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.791466 28454 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:15.793721 28290 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:15.795902 18018 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4636) took 2116 ms (client timeout 1686 ms). Trace:
W20260709 12:26:15.796420 18018 rpcz_store.cc:269] 0709 12:26:13.679714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.681490 (+  1776us) service_pool.cc:224] Handling call
0709 12:26:15.795608 (+2114118us) spinlock_profiling.cc:243] Waited 76.7 ms on lock 0x7b540010a4d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:15.795770 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176475392}
I20260709 12:26:15.799604 28403 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:15.800227 28403 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:15.805051 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.741453 28355 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.763271 28316 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.760877 28281 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:15.835716 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:15.837689 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:15.847378 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3662) took 1228 ms. Trace:
I20260709 12:26:15.849812 18813 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:15.801177 28403 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.732151 28436 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.848019 18164 rpcz_store.cc:276] 0709 12:26:14.619286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.619537 (+   251us) service_pool.cc:224] Handling call
0709 12:26:14.620096 (+   559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:14.620635 (+   539us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:14.620683 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:14.621375 (+   692us) log.cc:844] Serialized 26 byte log entry
0709 12:26:14.836459 (+215084us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:26:14.858653 (+ 22194us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.859191 (+   538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.859312 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:15.847062 (+987750us) raft_consensus.cc:1685] finished
0709 12:26:15.847090 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.847292 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":173,"prepare.queue_time_us":21836,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":10867,"raft.queue_time_us":430,"spinlock_wait_cycles":938752,"thread_start_us":168105,"threads_started":1,"wal-append.queue_time_us":395150}
W20260709 12:26:15.862910 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 89622.2us
I20260709 12:26:15.791469 27957 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:15.868295 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:15.868587 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 121179us
I20260709 12:26:15.784749 28404 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:15.791631 28443 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.732849 28313 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.849704 28396 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:15.763693 28447 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:15.879640 18137 connection.cc:256] RPC call timeout handler was delayed by -1.2411190800003169s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1876s for a response
I20260709 12:26:15.768440 28326 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.887212 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:15.857343 28432 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:15.894191 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.759543 28446 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.866796 28281 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:15.872356 28431 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.864106 28296 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
W20260709 12:26:15.894860 28340 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.078s	user 0.011s	sys 0.006s
I20260709 12:26:15.904438 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:15.886264 28403 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:15.897192 28443 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:26:15.904577 18780 connection.cc:256] RPC call timeout handler was delayed by -1.6854376060000504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
W20260709 12:26:15.899240 28280 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.746s	user 0.013s	sys 0.004s
I20260709 12:26:15.907042 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:15.902832 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:15.864189 28316 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.915328 28443 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.915495 18171 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:15.923758 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:15.924770 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:15.926148 18171 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 33: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:15.916196 28449 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:15.916750 28453 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:15.917016 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:15.917683 28463 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:15.918210 18172 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:15.919034 18173 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260709 12:26:15.919349 28359 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:26:15.920372 18098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4826) took 2016 ms (client timeout 1937 ms). Trace:
W20260709 12:26:15.909101 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.969796 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.925140 18170 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:15.971436 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:15.905898 28340 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.972182 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:15.946034 18173 pending_rounds.cc:85] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 19
I20260709 12:26:15.983377 18173 pending_rounds.cc:107] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260709 12:26:15.920264 28415 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:15.950201 28346 raft_consensus.cc:3055] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:15.922549 28454 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.967423 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:15.930295 28280 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:15.905328 28385 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.950991 28359 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:15.977738 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:15.978688 18098 rpcz_store.cc:269] 0709 12:26:13.903978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.904168 (+   190us) service_pool.cc:224] Handling call
0709 12:26:15.920100 (+2015932us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54002ca954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:15.920284 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3665603456}
I20260709 12:26:15.927313 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:15.904057 28431 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:15.980804 27957 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:15.965862 28453 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:15.984006 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:15.940347 28443 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:15.942329 28467 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:16.001794 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 115314us
I20260709 12:26:16.007532 18167 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:15.985435 28449 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.011919 18814 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:16.017383 18171 raft_consensus.cc:1240] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:16.035426 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:15.996436 28346 raft_consensus.cc:740] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:15.977742 28458 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:26:16.025172 18172 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:16.027302 27957 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.029914 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:16.033110 28159 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:16.031471 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.031800 18169 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:16.032526 18166 raft_consensus.cc:2393] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:15.983747 18173 raft_consensus.cc:2889] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:16.043416 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:16.037886 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 59840.7us
I20260709 12:26:16.036422 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.054306 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:16.059192 18167 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:26:16.045466 28377 consensus_meta.cc:287] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.452s	user 0.022s	sys 0.004s
I20260709 12:26:16.065898 18809 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 33: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:16.070031 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:16.034945 28422 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.073112 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:16.078671 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:16.019827 28385 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.076253 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.082221 28470 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:26:16.082778 28211 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:16.082901 28470 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.083174 28463 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:16.084810 18172 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:16.087363 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.087715 18809 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 33: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:16.053351 27975 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:16.083447 28211 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:16.012595 28281 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.089598 28211 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.090286 28454 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.061059 28313 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.091032 28454 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:16.080109 28296 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:16.092087 28296 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.091409 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.092551 28296 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:16.083370 28470 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:26:16.092797 18167 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:26:16.091055 18807 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:26:16.096153 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:16.095127 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.097229 28462 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:16.097857 28315 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:16.098971 27564 consensus_queue.cc:1059] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:16.092020 28454 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.009s
W20260709 12:26:16.101176 28461 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.059s	user 0.001s	sys 0.001s
I20260709 12:26:16.101641 27564 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:16.102224 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:16.106681 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:16.112290 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:16.069159 28435 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.097970 28462 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.115334 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:16.118673 28466 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:16.122823 28471 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.078544 28415 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:16.086784 28422 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:16.130729 18137 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 2.211s (SENT) [suppressed 25 similar messages]
I20260709 12:26:16.042593 28467 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.118978 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa"
dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
 from {username='slave'} at 127.0.0.1:58850
I20260709 12:26:16.152216 18812 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:16.152872 18812 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:16.142199 28463 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.114733 28476 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:16.075254 28377 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.158253 18170 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:16.119769 28326 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.163158 28472 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:16.165205 28467 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.139206 28436 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.155717 28447 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:26:16.070194 28359 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.169651 28476 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:16.206473 28460 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.009s
I20260709 12:26:16.208560 28477 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:16.218847 18170 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:26:16.182498 28326 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.171108 27564 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:16.168366 28440 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:26:16.168009 28422 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.203423 28476 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:16.221668 28446 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.220178 28415 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:16.251557 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 184164us
I20260709 12:26:16.211827 28444 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.244467 18812 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.152660 28471 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.133165 28453 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.262064 18138 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:16.242920 28280 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.244666 28476 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:16.252650 28431 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.246803 28377 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.269984 28466 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:16.272855 28462 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:16.273664 28469 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.203s	user 0.000s	sys 0.002s
I20260709 12:26:16.276544 28422 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:16.277464 28406 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.583s	user 0.004s	sys 0.012s
I20260709 12:26:16.282541 28385 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.264678 28316 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.286550 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3909) took 2331 ms. Trace:
I20260709 12:26:16.305330 28453 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:16.287384 18168 rpcz_store.cc:276] 0709 12:26:13.955446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.955796 (+   350us) service_pool.cc:224] Handling call
0709 12:26:13.956192 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:13.968069 (+ 11877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:13.968104 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:13.968696 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 12:26:14.131974 (+163278us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b18004997e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:26:14.462524 (+330550us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:14.498088 (+ 35564us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.498451 (+   363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.498566 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.145658 (+1647092us) spinlock_profiling.cc:243] Waited 7.13 ms on lock 0x7b18004997e0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c1b5e0328 00007f6c1b5d7901 00007f6c1b5d78a7 00007f6c1b6cf288 00007f6c1b6cef37 00007f6c1b6e355c 00007f6c1b6e3508
0709 12:26:16.246407 (+100749us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:16.285987 (+ 39580us) raft_consensus.cc:1685] finished
0709 12:26:16.286038 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.286407 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38810,"prepare.queue_time_us":85217,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":56,"raft.queue_time_us":38426,"raft.run_cpu_time_us":15738,"raft.run_wall_time_us":1649688,"spinlock_wait_cycles":80694656,"thread_start_us":303602,"threads_started":1,"wal-append.queue_time_us":645951}
I20260709 12:26:16.239854 28326 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.268975 18812 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:16.265777 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 135100us
I20260709 12:26:16.313387 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.314219 18016 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:26:16.318158 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:16.319097 18019 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:16.321705 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:16.256138 28415 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.322643 18015 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:16.269174 27957 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.333169 18020 raft_consensus.cc:1240] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [28.14-28.14]
I20260709 12:26:16.341987 28415 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.342228 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:16.351459 28433 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.639s	user 0.013s	sys 0.003s
W20260709 12:26:16.354110 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.354686 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.357177 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:16.358664 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:16.359504 18811 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
I20260709 12:26:16.360553 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4841) took 1896 ms. Trace:
I20260709 12:26:16.361089 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:16.364701 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:16.365797 18811 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:16.361243 18099 rpcz_store.cc:276] 0709 12:26:14.463827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.423081 (+959254us) service_pool.cc:224] Handling call
0709 12:26:15.423568 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.534344 (+110776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.360142 (+825798us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b5400291b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:16.360450 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":224984832}
I20260709 12:26:16.372637 28249 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:16.377173 18015 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:16.362228 18806 raft_consensus.cc:2376] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31. Re-sending same reply.
I20260709 12:26:16.387027 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:16.413180 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.298177 27564 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.415527 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:16.388581 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:16.377207 28377 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.352125 28433 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.425604 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:16.419788 18808 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:16.429349 18019 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 33.
I20260709 12:26:16.428300 18022 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:26:16.433854 28436 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:26:16.435247 18098 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:16.435041 28435 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:16.437070 28436 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:16.437554 28436 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:16.438635 28436 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.439845 28470 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.440400 28436 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:16.385797 28249 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:16.441987 28383 raft_consensus.cc:3055] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:16.442574 28383 raft_consensus.cc:740] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:16.443001 28249 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.443559 28383 consensus_queue.cc:260] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.444515 28249 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:16.448019 28433 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.422729 28467 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:16.450558 28467 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:16.450958 18138 leader_election.cc:336] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.686s (SENT)
I20260709 12:26:16.451887 28449 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.452714 28471 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.453085 28467 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:16.455389 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.456111 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.418128 28483 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.384824 28404 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.456666 28377 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:16.543468 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.480643 18063 leader_election.cc:400] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:16.561122 18063 leader_election.cc:403] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:26:16.415288 28280 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.544051 18015 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:26:16.414901 28462 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.610847 28462 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.423202 28466 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.414710 27957 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:16.614370 27957 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.616855 27957 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:16.456956 28403 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.009s
I20260709 12:26:16.422907 28431 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.618153 28431 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.548924 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.549367 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:16.549683 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.556722 28467 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.619976 18019 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:16.620416 18099 raft_consensus.cc:2410] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:16.551074 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.551364 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:16.621855 18090 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:16.555208 28383 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:16.562525 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.564002 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.624583 28353 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:16.624919 28431 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.625326 28430 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.625170 28353 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.627851 28353 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:16.628686 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.628873 28470 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.614372 28466 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.564498 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:16.569954 28326 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:16.570784 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 137988us
I20260709 12:26:16.619976 18022 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:16.571038 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.571861 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.572053 28280 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.571877 28435 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:16.573002 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:16.553627 28471 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.547518 28440 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.582331 28279 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:16.591707 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.598810 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4047) took 1761 ms. Trace:
I20260709 12:26:16.610016 28377 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.657461 18098 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 31 because replica has last-logged OpId of term: 29 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
I20260709 12:26:16.656972 18094 rpcz_store.cc:276] 0709 12:26:14.837352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.419963 (+582611us) service_pool.cc:224] Handling call
0709 12:26:15.420384 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.489777 (+ 69393us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:26:15.662624 (+172847us) log.cc:844] Serialized 14 byte log entry
0709 12:26:15.731775 (+ 69151us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:15.731836 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:15.731891 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.970545 (+238654us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18001bc4c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca557 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:16.598337 (+627792us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b5400215054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:16.598422 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.598683 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":224425,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":1600,"raft.queue_time_us":467,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8752,"spinlock_wait_cycles":163090560,"thread_start_us":225255,"threads_started":2,"wal-append.queue_time_us":119143}
W20260709 12:26:16.634130 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69779.9us
I20260709 12:26:16.635855 18090 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:16.644992 18093 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:16.648972 18015 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:26:16.579459 18097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4816) took 2708 ms (client timeout 1876 ms). Trace:
I20260709 12:26:16.653332 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:16.664605 18097 rpcz_store.cc:269] 0709 12:26:13.870729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.871758 (+  1029us) service_pool.cc:224] Handling call
0709 12:26:16.579129 (+2707371us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:16.579337 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3270632832}
I20260709 12:26:16.667462 28471 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.669020 28280 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.560441 28483 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.672828 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:16.649237 28470 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:16.680475 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:16.683774 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:16.681111 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:16.679064 28280 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.008s
I20260709 12:26:16.671352 28466 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.699723 28483 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.703495 28404 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.697178 28476 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:16.707955 28443 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.657s	user 0.014s	sys 0.000s
I20260709 12:26:16.708559 28443 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.709211 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35286 (request call id 1937) took 1122 ms. Trace:
I20260709 12:26:16.709403 18096 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:16.709776 18810 rpcz_store.cc:276] 0709 12:26:15.588835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.595856 (+  7021us) service_pool.cc:224] Handling call
0709 12:26:16.708969 (+1113113us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b5400590e54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:16.709127 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192581504}
I20260709 12:26:16.711966 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.688079 18063 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:16.690833 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:16.620831 28467 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.692487 28039 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:16.717670 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.718617 18096 raft_consensus.cc:2393] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:16.719336 28443 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:16.719642 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.720605 18096 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:16.722196 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.723278 18172 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:16.734407 18015 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 36.
I20260709 12:26:16.721740 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:16.750965 18096 raft_consensus.cc:2376] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25. Re-sending same reply.
W20260709 12:26:16.751771 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 71360.2us
I20260709 12:26:16.752774 18022 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:16.755031 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.754923 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.756486 18020 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:16.757997 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.759022 18167 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 26 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:26:16.761124 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.763947 28476 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:26:16.711877 28462 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.772018 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.782610 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.784550 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:16.787871 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.788297 27957 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:16.788749 18168 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
W20260709 12:26:16.787712 28296 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.695s	user 0.015s	sys 0.002s
W20260709 12:26:16.790072 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99383.1us
I20260709 12:26:16.685715 28440 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.785636 18172 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:16.718703 28424 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.814101 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.789804 28305 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:16.822806 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.825903 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.826390 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.827327 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:16.824262 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.830840 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.771325 28404 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:16.834879 18168 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:16.771641 28436 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:16.831660 18138 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.840279 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:16.843240 18168 raft_consensus.cc:2368] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:16.844556 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.841593 18138 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:16.848184 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.845463 18168 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:16.849424 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.850446 18096 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:16.850836 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.837752 28039 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:16.841370 28465 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 12:26:16.854307 28039 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:16.855655 27822 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.855412 28039 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:16.854300 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.859226 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.862021 28470 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:16.863406 28440 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.821290 28296 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.864218 28467 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:16.847060 28436 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:16.866207 28296 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.866930 28436 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:16.834898 28462 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:26:16.867527 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.868340 18094 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:16.870641 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.873003 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:16.869587 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.848330 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.853185 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.876904 18138 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.882007 18168 raft_consensus.cc:2393] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 25: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:16.883121 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.881065 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:16.852890 28280 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:16.883018 28255 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.173s	user 0.018s	sys 0.018s
I20260709 12:26:16.838574 28305 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:16.847283 28404 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:16.882506 28462 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:26:16.894471 28280 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.892483 28467 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:16.896428 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:16.930823 18018 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:16.924307 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:16.925486 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4829) took 2985 ms (client timeout 1844 ms). Trace:
I20260709 12:26:16.919250 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:17.008555 18092 rpcz_store.cc:269] 0709 12:26:13.939888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.940826 (+   938us) service_pool.cc:224] Handling call
0709 12:26:16.925169 (+2984343us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b540035e8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:16.925374 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":459677568}
I20260709 12:26:17.009398 18808 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:16.912377 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1928) took 1416 ms. Trace:
I20260709 12:26:16.914500 18172 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 13.
I20260709 12:26:16.923450 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:16.961170 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.011443 18022 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
W20260709 12:26:17.011693 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.010083 18813 rpcz_store.cc:276] 0709 12:26:15.496591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.496858 (+   267us) service_pool.cc:224] Handling call
0709 12:26:15.507948 (+ 11090us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.850440 (+342492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.912022 (+1061582us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400494154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:16.912301 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":785425152}
I20260709 12:26:16.894636 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:16.918190 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:17.016130 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 98020.6us
I20260709 12:26:17.017494 18063 leader_election.cc:400] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
I20260709 12:26:17.014552 18168 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:16.927140 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:16.913880 28277 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:16.911427 28456 raft_consensus.cc:3055] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:17.071096 28456 raft_consensus.cc:740] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:17.072613 18165 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 18. (term mismatch)
I20260709 12:26:17.073136 18165 pending_rounds.cc:85] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 17
I20260709 12:26:17.073408 18165 pending_rounds.cc:107] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:26:17.072854 28456 consensus_queue.cc:260] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:17.073652 18165 raft_consensus.cc:2889] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:17.074541 28456 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:17.035028 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:17.000404 28397 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.058074 18094 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:17.062335 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:16.898663 28462 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:17.068375 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.906528 28429 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:17.012928 28296 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.921401 28470 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:16.898099 28404 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:16.897445 28255 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.191s	user 0.018s	sys 0.018s
I20260709 12:26:16.918783 28280 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:17.026111 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.998268 28465 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:17.076079 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57287.6us
I20260709 12:26:17.074657 28253 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:16.904310 28441 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.321s	user 0.007s	sys 0.000s
I20260709 12:26:17.083356 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:17.154575 18093 raft_consensus.cc:2410] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 12.
I20260709 12:26:17.056929 18015 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:17.140026 28470 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.142215 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1935) took 1613 ms. Trace:
I20260709 12:26:16.927425 28467 raft_consensus.cc:697] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:17.146024 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:17.166724 18092 raft_consensus.cc:2410] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 13.
I20260709 12:26:17.162722 18814 rpcz_store.cc:276] 0709 12:26:15.528336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.528847 (+   511us) service_pool.cc:224] Handling call
0709 12:26:15.586848 (+ 58001us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.035106 (+448258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.141745 (+1106639us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54000e2ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:17.142095 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":976972032}
W20260709 12:26:17.151276 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.080840 28377 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:17.082170 28224 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:26:17.145423 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:17.163048 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4641) took 3471 ms (client timeout 1982 ms). Trace:
W20260709 12:26:17.169890 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.019665 28436 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:17.205543 18024 rpcz_store.cc:269] 0709 12:26:13.691284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.747661 (+ 56377us) service_pool.cc:224] Handling call
0709 12:26:17.162756 (+3415095us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001d1854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:17.162941 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2981468928}
I20260709 12:26:17.099937 28397 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:17.168419 28470 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:17.149251 28385 consensus_queue.cc:1059] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:26:17.211159 28436 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.211027 28224 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.143935 28296 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:17.099507 28305 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.157131 28253 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:17.374653 18092 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:17.375428 18092 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:17.154606 28465 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:17.351547 28279 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.679s	user 0.017s	sys 0.011s
I20260709 12:26:17.393350 18095 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:26:17.376602 18092 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.398849 18092 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:17.367246 28436 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:17.373353 28224 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:17.366801 28215 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:17.395108 28064 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:17.408687 28393 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.382s	user 0.015s	sys 0.020s
I20260709 12:26:17.334071 28462 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:17.423532 28393 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.408s	user 0.015s	sys 0.020s
I20260709 12:26:17.415014 28215 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:17.415489 28467 consensus_queue.cc:237] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.429240 28064 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:17.486826 28492 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.127s	user 0.001s	sys 0.001s
I20260709 12:26:17.502489 28465 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:17.549994 18016 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.012s	sys 0.058s
W20260709 12:26:17.551110 18016 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.012s	sys 0.059s
W20260709 12:26:17.551422 18023 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.026s	sys 0.038s
W20260709 12:26:17.552130 18023 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.026s	sys 0.038s
W20260709 12:26:17.565794 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.604800 27938 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:17.450348 27942 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.033s
W20260709 12:26:17.606374 27942 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.000s	sys 0.033s
W20260709 12:26:17.625748 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.513715 28470 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.676888 28470 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:17.678196 28290 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.679627 28290 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:17.759001 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.767064 28215 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:17.784235 28501 log.cc:927] Time spent T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:26:17.786253 28499 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:26:17.789929 28397 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:17.792507 28290 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:17.795498 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:17.810616 28496 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:26:17.813105 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4681) took 1263 ms. Trace:
I20260709 12:26:17.814129 28255 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:17.815574 28255 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.774206 28296 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:17.819782 28255 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:17.813709 18023 rpcz_store.cc:276] 0709 12:26:16.549303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.549432 (+   129us) service_pool.cc:224] Handling call
0709 12:26:16.569452 (+ 20020us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.569966 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:16.875752 (+305786us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.553073 (+677321us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.554666 (+  1593us) log.cc:844] Serialized 26 byte log entry
0709 12:26:17.554984 (+   318us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:17.555094 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.555310 (+   216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:17.812677 (+257367us) raft_consensus.cc:1685] finished
0709 12:26:17.812714 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.812938 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":826108,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":50924,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":535,"thread_start_us":960763,"threads_started":2,"wal-append.queue_time_us":600508}
W20260709 12:26:17.817549 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:17.825609 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1539) took 1879 ms (client timeout 1827 ms). Trace:
I20260709 12:26:17.817533 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3910) took 3785 ms. Trace:
I20260709 12:26:17.833438 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:17.833671 18170 rpcz_store.cc:269] 0709 12:26:15.946087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.972610 (+ 26523us) service_pool.cc:224] Handling call
0709 12:26:17.824960 (+1852350us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b540002a8d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:17.825261 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328237952}
I20260709 12:26:17.834067 18165 rpcz_store.cc:276] 0709 12:26:14.032107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.037128 (+  5021us) service_pool.cc:224] Handling call
0709 12:26:14.037526 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.074474 (+3036948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.816747 (+742273us) spinlock_profiling.cc:243] Waited 3.03 s on lock 0x7b54003016d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:17.817190 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6978950912}
I20260709 12:26:17.837106 18170 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:17.837301 18165 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:17.838290 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3670) took 1083 ms. Trace:
I20260709 12:26:17.838792 18165 rpcz_store.cc:276] 0709 12:26:16.754369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.835964 (+1081595us) service_pool.cc:224] Handling call
0709 12:26:17.836655 (+   691us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.837795 (+  1140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.838174 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20736}
I20260709 12:26:17.843963 18165 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:17.848654 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3672) took 1093 ms. Trace:
I20260709 12:26:17.854470 18165 rpcz_store.cc:276] 0709 12:26:16.754898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.842677 (+1087779us) service_pool.cc:224] Handling call
0709 12:26:17.843400 (+   723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.848218 (+  4818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.848542 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:17.853338 28431 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:17.814199 28465 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:17.858650 28431 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:17.859400 28431 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:17.858788 28465 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.846247 28462 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:17.861390 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4053) took 1207 ms. Trace:
W20260709 12:26:17.838919 18094 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.760s	user 0.022s	sys 0.010s
I20260709 12:26:17.864118 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3669) took 1106 ms. Trace:
I20260709 12:26:17.867208 18170 rpcz_store.cc:276] 0709 12:26:16.754082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.835755 (+1081673us) service_pool.cc:224] Handling call
0709 12:26:17.836606 (+   851us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.837722 (+  1116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.860380 (+ 22658us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22400}
I20260709 12:26:17.869997 28305 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:17.870654 28305 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:17.880506 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.864904 28462 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.889902 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:17.892345 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:17.895959 18814 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:17.897496 28039 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:17.898516 28039 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.900426 28359 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:17.899710 28305 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.892095 28465 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:17.917589 18094 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:26:17.864953 18091 rpcz_store.cc:276] 0709 12:26:16.653302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.670095 (+ 16793us) service_pool.cc:224] Handling call
0709 12:26:16.670617 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.671108 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:16.932374 (+261266us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.780390 (+848016us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.850543 (+ 70153us) log.cc:844] Serialized 26 byte log entry
0709 12:26:17.850785 (+   242us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:17.850835 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.851017 (+   182us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:17.859972 (+  8955us) raft_consensus.cc:1685] finished
0709 12:26:17.860009 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.861058 (+  1049us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":427897,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":64,"raft.queue_time_us":55020,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":11878,"thread_start_us":1001030,"threads_started":3,"wal-append.queue_time_us":694990}
I20260709 12:26:17.915597 28346 consensus_queue.cc:260] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:17.916643 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4861) took 1079 ms. Trace:
I20260709 12:26:17.916740 27938 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:17.919652 18096 rpcz_store.cc:276] 0709 12:26:16.837114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.851561 (+ 14447us) service_pool.cc:224] Handling call
0709 12:26:17.916150 (+1064589us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002c9a54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:17.916380 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2353376512}
I20260709 12:26:17.922037 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4056) took 1250 ms. Trace:
I20260709 12:26:17.922587 18092 rpcz_store.cc:276] 0709 12:26:16.672030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.346014 (+673984us) service_pool.cc:224] Handling call
0709 12:26:17.374448 (+ 28434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.921234 (+546786us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:17.921293 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.921369 (+    76us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:17.921410 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.921528 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.921798 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6606,"raft.queue_time_us":12112,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":229}
I20260709 12:26:17.924465 18170 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:17.893584 28462 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:17.925316 18096 raft_consensus.cc:1275] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:26:17.928980 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3674) took 1170 ms. Trace:
I20260709 12:26:17.929718 18170 rpcz_store.cc:276] 0709 12:26:16.755432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.879160 (+1123728us) service_pool.cc:224] Handling call
0709 12:26:17.923927 (+ 44767us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.925178 (+  1251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.925499 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:17.928958 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4857) took 1124 ms. Trace:
I20260709 12:26:17.894176 28296 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.958446 28296 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:17.931193 18096 rpcz_store.cc:276] 0709 12:26:16.804328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.923585 (+1119257us) service_pool.cc:224] Handling call
0709 12:26:17.924664 (+  1079us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.928170 (+  3506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.928863 (+   693us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:17.971231 28397 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:17.973169 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4850) took 1421 ms. Trace:
I20260709 12:26:17.973857 18095 rpcz_store.cc:276] 0709 12:26:16.551305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.551857 (+   552us) service_pool.cc:224] Handling call
0709 12:26:17.972532 (+1420675us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b540032c154. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:17.972877 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1748729600}
I20260709 12:26:17.938584 28509 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:17.942128 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:17.943531 28305 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:17.949086 28423 raft_consensus.cc:3055] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:17.953419 28471 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:17.955940 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3918) took 2064 ms. Trace:
I20260709 12:26:17.957072 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45032 (request call id 4068) took 1074 ms. Trace:
I20260709 12:26:17.978644 28397 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:17.989027 18094 rpcz_store.cc:276] 0709 12:26:16.882586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.895118 (+ 12532us) service_pool.cc:224] Handling call
0709 12:26:17.956948 (+1061830us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:17.935714 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4069) took 1050 ms. Trace:
I20260709 12:26:17.980681 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:17.985989 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4854) took 1030 ms. Trace:
I20260709 12:26:17.992444 18091 rpcz_store.cc:276] 0709 12:26:16.885776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.934298 (+1048522us) service_pool.cc:224] Handling call
0709 12:26:17.934840 (+   542us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.935270 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.935300 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.935378 (+    78us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:17.935418 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.935485 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.935632 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:18.000047 18165 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: [28.15-28.15]
W20260709 12:26:17.973173 28456 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.896s	user 0.021s	sys 0.007s
I20260709 12:26:17.987910 28296 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.019692 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3939) took 1295 ms. Trace:
I20260709 12:26:17.987877 18164 rpcz_store.cc:276] 0709 12:26:15.891962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.892185 (+   223us) service_pool.cc:224] Handling call
0709 12:26:15.892818 (+   633us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:16.024446 (+131628us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.024485 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:16.038979 (+ 14494us) log.cc:844] Serialized 52 byte log entry
0709 12:26:16.151894 (+112915us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:16.902137 (+750243us) spinlock_profiling.cc:243] Waited 892 us on lock 0x7b18000e0380. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c1a0af3c7 00007f6c166c19a7 00007f6c166d10bf 00007f6c166d1038
0709 12:26:17.360590 (+458453us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.878118 (+517528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.892873 (+ 14755us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:17.901253 (+  8380us) raft_consensus.cc:1685] finished
0709 12:26:17.912370 (+ 11117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.955673 (+ 43303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":156290,"prepare.queue_time_us":911187,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":108540,"raft.run_cpu_time_us":625,"raft.run_wall_time_us":170590,"spinlock_wait_cycles":2051456,"thread_start_us":1035561,"threads_started":4,"wal-append.queue_time_us":574122,"wal-append.run_cpu_time_us":3311,"wal-append.run_wall_time_us":1070842}
I20260709 12:26:18.026328 27938 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:18.027015 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 46656.3us
I20260709 12:26:17.992578 18099 rpcz_store.cc:276] 0709 12:26:16.801607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.585414 (+783807us) service_pool.cc:224] Handling call
0709 12:26:17.586192 (+   778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.586826 (+   634us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:17.608096 (+ 21270us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.830546 (+222450us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.830615 (+    69us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:17.830668 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.830751 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.831836 (+  1085us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14288,"prepare.queue_time_us":11822,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":9022,"raft.queue_time_us":57261,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":229,"thread_start_us":170214,"threads_started":1,"wal-append.queue_time_us":258589}
I20260709 12:26:18.030627 28279 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:17.985914 28423 raft_consensus.cc:740] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:18.006807 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 64888.4us
W20260709 12:26:18.017107 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 159563us
I20260709 12:26:18.033056 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3919) took 2140 ms. Trace:
I20260709 12:26:18.045322 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4855) took 1080 ms. Trace:
I20260709 12:26:18.034307 18167 rpcz_store.cc:276] 0709 12:26:16.724134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.819058 (+1094924us) service_pool.cc:224] Handling call
0709 12:26:17.819863 (+   805us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.820654 (+   791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.820696 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.833099 (+ 12403us) log.cc:844] Serialized 26 byte log entry
0709 12:26:17.884077 (+ 50978us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:17.884139 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.884233 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.019169 (+134936us) raft_consensus.cc:1685] finished
0709 12:26:18.019196 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.019412 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4460,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":252,"thread_start_us":47799,"threads_started":1,"wal-append.queue_time_us":93750}
I20260709 12:26:18.041316 18169 rpcz_store.cc:276] 0709 12:26:15.892361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.892511 (+   150us) service_pool.cc:224] Handling call
0709 12:26:15.892837 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.050175 (+157338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.032184 (+1982009us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54000433d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:18.032764 (+   580us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318866816}
I20260709 12:26:18.039351 28279 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:18.073465 28280 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.919s	user 0.018s	sys 0.011s
I20260709 12:26:18.085016 18169 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:18.085943 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3943) took 1307 ms. Trace:
I20260709 12:26:18.086453 28346 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:18.087114 28039 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.088889 28377 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:18.089529 28377 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:18.086318 18169 rpcz_store.cc:276] 0709 12:26:16.778705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.054658 (+1275953us) service_pool.cc:224] Handling call
0709 12:26:18.081175 (+ 26517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.085608 (+  4433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.085875 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:18.057261 28279 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:18.047662 18098 rpcz_store.cc:276] 0709 12:26:16.802401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.803139 (+1000738us) service_pool.cc:224] Handling call
0709 12:26:17.803946 (+   807us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.804547 (+   601us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:17.805120 (+   573us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.881792 (+ 76672us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.882385 (+   593us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:17.882449 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.882530 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.882700 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":288,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":86,"raft.queue_time_us":8882,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":1286,"thread_start_us":67365,"threads_started":1,"wal-append.queue_time_us":233386}
W20260709 12:26:18.095449 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45042.1us
I20260709 12:26:18.109166 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3677) took 1352 ms. Trace:
I20260709 12:26:18.111665 18164 rpcz_store.cc:276] 0709 12:26:16.756275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.053123 (+1296848us) service_pool.cc:224] Handling call
0709 12:26:18.108135 (+ 55012us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.108635 (+   500us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:18.108669 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:18.108756 (+    87us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:18.108794 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.108872 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.109059 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:18.116124 18167 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: [29.23-29.23]
I20260709 12:26:18.117214 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3678) took 1360 ms. Trace:
I20260709 12:26:18.117602 18167 rpcz_store.cc:276] 0709 12:26:16.756516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.053470 (+1296954us) service_pool.cc:224] Handling call
0709 12:26:18.113206 (+ 59736us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:18.116769 (+  3563us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.117127 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:18.119362 28436 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.703s	user 0.013s	sys 0.009s
I20260709 12:26:18.119840 28467 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.120440 28253 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.120085 28436 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.120709 28467 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.121625 18092 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [28.21-28.21]
W20260709 12:26:18.124209 28517 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:26:18.124387 28467 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.119958 28473 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.126828 28473 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:18.132931 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.129580 28473 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.137121 28423 consensus_queue.cc:260] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.139810 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4691) took 1215 ms. Trace:
I20260709 12:26:18.145022 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3675) took 1389 ms. Trace:
I20260709 12:26:18.145649 18170 rpcz_store.cc:276] 0709 12:26:16.755709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.969098 (+1213389us) service_pool.cc:224] Handling call
0709 12:26:17.969871 (+   773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.970601 (+   730us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.970630 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.971532 (+   902us) log.cc:844] Serialized 26 byte log entry
0709 12:26:18.037439 (+ 65907us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:18.037482 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.037568 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.144615 (+107047us) raft_consensus.cc:1685] finished
0709 12:26:18.144652 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.144921 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":65133,"threads_started":1,"wal-append.queue_time_us":132860}
I20260709 12:26:18.147578 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3923) took 2243 ms. Trace:
I20260709 12:26:18.148132 18166 rpcz_store.cc:276] 0709 12:26:15.903962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.904111 (+   149us) service_pool.cc:224] Handling call
0709 12:26:18.147057 (+2242946us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54003020d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:18.147341 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277104384}
I20260709 12:26:18.150868 28436 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.151873 28431 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.153405 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3673) took 1398 ms. Trace:
W20260709 12:26:18.154908 28224 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.740s	user 0.010s	sys 0.010s
I20260709 12:26:18.155369 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.155649 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:18.156029 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.156154 28518 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:26:18.156966 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.157251 18024 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:18.157663 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.140368 18020 rpcz_store.cc:276] 0709 12:26:16.924571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.924708 (+   137us) service_pool.cc:224] Handling call
0709 12:26:17.000228 (+ 75520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.000784 (+   556us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:17.550020 (+549236us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.987028 (+437008us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.043458 (+ 56430us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.050057 (+  6599us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:18.072369 (+ 22312us) log.cc:844] Serialized 26 byte log entry
0709 12:26:18.072616 (+   247us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:18.072672 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.072844 (+   172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.139277 (+ 66433us) raft_consensus.cc:1685] finished
0709 12:26:18.139304 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.139548 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108264,"prepare.queue_time_us":553648,"prepare.run_cpu_time_us":220,"prepare.run_wall_time_us":2607,"raft.queue_time_us":98927,"raft.run_cpu_time_us":647,"raft.run_wall_time_us":22554,"spinlock_wait_cycles":9859328,"thread_start_us":580220,"threads_started":2,"wal-append.queue_time_us":266103}
I20260709 12:26:18.158442 18018 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:18.170437 18165 rpcz_store.cc:276] 0709 12:26:16.755164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.866659 (+1111495us) service_pool.cc:224] Handling call
0709 12:26:17.982626 (+115967us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:18.001588 (+ 18962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.152980 (+151392us) spinlock_profiling.cc:243] Waited 7.99 ms on lock 0x7b540007bf54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:18.153263 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18381568}
I20260709 12:26:18.158921 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:18.174348 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52540.1us
W20260709 12:26:18.177843 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.181226 18016 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:18.173887 28039 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.182092 28224 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.182655 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.183408 18810 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:18.185101 28224 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.186353 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.187215 18093 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:18.188086 28340 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:18.189188 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.189407 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4055) took 1529 ms. Trace:
I20260709 12:26:18.182121 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.191259 18021 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:18.094405 28280 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.189908 18093 rpcz_store.cc:276] 0709 12:26:16.660435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.163899 (+503464us) service_pool.cc:224] Handling call
0709 12:26:17.164426 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.187679 (+1023253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.189061 (+  1382us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:18.189338 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2350900608}
I20260709 12:26:18.199112 18164 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:18.182124 28039 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.207835 28385 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.208714 28385 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.211503 28385 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:18.212924 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 1.25554e+06us
W20260709 12:26:18.214105 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.215202 28280 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.218036 28404 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:18.218777 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.219457 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.219084 28404 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.220067 28039 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.221800 28429 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:18.222436 28456 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.222517 28429 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.223152 28456 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.223436 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.224347 18813 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:18.226263 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.226379 28340 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.179352 28471 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.228269 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.228135 28471 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.229344 18093 raft_consensus.cc:2393] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:18.227335 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.221884 28359 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:18.231055 28359 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.225644 28456 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.227051 28340 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.235816 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.236088 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.237178 18808 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 31: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:18.225530 28429 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.238391 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.238595 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.239415 18097 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:18.239873 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3944) took 1460 ms. Trace:
I20260709 12:26:18.240418 18169 rpcz_store.cc:276] 0709 12:26:16.779192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.101619 (+1322427us) service_pool.cc:224] Handling call
0709 12:26:18.205488 (+103869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.239470 (+ 33982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:18.239500 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:18.239570 (+    70us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:18.239607 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.239674 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.239803 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:18.241242 18017 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
W20260709 12:26:18.242496 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 100109us
I20260709 12:26:18.244536 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.245784 18812 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 31: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:18.244813 18170 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:26:18.246532 18166 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:18.204509 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3686) took 1405 ms. Trace:
I20260709 12:26:18.247046 18164 rpcz_store.cc:276] 0709 12:26:16.798651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.161955 (+1363304us) service_pool.cc:224] Handling call
0709 12:26:18.165339 (+  3384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.204045 (+ 38706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.204407 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:18.246409 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.258446 18021 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:26:18.272436 28505 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.553s	user 0.000s	sys 0.005s
W20260709 12:26:18.274168 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.274935 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.274725 28215 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:18.275513 28215 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:18.276903 28215 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.279031 28215 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:18.280215 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3937) took 1566 ms. Trace:
I20260709 12:26:18.197304 28522 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.108s
I20260709 12:26:18.280696 18168 rpcz_store.cc:276] 0709 12:26:16.713493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.051060 (+337567us) service_pool.cc:224] Handling call
0709 12:26:17.051667 (+   607us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.052303 (+   636us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.052350 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.053130 (+   780us) log.cc:844] Serialized 26 byte log entry
0709 12:26:17.422437 (+369307us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:17.580086 (+157649us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.580754 (+   668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.580918 (+   164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.271596 (+690678us) spinlock_profiling.cc:243] Waited 7.26 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:18.279581 (+  7985us) raft_consensus.cc:1685] finished
0709 12:26:18.279727 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.279985 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":205024,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":110,"raft.queue_time_us":29389,"raft.run_cpu_time_us":462,"raft.run_wall_time_us":463,"spinlock_wait_cycles":16701696,"thread_start_us":358572,"threads_started":1,"wal-append.queue_time_us":652479}
I20260709 12:26:18.264107 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.286247 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.287215 18097 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:18.287922 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.288712 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.289737 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.290292 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.291019 18015 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:18.291738 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.289903 18023 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:18.292696 18018 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 30: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:18.293313 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.294299 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.295279 18023 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:18.295436 18016 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:18.296247 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.286310 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.297473 18815 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:18.298247 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.286621 18808 raft_consensus.cc:2410] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:26:18.298857 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.299058 18166 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:18.293648 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.298635 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.300359 28431 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.300849 18807 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:18.301383 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:18.302253 18169 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:18.303227 28404 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.303898 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:18.304432 18017 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:18.307299 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.307771 28340 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:18.309079 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.309430 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.310398 28471 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.310992 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.311842 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.326676 18813 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:18.334872 27938 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:18.341041 18168 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:18.350098 18815 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:18.352365 27942 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:18.354451 27942 consensus_queue.cc:1059] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:18.356241 28521 raft_consensus.cc:3055] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:18.356700 28521 raft_consensus.cc:740] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:18.359222 28359 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.359702 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.360404 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.361279 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.361567 18097 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:18.362120 18099 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
I20260709 12:26:18.359251 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:18.363721 18095 raft_consensus.cc:2376] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25. Re-sending same reply.
I20260709 12:26:18.360765 18094 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:18.365993 18093 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 22.
I20260709 12:26:18.366400 28296 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:18.367012 18813 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:26:18.368474 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.369091 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:18.360774 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.373466 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.373967 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3941) took 1623 ms. Trace:
I20260709 12:26:18.374238 18806 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:18.374414 18171 rpcz_store.cc:276] 0709 12:26:16.750817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.822696 (+1071879us) service_pool.cc:224] Handling call
0709 12:26:17.823324 (+   628us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.886228 (+ 62904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.886265 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.886349 (+    84us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:17.886396 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.373536 (+487140us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b540029e0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:18.373590 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.373738 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139559296}
I20260709 12:26:18.377394 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3942) took 1607 ms. Trace:
I20260709 12:26:18.373229 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4693) took 1443 ms. Trace:
I20260709 12:26:18.342007 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.378346 28234 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.381481 18164 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:18.374531 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.369374 18813 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:18.371384 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.372287 18811 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:18.372921 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.387806 28521 consensus_queue.cc:260] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.323235 28462 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.390838 28234 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.393510 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.380281 28448 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:18.391902 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.380589 18172 rpcz_store.cc:276] 0709 12:26:16.770225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.858907 (+ 88682us) service_pool.cc:224] Handling call
0709 12:26:18.376924 (+1518017us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b54002026d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:18.377172 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103350656}
I20260709 12:26:18.398458 18812 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:18.384636 18808 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:18.385725 18810 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:18.398862 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:18.393652 28234 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.396342 28423 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:18.413208 28064 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.414242 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.415208 18815 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:18.416922 18815 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:18.419437 18093 raft_consensus.cc:2410] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 25 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:26:18.399533 18063 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:18.425635 28279 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.394532 18136 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:18.428961 18169 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:18.373008 28436 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:18.436584 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47497.3us
I20260709 12:26:18.380815 18019 rpcz_store.cc:276] 0709 12:26:16.929565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.929842 (+   277us) service_pool.cc:224] Handling call
0709 12:26:16.930333 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.930667 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:17.350212 (+419545us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.371939 (+ 21727us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.144258 (+772319us) spinlock_profiling.cc:243] Waited 9.25 ms on lock 0x7b180012f1c0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:26:18.265040 (+120782us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:18.372607 (+107567us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:18.372720 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.372854 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.373018 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83680,"prepare.queue_time_us":596606,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":1238,"raft.queue_time_us":3475,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":24038,"spinlock_wait_cycles":21278080,"thread_start_us":399623,"threads_started":2}
I20260709 12:26:18.391923 28521 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:18.398315 28473 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.453229 18018 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:18.473804 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4860) took 1644 ms. Trace:
I20260709 12:26:18.477015 28279 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.438479 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:18.459317 28234 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:26:18.462013 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.470942 17914 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 18 to 20, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 20 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:18.426684 28462 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.477367 18092 rpcz_store.cc:276] 0709 12:26:16.829097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.924269 (+1095172us) service_pool.cc:224] Handling call
0709 12:26:17.924937 (+   668us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:18.122231 (+197294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.473395 (+351164us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400291b54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:18.473701 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":451521280}
I20260709 12:26:18.476712 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:18.458413 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64015.1us
I20260709 12:26:18.481648 18024 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:18.475759 17989 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:18.486735 18807 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:18.493177 18098 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:18.495942 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.474080 28064 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:18.496191 18063 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:18.499106 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.500198 18815 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:18.512205 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:18.519011 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.476022 28325 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:26:18.514485 18809 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:18.523295 27949 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:18.524031 27949 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.530107 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:18.531656 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:18.532634 28467 raft_consensus.cc:2764] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:26:18.533779 28467 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:18.534267 28467 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:18.534359 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58832.2us
I20260709 12:26:18.534782 28467 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:18.537618 27822 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.538347 27822 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.495720 28436 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.542040 28274 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:18.545157 28436 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:18.534544 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102664us
I20260709 12:26:18.547988 17987 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:18.503849 28234 raft_consensus.cc:697] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:18.550981 28234 consensus_queue.cc:237] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.521955 18809 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:18.546008 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 2.5933e+06us
I20260709 12:26:18.476356 28473 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.506839 28279 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:18.483176 28377 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.524575 17914 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 14 to 18, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 18 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:18.584950 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.510288 28064 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:18.592732 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.593125 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:18.595791 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 85749.8us
I20260709 12:26:18.598452 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.602383 18169 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
W20260709 12:26:18.604460 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.589270 27822 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:18.601069 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.607749 18019 raft_consensus.cc:1240] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:18.569594 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 57533.8us
I20260709 12:26:18.599259 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:18.610524 18166 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:26:18.607080 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.609412 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:18.613011 18172 raft_consensus.cc:3055] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:18.613400 18172 raft_consensus.cc:740] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:18.614543 18172 consensus_queue.cc:260] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.615759 18172 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:18.618503 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.606139 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.621471 18021 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 25. Ops: []
I20260709 12:26:18.621753 18024 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 33: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:18.618017 28436 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.624745 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.625276 28436 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.625805 18165 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:18.626304 27822 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:18.627987 28436 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.629232 27949 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:18.630156 28448 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:18.625656 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69383.9us
I20260709 12:26:18.633950 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:18.634418 27564 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:18.635051 28280 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:18.637029 18173 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 35. Current term is 36. Ops: []
I20260709 12:26:18.631513 28467 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:18.638062 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.638276 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:18.638935 18807 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:18.640245 28280 consensus_queue.cc:1048] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:18.640918 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:18.642601 28530 log.cc:927] Time spent T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:26:18.643033 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:18.644443 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.650755 28346 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:18.651428 28346 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.651830 28346 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:18.651671 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:18.653126 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:18.650635 27938 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:18.653949 18169 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 30 index: 24. (index mismatch)
I20260709 12:26:18.654263 18165 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:18.655246 28431 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.655403 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.655882 28431 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.597018 28404 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:18.622123 18164 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:26:18.597368 28522 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:18.651696 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:18.664422 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: true dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:18.669206 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.562497 28296 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:26:18.586889 28397 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:26:18.673940 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.674302 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.608948 28279 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:18.621259 18809 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.675069 18168 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:18.683210 28397 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:18.683843 28397 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.684244 28397 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:18.684348 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.686730 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.687333 28473 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.688112 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.688638 28064 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.689420 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.690608 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:18.694008 18814 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:18.663990 28404 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.692428 18809 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:18.640252 28467 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:18.671469 28465 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.055s
W20260709 12:26:18.708675 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 101606us
I20260709 12:26:18.727207 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:18.668378 28522 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.702044 28377 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.728603 28522 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:18.698799 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 47194.6us
I20260709 12:26:18.703004 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.731489 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.694943 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.714911 18811 raft_consensus.cc:1240] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:18.672676 28325 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
W20260709 12:26:18.695621 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.703994 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.765178 18815 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:18.698110 18806 raft_consensus.cc:2468] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:18.703495 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.712200 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:18.735589 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.732754 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.706557 18017 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:18.770982 18808 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:26:18.711014 18173 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:18.717167 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.710058 18023 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 22.
I20260709 12:26:18.715567 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.781772 18024 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:18.778613 18016 raft_consensus.cc:2393] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 33: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:18.782014 18813 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
W20260709 12:26:18.787930 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.718240 28431 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.703204 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:18.769326 27822 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:18.787338 18019 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
W20260709 12:26:18.808569 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95382.6us
I20260709 12:26:18.809841 18136 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:18.810159 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.810706 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.809125 18812 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:26:18.769920 28404 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:18.798628 17914 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 15 to 17, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 17 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:18.705015 28279 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.772212 28296 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:26:18.811923 18812 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:18.827353 27942 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:18.827941 27942 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:18.827598 18812 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.829152 18812 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:26:18.813567 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.814450 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.823854 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:18.830746 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.814673 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.815743 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.817615 28346 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.822451 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:18.823414 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.844619 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.841547 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.833217 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:18.845520 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.840083 18809 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 20. (term mismatch)
I20260709 12:26:18.846180 18809 pending_rounds.cc:85] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 19
W20260709 12:26:18.847538 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 64412.2us
I20260709 12:26:18.849984 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:18.836062 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:18.853104 18063 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:18.847189 28397 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.857656 28377 raft_consensus.cc:2804] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:18.858856 28397 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.851781 28280 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:18.859941 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:18.861047 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:18.862071 28465 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:18.862537 28465 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.862912 28465 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:18.863917 28535 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.063s	user 0.002s	sys 0.000s
I20260709 12:26:18.846465 18809 pending_rounds.cc:107] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 29.20
I20260709 12:26:18.866179 18809 raft_consensus.cc:2889] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:18.866890 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:18.839591 28234 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:18.868558 28234 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.868966 28234 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:18.854336 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.829058 27942 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.804555 28274 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:18.883488 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.883893 28346 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:18.885155 18020 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 35. Current term is 36. Ops: []
I20260709 12:26:18.885753 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:18.886554 18021 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:18.886431 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.880057 28467 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:18.880280 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.880889 27949 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:18.888417 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:18.887571 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.887694 18015 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:18.892216 18811 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:18.892966 28521 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:18.893638 28521 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:18.894698 28521 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:18.896186 28521 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:18.898777 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.900926 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.903139 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.904508 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.904742 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.906337 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.908022 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.908537 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.909138 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:18.879747 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:18.909363 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.909262 18809 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:19.026355 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 164037us
W20260709 12:26:18.972417 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.016153 18812 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 36 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 23. (term mismatch)
I20260709 12:26:19.037798 18812 pending_rounds.cc:85] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 22
I20260709 12:26:19.038252 18812 pending_rounds.cc:107] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 35.23
I20260709 12:26:19.038610 18812 raft_consensus.cc:2889] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:18.984185 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:19.039918 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 55907.6us
I20260709 12:26:18.996506 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.954129 18172 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:18.914261 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.035938 18808 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:18.914880 18019 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:18.997190 18022 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
W20260709 12:26:18.996084 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.925086 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 75311.6us
I20260709 12:26:19.018695 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:19.087239 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.113080 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 233398us
W20260709 12:26:19.074787 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.061859 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.118616 18021 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:26:19.120222 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:19.085624 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 299) took 35203 ms (client timeout 15000 ms). Trace:
I20260709 12:26:19.142833 28340 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:19.143491 28340 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.143913 28340 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.017241 28404 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.146911 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.063290 17914 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 12 to 18. New cstate: current_term: 18 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:19.148432 18815 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:19.047245 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:19.158028 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.159323 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.161039 18020 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:18.874305 28431 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:19.169003 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.919032 28280 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:19.164307 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.165622 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.923133 28471 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:19.162853 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 115788us
I20260709 12:26:19.163498 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:18.919711 28462 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:19.102032 28279 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:19.094652 27942 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:19.010764 28448 consensus_queue.cc:1059] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.157s
I20260709 12:26:18.919392 28467 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:19.183653 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:19.185392 18019 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
W20260709 12:26:19.185555 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.173527 28280 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:19.193552 28280 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:18.925143 28234 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.198122 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.199959 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.204469 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.064824 28473 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.119s
I20260709 12:26:19.224074 28234 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:19.207401 18022 raft_consensus.cc:1240] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:19.188994 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.211606 28404 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:19.212206 28471 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.212958 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:19.169008 28431 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.908499 28346 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.265s
I20260709 12:26:19.160589 28522 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.237890 28448 raft_consensus.cc:2764] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
W20260709 12:26:19.225975 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.207144 28280 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.216533 17914 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 13 to 16, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 16 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:19.125901 17920 rpcz_store.cc:269] 0709 12:25:43.887620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.564389 (+13676769us) service_pool.cc:224] Handling call
0709 12:26:19.085208 (+21520819us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:06.913453 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:06.922102 (+  8649us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:06.922157 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:26:06.922186 (+    29us) tablet.cc:662] Decoding operations
0709 12:26:06.925200 (+  3014us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:06.925314 (+   114us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:06.925361 (+    47us) tablet.cc:685] Acquiring locks for 28 operations
0709 12:26:06.931529 (+  6168us) tablet.cc:701] Row locks acquired
0709 12:26:06.931587 (+    58us) write_op.cc:260] PREPARE: finished
0709 12:26:06.931802 (+   215us) write_op.cc:270] Start()
0709 12:26:07.097596 (+165794us) write_op.cc:276] Timestamp: P: 1783599966931737 usec, L: 0
0709 12:26:07.111369 (+ 13773us) op_driver.cc:348] REPLICATION: starting
0709 12:26:07.171225 (+ 59856us) log.cc:844] Serialized 10936 byte log entry
0709 12:26:07.467773 (+296548us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c135ad0ac 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b
0709 12:26:08.348110 (+880337us) op_driver.cc:464] REPLICATION: finished
0709 12:26:08.755421 (+407311us) write_op.cc:301] APPLY: starting
0709 12:26:08.756181 (+   760us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:09.151275 (+395094us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:09.151324 (+    49us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:11.766581 (+2615257us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:11.767116 (+   535us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=56,key_file_lookups=56,delta_file_lookups=0,mrs_lookups=0
0709 12:26:11.767184 (+    68us) write_op.cc:312] APPLY: finished
0709 12:26:11.767861 (+   677us) log.cc:844] Serialized 244 byte log entry
0709 12:26:11.871523 (+103662us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:11.872181 (+   658us) write_op.cc:454] Released schema lock
0709 12:26:11.872319 (+   138us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:11.957979 (+ 85660us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:11.958043 (+    64us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5387683,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":354449,"apply.run_cpu_time_us":90756,"apply.run_wall_time_us":3155759,"cfile_cache_hit":114,"cfile_cache_hit_bytes":121335,"num_ops":28,"prepare.queue_time_us":416993,"prepare.run_cpu_time_us":47251,"prepare.run_wall_time_us":940181,"raft.queue_time_us":143989,"raft.run_cpu_time_us":10917,"raft.run_wall_time_us":378697,"replication_time_us":1206884,"spinlock_wait_cycles":13220224,"thread_start_us":1525541,"threads_started":5,"wal-append.queue_time_us":856990,"wal-append.run_cpu_time_us":18646,"wal-append.run_wall_time_us":2718751}]]}
I20260709 12:26:19.243001 18018 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 27. Ops: []
I20260709 12:26:19.231902 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.232689 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.234527 18812 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.172585 28542 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:19.235126 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:19.239177 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55638.5us
I20260709 12:26:19.189219 28279 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.242694 18023 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
W20260709 12:26:19.246001 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 80939.3us
I20260709 12:26:19.248903 18809 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:19.250727 18017 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 23: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:19.257433 18019 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:26:19.239306 28431 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:19.259565 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 248009us
I20260709 12:26:19.251751 28279 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:19.266183 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.247383 28542 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.272176 28423 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:26:19.273586 17914 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 12 to 24, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 24 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:19.256685 28234 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:19.269364 18098 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
W20260709 12:26:19.280268 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.287822 18092 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:19.290495 18094 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:26:19.295590 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.319808 18094 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:19.279050 18022 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:26:19.319480 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:19.277673 18808 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:19.323516 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.317951 28377 raft_consensus.cc:697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:19.330577 18095 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:19.330332 28377 consensus_queue.cc:237] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.332168 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.343703 18021 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:19.346905 18812 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:19.322690 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:19.277274 28542 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.352420 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.355154 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.358379 18098 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:19.283425 28234 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:19.360141 18093 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
W20260709 12:26:19.379870 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 117915us
I20260709 12:26:19.381072 18097 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:26:19.374441 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:19.389037 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.381489 18097 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:19.389717 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.393002 18063 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:19.394122 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.395157 18170 raft_consensus.cc:2393] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:19.360955 27564 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:19.395948 27564 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:19.396140 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:19.356860 28471 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:19.382678 18091 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:19.279934 28465 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.404695 18091 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:19.397059 27564 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.408851 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.375285 28521 raft_consensus.cc:3055] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:19.418716 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:19.414614 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.416363 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.379107 27949 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.424876 18093 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:19.423390 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.442385 18090 raft_consensus.cc:1240] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:19.443329 28280 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:19.444842 18090 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:19.295208 28467 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.450248 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.450692 18022 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:19.425328 18093 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:19.420284 28404 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:19.426708 18094 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.469126 18166 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:19.418716 28521 raft_consensus.cc:740] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:19.472487 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:19.474779 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.462360 18091 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.475965 18171 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:19.476152 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.481949 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.484107 18098 raft_consensus.cc:2468] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:19.486328 18097 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.495138 18097 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:26:19.496795 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.486521 18093 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.502195 28547 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:19.502961 18093 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:19.504752 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.487977 18812 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:26:19.483080 18168 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:19.468029 27949 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.490096 18172 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:19.492101 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.515353 18094 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:19.488118 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.493566 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:19.493712 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:19.516467 18092 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:19.489387 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:19.517192 18164 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
W20260709 12:26:19.510718 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.513327 18173 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:26:19.514683 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.481647 18090 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
I20260709 12:26:19.512104 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:19.530857 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:19.535476 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.468909 28522 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:19.541308 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 87928.1us
I20260709 12:26:19.540442 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:19.553153 18090 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:19.556671 28528 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.563612 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:19.567644 28064 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:26:19.568918 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.563534 28528 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.572257 28528 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:19.576622 18097 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 19. (term mismatch)
I20260709 12:26:19.577297 18097 pending_rounds.cc:85] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 18
I20260709 12:26:19.577673 18097 pending_rounds.cc:107] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 32.19
I20260709 12:26:19.578034 18097 raft_consensus.cc:2889] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.598198 18090 raft_consensus.cc:2468] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:19.599510 18099 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:19.600468 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.603082 18093 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:26:19.604276 18093 pending_rounds.cc:85] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 17
I20260709 12:26:19.604544 18093 pending_rounds.cc:107] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:26:19.604861 18093 raft_consensus.cc:2889] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.606407 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:19.606824 18090 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:19.607206 18095 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:19.538141 28547 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.508009 28340 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:19.535209 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 186173us
I20260709 12:26:19.613004 18091 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:19.614418 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.638128 18090 pending_rounds.cc:85] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 15
I20260709 12:26:19.638723 18090 pending_rounds.cc:107] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:26:19.639066 18090 raft_consensus.cc:2889] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.644111 18092 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:19.644977 18094 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 17. (term mismatch)
I20260709 12:26:19.647228 18094 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 16
I20260709 12:26:19.647765 18094 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:26:19.648092 18094 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:19.614619 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 367559us
W20260709 12:26:19.665638 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 267300us
W20260709 12:26:19.682723 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 164447us
W20260709 12:26:19.685173 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.687186 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:19.688663 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.689988 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.696712 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.697213 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.694664 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.539882 28234 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:19.706431 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.709056 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.711750 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 523119us
W20260709 12:26:19.715607 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 203526us
W20260709 12:26:19.717473 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.615166 28547 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.722772 18093 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:19.723325 18093 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:19.725540 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.725319 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.726363 18022 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:19.727717 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:19.728560 18808 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:19.733121 18812 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:26:19.726236 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:19.741410 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:19.742257 18092 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:19.744969 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146479us
I20260709 12:26:19.745080 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:19.746073 18809 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 33.
W20260709 12:26:19.750870 18015 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.001s
I20260709 12:26:19.753301 18015 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
W20260709 12:26:19.726047 18064 connection.cc:256] RPC call timeout handler was delayed by -2.5340942220000215s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
I20260709 12:26:19.754317 18023 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:19.754621 18064 connection.cc:256] RPC call timeout handler was delayed by -2.2631942220000383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20830000000000004s for a response
W20260709 12:26:19.755067 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1308942219998244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
W20260709 12:26:19.758212 18064 connection.cc:256] RPC call timeout handler was delayed by -2.1308942219998244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
W20260709 12:26:19.758738 18064 connection.cc:256] RPC call timeout handler was delayed by -1.905411090999678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
W20260709 12:26:19.759164 18064 connection.cc:256] RPC call timeout handler was delayed by -1.905411090999678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
W20260709 12:26:19.751490 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 187991us
I20260709 12:26:19.680063 28521 consensus_queue.cc:260] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.761706 18093 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:19.767877 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.768890 18022 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:26:19.770310 18064 connection.cc:256] RPC call timeout handler was delayed by -1.520082420000108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
I20260709 12:26:19.782613 18808 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
W20260709 12:26:19.794443 18064 connection.cc:256] RPC call timeout handler was delayed by -1.520082420000108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260709 12:26:19.796591 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 53583.8us
I20260709 12:26:19.741338 28465 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:19.750883 28555 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.795548 28551 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:19.803923 18022 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:19.810164 28565 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:19.784299 28215 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:19.813175 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.809634 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:19.742699 28404 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:19.826227 18064 connection.cc:256] RPC call timeout handler was delayed by -1.465287268999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
W20260709 12:26:19.838078 18064 connection.cc:256] RPC call timeout handler was delayed by -1.465287268999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
W20260709 12:26:19.838503 18064 connection.cc:256] RPC call timeout handler was delayed by -1.4157012480000049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20260709 12:26:19.838881 18064 connection.cc:256] RPC call timeout handler was delayed by -1.269987269000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
W20260709 12:26:19.839211 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2339872689999538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260709 12:26:19.839493 18064 connection.cc:256] RPC call timeout handler was delayed by -1.2339872689999538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260709 12:26:19.839749 18064 connection.cc:256] RPC call timeout handler was delayed by -1.169187268999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260709 12:26:19.840006 18064 connection.cc:256] RPC call timeout handler was delayed by -1.169187268999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260709 12:26:19.840250 18064 connection.cc:256] RPC call timeout handler was delayed by -1.1277872690002368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
I20260709 12:26:19.801964 28521 raft_consensus.cc:3060] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:19.848650 18093 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:19.826169 18092 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:19.855051 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.857146 18099 raft_consensus.cc:1240] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:19.858688 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:19.855953 18022 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 24.
W20260709 12:26:19.862887 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.863485 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.831547 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:19.817451 28234 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:19.866111 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:19.871613 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.871657 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:19.873303 18815 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 36.
I20260709 12:26:19.848795 18095 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:19.858523 28215 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.875973 28215 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:19.832940 28340 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:19.866219 27949 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:19.878469 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.523s (SENT)
I20260709 12:26:19.878298 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:19.878392 28473 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:19.879243 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:19.880311 28340 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:19.881747 28541 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.882404 28541 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:19.883955 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.885232 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.885542 18021 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:26:19.886045 18018 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:19.886163 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.834136 28555 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:19.891659 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.894300 28546 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 12:26:19.896510 18169 raft_consensus.cc:2410] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 29 because replica has last-logged OpId of term: 28 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260709 12:26:19.899066 18018 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:19.855667 27822 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.830658 28565 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.885543 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.919616 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:19.848979 28522 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:19.849186 28471 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:19.922714 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:19.934113 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.935444 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:19.918594 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87372.1us
I20260709 12:26:19.931536 18017 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:19.929179 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.941803 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 222723us
I20260709 12:26:19.948829 18022 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:26:19.952245 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:19.925791 28522 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.927070 28565 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.973562 18136 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:19.955401 28215 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:19.963372 28340 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:19.950771 28548 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:19.940896 28567 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:19.987112 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.980576 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 122008us
I20260709 12:26:19.986462 18093 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 15. (term mismatch)
W20260709 12:26:19.987957 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 128938us
I20260709 12:26:19.978993 18136 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:19.987929 18093 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:19.970902 28522 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:20.002198 18093 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 29.15
I20260709 12:26:20.004218 18022 raft_consensus.cc:1275] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:20.008014 18093 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:20.016394 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:20.017174 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 151191us
W20260709 12:26:19.979206 28279 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.707s	user 0.013s	sys 0.002s
I20260709 12:26:20.020328 18022 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:20.022003 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:19.955920 27822 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:20.027276 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48394.6us
I20260709 12:26:20.001569 28471 raft_consensus.cc:2804] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:26:20.031275 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.033071 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:19.985616 28567 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:19.986013 28548 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.050575 28548 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:20.034615 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.978711 28561 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
W20260709 12:26:20.041643 28572 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:26:20.055620 28234 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:20.055938 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.056331 28234 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:20.058929 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:20.044075 28567 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:20.045894 28566 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:26:20.037557 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 94233.1us
I20260709 12:26:20.077397 18094 raft_consensus.cc:1240] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:20.058041 28215 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:20.060345 18020 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:20.064417 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:20.079986 18093 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 26
W20260709 12:26:20.064915 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.065371 28574 raft_consensus.cc:2804] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:20.065624 28563 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:20.066227 28280 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:26:20.067049 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:20.086156 18098 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
W20260709 12:26:20.068648 28542 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.000s
W20260709 12:26:20.070154 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.072027 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:20.078639 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.069675 28462 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.880s	user 0.013s	sys 0.000s
I20260709 12:26:20.087749 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:20.104682 18808 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:20.088560 28574 raft_consensus.cc:697] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:20.106824 28574 consensus_queue.cc:237] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.074158 28529 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:26:20.036098 28465 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:20.037503 28547 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:20.080925 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.112157 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:20.113801 18093 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:20.118988 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:20.122099 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.124522 27942 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.890s	user 0.016s	sys 0.000s
I20260709 12:26:20.137111 28215 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:20.126060 18091 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:26:20.142820 18091 pending_rounds.cc:85] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 17
I20260709 12:26:20.082751 28563 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.073722 28541 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:20.143167 18091 pending_rounds.cc:107] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:26:20.148224 18091 raft_consensus.cc:2889] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:20.081447 28471 raft_consensus.cc:697] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:20.027027 28279 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.127676 28346 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:20.143842 18016 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
W20260709 12:26:20.130054 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51920.2us
I20260709 12:26:20.139972 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:20.135035 28462 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:20.113924 28542 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.145480 28215 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:20.103360 27949 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.057592 28234 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.134444 28529 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.030346 28404 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.160310 18136 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:20.162159 18780 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:20.149062 28563 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:20.150841 28377 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:20.166479 18093 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:20.151619 28556 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:20.176362 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 154568us
I20260709 12:26:20.167156 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:20.170097 18015 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:20.143821 28465 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.161682 28529 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:26:20.166900 28547 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:20.163858 28462 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:20.177567 18022 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:20.180244 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:20.171633 28377 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.164848 27949 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.161877 28346 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.178049 28340 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:20.174579 28280 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260709 12:26:20.205715 28279 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:20.208537 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47637.1us
I20260709 12:26:20.223785 18809 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260709 12:26:20.224784 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.183091 28556 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.184329 28462 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.178983 28215 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.190862 28555 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:20.193269 28551 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:26:20.235185 28462 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:20.225291 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:20.218160 28467 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:20.197584 28547 raft_consensus.cc:697] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:20.233587 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:20.225733 18806 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:26:20.237578 18091 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:20.237720 18812 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:20.242031 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.244046 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 66398.1us
I20260709 12:26:20.191535 28377 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:20.249368 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.260372 27949 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:20.262081 27822 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:20.263412 28547 consensus_queue.cc:237] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:20.268047 28579 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:26:20.270215 28556 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:20.271209 28522 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.272310 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:20.273084 18815 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:20.274111 18017 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34. Ops: []
I20260709 12:26:20.274716 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:20.275084 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:20.254923 28541 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 12:26:20.279770 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.286120 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:20.296335 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.315282 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.320113 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.334106 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88032.5us
W20260709 12:26:20.315119 18780 connection.cc:256] RPC call timeout handler was delayed by -1.2161883929998112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32620000000000005s for a response
W20260709 12:26:20.337602 18780 connection.cc:256] RPC call timeout handler was delayed by -1.1576883930001713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3327s for a response
I20260709 12:26:20.340307 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.238481 27564 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.069s
I20260709 12:26:20.287936 28556 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:20.351437 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:20.297674 28064 raft_consensus.cc:2764] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:26:20.324995 28431 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.029s	user 0.008s	sys 0.011s
I20260709 12:26:20.384258 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.348949 18812 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:20.306428 28555 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:20.317894 28522 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:20.322173 28340 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:20.237183 28471 consensus_queue.cc:237] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.387051 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:20.401425 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.352236 28556 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.388820 28555 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.421993 28280 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:20.358016 28589 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:20.391911 28431 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:20.429190 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 151453us
W20260709 12:26:20.417936 28569 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:26:20.358582 27822 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:26:20.418569 28556 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:20.443921 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 59822.9us
I20260709 12:26:20.411420 28586 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:20.412700 28542 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:20.451424 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.428548 28555 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:20.454412 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1520) took 1561 ms (client timeout 1670 ms). Trace:
I20260709 12:26:20.422216 28576 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:20.455191 18022 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:20.437342 28064 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:26:20.461777 18016 rpcz_store.cc:269] 0709 12:26:18.881807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.888073 (+  6266us) service_pool.cc:224] Handling call
0709 12:26:20.436912 (+1548839us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:20.442808 (+  5896us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2884604672}
I20260709 12:26:20.426179 28280 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.468796 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:20.476251 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:20.448143 28586 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:20.479164 28561 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:26:20.480568 28581 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:26:20.481925 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:20.435002 28565 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:20.514607 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.515617 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.516180 18815 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:20.525230 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:20.534850 18812 raft_consensus.cc:2368] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30.
W20260709 12:26:20.489153 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.464174 28551 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.066s
I20260709 12:26:20.483825 28064 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:20.548377 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.526011 28586 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:20.544246 28582 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.285s	user 0.002s	sys 0.000s
W20260709 12:26:20.554813 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80085.7us
I20260709 12:26:20.562589 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.557773 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:20.570021 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:20.571148 18016 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:26:20.573467 28064 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.578651 28565 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:20.569662 28577 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:20.582620 28577 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.476363 28576 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.582739 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.589553 18019 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:20.594524 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.585508 28577 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:20.603852 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:20.596334 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.607956 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:20.608757 18094 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:26:20.554720 28465 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:20.659224 18096 raft_consensus.cc:3055] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:20.604939 18019 raft_consensus.cc:2376] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26. Re-sending same reply.
I20260709 12:26:20.487082 28340 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.661289 18096 raft_consensus.cc:740] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:20.663057 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 200591us
I20260709 12:26:20.666762 28601 raft_consensus.cc:2764] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:20.617875 18815 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 34.
I20260709 12:26:20.670797 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.614064 28589 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:20.714418 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.714617 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.715189 18019 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:20.719321 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 156817us
I20260709 12:26:20.721630 18064 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.725728 18094 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:20.727658 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:20.734836 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.724265 28584 log.cc:927] Time spent T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.262s	user 0.003s	sys 0.012s
I20260709 12:26:20.745139 18090 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:20.763554 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.729658 18096 consensus_queue.cc:260] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.793421 28279 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:26:20.781893 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:20.835404 18096 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:20.782538 28590 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.300s	user 0.002s	sys 0.000s
I20260709 12:26:20.782757 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:20.783006 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:20.842324 18092 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:20.786522 28588 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
W20260709 12:26:20.788080 28597 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
W20260709 12:26:20.789326 28603 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 12:26:20.790760 28548 consensus_meta.cc:287] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.739s	user 0.010s	sys 0.009s
W20260709 12:26:20.791338 28575 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.662s	user 0.005s	sys 0.000s
I20260709 12:26:20.866705 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:20.792186 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:20.793489 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:20.868615 18815 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:20.812897 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.745343 28577 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:20.825290 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.873854 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.874547 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:20.875152 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:20.875931 18023 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:20.742125 28431 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:20.756892 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.844509 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 129997us
I20260709 12:26:20.878875 18016 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:20.846387 18092 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:20.830027 28462 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.594s	user 0.016s	sys 0.000s
I20260709 12:26:20.815249 28589 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:20.886004 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.871163 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.871282 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:20.877606 18022 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [23.17-25.18]
I20260709 12:26:20.875339 18019 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:26:20.844379 18094 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:20.879988 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:20.881387 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:20.860805 28377 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.007s
I20260709 12:26:20.895076 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:20.880925 28606 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.926621 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:20.928689 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:20.929471 18170 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:20.929906 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 58799us
I20260709 12:26:20.887847 28462 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.880048 27949 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.879642 28280 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:20.933907 18166 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:20.934355 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.939369 18815 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
W20260709 12:26:20.934960 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55062.2us
I20260709 12:26:20.941730 17988 leader_election.cc:400] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30.
I20260709 12:26:20.942399 17988 leader_election.cc:403] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:26:20.893096 28567 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.809s	user 0.013s	sys 0.005s
I20260709 12:26:20.943392 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:20.944222 18815 raft_consensus.cc:2368] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34.
I20260709 12:26:20.944105 18813 raft_consensus.cc:1217] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Deduplicated request from leader. Original: 26.20->[28.21-28.21]   Dedup: 28.21->[]
I20260709 12:26:20.944645 18813 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:26:20.945312 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:20.872895 28548 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:20.946185 18172 raft_consensus.cc:2376] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30. Re-sending same reply.
W20260709 12:26:20.946978 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53388.8us
I20260709 12:26:20.889271 28577 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:20.947912 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:20.949522 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:20.932440 28563 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.747s	user 0.014s	sys 0.001s
W20260709 12:26:20.932909 28574 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.010s	sys 0.000s
I20260709 12:26:20.951934 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:21.002290 18809 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:20.955355 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:20.931937 28606 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.005870 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.006767 18172 raft_consensus.cc:2368] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30.
I20260709 12:26:20.935441 27942 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:26:21.001993 18780 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.670s (SENT)
W20260709 12:26:21.009933 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52716.1us
I20260709 12:26:21.014299 18812 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34. Ops: []
I20260709 12:26:20.894618 28551 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.105s
I20260709 12:26:21.016966 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.018187 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.018869 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.019886 18172 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260709 12:26:21.022614 18810 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:21.023414 28602 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:21.024823 28570 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:21.025676 28570 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:21.034225 28607 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
W20260709 12:26:21.039158 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.888613 28555 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.056689 18164 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:20.892035 28556 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:20.956243 28522 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30.
I20260709 12:26:21.073523 18170 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:21.070387 18165 raft_consensus.cc:2393] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:20.930681 28377 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.001751 28574 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.011s	sys 0.000s
I20260709 12:26:21.076510 28616 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.124s
I20260709 12:26:21.069794 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:21.079716 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.104153 18169 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:21.084895 28346 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.087467 28577 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.091456 28570 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:20.951318 28567 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.096602 28548 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:21.099640 28556 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.041193 28576 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.127413 28555 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:21.109292 18172 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: [30.17-30.17]
I20260709 12:26:20.892203 28589 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.126191 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:21.127436 28606 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:21.137185 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.148757 18096 raft_consensus.cc:1275] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 15. (term mismatch)
I20260709 12:26:20.951931 27949 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.103101 18095 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:21.155239 18780 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.827s (SENT)
I20260709 12:26:21.166473 18170 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31. Re-sending same reply.
I20260709 12:26:21.166716 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:21.159536 18167 raft_consensus.cc:1240] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:21.108417 27942 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:21.164039 18096 pending_rounds.cc:85] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:21.170683 18096 pending_rounds.cc:107] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 28.15
I20260709 12:26:21.171124 18096 raft_consensus.cc:2889] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:21.150321 28581 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.259s	user 0.005s	sys 0.000s
I20260709 12:26:21.175590 28377 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.167848 18173 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:21.172432 27942 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:21.006892 28563 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.163166 28576 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:21.187387 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 239578us
I20260709 12:26:21.189064 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:21.189767 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 468223us
I20260709 12:26:21.190092 18170 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:21.181734 28551 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:26:21.192896 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.884s (SENT) [suppressed 54 similar messages]
W20260709 12:26:21.212702 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 269349us
W20260709 12:26:21.244081 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.216012 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:21.259097 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 118694us
I20260709 12:26:21.258853 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:21.266090 18780 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.961s (SENT)
I20260709 12:26:21.266786 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:21.269552 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78694.2us
W20260709 12:26:21.272364 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.212885 28529 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.022s	user 0.015s	sys 0.003s
I20260709 12:26:21.283252 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:21.283088 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:21.286624 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 181418us
I20260709 12:26:21.289582 18064 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:21.277933 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 62059.7us
W20260709 12:26:21.296432 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.296381 28431 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:21.298195 28606 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:26:21.302435 28548 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:26:21.303256 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.284174 28555 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:21.304849 28555 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.305250 28555 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:21.280063 28465 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.306286 28465 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.207409 28576 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.210378 28586 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.637s	user 0.013s	sys 0.002s
I20260709 12:26:21.299576 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:21.299762 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:21.311098 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 45072.4us
W20260709 12:26:21.311887 18780 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 1.984s (SENT)
I20260709 12:26:21.312307 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:21.313351 18780 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Timed out: RequestConsensusVote RPC to 127.16.102.3:39905 timed out after 2.074s (SENT)
I20260709 12:26:21.313799 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.316691 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:21.329120 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.297508 28462 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:21.296618 28556 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:21.384506 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84831.9us
W20260709 12:26:21.385187 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 85685.2us
W20260709 12:26:21.384639 18023 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.508s	user 0.017s	sys 0.007s
I20260709 12:26:21.388782 28617 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:21.386709 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47851.3us
I20260709 12:26:21.388787 18023 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:21.315707 28551 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:21.314778 28340 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:21.392776 18138 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.394711 18137 leader_election.cc:400] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Vote denied by peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307) with higher term. Message: Invalid argument: T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34.
I20260709 12:26:21.389220 28626 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:21.402746 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 187666us
W20260709 12:26:21.405678 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84148.1us
I20260709 12:26:21.396179 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:21.337663 28605 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:26:21.382141 28608 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.288460 28529 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.410112 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.410631 28462 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:21.410966 28548 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:21.411384 27949 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.411878 28471 consensus_queue.cc:1048] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:26:21.412495 28576 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.393952 28467 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:21.387965 28556 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:21.406926 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:21.420131 18098 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:26:21.431960 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:21.299522 27942 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.433954 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.449457 18780 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:21.317318 28619 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.108s
I20260709 12:26:21.428858 28617 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:26:21.466921 28529 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 37 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:21.442222 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.468194 28589 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:21.429355 28608 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.446954 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:21.469974 28604 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:21.471231 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.471616 28560 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:21.472426 28560 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:21.474013 28623 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:21.474689 28628 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:26:21.475437 28563 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.476981 28608 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:21.463723 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:21.454227 18091 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:21.476694 28340 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:21.451189 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:21.417245 28567 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:21.451789 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:21.463404 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 56586.5us
I20260709 12:26:21.488669 18092 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 30
W20260709 12:26:21.465917 28625 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:26:21.447158 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:21.491185 18094 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:21.393685 28215 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:26:21.479564 18090 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:21.480698 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:21.494278 18809 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:26:21.494016 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 103967us
I20260709 12:26:21.319576 28586 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.398628 28547 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.048s
I20260709 12:26:21.485284 18811 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:21.463104 27949 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.495291 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:21.499217 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:21.506897 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52226.3us
I20260709 12:26:21.499209 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:21.463218 28556 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:21.508158 18091 raft_consensus.cc:2410] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 31 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:26:21.482419 28529 raft_consensus.cc:2764] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:26:21.478717 28551 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.514487 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:21.509418 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.516465 18094 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:21.518399 18092 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:21.538050 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:21.417621 28626 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:21.522037 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 129984us
I20260709 12:26:21.520181 18808 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:21.538604 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.518772 18090 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:21.500952 28615 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:21.540779 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:21.543974 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:21.547122 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.446612 28574 consensus_queue.cc:1048] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:26:21.490630 28563 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:26:21.551476 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:21.554636 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:21.563751 18091 raft_consensus.cc:2410] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 31 because replica has last-logged OpId of term: 29 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 22.
W20260709 12:26:21.553175 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.500s (SENT)
W20260709 12:26:21.564469 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.564731 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:21.473762 28560 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:21.566380 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77088.7us
I20260709 12:26:21.500577 28340 raft_consensus.cc:697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:21.570142 18809 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 37.
I20260709 12:26:21.561144 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.541873 28608 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:21.537494 28589 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:21.543864 28561 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:21.514106 28606 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.067s
I20260709 12:26:21.607401 18094 raft_consensus.cc:2468] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:21.542583 28615 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:21.613921 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60774.8us
I20260709 12:26:21.576400 28462 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:21.514680 27949 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.548666 28616 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:21.558069 28637 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.560531 28576 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:26:21.578855 28635 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:26:21.600131 28529 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:26:21.624158 28615 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:21.676124 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96386.9us
I20260709 12:26:21.600076 27822 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:26:21.626763 28462 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:21.684705 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 118834us
W20260709 12:26:21.693449 18778 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.636s (SENT)
I20260709 12:26:21.629138 28589 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:21.675238 28624 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.285s	user 0.000s	sys 0.002s
I20260709 12:26:21.655659 28563 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:26:21.682001 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.700459 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 149074us
I20260709 12:26:21.642695 28567 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.056s
W20260709 12:26:21.707759 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 146721us
I20260709 12:26:21.686408 28462 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:21.681974 28467 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:21.710670 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:21.686650 28215 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.692736 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:21.695472 28576 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:21.724290 18138 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.724846 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:21.730502 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50282.1us
I20260709 12:26:21.734368 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:21.736698 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:21.739952 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:21.740942 18812 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:21.742753 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.728664 28563 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:21.745263 28586 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:21.751223 28639 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:21.750857 28563 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.753908 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 215054us
W20260709 12:26:21.754573 28631 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.260s	user 0.002s	sys 0.000s
I20260709 12:26:21.755115 28589 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.757565 28644 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:21.540324 28551 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.740007 27942 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
W20260709 12:26:21.747989 28622 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.293s	user 0.002s	sys 0.000s
I20260709 12:26:21.731482 28561 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:26:21.657308 28616 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:21.768785 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76183.2us
I20260709 12:26:21.705565 28340 consensus_queue.cc:237] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.738627 28215 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:21.780522 28467 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.792629 28215 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:21.666715 28637 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.783919 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.795763 28637 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:21.796149 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.784785 18812 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
W20260709 12:26:21.787288 18778 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.670s (SENT)
I20260709 12:26:21.798318 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.787001 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:21.773273 28644 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:21.808761 28377 raft_consensus.cc:2764] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
W20260709 12:26:21.820375 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.798067 28604 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:21.802853 28626 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:21.823233 28647 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:21.848846 28647 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:21.849294 28647 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:21.850232 18063 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:21.850617 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3958) took 1019 ms. Trace:
I20260709 12:26:21.851008 18173 rpcz_store.cc:276] 0709 12:26:20.831146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.832125 (+   979us) service_pool.cc:224] Handling call
0709 12:26:20.832534 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.181574 (+349040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.850329 (+668755us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b540002a3d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:21.850541 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":762590464}
I20260709 12:26:21.851666 28541 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:21.852257 28541 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:21.852398 28616 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:21.852797 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4098) took 1071 ms. Trace:
I20260709 12:26:21.853241 18095 rpcz_store.cc:276] 0709 12:26:20.781028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.781165 (+   137us) service_pool.cc:224] Handling call
0709 12:26:20.781516 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.166595 (+385079us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.852505 (+685910us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54002cae54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:21.852720 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":738813184}
I20260709 12:26:21.822770 18136 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:21.819548 28637 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:21.845520 28650 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:26:21.853404 28541 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.843482 28652 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.094s
I20260709 12:26:21.875111 28529 consensus_queue.cc:1059] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:21.843567 28642 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:21.876480 28522 raft_consensus.cc:3055] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:21.846326 28648 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:21.877000 28522 raft_consensus.cc:740] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:21.877349 28648 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:21.877912 28648 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:21.879190 28653 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.879895 28653 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.846091 28640 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:26:21.867070 28565 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.847584 27949 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:21.891151 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.878129 28522 consensus_queue.cc:260] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:21.907938 28565 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:21.911285 28600 log.cc:927] Time spent T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.187s	user 0.009s	sys 0.005s
W20260709 12:26:21.922885 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 135464us
W20260709 12:26:21.922374 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 126336us
W20260709 12:26:21.923139 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.931440 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:21.932294 18815 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:21.936280 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:21.942366 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 123601us
I20260709 12:26:21.944046 18136 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:21.925691 28522 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:21.945103 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 173067us
I20260709 12:26:21.950739 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:21.962551 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3956) took 1132 ms. Trace:
I20260709 12:26:21.990028 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:21.996311 18812 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:21.966848 28596 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:21.968585 18172 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3967) took 1112 ms. Trace:
I20260709 12:26:21.995731 18169 rpcz_store.cc:276] 0709 12:26:20.830297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.830472 (+   175us) service_pool.cc:224] Handling call
0709 12:26:20.849675 (+ 19203us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.104690 (+255015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.962205 (+857515us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400383954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:21.962468 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584728832}
W20260709 12:26:21.988297 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44350.5us
W20260709 12:26:21.949525 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 222676us
I20260709 12:26:21.971714 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 3975) took 1043 ms. Trace:
I20260709 12:26:22.000900 18165 rpcz_store.cc:276] 0709 12:26:20.931301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.954988 (+ 23687us) service_pool.cc:224] Handling call
0709 12:26:21.971415 (+1016427us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b54002026d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:21.971610 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87867264}
I20260709 12:26:21.999487 18172 rpcz_store.cc:276] 0709 12:26:20.856490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.020524 (+164034us) service_pool.cc:224] Handling call
0709 12:26:21.020906 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.157026 (+136120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.968227 (+811201us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b5400274cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:21.968487 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":202279168}
I20260709 12:26:22.006148 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:21.978423 28555 consensus_meta.cc:287] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.673s	user 0.020s	sys 0.007s
I20260709 12:26:22.017258 28633 consensus_queue.cc:1059] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:22.021605 28542 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:22.023526 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:22.022982 28644 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.055130 18098 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: [30.15-30.15]
I20260709 12:26:22.006031 28596 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.053156 18815 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
I20260709 12:26:22.028239 28562 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:21.962852 28565 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.029258 28657 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:22.063179 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:22.066545 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.033615 28601 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:22.034907 28473 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:26:22.063894 18095 raft_consensus.cc:1240] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: [30.17-30.17]
W20260709 12:26:21.975911 28623 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.501s	user 0.019s	sys 0.004s
I20260709 12:26:21.958117 28551 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.072532 18019 raft_consensus.cc:2393] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:22.078115 28657 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:22.078303 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.078754 28657 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:22.079149 18021 raft_consensus.cc:3060] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:22.079656 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.079821 28657 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.081286 28657 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.061926 28542 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.082137 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.083581 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.084595 28542 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.084892 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.085299 28562 raft_consensus.cc:697] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.086169 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:22.086617 28562 consensus_queue.cc:237] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.078310 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.089007 18015 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:22.090127 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.091337 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:21.977098 28609 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.114s	user 0.000s	sys 0.003s
I20260709 12:26:22.042788 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:22.045575 18093 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:22.046268 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:21.969148 28556 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.066766 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.064772 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:22.063060 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 112454us
I20260709 12:26:22.063956 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.068871 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.065177 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:22.008272 28560 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.033573 18811 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:26:22.069706 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.070734 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:21.951311 28642 raft_consensus.cc:2804] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:22.081349 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.085610 18024 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:22.134090 18019 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
W20260709 12:26:22.101342 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58441.5us
W20260709 12:26:22.088955 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.580s (SENT)
I20260709 12:26:22.136893 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:22.138058 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172952us
W20260709 12:26:22.139453 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.144129 18092 raft_consensus.cc:1240] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:26:22.143129 18019 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:26:22.147678 18092 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:22.149437 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:22.149145 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:22.063060 28662 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:22.155742 18022 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34. Ops: [33.21-33.21]
W20260709 12:26:22.156605 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.070317 28473 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.155841 18815 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:26:22.160406 18096 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:22.162839 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.097280 28623 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:22.165364 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:22.075309 28651 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.070514 28601 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:22.046316 28555 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.162907 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:22.163426 18016 raft_consensus.cc:3055] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:22.171784 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.173688 18016 raft_consensus.cc:740] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:22.158248 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69370.4us
I20260709 12:26:22.178042 18021 raft_consensus.cc:2468] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
W20260709 12:26:22.179468 18778 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.961s (SENT)
I20260709 12:26:22.143947 28528 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.151669 28551 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.159471 28644 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:26:22.196393 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.196393 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:22.157021 28642 raft_consensus.cc:697] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.199179 18091 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 23.
I20260709 12:26:22.182617 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.207402 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1565) took 1019 ms. Trace:
W20260709 12:26:22.228366 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45846.4us
I20260709 12:26:22.183362 18016 consensus_queue.cc:260] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.228257 18167 rpcz_store.cc:276] 0709 12:26:21.187880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.470885 (+283005us) service_pool.cc:224] Handling call
0709 12:26:22.198882 (+727997us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b5400031bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:22.207259 (+  8377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1222299392}
I20260709 12:26:22.228981 18169 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:22.230189 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:22.230865 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78915.5us
I20260709 12:26:22.231928 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:22.233101 28642 consensus_queue.cc:237] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.234102 28606 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.234902 28547 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:22.235538 28547 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.236129 28547 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.246184 28471 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:22.247503 28471 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:22.252206 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72803.2us
W20260709 12:26:22.253163 18778 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.965s (SENT)
I20260709 12:26:22.253338 28473 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:22.254243 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.978s (SENT)
I20260709 12:26:22.254930 28473 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:22.261494 28471 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.197077 28651 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.264627 28651 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.266253 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.181681 28565 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:22.278234 28565 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:22.278512 28340 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:22.279084 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.022s (SENT)
I20260709 12:26:22.211841 28528 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.279719 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.280646 28340 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:22.181500 28662 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.283538 28547 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.284579 28662 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.211324 28560 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:22.289654 28601 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.290992 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:22.291426 28643 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.511s	user 0.000s	sys 0.003s
I20260709 12:26:22.291718 28601 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.288335 18024 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:22.295910 28555 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.183233 28467 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:22.164613 28596 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:22.306572 28615 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.015s
I20260709 12:26:22.307262 28615 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:22.312315 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74493.1us
I20260709 12:26:22.314929 18018 raft_consensus.cc:3055] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:22.315431 18018 raft_consensus.cc:740] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:22.316327 18018 consensus_queue.cc:260] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.317714 18018 raft_consensus.cc:3060] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:22.320663 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:22.324791 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.326134 18136 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.199342 28563 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.328351 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:22.329012 18170 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:22.325982 18063 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:22.332152 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:22.335763 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.278785 28565 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:22.328339 28528 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:22.343446 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 147469us
I20260709 12:26:22.346362 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:22.346966 18093 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:22.347360 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.348196 18024 raft_consensus.cc:2393] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:22.328039 28560 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:22.371902 28528 consensus_queue.cc:1059] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:22.373944 28638 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:26:22.375052 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.376546 18094 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:22.381021 18022 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:22.386523 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:22.365409 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.852s (SENT)
I20260709 12:26:22.389377 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:22.385329 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.390980 18020 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:22.390061 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.336968 28574 raft_consensus.cc:2804] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:26:22.394345 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.394938 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:22.395622 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.337983 28639 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:22.351123 28462 consensus_meta.cc:287] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.615s	user 0.017s	sys 0.008s
I20260709 12:26:22.348718 28064 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.380223 28346 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:22.402974 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.338017 27942 consensus_queue.cc:1059] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:26:22.340312 27822 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:26:22.394318 28340 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.395051 28645 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:22.402766 28638 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:22.422021 28064 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:22.399528 28521 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 29 current term: 29 is higher.
W20260709 12:26:22.444057 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 212218us
W20260709 12:26:22.429386 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 266548us
W20260709 12:26:22.406739 28608 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.793s	user 0.020s	sys 0.006s
I20260709 12:26:22.261513 28648 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.349382 28652 consensus_queue.cc:1059] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:26:22.394001 28551 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:22.399981 28279 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:22.402177 28637 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:22.402587 28586 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:22.463625 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:22.466689 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.468686 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:22.478596 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.464772 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:22.481060 18137 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.408480 28574 raft_consensus.cc:697] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.446895 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:22.385596 28563 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:26:22.417290 28346 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:22.415298 28462 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.489243 18809 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:22.415534 28639 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.485587 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.453409 28615 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:22.455237 28608 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:22.496573 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49875.2us
I20260709 12:26:22.457775 28556 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.456794 28404 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:22.499857 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 105006us
I20260709 12:26:22.500388 17989 leader_election.cc:400] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30.
I20260709 12:26:22.478036 28645 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:22.440827 28340 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.470737 28551 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.474040 28279 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.479956 28586 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.505223 18098 raft_consensus.cc:1275] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 12:26:22.537279 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 80257.7us
I20260709 12:26:22.636088 18813 raft_consensus.cc:2410] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
W20260709 12:26:22.502933 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.052s (SENT)
I20260709 12:26:22.584188 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 2036) took 1294 ms. Trace:
I20260709 12:26:22.521507 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:22.558465 28669 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:26:22.678458 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.679299 18019 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.679435 18020 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:22.502628 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:22.680632 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.488895 28280 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:22.646359 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.503130 18137 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:22.680413 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:22.646931 18808 rpcz_store.cc:276] 0709 12:26:21.290227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.290455 (+   228us) service_pool.cc:224] Handling call
0709 12:26:21.304357 (+ 13902us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.539297 (+234940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.583765 (+1044468us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54004df654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:22.584058 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":495623040}
W20260709 12:26:22.569054 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.581154 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4909) took 1090 ms. Trace:
I20260709 12:26:22.650874 28639 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:22.654428 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.657673 28648 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:22.659014 28234 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.660610 28586 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:22.661981 28596 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.662530 28645 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.535708 28660 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:22.559530 28672 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:26:22.663704 28551 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.536792 28638 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:22.574849 28671 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:26:22.728469 18064 leader_election.cc:400] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25.
W20260709 12:26:22.681185 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 178657us
I20260709 12:26:22.687996 18096 rpcz_store.cc:276] 0709 12:26:21.490232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.490411 (+   179us) service_pool.cc:224] Handling call
0709 12:26:21.505004 (+ 14593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:22.162595 (+657591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.580690 (+418095us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400213c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:22.581048 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1506678528}
I20260709 12:26:22.681349 18020 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:22.689538 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 186614us
I20260709 12:26:22.491499 28595 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:22.578378 28647 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.728s	user 0.015s	sys 0.008s
I20260709 12:26:22.583962 28064 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.635347 28547 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:22.502137 28606 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:22.701920 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:22.507241 28471 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:26:22.519117 28563 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:26:22.717111 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.754197 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3961) took 1907 ms. Trace:
I20260709 12:26:22.754630 18168 rpcz_store.cc:276] 0709 12:26:20.847151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.847356 (+   205us) service_pool.cc:224] Handling call
0709 12:26:20.847888 (+   532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.146911 (+299023us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:21.146950 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:21.155323 (+  8373us) log.cc:844] Serialized 26 byte log entry
0709 12:26:21.310120 (+154797us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:21.310193 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.750111 (+1439918us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:22.750176 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:22.750247 (+    71us) raft_consensus.cc:1685] finished
0709 12:26:22.750265 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:22.754096 (+  3831us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31612,"raft.queue_time_us":19769,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":14369,"spinlock_wait_cycles":598333312,"thread_start_us":146511,"threads_started":1,"wal-append.queue_time_us":235963}
I20260709 12:26:22.500744 28542 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:26:22.703088 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 200152us
I20260709 12:26:22.724359 18015 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:26:22.480618 28473 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:22.693902 28644 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.730675 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:22.699738 28604 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:22.721719 28629 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.740512 27564 raft_consensus.cc:3055] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:22.746999 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 3959) took 1915 ms. Trace:
I20260709 12:26:22.777936 27564 raft_consensus.cc:740] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:22.746930 18173 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.019s	sys 0.008s
I20260709 12:26:22.724498 28639 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:22.748651 18020 raft_consensus.cc:2468] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:22.697721 28576 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:26:22.642057 28404 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.790314 28404 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.705746 18811 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:26:22.758713 18778 leader_election.cc:336] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.152s (SENT)
I20260709 12:26:22.569669 28346 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.793941 28346 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.692916 28574 consensus_queue.cc:237] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.798017 28280 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:22.798297 18778 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:22.799220 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75277.8us
I20260709 12:26:22.800203 28551 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:22.800359 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.803139 28279 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:22.802700 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 43999.5us
W20260709 12:26:22.804466 28522 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.802s	user 0.010s	sys 0.005s
W20260709 12:26:22.760581 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58780.3us
I20260709 12:26:22.747052 28647 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.793985 28529 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:22.805255 18778 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.049s (SENT)
I20260709 12:26:22.812564 28647 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:22.812793 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:22.813738 28604 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:22.815918 28340 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.823609 28404 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:22.824235 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:22.766639 28473 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.753693 28465 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.827764 28473 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.827786 28565 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:22.779899 18173 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.019s	sys 0.009s
I20260709 12:26:22.834306 28629 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:22.837075 28629 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:22.723845 28660 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:22.837702 28565 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.840829 28567 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:22.855971 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.856233 28660 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.855839 28567 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:22.857091 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:22.857575 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.858762 28567 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.864212 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:22.847806 28653 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:22.846089 28639 raft_consensus.cc:697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.870090 28462 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:22.877090 27564 consensus_queue.cc:260] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:22.854156 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.752725 28471 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.850747 18016 raft_consensus.cc:3060] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:22.935501 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:22.995679 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.865902 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:22.944404 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.866508 18164 rpcz_store.cc:276] 0709 12:26:20.831436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.842040 (+ 10604us) service_pool.cc:224] Handling call
0709 12:26:20.844995 (+  2955us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.065259 (+220264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.746635 (+1681376us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400300cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:22.746916 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264782720}
I20260709 12:26:22.903514 28593 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:22.769788 17989 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:23.014168 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 148397us
W20260709 12:26:22.967123 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.017429 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.018313 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:22.968144 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 167849us
W20260709 12:26:22.974195 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 209187us
I20260709 12:26:22.975404 27564 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:22.977686 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 168443us
W20260709 12:26:22.980504 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.022145 18164 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:26:23.023749 18093 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20260709 12:26:23.023591 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:23.024518 18172 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:26:23.025239 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:23.027094 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 257466us
I20260709 12:26:23.028407 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:23.030386 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.030966 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:23.032140 18019 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:22.981222 28576 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.002s
I20260709 12:26:22.971336 28280 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:22.966104 28346 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:22.948597 27822 raft_consensus.cc:3055] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:23.059841 27822 raft_consensus.cc:740] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:23.060465 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:23.061280 27822 consensus_queue.cc:260] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:23.062775 27822 raft_consensus.cc:3060] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:23.028684 18099 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:26:23.033594 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.033987 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:23.034348 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:23.031566 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:23.067859 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:23.038019 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:22.966398 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1793) took 1218 ms. Trace:
I20260709 12:26:23.040931 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:23.070492 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:23.071164 18167 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:23.061376 18171 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:23.042196 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:23.045089 28639 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.047171 28652 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:23.047637 28542 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:23.048899 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.049633 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:23.098789 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49288us
I20260709 12:26:23.039058 28633 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:23.048902 28522 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.003s
I20260709 12:26:23.054311 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:23.105433 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:23.075953 18810 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 25: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:23.083455 18164 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 35 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
I20260709 12:26:23.084347 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:23.070006 18809 rpcz_store.cc:276] 0709 12:26:21.724009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.724521 (+   512us) service_pool.cc:224] Handling call
0709 12:26:22.876309 (+1151788us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b540064e454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:22.941933 (+ 65624us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1753049984}
I20260709 12:26:23.093444 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.095556 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:23.094602 18137 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:23.102212 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65518.1us
I20260709 12:26:23.024642 28528 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:26:23.108635 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:23.106292 18091 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:23.110321 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:23.107483 18808 raft_consensus.cc:1275] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:26:23.111228 18808 pending_rounds.cc:85] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 17
I20260709 12:26:23.111558 18808 pending_rounds.cc:107] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:26:23.111887 18808 raft_consensus.cc:2889] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:23.114815 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.116499 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:23.118570 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.116146 27949 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:23.040345 28346 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.129534 18814 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 32: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:23.125931 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:23.132099 18016 raft_consensus.cc:1275] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:26:23.140689 28632 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:26:23.140800 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4747) took 1068 ms. Trace:
I20260709 12:26:23.142118 28562 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:23.097474 28593 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.109421 28462 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:23.125895 28653 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:23.133725 28551 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:23.133087 28346 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:23.132400 28648 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:26:23.174410 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 57390us
I20260709 12:26:23.161432 28593 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.145112 27949 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:26:23.146944 28565 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:23.178360 28606 consensus_queue.cc:1048] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:23.206784 28542 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:23.207374 28542 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:23.208582 28542 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.144459 28633 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:23.210783 28633 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.198709 28280 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:23.117959 28647 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
W20260709 12:26:23.195698 18170 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1566) took 2002 ms (client timeout 1938 ms). Trace:
I20260709 12:26:23.209389 28593 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:23.141487 18016 rpcz_store.cc:276] 0709 12:26:22.072781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.096583 (+ 23802us) service_pool.cc:224] Handling call
0709 12:26:22.156861 (+ 60278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:22.849910 (+693049us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b58001421d0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166130aa 00007f6c16695447 00007f6c166adfe7 00007f6c166af7de 00007f6c166b9b49 00007f6c166ba416 00007f6c166b6d8f 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4
0709 12:26:23.136744 (+286834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.140726 (+  3982us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261385088}
I20260709 12:26:23.143961 28528 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:23.147910 28601 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:26:23.243360 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.241519 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.237788 18091 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:23.325451 18167 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
W20260709 12:26:23.325949 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.302258 18170 rpcz_store.cc:269] 0709 12:26:21.192840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.279813 (+ 86973us) service_pool.cc:224] Handling call
0709 12:26:23.187267 (+1907454us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54002742d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:23.195568 (+  8301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2136958720}
I20260709 12:26:23.238651 18136 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:23.239233 28521 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:23.317934 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:23.335502 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96958.6us
W20260709 12:26:23.318229 28675 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.733s	user 0.002s	sys 0.004s
I20260709 12:26:23.231710 28565 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.088s
I20260709 12:26:23.238051 28570 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:26:23.221707 28471 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.323000 28528 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.237370 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:23.385450 18809 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 21.
W20260709 12:26:23.324837 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 285042us
I20260709 12:26:23.190593 28551 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:23.336575 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:23.637784 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.727587 18173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.009s	sys 0.009s
I20260709 12:26:23.388063 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:23.339999 28576 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:23.642493 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.335638 28521 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:23.728206 18173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.010s	sys 0.010s
W20260709 12:26:23.730779 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 343022us
I20260709 12:26:23.643446 28662 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.644933 28548 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:23.646533 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 328739us
W20260709 12:26:23.649325 28686 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.399s	user 0.004s	sys 0.010s
I20260709 12:26:23.656258 18016 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:26:23.657925 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:23.744451 18020 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:26:23.659659 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:23.665212 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.670562 28234 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.965s	user 0.013s	sys 0.002s
I20260709 12:26:23.671741 28615 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:23.677642 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.387233 28647 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:26:23.687755 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.690996 28567 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.742527 28648 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:23.758841 28567 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.755791 28647 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:23.763597 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:23.698412 18165 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:26:23.704245 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.332655 28601 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.795621 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:23.591290 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 459867us
W20260709 12:26:23.684162 28596 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.961s	user 0.020s	sys 0.009s
I20260709 12:26:23.393656 18814 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:26:23.684397 28619 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:23.733439 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4754) took 1655 ms. Trace:
I20260709 12:26:23.802739 18016 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:23.251430 28462 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:23.730268 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.685390 28551 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:23.729506 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.737234 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 400753us
W20260709 12:26:23.738430 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.748044 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.706784 27949 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:23.757741 28615 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:23.759292 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4760) took 1083 ms. Trace:
I20260709 12:26:23.763399 28647 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:23.696143 28471 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:26:23.640233 27564 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.620s	user 0.015s	sys 0.004s
W20260709 12:26:23.750350 28473 consensus_meta.cc:287] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.922s	user 0.017s	sys 0.009s
I20260709 12:26:23.801476 18023 rpcz_store.cc:276] 0709 12:26:22.141537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.157434 (+ 15897us) service_pool.cc:224] Handling call
0709 12:26:22.158015 (+   581us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:22.163389 (+  5374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:22.163433 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:22.181172 (+ 17739us) log.cc:844] Serialized 26 byte log entry
0709 12:26:22.478446 (+297274us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:22.674086 (+195640us) log.cc:844] Serialized 14 byte log entry
0709 12:26:22.674402 (+   316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.674499 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:23.317379 (+642880us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:23.710870 (+393491us) raft_consensus.cc:1685] finished
0709 12:26:23.733165 (+ 22295us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:23.733362 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":200842,"prepare.queue_time_us":204776,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":9809,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":15488,"spinlock_wait_cycles":6638464,"thread_start_us":216546,"threads_started":1,"wal-append.queue_time_us":344521}
I20260709 12:26:23.742560 28589 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:23.732345 28633 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:23.807615 28559 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:23.736263 28404 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:23.845448 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:23.809234 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:23.829666 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:23.804131 28601 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:23.852758 18018 rpcz_store.cc:276] 0709 12:26:22.676380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.677205 (+   825us) service_pool.cc:224] Handling call
0709 12:26:22.677653 (+   448us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:22.701173 (+ 23520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:22.701212 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:22.701927 (+   715us) log.cc:844] Serialized 52 byte log entry
0709 12:26:23.010622 (+308695us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:23.044318 (+ 33696us) log.cc:844] Serialized 14 byte log entry
0709 12:26:23.050419 (+  6101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.050516 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:23.758981 (+708465us) raft_consensus.cc:1685] finished
0709 12:26:23.759009 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:23.759217 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5722,"prepare.queue_time_us":33410,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":13298,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":18974,"thread_start_us":270239,"threads_started":1,"wal-append.queue_time_us":391732}
I20260709 12:26:23.827018 18138 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:23.879513 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52633.9us
I20260709 12:26:23.880565 18138 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:23.883386 28340 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.884090 28340 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:23.885198 27822 consensus_meta.cc:287] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.822s	user 0.011s	sys 0.002s
I20260709 12:26:23.828411 28547 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:23.886296 28547 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:23.889088 28547 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:23.827767 28574 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.890048 28596 raft_consensus.cc:2804] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:23.890825 28596 raft_consensus.cc:697] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:23.872428 28647 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:23.796451 28567 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:23.803579 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:23.765574 28561 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:23.854470 28675 log.cc:927] Time spent T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:26:23.749486 28662 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.754724 28548 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:23.862493 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98993.5us
I20260709 12:26:23.808380 28619 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:23.825753 28691 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:23.949117 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 119559us
I20260709 12:26:24.048403 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:23.826048 28562 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.952592 18094 raft_consensus.cc:1240] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: [30.15-30.15]
I20260709 12:26:23.954593 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:23.954881 28340 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:23.955906 28629 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:23.956429 28462 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:23.890089 28574 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.946343 28570 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:23.803865 28604 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:23.960784 28645 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:23.960831 27564 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:23.804593 28280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.019s	sys 0.004s
I20260709 12:26:23.835659 28465 consensus_queue.cc:1048] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.243s
I20260709 12:26:23.991294 18022 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:24.049912 28619 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:24.048890 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:23.950510 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 147002us
W20260709 12:26:24.064172 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.964121 28560 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:24.146502 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97746.6us
I20260709 12:26:24.147657 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:23.980563 28567 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:23.951145 28547 consensus_queue.cc:1048] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.123s
I20260709 12:26:23.878711 28633 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:24.085238 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122000us
W20260709 12:26:24.119189 28280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.026s	sys 0.005s
I20260709 12:26:23.877122 28404 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:23.959389 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 150228us
W20260709 12:26:24.103524 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.078728 28684 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:24.109810 27564 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:23.962925 28595 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.978792 18017 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:26:24.184407 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:24.045495 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.087278 28346 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.004s
I20260709 12:26:23.864295 28473 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.079077 28559 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:24.087320 28596 consensus_queue.cc:237] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:23.864639 27949 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.193460 27949 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.156512 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:23.947044 28593 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.083765 28691 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.159610 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:24.261695 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 102234us
I20260709 12:26:24.160537 28680 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:24.162335 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 1995) took 1474 ms. Trace:
I20260709 12:26:24.166424 28701 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:24.110997 28340 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.114135 28645 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.872337 28589 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.189311 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:24.155057 28563 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:24.187062 28404 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:24.087617 28562 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.190541 28595 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.153978 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:24.194026 28529 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:24.194623 28644 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:24.206120 28548 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.208280 28619 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.212877 28473 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:24.376166 18781 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.026s	sys 0.058s
W20260709 12:26:24.376659 18781 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.026s	sys 0.058s
I20260709 12:26:24.315961 28648 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:24.214824 28346 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.384135 28632 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:24.386408 28346 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.219375 28542 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:24.220425 18168 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:24.225415 28688 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.575s	user 0.005s	sys 0.006s
W20260709 12:26:24.228782 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.392328 18168 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60850 (request call id 1571) took 1371 ms (client timeout 1569 ms). Trace:
W20260709 12:26:24.235152 28695 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:26:24.235476 28615 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:24.048918 28561 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.238946 27564 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:24.250005 28692 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.181s	user 0.002s	sys 0.000s
I20260709 12:26:24.355253 28522 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:24.251170 28694 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.176s	user 0.000s	sys 0.002s
I20260709 12:26:24.394393 28522 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.395076 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2002) took 1395 ms. Trace:
I20260709 12:26:24.395730 28561 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:24.395578 18811 rpcz_store.cc:276] 0709 12:26:22.999641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.000350 (+   709us) service_pool.cc:224] Handling call
0709 12:26:23.035878 (+ 35528us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:23.036356 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:23.036388 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:23.039595 (+  3207us) log.cc:844] Serialized 26 byte log entry
0709 12:26:23.338123 (+298528us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:23.338183 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.338280 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.249094 (+910814us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:24.394680 (+145586us) raft_consensus.cc:1685] finished
0709 12:26:24.394708 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.394988 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":779,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":47051,"spinlock_wait_cycles":23714048,"thread_start_us":284809,"threads_started":1,"wal-append.queue_time_us":892874}
I20260709 12:26:24.229099 28521 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.398340 28521 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:24.401078 28633 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:24.255290 17914 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 184) took 71867 ms (client timeout 15000 ms). Trace:
I20260709 12:26:24.269363 18099 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:24.298853 28593 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.406085 18099 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:24.407130 18099 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.408432 18099 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:24.408528 28593 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.409386 28593 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.318813 18095 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:24.319072 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.426394 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.329643 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.332434 18091 raft_consensus.cc:1275] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:24.158106 28560 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.328537 28700 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:24.372946 28576 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.640s	user 0.012s	sys 0.000s
I20260709 12:26:24.287528 28574 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.375424 28619 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.120220 28604 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:24.377367 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.041s	sys 0.039s
W20260709 12:26:24.443172 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.042s	sys 0.039s
I20260709 12:26:24.308655 28645 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:24.213615 28684 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:24.379859 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 223484us
I20260709 12:26:24.380918 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.312721 27949 raft_consensus.cc:2764] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:24.445170 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:24.383442 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 229618us
I20260709 12:26:24.444986 18093 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
W20260709 12:26:24.354135 28638 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.025s	sys 0.058s
W20260709 12:26:24.397099 28690 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.597s	user 0.005s	sys 0.004s
W20260709 12:26:24.455547 28471 consensus_meta.cc:287] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.002s
W20260709 12:26:24.309935 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 121063us
I20260709 12:26:24.456094 28662 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.124022 28570 raft_consensus.cc:697] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:24.458655 28570 consensus_queue.cc:237] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.430349 28563 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.460733 28563 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.310009 18815 rpcz_store.cc:276] 0709 12:26:22.687589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.699032 (+ 11443us) service_pool.cc:224] Handling call
0709 12:26:22.726704 (+ 27672us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:22.726993 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:22.727021 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:22.734210 (+  7189us) log.cc:844] Serialized 26 byte log entry
0709 12:26:23.242421 (+508211us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:23.242481 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.242579 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.161936 (+919357us) raft_consensus.cc:1685] finished
0709 12:26:24.161973 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.162200 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":468271,"threads_started":1,"wal-append.queue_time_us":1364806}
I20260709 12:26:24.439613 28593 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.464094 28563 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.411891 28691 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.412086 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1568) took 1391 ms. Trace:
W20260709 12:26:24.402163 17914 rpcz_store.cc:269] 0709 12:25:12.387884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.155616 (+2767732us) service_pool.cc:224] Handling call
0709 12:26:24.255169 (+69099553us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:19.897420 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:19.898358 (+   938us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:19.898429 (+    71us) write_op.cc:435] Acquired schema lock
0709 12:26:19.898451 (+    22us) tablet.cc:662] Decoding operations
0709 12:26:20.033781 (+135330us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:20.042035 (+  8254us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:20.042088 (+    53us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:26:20.042911 (+   823us) tablet.cc:701] Row locks acquired
0709 12:26:20.042971 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:26:20.043200 (+   229us) write_op.cc:270] Start()
0709 12:26:20.043461 (+   261us) write_op.cc:276] Timestamp: P: 1783599980043119 usec, L: 0
0709 12:26:20.043506 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:26:20.044413 (+   907us) log.cc:844] Serialized 2430 byte log entry
0709 12:26:20.585094 (+540681us) op_driver.cc:464] REPLICATION: finished
0709 12:26:20.790449 (+205355us) write_op.cc:301] APPLY: starting
0709 12:26:20.790593 (+   144us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:21.024690 (+234097us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:21.024839 (+   149us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:21.481328 (+456489us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:21.481644 (+   316us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=0,mrs_lookups=0
0709 12:26:21.481702 (+    58us) write_op.cc:312] APPLY: finished
0709 12:26:21.493325 (+ 11623us) log.cc:844] Serialized 65 byte log entry
0709 12:26:21.519057 (+ 25732us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:21.537873 (+ 18816us) write_op.cc:454] Released schema lock
0709 12:26:21.548467 (+ 10594us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:21.622532 (+ 74065us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:21.626586 (+  4054us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":293589,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":204860,"apply.run_cpu_time_us":19168,"apply.run_wall_time_us":809448,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29243,"num_ops":6,"prepare.queue_time_us":330256,"prepare.run_cpu_time_us":50094,"prepare.run_wall_time_us":260854,"raft.queue_time_us":202216,"raft.run_cpu_time_us":62103,"raft.run_wall_time_us":121550,"replication_time_us":527919,"spinlock_wait_cycles":2944,"thread_start_us":479480,"threads_started":4,"wal-append.queue_time_us":131827,"wal-append.run_cpu_time_us":13190,"wal-append.run_wall_time_us":2443398}]]}
I20260709 12:26:24.445408 28576 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:24.466660 18165 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.467036 18172 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 21. (term mismatch)
I20260709 12:26:24.467499 18172 pending_rounds.cc:85] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 20
I20260709 12:26:24.467749 18172 pending_rounds.cc:107] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 33.21
I20260709 12:26:24.468027 18172 raft_consensus.cc:2889] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:24.415581 28595 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:24.392795 18168 rpcz_store.cc:269] 0709 12:26:23.021205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.035429 (+ 14224us) service_pool.cc:224] Handling call
0709 12:26:24.392065 (+1356636us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000325d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:24.392238 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2560156928}
I20260709 12:26:24.476315 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.355692 28529 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.379741 28064 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:24.450985 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.443508 28700 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.407522 28589 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.455319 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.325165 28340 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.465063 18173 rpcz_store.cc:276] 0709 12:26:23.020231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.035155 (+ 14924us) service_pool.cc:224] Handling call
0709 12:26:23.048196 (+ 13041us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:23.091089 (+ 42893us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:23.091130 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:23.110131 (+ 19001us) log.cc:844] Serialized 52 byte log entry
0709 12:26:23.734042 (+623911us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:26:23.754946 (+ 20904us) log.cc:844] Serialized 14 byte log entry
0709 12:26:23.755247 (+   301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.755351 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.236102 (+480751us) spinlock_profiling.cc:243] Waited 8.99 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:24.305925 (+ 69823us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b180045d360. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:24.411781 (+105856us) raft_consensus.cc:1685] finished
0709 12:26:24.411819 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.412007 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23565,"prepare.queue_time_us":23709,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":5157,"raft.queue_time_us":24439,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":13018,"spinlock_wait_cycles":24369536,"thread_start_us":583350,"threads_started":1,"wal-append.queue_time_us":937760}
I20260709 12:26:24.479159 18167 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.498373 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:24.476331 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.499330 18169 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:24.474520 28648 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:24.318881 28701 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.483007 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.483798 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.480294 28595 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:24.487263 18095 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:24.439738 18024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.044s	sys 0.050s
I20260709 12:26:24.476432 28473 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.502575 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.501636 18168 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:26:24.507519 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.479053 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:24.505852 18170 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:24.486240 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.507519 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:24.512889 18091 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 28
W20260709 12:26:24.513392 18024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.045s	sys 0.051s
I20260709 12:26:24.514842 18166 raft_consensus.cc:2393] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:24.516484 18809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.013s	sys 0.006s
I20260709 12:26:24.497112 28562 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:24.491724 28522 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.500337 28593 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.476902 28559 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.032s
I20260709 12:26:24.503381 28574 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:24.631948 28574 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:24.488746 28563 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
W20260709 12:26:24.635972 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.504824 28606 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:26:24.540501 18097 raft_consensus.cc:3055] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:24.542269 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.637835 18097 raft_consensus.cc:740] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:24.639957 18097 consensus_queue.cc:260] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.544149 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:24.633098 28574 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.570158 18091 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:24.582404 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:24.602695 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 123726us
I20260709 12:26:24.641793 18097 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:24.609719 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:24.517206 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.053s	sys 0.067s
I20260709 12:26:24.650537 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.545955 18173 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
W20260709 12:26:24.576668 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.652379 18091 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 29 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:24.546772 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:24.651494 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.053s	sys 0.067s
W20260709 12:26:24.516945 28639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.865s	user 0.012s	sys 0.011s
W20260709 12:26:24.615948 28528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.006s	sys 0.013s
I20260709 12:26:24.629031 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.552386 28595 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:24.648736 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:24.592110 28560 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.577641 28700 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:24.622645 28608 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.952s	user 0.028s	sys 0.019s
W20260709 12:26:24.662577 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115945us
I20260709 12:26:24.660439 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:24.663645 28639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.012s	sys 0.012s
W20260709 12:26:24.630249 18809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.013s	sys 0.006s
I20260709 12:26:24.582885 28684 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.606190 28706 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:24.657275 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.515705 28473 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.577486 28529 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.574757 28604 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.539104 28615 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:24.601184 28702 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:24.584030 28340 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:26:24.517620 28561 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:24.668097 18091 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
I20260709 12:26:24.735747 28700 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.713956 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:24.794184 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:24.795025 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 81339.5us
I20260709 12:26:24.790274 18169 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:24.796535 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:24.707165 28596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.004s	sys 0.009s
W20260709 12:26:24.664009 28528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.007s	sys 0.013s
W20260709 12:26:24.723845 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 332056us
I20260709 12:26:24.734027 18815 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.748725 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:24.552385 28346 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.157s
I20260709 12:26:24.668866 28521 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:24.751188 18095 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
I20260709 12:26:24.760222 18021 raft_consensus.cc:1275] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 17. (term mismatch)
I20260709 12:26:24.812276 18021 pending_rounds.cc:85] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 16
I20260709 12:26:24.812665 18021 pending_rounds.cc:107] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 30.17
I20260709 12:26:24.762229 18017 raft_consensus.cc:1275] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 31 index: 15. (term mismatch)
I20260709 12:26:24.813206 18017 pending_rounds.cc:85] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting all ops after (but not including) 14
I20260709 12:26:24.813490 18017 pending_rounds.cc:107] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting uncommitted NO_OP operation due to leader change: 30.15
I20260709 12:26:24.804934 18023 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 LEADER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:26:24.813839 18017 raft_consensus.cc:2889] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:24.763183 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.816033 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.764029 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.816890 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.817588 18022 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:26:24.817755 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.623180 28404 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:24.718473 28551 consensus_meta.cc:287] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.838s	user 0.007s	sys 0.008s
I20260709 12:26:24.824949 28716 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.824802 28551 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.825636 28716 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.827656 28551 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.828449 28716 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.828624 18814 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
W20260709 12:26:24.710233 28608 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.029s	sys 0.020s
I20260709 12:26:24.830480 28718 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.832029 28340 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.831513 28718 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.832752 28340 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.836556 28340 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.649232 28280 consensus_queue.cc:1048] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.101s
I20260709 12:26:24.764945 27949 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.838426 27949 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.839075 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.815047 27822 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.781978 28615 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.787541 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.735569 28560 raft_consensus.cc:2764] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:24.841334 27822 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.792001 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.842725 18015 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:24.843047 28615 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:24.711167 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 62557.7us
I20260709 12:26:24.843397 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.844163 18091 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:24.663405 28593 consensus_queue.cc:1059] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.094s
I20260709 12:26:24.845748 28593 raft_consensus.cc:3055] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:24.841276 27949 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.840502 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.808035 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.812110 28691 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:26:24.848505 18097 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 23: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:24.765292 28465 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:24.665529 28648 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.767640 28706 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.824071 28404 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:24.840731 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 2050) took 1138 ms. Trace:
I20260709 12:26:24.845304 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.808036 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:24.819926 18021 raft_consensus.cc:2889] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:24.851709 18809 rpcz_store.cc:276] 0709 12:26:23.702139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.702372 (+   233us) service_pool.cc:224] Handling call
0709 12:26:23.702846 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:23.892663 (+189817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:23.892731 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:23.893961 (+  1230us) log.cc:844] Serialized 26 byte log entry
0709 12:26:24.719124 (+825163us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:24.719179 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.719265 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.840464 (+121199us) raft_consensus.cc:1685] finished
0709 12:26:24.840494 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.840659 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":179842,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":108,"thread_start_us":937097,"threads_started":2,"wal-append.queue_time_us":709565}
W20260709 12:26:24.804860 28596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.005s	sys 0.009s
I20260709 12:26:24.849747 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.860277 18097 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:24.852029 18091 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:24.790064 28604 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.855684 18017 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:24.794222 28704 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:24.807641 28529 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:24.871109 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.846376 28593 raft_consensus.cc:740] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:24.873145 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.875669 18091 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:26:24.879647 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.883922 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.887115 18092 raft_consensus.cc:2368] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23.
W20260709 12:26:24.869676 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61742.8us
I20260709 12:26:24.883700 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:24.813060 28528 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:24.906944 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.914775 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:24.864373 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 204004us
I20260709 12:26:24.846438 28701 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:24.811085 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.900714 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.901588 28340 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:24.807349 28561 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:24.901774 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:24.815558 28722 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:24.800348 28702 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.924690 18136 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:24.896610 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:24.807049 28700 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:24.985224 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.966825 18097 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
W20260709 12:26:24.969137 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.853451 28465 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:24.852385 28706 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:24.927989 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:24.992945 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:24.996747 18017 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:26:24.871588 28529 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:25.002616 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 106161us
I20260709 12:26:25.003571 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:25.004635 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:24.927968 28691 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:24.929631 28528 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:25.029325 18098 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:25.021343 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:25.033488 18017 raft_consensus.cc:1275] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:26:25.034490 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:25.033952 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 109422us
I20260709 12:26:25.021296 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.037160 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.038154 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.040580 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.041567 18808 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:24.965854 28340 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:25.042181 28340 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:25.043449 28718 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.051725 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:24.903924 28551 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.104s
W20260709 12:26:25.063828 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 85909.3us
W20260709 12:26:25.068120 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.005888 28529 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:25.079159 18098 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:25.001240 28465 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:25.020630 28732 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.087414 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:24.904209 28604 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.005275 28701 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:25.055475 28528 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:25.097455 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:24.892587 28704 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.060458 28706 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:25.105768 18808 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:26:25.008344 28730 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:25.065888 28548 raft_consensus.cc:2804] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:25.073001 28718 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:25.069073 28702 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.119834 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:25.084890 28473 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.132912 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.156607 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4775) took 1377 ms. Trace:
I20260709 12:26:25.161868 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.160673 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.162393 18023 rpcz_store.cc:276] 0709 12:26:23.778689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.857885 (+ 79196us) service_pool.cc:224] Handling call
0709 12:26:23.858647 (+   762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:24.814057 (+955410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.156254 (+342197us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b540004bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:25.156521 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2175507328}
I20260709 12:26:25.000011 28722 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:25.158707 28708 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.378s	user 0.002s	sys 0.000s
I20260709 12:26:25.172784 28704 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.172015 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.174075 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.174968 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:25.175544 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.175717 18807 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:25.161140 28548 raft_consensus.cc:697] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:25.179095 28548 consensus_queue.cc:237] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:25.151517 28725 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.251s	user 0.004s	sys 0.000s
I20260709 12:26:25.172205 28700 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:25.207680 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 130246us
I20260709 12:26:25.196535 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: true dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:25.209486 18098 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:25.115177 28706 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:25.141491 28542 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.754s	user 0.016s	sys 0.012s
I20260709 12:26:25.226627 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.111085 28593 consensus_queue.cc:260] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.140265 27949 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:25.236315 18098 raft_consensus.cc:2368] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:25.114557 28563 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.116s
W20260709 12:26:25.200591 28601 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.237s	user 0.021s	sys 0.011s
I20260709 12:26:25.114181 28701 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.185719 28604 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.189452 28561 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.231030 28706 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:25.263777 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.186744 28722 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.161439 28702 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:26:25.111459 28732 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.232034 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:25.274984 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.279280 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.282361 18812 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:25.283620 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.283741 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.285341 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.284886 18023 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:26:25.262243 27949 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.288028 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.288961 18023 raft_consensus.cc:2393] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:25.257264 28551 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:25.291548 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.329679 18098 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:25.331339 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:25.332703 18166 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.016s	sys 0.011s
I20260709 12:26:25.313330 18808 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
W20260709 12:26:25.318041 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 138727us
I20260709 12:26:25.269632 28601 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.291186 18097 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
W20260709 12:26:25.292186 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60242.5us
I20260709 12:26:25.394443 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.296906 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.395673 18023 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:26:25.397439 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 38 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.400259 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.322435 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.409780 18092 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:26:25.323493 28711 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20260709 12:26:25.428496 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.300246 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.431650 18098 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
W20260709 12:26:25.429383 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 324700us
W20260709 12:26:25.333727 18166 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.016s	sys 0.011s
I20260709 12:26:25.433658 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:25.331462 28473 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:25.296631 28740 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.297749 28662 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:25.460750 28701 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.440943 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:25.323239 28639 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.296289 28722 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:25.440253 18097 raft_consensus.cc:1240] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 25. Current term is 27. Ops: []
I20260709 12:26:25.443872 18098 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 29 index: 23. Preceding OpId from leader: term: 30 index: 24. (index mismatch)
I20260709 12:26:25.393082 28563 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.439962 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:25.450292 28542 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:25.296192 28718 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:25.476941 18806 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:25.481937 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:25.527546 18808 raft_consensus.cc:3060] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.521073 28662 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:25.529224 27949 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:25.497521 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.485982 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45132.2us
I20260709 12:26:25.302922 28551 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.520015 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:25.523733 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.475167 28529 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.449736 28562 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:25.536660 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:25.541417 18807 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:25.547361 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:25.526378 28542 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.553802 18808 raft_consensus.cc:2468] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:25.448891 28574 consensus_queue.cc:1048] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.095s
W20260709 12:26:25.544828 28739 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:26:25.565405 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:25.567655 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4003) took 1051 ms. Trace:
I20260709 12:26:25.570976 28732 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:25.572206 28346 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:25.613080 18168 rpcz_store.cc:276] 0709 12:26:24.516604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.516773 (+   169us) service_pool.cc:224] Handling call
0709 12:26:24.542829 (+ 26056us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.035244 (+492415us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:25.075993 (+ 40749us) log.cc:844] Serialized 14 byte log entry
0709 12:26:25.297996 (+222003us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:25.298078 (+    82us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:25.298291 (+   213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.567250 (+268959us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b5400043dd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:25.567338 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.567557 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46553,"prepare.queue_time_us":63327,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":54153,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":13524,"spinlock_wait_cycles":1131816704,"thread_start_us":167418,"threads_started":1,"wal-append.queue_time_us":210746}
W20260709 12:26:25.614809 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 49652.6us
I20260709 12:26:25.615252 28346 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:25.615847 28633 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:25.616289 28633 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:25.557075 28542 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:25.618513 28742 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
W20260709 12:26:25.621080 28733 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.718s	user 0.000s	sys 0.007s
W20260709 12:26:25.574595 28340 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.532s	user 0.021s	sys 0.025s
I20260709 12:26:25.622263 28340 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.622867 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45040 (request call id 4957) took 1081 ms. Trace:
I20260709 12:26:25.623420 18095 rpcz_store.cc:276] 0709 12:26:24.541511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.541748 (+   237us) service_pool.cc:224] Handling call
0709 12:26:25.622623 (+1080875us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54002968d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:25.622780 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":258566016}
I20260709 12:26:25.625169 28340 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.625603 18090 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 15. (term mismatch)
I20260709 12:26:25.626148 18815 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:26:25.628808 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.629292 18095 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:26:25.552194 28756 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:25.629894 18095 pending_rounds.cc:85] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 15
I20260709 12:26:25.630270 18095 pending_rounds.cc:107] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:26:25.630707 18095 raft_consensus.cc:2889] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.630402 28756 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:25.631512 28724 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.848s	user 0.004s	sys 0.000s
I20260709 12:26:25.526901 28563 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.575516 28608 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:25.573732 28750 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.635336 28750 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.636264 28563 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:25.636509 28707 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.029s	user 0.002s	sys 0.002s
I20260709 12:26:25.636656 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4942) took 1368 ms. Trace:
I20260709 12:26:25.538708 28722 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.637370 28718 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.637135 18096 rpcz_store.cc:276] 0709 12:26:24.267778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.271490 (+  3712us) service_pool.cc:224] Handling call
0709 12:26:24.317650 (+ 46160us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:24.725634 (+407984us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:24.725712 (+    78us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:24.727767 (+  2055us) log.cc:844] Serialized 26 byte log entry
0709 12:26:24.799213 (+ 71446us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:24.799280 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.799397 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:25.620232 (+820835us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:25.636300 (+ 16068us) raft_consensus.cc:1685] finished
0709 12:26:25.636330 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.636593 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60982,"raft.queue_time_us":122203,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":4842,"spinlock_wait_cycles":58205568,"thread_start_us":416340,"threads_started":2,"wal-append.queue_time_us":174783}
I20260709 12:26:25.639863 28722 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.640188 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4770) took 1865 ms. Trace:
I20260709 12:26:25.635378 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4774) took 1857 ms. Trace:
I20260709 12:26:25.642132 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.643546 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.579631 28601 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:25.582032 28755 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:26:25.585196 28706 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.585924 28632 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:25.586570 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.589286 28662 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:25.671883 18808 raft_consensus.cc:2410] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 35 because replica has last-logged OpId of term: 34 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
W20260709 12:26:25.672909 28729 log.cc:927] Time spent T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.769s	user 0.000s	sys 0.002s
I20260709 12:26:25.675567 18810 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 31: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:25.676644 28706 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.617276 28346 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
W20260709 12:26:25.622917 28744 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:26:25.607976 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:25.633636 28756 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:25.605739 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:25.685082 18169 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 4005) took 1142 ms. Trace:
I20260709 12:26:25.640689 18024 rpcz_store.cc:276] 0709 12:26:23.774805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.775789 (+   984us) service_pool.cc:224] Handling call
0709 12:26:23.776187 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:23.776482 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:23.776509 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:23.796307 (+ 19798us) log.cc:844] Serialized 26 byte log entry
0709 12:26:24.513962 (+717655us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:24.514023 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.516315 (+  2292us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:25.635801 (+1119486us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:25.639808 (+  4007us) raft_consensus.cc:1685] finished
0709 12:26:25.639856 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.640116 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37867904,"thread_start_us":641872,"threads_started":1,"wal-append.queue_time_us":763162}
I20260709 12:26:25.626111 18090 pending_rounds.cc:85] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:25.686070 28601 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:25.686390 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:25.686769 28601 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.687305 28601 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:25.688153 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.689097 18095 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28.
I20260709 12:26:25.646538 28652 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:25.690430 28652 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.691700 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.691972 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.692665 18093 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:25.692830 18092 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:25.692970 28652 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.617373 28633 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.680234 28615 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.646819 28750 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.693755 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:25.647344 28465 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.539s	user 0.022s	sys 0.000s
I20260709 12:26:25.695391 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.694659 18091 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:26:25.696136 18099 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:25.483850 28639 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.484977 28740 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.698513 28639 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:25.693755 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.699582 28551 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.699769 18096 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:25.642428 18016 rpcz_store.cc:276] 0709 12:26:23.778434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.803728 (+ 25294us) service_pool.cc:224] Handling call
0709 12:26:23.804099 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.259851 (+455752us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:24.722077 (+462226us) log.cc:844] Serialized 14 byte log entry
0709 12:26:25.117470 (+395393us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:25.117562 (+    92us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:25.117649 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.635109 (+517460us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:25.635165 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.635312 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":524246,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":110,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":16218,"spinlock_wait_cycles":1047306496,"thread_start_us":841370,"threads_started":2,"wal-append.queue_time_us":521095}
I20260709 12:26:25.697755 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.695411 28615 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.700672 28740 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:25.701344 18019 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.695854 28465 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:25.685930 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 80308.7us
I20260709 12:26:25.703346 28615 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:25.703473 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:25.703733 28465 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:25.694833 28633 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:25.707055 28528 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.579s	user 0.014s	sys 0.000s
W20260709 12:26:25.602674 28735 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.306s	user 0.000s	sys 0.001s
I20260709 12:26:25.554986 27949 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:25.561722 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.681710 18020 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:25.478611 28473 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.685645 18169 rpcz_store.cc:276] 0709 12:26:24.543133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.543823 (+   690us) service_pool.cc:224] Handling call
0709 12:26:25.684835 (+1141012us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b540029e0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:25.684994 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":336872320}
I20260709 12:26:25.686530 18090 pending_rounds.cc:107] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 28.15
I20260709 12:26:25.735778 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.708652 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.708832 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:25.711061 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.712294 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.652834 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.682667 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.739044 18097 raft_consensus.cc:2368] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 22. Current term is 23.
I20260709 12:26:25.719285 27822 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:25.715641 28632 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:25.724089 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 284237us
I20260709 12:26:25.727596 18812 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:25.731647 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.732372 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.732714 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.735954 18090 raft_consensus.cc:2889] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.708281 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.769898 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.709371 28608 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.763240 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:25.797479 18167 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:25.769196 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.771405 18171 raft_consensus.cc:2393] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:25.771668 18172 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:26:25.771421 18165 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:26:25.765561 18170 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:25.748064 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:25.721206 28645 consensus_meta.cc:287] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.246s	user 0.012s	sys 0.019s
I20260709 12:26:25.774924 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.745241 18019 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:25.807185 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:25.807946 18170 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.723542 28551 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:25.793032 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 38 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.794099 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.795679 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:25.722412 28691 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.688s	user 0.017s	sys 0.000s
I20260709 12:26:25.793540 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.810560 18169 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33.
I20260709 12:26:25.803915 18166 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:25.737419 27949 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.727550 28652 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:25.812106 28465 consensus_queue.cc:1048] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20260709 12:26:25.817889 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.820214 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.820520 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.822305 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.822707 28756 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:25.823740 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.824719 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.825557 28473 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:25.831374 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2012) took 1096 ms. Trace:
I20260709 12:26:25.831631 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:25.835655 28562 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:25.848376 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.726423 28528 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:25.821386 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:25.721171 28340 consensus_queue.cc:1048] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.088s
I20260709 12:26:25.851193 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:25.861922 18090 raft_consensus.cc:3055] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:25.864097 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:25.903136 18170 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:25.988603 18090 raft_consensus.cc:740] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:25.901738 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.952641 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:25.990319 18173 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:26:25.912107 18167 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
W20260709 12:26:25.993610 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 3.262s (SENT)
I20260709 12:26:25.993960 18092 pending_rounds.cc:85] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:26:25.994529 18092 pending_rounds.cc:107] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:26:25.994880 18092 raft_consensus.cc:2889] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.995306 18092 pending_rounds.cc:107] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:26:25.995616 18092 raft_consensus.cc:2889] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.980115 18097 raft_consensus.cc:2393] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:25.931316 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.983278 18095 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:25.981178 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:25.984750 18096 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 24: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:26.001210 18094 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:26.004505 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.004503 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:26.010095 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.989955 18090 consensus_queue.cc:260] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.010708 18093 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 31: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:26.011032 18090 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:26.013145 18097 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:26:25.977893 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 146351us
I20260709 12:26:25.850219 28691 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:25.862792 27949 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:25.814137 28551 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:25.985213 28756 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.028741 18808 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:26:25.856478 28652 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:25.812397 28615 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.210s
I20260709 12:26:25.992476 28562 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:26.033389 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:25.804474 28542 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.067860 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:26.037865 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:25.967315 28595 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:26.000912 28574 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:26.060880 18171 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:25.848277 28645 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:25.981067 18811 rpcz_store.cc:276] 0709 12:26:24.734882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.756829 (+ 21947us) service_pool.cc:224] Handling call
0709 12:26:24.777421 (+ 20592us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.033532 (+256111us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:26:25.335033 (+301501us) log.cc:844] Serialized 14 byte log entry
0709 12:26:25.558379 (+223346us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:25.558476 (+    97us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:26:25.558553 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.831048 (+272495us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54005f1c54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:25.831110 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.831273 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":489143,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":139,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":150,"spinlock_wait_cycles":587700992,"thread_start_us":422616,"threads_started":2,"wal-append.queue_time_us":242759}
I20260709 12:26:25.809104 28608 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.070287 18808 raft_consensus.cc:2393] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 30: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:26.018373 18095 pending_rounds.cc:85] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 18
I20260709 12:26:26.070992 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:26.077335 18095 pending_rounds.cc:107] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 31.19
I20260709 12:26:26.078007 18095 raft_consensus.cc:2889] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:26.080169 28722 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:26.080636 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.081372 18093 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:26:26.082147 28608 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:26.097528 18017 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29.
I20260709 12:26:26.083465 18168 raft_consensus.cc:2410] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 38 because replica has last-logged OpId of term: 36 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 22.
I20260709 12:26:26.098764 28542 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:26.080638 28722 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:26.100114 28722 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:26.102581 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4791) took 1303 ms. Trace:
I20260709 12:26:26.015686 28632 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.036052 28529 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:26.091601 28652 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.057204 28756 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:26.095216 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:26.118496 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:26.119653 18808 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:26.103116 18020 rpcz_store.cc:276] 0709 12:26:24.799211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.817474 (+ 18263us) service_pool.cc:224] Handling call
0709 12:26:26.102313 (+1284839us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b540010a9d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:26.102506 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1984722176}
I20260709 12:26:26.079120 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:26.106159 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.049624 28551 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:26.105577 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 76540.3us
I20260709 12:26:26.123912 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:26.136490 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.079756 28730 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:26.074254 28574 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.144752 18023 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:26:26.145115 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4785) took 1381 ms. Trace:
I20260709 12:26:26.145378 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.145548 18018 rpcz_store.cc:276] 0709 12:26:24.763597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.763743 (+   146us) service_pool.cc:224] Handling call
0709 12:26:25.671783 (+908040us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b5800002d98. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c13592a22 00007f6c1c4ab8d9 00007f6c1b605ee0 00007f6c1b6cf0ab 00007f6c1b6cf131 00007f6c1b690fc0 00007f6c1b67b929 00007f6c1b67b3d2 00007f6c14bf2bd4
0709 12:26:25.672064 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.135498 (+463434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.135555 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:26.144745 (+  9190us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:26.144805 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.144898 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.145049 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62309,"raft.queue_time_us":43553,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":145,"spinlock_wait_cycles":4074240,"thread_start_us":427241,"threads_started":1}
I20260709 12:26:26.160125 18811 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:26.135527 28529 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.164671 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.165534 18024 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:26.166612 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.168529 18020 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:26.144299 28730 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.176352 18093 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:26:26.181896 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.188041 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.191004 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.144627 28608 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:26.104691 28595 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.195179 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:26.286984 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.000s	sys 0.020s
I20260709 12:26:26.287236 18166 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:26.287684 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.000s	sys 0.021s
W20260709 12:26:26.288317 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 607469us
I20260709 12:26:26.289264 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.237062 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.148741 28574 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.195076 18090 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 15. (term mismatch)
I20260709 12:26:26.294622 18090 pending_rounds.cc:85] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:26.295079 18090 pending_rounds.cc:107] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:26:26.295471 18090 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:26.297863 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.230096 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.271845 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.342715 18169 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
W20260709 12:26:26.209651 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.263641 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.221093 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.154436 28706 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:26.259637 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:26.161324 28652 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:26.339648 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:26.377194 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.340106 18015 raft_consensus.cc:1275] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:26:26.154863 27822 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.348273 18090 raft_consensus.cc:2410] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 31 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 13.
I20260709 12:26:26.195016 28608 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:26.376597 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.383451 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:26.171571 28346 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.190838 28762 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.389990 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.377746 28718 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.013s	sys 0.014s
I20260709 12:26:26.391638 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.453622 18099 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:26.454224 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:26.454981 18812 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 29
W20260709 12:26:26.455778 18806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.008s	sys 0.021s
W20260709 12:26:26.399142 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.456784 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.458047 28064 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.458451 28710 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 1.672s	user 0.010s	sys 0.006s
I20260709 12:26:26.400003 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.458864 28064 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.400789 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2021) took 1292 ms. Trace:
I20260709 12:26:26.462292 18814 raft_consensus.cc:1240] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:26.402016 28595 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:26.402114 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 2055) took 1665 ms. Trace:
I20260709 12:26:26.407621 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:26.410810 18017 raft_consensus.cc:1275] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
W20260709 12:26:26.397694 28633 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.003s
I20260709 12:26:26.413544 18091 raft_consensus.cc:3055] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:26.508522 18091 raft_consensus.cc:740] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:26.415138 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1549) took 1119 ms. Trace:
I20260709 12:26:26.415580 28542 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:26.396144 27822 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.432953 28346 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.195839 28662 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.230s
I20260709 12:26:26.197103 28529 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.539994 28645 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:26.541659 28340 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:26:26.542634 28529 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:26.426172 28528 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:26.543727 28340 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:26.195677 28561 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.232s
W20260709 12:26:26.429193 28766 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:26:26.435349 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.548435 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.549383 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:26.577997 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:26.377998 28706 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:26.446197 18138 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.020s (SENT) [suppressed 7 similar messages]
W20260709 12:26:26.580312 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 134176us
I20260709 12:26:26.397264 28608 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:26.202021 28730 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:26.456701 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58848 (request call id 2060) took 1282 ms. Trace:
I20260709 12:26:26.411746 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:26.591370 18097 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:26.461999 18809 rpcz_store.cc:276] 0709 12:26:25.108278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.110761 (+  2483us) service_pool.cc:224] Handling call
0709 12:26:25.111313 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.178941 (+ 67628us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.179009 (+    68us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:25.179070 (+    61us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:26:25.179125 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.400492 (+1221367us) spinlock_profiling.cc:243] Waited 67.3 ms on lock 0x7b540009bfd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:26.400552 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.400718 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":154824960}
I20260709 12:26:26.593415 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:26.466135 18815 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.014s	sys 0.014s
W20260709 12:26:26.595528 18815 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.015s	sys 0.014s
I20260709 12:26:26.596913 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2023) took 1397 ms. Trace:
W20260709 12:26:26.471398 18806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.008s	sys 0.021s
W20260709 12:26:26.477900 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.598892 18809 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:26.463317 18813 rpcz_store.cc:276] 0709 12:26:24.737181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.156951 (+419770us) service_pool.cc:224] Handling call
0709 12:26:25.157326 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.165895 (+  8569us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.165932 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:25.165982 (+    50us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:25.166015 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.401858 (+1235843us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b5400544054. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:26.401909 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.402049 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19054336}
W20260709 12:26:26.581439 28601 consensus_meta.cc:287] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.894s	user 0.017s	sys 0.000s
I20260709 12:26:26.439612 28740 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:26.445906 28563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.015s	sys 0.030s
I20260709 12:26:26.604672 18167 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 23. (term mismatch)
I20260709 12:26:26.605176 18167 pending_rounds.cc:85] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 22
I20260709 12:26:26.605463 18167 pending_rounds.cc:107] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 36.23
I20260709 12:26:26.579664 28706 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:26.509685 18023 rpcz_store.cc:276] 0709 12:26:25.295653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.401646 (+105993us) service_pool.cc:224] Handling call
0709 12:26:25.432980 (+ 31334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.145206 (+712226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.414863 (+269657us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54001d1d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:26.415069 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1623690624}
I20260709 12:26:26.422350 18171 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: []
W20260709 12:26:26.605008 28563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.021s	sys 0.033s
W20260709 12:26:26.455376 28718 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.013s	sys 0.015s
I20260709 12:26:26.424333 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:26.617913 18164 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:26.538282 18812 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:26.538576 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:26.611873 28601 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.620364 18811 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:26.620597 28593 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.437493 18166 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:26.623037 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:26.550026 28589 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:26.551476 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:26.619879 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 81404.4us
I20260709 12:26:26.627094 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.554050 18091 consensus_queue.cc:260] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.462242 28064 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:26.438772 28762 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.570331 28763 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:26:26.572449 28760 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:26:26.575448 27822 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:26.439447 18019 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:26:26.537745 28542 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.591004 18810 rpcz_store.cc:276] 0709 12:26:25.174848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.175278 (+   430us) service_pool.cc:224] Handling call
0709 12:26:26.456435 (+1281157us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b540034fdd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:26.456634 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147772672}
I20260709 12:26:26.603200 28726 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:26.603673 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.605736 18167 raft_consensus.cc:2889] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:26.438409 28751 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:26.602106 27949 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.537s	user 0.013s	sys 0.000s
I20260709 12:26:26.621656 28601 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965)
I20260709 12:26:26.630326 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:26.675364 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45575us
I20260709 12:26:26.677713 18778 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:26.635043 18096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.004s	sys 0.028s
I20260709 12:26:26.638556 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1809) took 1429 ms. Trace:
I20260709 12:26:26.648916 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:26.629021 18170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.003s	sys 0.020s
W20260709 12:26:26.652257 18092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.002s	sys 0.031s
I20260709 12:26:26.667965 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:26.682130 18092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.002s	sys 0.031s
I20260709 12:26:26.681160 18169 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:26.578768 28595 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:26.673978 27949 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.689507 28595 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:26.652842 28064 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:26.690747 28595 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:26.691367 28064 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.691792 28064 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:26.655491 28340 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:26.695271 28340 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.695698 28340 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.696535 28762 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:26.697503 28576 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:26.697997 28576 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.698137 28528 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:26.698428 28576 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:26.698637 28528 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:26.699719 28528 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.696592 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:26.681735 18170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.003s	sys 0.020s
I20260709 12:26:26.679623 18814 rpcz_store.cc:276] 0709 12:26:25.209347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.841319 (+631972us) service_pool.cc:224] Handling call
0709 12:26:26.462078 (+620759us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.497102 (+ 35024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.638265 (+141163us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b540034fdd0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:26.638471 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1426563456}
W20260709 12:26:26.647459 28596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.032s	sys 0.039s
W20260709 12:26:26.681735 18096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.004s	sys 0.028s
I20260709 12:26:26.688326 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.700531 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:26.672843 18095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.000s	sys 0.076s
W20260709 12:26:26.713291 18095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.000s	sys 0.077s
I20260709 12:26:26.722721 18097 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:26.747620 18172 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
I20260709 12:26:26.751168 18169 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
W20260709 12:26:26.752219 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.012s	sys 0.029s
W20260709 12:26:26.753453 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.013s	sys 0.029s
I20260709 12:26:26.756449 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:26.756647 18171 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:26.772045 18021 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:26.732147 18136 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:26.759337 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:26.759724 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 680621us
I20260709 12:26:26.775180 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:26.758385 18091 raft_consensus.cc:3060] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:26.768445 18173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.006s	sys 0.011s
W20260709 12:26:26.805074 18173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.006s	sys 0.012s
W20260709 12:26:26.685606 28719 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.012s	sys 0.015s
W20260709 12:26:26.743631 28596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.032s	sys 0.039s
I20260709 12:26:26.799016 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.704378 28718 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:26.811975 27822 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:26.829473 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:26.830642 18166 raft_consensus.cc:2410] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 26 because replica has last-logged OpId of term: 25 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 19.
I20260709 12:26:26.653546 28589 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.743482 28601 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:26.830487 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 130055us
W20260709 12:26:26.813019 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60083.6us
W20260709 12:26:26.821054 28719 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.012s	sys 0.016s
I20260709 12:26:26.769610 28521 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.832906 18018 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.024s	sys 0.021s
I20260709 12:26:26.858879 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:26.859297 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 127315us
I20260709 12:26:26.887149 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:26.888062 18167 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:26.892690 18167 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:26.895203 18018 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.027s	sys 0.021s
I20260709 12:26:26.900589 18164 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
W20260709 12:26:26.917218 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.000s	sys 0.026s
I20260709 12:26:26.756762 28570 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.124s
W20260709 12:26:26.965657 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.990813 18167 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:26:26.989467 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.000s	sys 0.028s
I20260709 12:26:26.901031 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:26.872018 28528 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:26.992616 18167 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 24. Preceding OpId from leader: term: 32 index: 25. (index mismatch)
I20260709 12:26:27.000036 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:26.951164 28722 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.833s	user 0.012s	sys 0.004s
I20260709 12:26:27.003059 18023 raft_consensus.cc:2368] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30.
I20260709 12:26:26.898429 28606 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:26:27.011392 18022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.016s	sys 0.016s
W20260709 12:26:27.012048 18022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.017s	sys 0.017s
I20260709 12:26:26.891512 28732 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:27.017418 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:26.953517 28718 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:26.901605 28633 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:27.024689 18090 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.015s	sys 0.020s
W20260709 12:26:27.052989 18090 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.015s	sys 0.020s
I20260709 12:26:27.040558 18091 raft_consensus.cc:1275] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 15. (term mismatch)
I20260709 12:26:26.831194 28574 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.993499 18164 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:27.049531 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.051021 18023 raft_consensus.cc:2368] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29.
I20260709 12:26:27.055749 18164 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:26:27.057607 18091 pending_rounds.cc:85] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:27.058225 18091 pending_rounds.cc:107] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 29.15
I20260709 12:26:27.058615 18091 raft_consensus.cc:2889] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:27.003772 28662 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.063426 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:26.901050 28740 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.140s
W20260709 12:26:27.027530 28648 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.007s	sys 0.019s
W20260709 12:26:27.082829 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.000s	sys 0.045s
W20260709 12:26:27.083283 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.000s	sys 0.045s
W20260709 12:26:27.086913 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.097697 18171 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:27.098261 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:27.083657 28648 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.008s	sys 0.020s
W20260709 12:26:27.071991 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.904417 28521 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:27.099795 18024 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.013s	sys 0.017s
W20260709 12:26:27.077473 28632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.005s	sys 0.022s
W20260709 12:26:27.110610 18137 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.982s	user 0.006s	sys 0.033s
W20260709 12:26:27.146401 18137 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.018s	user 0.007s	sys 0.033s
W20260709 12:26:27.146953 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.01885e+06us
I20260709 12:26:27.070747 18167 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:26:27.017334 28722 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:27.153622 18017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.011s	sys 0.014s
W20260709 12:26:27.154103 18017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.011s	sys 0.014s
I20260709 12:26:27.164880 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:27.144528 18024 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.014s	sys 0.017s
W20260709 12:26:27.141088 18015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.009s	sys 0.020s
W20260709 12:26:27.171092 18015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.009s	sys 0.021s
W20260709 12:26:27.149184 18093 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.020s	sys 0.039s
W20260709 12:26:27.172359 18093 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.021s	sys 0.040s
I20260709 12:26:27.151552 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:27.175856 18020 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.015s	sys 0.045s
W20260709 12:26:27.176465 18020 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.016s	sys 0.045s
I20260709 12:26:27.088793 28346 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:27.173997 18023 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
I20260709 12:26:27.201941 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:27.085103 28662 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:27.078713 28783 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:27.145932 28632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.006s	sys 0.022s
I20260709 12:26:27.105644 28771 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:27.207000 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:27.187788 18098 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.020s	sys 0.010s
W20260709 12:26:27.208244 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 43519.3us
W20260709 12:26:27.221230 18098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.020s	sys 0.011s
I20260709 12:26:27.222232 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:27.211143 18808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.953s	user 0.006s	sys 0.019s
W20260709 12:26:27.225661 18808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s	user 0.006s	sys 0.019s
I20260709 12:26:27.260573 18808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 2064) took 1649 ms. Trace:
I20260709 12:26:27.134675 28762 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.258051 18019 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:27.148034 28718 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:27.228634 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.008s	sys 0.028s
W20260709 12:26:27.269743 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.008s	sys 0.028s
W20260709 12:26:27.270982 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 1.01138e+06us
I20260709 12:26:27.272357 17989 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:27.275565 18065 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.000s	sys 0.025s
W20260709 12:26:27.277221 18065 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.001s	sys 0.026s
W20260709 12:26:27.279336 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.081712 28633 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.209724 27949 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.208525 28570 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:27.255813 28756 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.091s	user 0.010s	sys 0.006s
I20260709 12:26:27.254976 28760 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:27.211231 28783 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:27.261297 18808 rpcz_store.cc:276] 0709 12:26:25.611446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.159830 (+548384us) service_pool.cc:224] Handling call
0709 12:26:26.160129 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.160365 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:26:26.202650 (+ 42285us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.260002 (+1057352us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.260110 (+   108us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:26:27.260186 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.260295 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.260480 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":181792,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":3202,"thread_start_us":1001544,"threads_started":1}
W20260709 12:26:27.303228 18778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.005s	sys 0.025s
W20260709 12:26:27.328797 18778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.006s	sys 0.026s
W20260709 12:26:27.314813 18064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.010s	sys 0.016s
I20260709 12:26:27.266215 28542 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:27.329205 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 569884us
W20260709 12:26:27.334599 18064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.011s	sys 0.016s
I20260709 12:26:27.346881 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:27.352460 28718 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:27.353156 28718 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:27.356554 28778 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
W20260709 12:26:27.358413 28777 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:26:27.359334 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4969) took 1498 ms. Trace:
I20260709 12:26:27.359854 18092 rpcz_store.cc:276] 0709 12:26:25.860714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.993312 (+132598us) service_pool.cc:224] Handling call
0709 12:26:25.993701 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:25.996291 (+  2590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.996326 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:25.996813 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.686209 (+689396us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:26.686278 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.686414 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.359055 (+672641us) raft_consensus.cc:1685] finished
0709 12:26:27.359084 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.359244 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50560,"thread_start_us":688773,"threads_started":1,"wal-append.queue_time_us":1137904}
I20260709 12:26:27.362602 28574 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:27.363258 28774 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.305s	user 0.000s	sys 0.001s
W20260709 12:26:27.364425 28784 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.229s	user 0.000s	sys 0.000s
W20260709 12:26:27.365216 28551 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.218s	user 0.012s	sys 0.000s
I20260709 12:26:27.365949 18096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4971) took 1504 ms. Trace:
I20260709 12:26:27.366400 18096 rpcz_store.cc:276] 0709 12:26:25.861542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.999615 (+138073us) service_pool.cc:224] Handling call
0709 12:26:26.017325 (+ 17710us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.065782 (+ 48457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.066511 (+   729us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.070465 (+  3954us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.706049 (+635584us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:26.706157 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.706280 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.365607 (+659327us) raft_consensus.cc:1685] finished
0709 12:26:27.365638 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.365870 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17256,"raft.queue_time_us":49086,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":11324,"thread_start_us":603278,"threads_started":1,"wal-append.queue_time_us":936959}
W20260709 12:26:27.359519 28776 log.cc:927] Time spent T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 12:26:27.264937 28771 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:27.386889 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:27.387801 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.004s	sys 0.026s
W20260709 12:26:27.391516 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.004s	sys 0.026s
W20260709 12:26:27.371827 28772 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.455s	user 0.000s	sys 0.002s
W20260709 12:26:27.362680 28781 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:26:27.282837 28762 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:26.678232 28751 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:27.386512 28064 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.694s	user 0.013s	sys 0.003s
I20260709 12:26:27.327445 28756 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:27.402966 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:27.401749 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:27.350858 28734 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:27.381726 28589 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:27.409338 28775 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.395s	user 0.001s	sys 0.001s
I20260709 12:26:27.398692 28800 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:27.331173 28783 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:27.398262 28730 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.793s	user 0.010s	sys 0.004s
I20260709 12:26:27.414419 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:27.586800 18091 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:27.365680 28551 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:27.487774 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.391566 18094 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:27.574667 28718 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:27.575925 28792 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.184s	user 0.000s	sys 0.000s
W20260709 12:26:27.576860 28773 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
W20260709 12:26:27.397795 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 620179us
I20260709 12:26:27.590438 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4970) took 1729 ms. Trace:
W20260709 12:26:27.565966 28652 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.179s	user 0.016s	sys 0.002s
I20260709 12:26:27.602836 18095 rpcz_store.cc:276] 0709 12:26:25.861145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.998683 (+137538us) service_pool.cc:224] Handling call
0709 12:26:26.018120 (+ 19437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.078916 (+ 60796us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:26.078973 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.079651 (+   678us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.714960 (+635309us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:26.715036 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.715190 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.590159 (+874969us) raft_consensus.cc:1685] finished
0709 12:26:27.590184 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.590362 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":633094,"threads_started":1,"wal-append.queue_time_us":1073920}
I20260709 12:26:27.605897 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:27.606276 28801 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:27.607015 28801 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:27.609985 28780 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.512s	user 0.000s	sys 0.002s
I20260709 12:26:27.612337 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4009) took 1693 ms. Trace:
I20260709 12:26:27.613751 28551 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.612769 18170 rpcz_store.cc:276] 0709 12:26:25.919027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.004866 (+ 85839us) service_pool.cc:224] Handling call
0709 12:26:26.005255 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.010136 (+  4881us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:26:26.065950 (+ 55814us) log.cc:844] Serialized 14 byte log entry
0709 12:26:26.829120 (+763170us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.858667 (+ 29547us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.859724 (+  1057us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:26.859771 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.859869 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.612038 (+752169us) raft_consensus.cc:1685] finished
0709 12:26:27.612074 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.612241 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":220587,"prepare.queue_time_us":51072,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":16006,"raft.queue_time_us":159472,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":29864,"thread_start_us":624824,"threads_started":1,"wal-append.queue_time_us":886625}
I20260709 12:26:27.614609 18807 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:27.618517 18016 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:27.616472 18806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1808) took 2407 ms. Trace:
I20260709 12:26:27.627332 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:27.639010 18091 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:27.620102 18806 rpcz_store.cc:276] 0709 12:26:25.209004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.825850 (+616846us) service_pool.cc:224] Handling call
0709 12:26:25.826287 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:25.826786 (+   499us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.826819 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:25.827387 (+   568us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.601294 (+773907us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:26.601342 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.601423 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.616118 (+1014695us) raft_consensus.cc:1685] finished
0709 12:26:27.616156 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.616376 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3719,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":153,"thread_start_us":773442,"threads_started":1,"wal-append.queue_time_us":1435935}
W20260709 12:26:27.597437 28562 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.488s	user 0.017s	sys 0.000s
I20260709 12:26:27.597889 28730 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.591985 28800 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:27.642802 18094 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:27.423399 28064 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.609619 28722 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.649294 28801 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.650876 28521 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.655185 28551 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:27.766654 28551 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:27.767195 28551 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:27.769145 28528 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.753s	user 0.013s	sys 0.000s
I20260709 12:26:27.656172 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:27.770943 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114849us
I20260709 12:26:27.772017 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:27.774215 28521 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:27.663012 28542 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.114s
W20260709 12:26:27.668947 18063 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.089s	user 0.007s	sys 0.044s
I20260709 12:26:27.774832 28521 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:27.669025 18097 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.019s	sys 0.059s
W20260709 12:26:27.775614 18097 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.019s	sys 0.059s
I20260709 12:26:27.775964 28521 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:27.669304 28595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.015s	sys 0.034s
W20260709 12:26:27.777644 28595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.015s	sys 0.034s
I20260709 12:26:27.690560 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:27.775048 18063 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.195s	user 0.007s	sys 0.044s
I20260709 12:26:27.779059 18091 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:27.779348 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 1.20135e+06us
I20260709 12:26:27.781970 28804 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:27.691349 28562 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:27.783023 28804 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:27.648546 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:27.783454 28804 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:27.784368 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 135968us
I20260709 12:26:27.784461 28804 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.785008 28562 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:27.785887 28804 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:27.700469 18807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1817) took 1985 ms (client timeout 2015 ms). Trace:
W20260709 12:26:27.786521 18807 rpcz_store.cc:269] 0709 12:26:25.714688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.814480 (+ 99792us) service_pool.cc:224] Handling call
0709 12:26:27.700121 (+1885641us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54004d17d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:27.700349 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3717401344}
I20260709 12:26:27.702255 28633 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:27.787837 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.788048 28722 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:27.788636 28722 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:27.705096 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:27.789739 28722 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.790561 18092 raft_consensus.cc:2410] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:26:27.791623 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:27.792245 18092 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:27.786238 18168 raft_consensus.cc:2393] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 31: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:27.712574 28751 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:27.733371 28771 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.795317 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3712) took 1535 ms. Trace:
I20260709 12:26:27.736054 28756 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:27.740783 28802 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.160s	user 0.000s	sys 0.002s
W20260709 12:26:27.796602 28788 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:26:27.797717 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4993) took 1211 ms. Trace:
I20260709 12:26:27.798019 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4806) took 1629 ms. Trace:
I20260709 12:26:27.798179 18098 rpcz_store.cc:276] 0709 12:26:26.586623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.586803 (+   180us) service_pool.cc:224] Handling call
0709 12:26:26.587264 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.587563 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.587605 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.588379 (+   774us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.221988 (+633609us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:27.222062 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.222178 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.797441 (+575263us) raft_consensus.cc:1685] finished
0709 12:26:27.797470 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.797642 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":599605,"threads_started":1,"wal-append.queue_time_us":793196}
W20260709 12:26:27.746490 28796 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:26:27.754048 28762 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:27.649227 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.018s	sys 0.031s
W20260709 12:26:27.801385 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.018s	sys 0.031s
I20260709 12:26:27.650267 18095 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:27.802251 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4820) took 1384 ms. Trace:
W20260709 12:26:27.746934 28787 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:26:27.709026 18813 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:26:27.802836 18017 rpcz_store.cc:276] 0709 12:26:26.418051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.497304 (+ 79253us) service_pool.cc:224] Handling call
0709 12:26:26.535860 (+ 38556us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:26.539735 (+  3875us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.539770 (+    35us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:26.540749 (+   979us) log.cc:844] Serialized 78 byte log entry
0709 12:26:27.154638 (+613889us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:27.154707 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.154832 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.801910 (+647078us) raft_consensus.cc:1685] finished
0709 12:26:27.801947 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.802157 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4526,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"thread_start_us":613338,"threads_started":1,"wal-append.queue_time_us":1024857}
I20260709 12:26:27.806293 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:27.807096 18017 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:27.807802 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2025) took 1371 ms. Trace:
I20260709 12:26:27.808001 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:27.808324 18813 rpcz_store.cc:276] 0709 12:26:26.436532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.603936 (+167404us) service_pool.cc:224] Handling call
0709 12:26:26.604296 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.805005 (+1200709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.807476 (+  2471us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000e2ed4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:27.807721 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2539603712}
I20260709 12:26:27.808912 18017 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 33.
W20260709 12:26:27.658610 18139 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.020s	sys 0.037s
W20260709 12:26:27.809926 18139 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.020s	sys 0.037s
I20260709 12:26:27.810350 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:27.811833 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:27.665876 28662 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:27.813869 28808 log.cc:927] Time spent T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:26:27.716732 18099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.008s	sys 0.024s
I20260709 12:26:27.819329 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4992) took 1233 ms. Trace:
W20260709 12:26:27.821151 18099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.008s	sys 0.024s
I20260709 12:26:27.821236 18093 rpcz_store.cc:276] 0709 12:26:26.586179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.586397 (+   218us) service_pool.cc:224] Handling call
0709 12:26:26.586998 (+   601us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.172981 (+585983us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.173035 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.248417 (+ 75382us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.709995 (+461578us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:27.710124 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.710340 (+   216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.819034 (+108694us) raft_consensus.cc:1685] finished
0709 12:26:27.819066 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.819246 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3259,"raft.queue_time_us":805815,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":15467,"thread_start_us":961610,"threads_started":2,"wal-append.queue_time_us":517362}
I20260709 12:26:27.785528 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:27.823891 28806 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:27.824542 28806 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:27.795782 18168 rpcz_store.cc:276] 0709 12:26:26.260256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.675916 (+415660us) service_pool.cc:224] Handling call
0709 12:26:27.795064 (+1119148us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000505d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:27.795241 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2484420992}
W20260709 12:26:27.720438 18062 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.003s	sys 0.002s
W20260709 12:26:27.835909 18062 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.004s	sys 0.002s
I20260709 12:26:27.825064 28806 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:27.758002 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:27.846591 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 945622us
I20260709 12:26:27.798575 18022 rpcz_store.cc:276] 0709 12:26:26.168671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.168819 (+   148us) service_pool.cc:224] Handling call
0709 12:26:26.230349 (+ 61530us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.257523 (+ 27174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.257580 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.258413 (+   833us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.012582 (+754169us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:27.012650 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.012771 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.797752 (+784981us) raft_consensus.cc:1685] finished
0709 12:26:27.797793 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.797951 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26512,"raft.queue_time_us":169474,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"spinlock_wait_cycles":330112,"thread_start_us":753636,"threads_started":1,"wal-append.queue_time_us":1091784}
I20260709 12:26:27.852022 18094 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:27.850827 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:27.853008 18020 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.023s	sys 0.033s
W20260709 12:26:27.853515 18020 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.023s	sys 0.033s
W20260709 12:26:27.846460 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61006.1us
I20260709 12:26:27.812433 18017 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:27.855362 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:27.777716 28340 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.082s	user 0.008s	sys 0.008s
I20260709 12:26:27.865633 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:27.781695 28576 consensus_meta.cc:287] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.083s	user 0.012s	sys 0.001s
I20260709 12:26:27.868836 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:27.870182 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.871934 18807 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:27.880318 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:27.880987 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.878121 18091 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 24.
W20260709 12:26:27.858273 28608 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.256s	user 0.013s	sys 0.003s
I20260709 12:26:26.597728 18815 rpcz_store.cc:276] 0709 12:26:25.199517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.681075 (+481558us) service_pool.cc:224] Handling call
0709 12:26:25.717145 (+ 36070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.717436 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:26:25.721710 (+  4274us) log.cc:844] Serialized 14 byte log entry
0709 12:26:26.596299 (+874589us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:26.596397 (+    98us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:26.596480 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.596589 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.596815 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":4703,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":1419,"thread_start_us":873930,"threads_started":1}
W20260709 12:26:27.857861 28706 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.211s	user 0.013s	sys 0.003s
I20260709 12:26:27.898053 18808 raft_consensus.cc:1275] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:27.901536 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:27.904508 18780 leader_election.cc:400] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29.
I20260709 12:26:27.905161 18780 leader_election.cc:403] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:27.906888 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:27.907269 18024 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.000s	sys 0.015s
I20260709 12:26:27.907012 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:27.907842 18024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.000s	sys 0.015s
I20260709 12:26:27.906931 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4814) took 1508 ms. Trace:
W20260709 12:26:27.921682 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.924681 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:27.925865 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:27.927031 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:27.923974 18092 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
W20260709 12:26:27.895896 28606 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.794s	user 0.007s	sys 0.007s
W20260709 12:26:27.931293 28606 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.007s	sys 0.007s
I20260709 12:26:27.868229 28662 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.933121 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:27.938596 18807 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:27.863307 28756 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:27.931779 28606 raft_consensus.cc:697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:27.977389 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4817) took 1560 ms. Trace:
I20260709 12:26:27.908704 18020 rpcz_store.cc:276] 0709 12:26:26.398214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.398430 (+   216us) service_pool.cc:224] Handling call
0709 12:26:26.448052 (+ 49622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.448420 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:27.177046 (+728626us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.906351 (+729305us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.906442 (+    91us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:27.906514 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.906708 (+   194us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.906862 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40184,"prepare.queue_time_us":943793,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":52178,"thread_start_us":1389494,"threads_started":2}
I20260709 12:26:27.936002 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:27.936928 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:27.875231 28528 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:27.897785 28064 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:27.895766 28563 consensus_queue.cc:1048] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:26:27.897550 28340 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.911929 28740 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:27.910225 28718 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.873098 28805 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:27.938608 28719 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.947s	user 0.000s	sys 0.016s
I20260709 12:26:27.933911 28722 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29.
I20260709 12:26:27.910984 28706 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.935122 28760 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.049309 28760 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:27.898169 28576 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.054051 28756 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.054839 28756 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:28.055717 28823 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:26:27.909510 28608 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.002645 28800 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:27.986711 28542 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:27.991231 28730 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.038919 18807 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:27.965992 28811 consensus_queue.cc:1048] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.137s
W20260709 12:26:28.043960 28821 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:26:28.043819 18806 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 37.
W20260709 12:26:28.044584 28719 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.000s	sys 0.017s
I20260709 12:26:27.934063 28751 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:26:27.994066 28763 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:27.965052 28662 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.048230 28706 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.004896 28528 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:28.210755 28818 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260709 12:26:28.211122 28765 log.cc:927] Time spent T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.946s	user 0.000s	sys 0.018s
I20260709 12:26:28.216315 28756 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:28.218688 28608 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:28.219837 28760 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:28.220589 28783 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.611s	user 0.011s	sys 0.006s
W20260709 12:26:28.221763 28593 consensus_meta.cc:287] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.601s	user 0.018s	sys 0.002s
I20260709 12:26:28.244777 28662 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:28.245570 28817 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.246258 28817 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.231901 28763 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.249493 28763 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:28.056473 28718 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.252048 28806 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.252923 28718 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:28.249970 28783 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.255792 28783 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:28.256446 28806 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:28.257169 28779 log.cc:927] Time spent T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 1.256s	user 0.009s	sys 0.019s
I20260709 12:26:28.311302 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4819) took 1893 ms. Trace:
W20260709 12:26:28.318496 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.312662 18015 rpcz_store.cc:276] 0709 12:26:26.417754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.461316 (+ 43562us) service_pool.cc:224] Handling call
0709 12:26:26.461662 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.544325 (+ 82663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.544382 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.545057 (+   675us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.171828 (+626771us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:27.687878 (+516050us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.689337 (+  1459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.309420 (+620083us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b54000ff854. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:28.309598 (+   178us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.309843 (+   245us) raft_consensus.cc:1685] finished
0709 12:26:28.310163 (+   320us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.310790 (+   627us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3300,"prepare.queue_time_us":384702,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"raft.queue_time_us":4429,"raft.run_cpu_time_us":423,"raft.run_wall_time_us":11878,"spinlock_wait_cycles":188810496,"thread_start_us":1064901,"threads_started":2,"wal-append.queue_time_us":870219}
I20260709 12:26:28.326776 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:27.991722 18017 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:28.339171 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3632) took 1198 ms. Trace:
W20260709 12:26:28.339737 18016 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4801) took 2176 ms (client timeout 2327 ms). Trace:
I20260709 12:26:28.339812 18017 rpcz_store.cc:276] 0709 12:26:27.140596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.936560 (+795964us) service_pool.cc:224] Handling call
0709 12:26:28.338857 (+402297us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:28.340377 18016 rpcz_store.cc:269] 0709 12:26:26.163259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.166241 (+  2982us) service_pool.cc:224] Handling call
0709 12:26:28.339262 (+2173021us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54001cfa54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:28.339516 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3336843520}
I20260709 12:26:28.342473 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:28.343712 18016 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:28.343987 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:27.978142 18024 rpcz_store.cc:276] 0709 12:26:26.417166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.435133 (+ 17967us) service_pool.cc:224] Handling call
0709 12:26:26.435555 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.435931 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:26:27.187393 (+751462us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.976737 (+789344us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.976836 (+    99us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:27.976919 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.977043 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.977274 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68233,"prepare.queue_time_us":891728,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":34183,"thread_start_us":1443754,"threads_started":2}
I20260709 12:26:28.346460 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:28.369364 18024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3613) took 1364 ms. Trace:
W20260709 12:26:28.349695 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.353418 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.374944 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3610) took 1376 ms. Trace:
I20260709 12:26:28.373647 18024 rpcz_store.cc:276] 0709 12:26:27.005440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.367129 (+1361689us) service_pool.cc:224] Handling call
0709 12:26:28.368032 (+   903us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.368615 (+   583us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:28.368676 (+    61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:28.368760 (+    84us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:28.368834 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.368936 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.369099 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.355319 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49542 (request call id 4800) took 2192 ms. Trace:
I20260709 12:26:28.376531 18021 rpcz_store.cc:276] 0709 12:26:26.162901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.190688 (+ 27787us) service_pool.cc:224] Handling call
0709 12:26:28.354641 (+2163953us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b54001fd954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:28.355067 (+   426us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326822144}
I20260709 12:26:28.347784 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:28.378278 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:28.378783 18023 raft_consensus.cc:2393] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 24: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:28.379604 18021 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:28.380585 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:28.375909 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.375495 18019 rpcz_store.cc:276] 0709 12:26:26.998722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.999248 (+   526us) service_pool.cc:224] Handling call
0709 12:26:26.999695 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:27.262490 (+262795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.374248 (+1111758us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b540004bad4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:28.374701 (+   453us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":593380224}
W20260709 12:26:28.383258 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.383575 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3614) took 1372 ms. Trace:
I20260709 12:26:28.384419 18019 raft_consensus.cc:1240] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:26:28.385380 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3617) took 1351 ms. Trace:
I20260709 12:26:28.385803 18019 rpcz_store.cc:276] 0709 12:26:27.034281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.383276 (+1348995us) service_pool.cc:224] Handling call
0709 12:26:28.383891 (+   615us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:28.385039 (+  1148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.385309 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:28.389676 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.389510 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:28.007519 18090 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.954s	user 0.009s	sys 0.013s
I20260709 12:26:28.398654 18019 raft_consensus.cc:1240] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30. Ops: [27.14-29.15]
W20260709 12:26:28.398751 18090 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.345s	user 0.013s	sys 0.016s
I20260709 12:26:28.399097 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4995) took 1698 ms. Trace:
I20260709 12:26:28.428078 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4986) took 2035 ms. Trace:
I20260709 12:26:28.427019 18099 rpcz_store.cc:276] 0709 12:26:26.700255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.700420 (+   165us) service_pool.cc:224] Handling call
0709 12:26:26.700905 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.038688 (+337783us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.038729 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.039396 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.826154 (+786758us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:27.826241 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.826352 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.055003 (+228651us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:28.398458 (+343455us) raft_consensus.cc:1685] finished
0709 12:26:28.398514 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.398862 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11751,"raft.queue_time_us":837948,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":15066,"spinlock_wait_cycles":14063488,"thread_start_us":1119297,"threads_started":2,"wal-append.queue_time_us":856936}
I20260709 12:26:28.399909 18092 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9"
dest_uuid: "768723648a9444fda53659e6a8cc3091"
mode: GRACEFUL
new_leader_uuid: "ae9641d5c3184572a102d51a75f670ee"
 from {username='slave'} at 127.0.0.1:45562
I20260709 12:26:28.430238 18092 raft_consensus.cc:606] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Received request to transfer leadership to TS ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:28.397760 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 4996) took 1509 ms. Trace:
I20260709 12:26:28.428712 18090 rpcz_store.cc:276] 0709 12:26:26.392638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.414792 (+ 22154us) service_pool.cc:224] Handling call
0709 12:26:26.415097 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.415346 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:26.415687 (+   341us) log.cc:844] Serialized 14 byte log entry
0709 12:26:28.427166 (+2011479us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:28.427321 (+   155us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:28.427466 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.427626 (+   160us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.427948 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5442,"prepare.queue_time_us":684,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":543400,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":15649,"thread_start_us":2010820,"threads_started":2,"wal-append.queue_time_us":896259}
W20260709 12:26:28.397519 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.430796 18092 raft_consensus.cc:614] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:28.430959 18097 rpcz_store.cc:276] 0709 12:26:26.888655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.888798 (+   143us) service_pool.cc:224] Handling call
0709 12:26:26.889196 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.775991 (+886795us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:27.776027 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.776627 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.987950 (+211323us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:27.987990 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.988070 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.397147 (+409077us) raft_consensus.cc:1685] finished
0709 12:26:28.397189 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.397520 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":695917,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":5852,"thread_start_us":1097072,"threads_started":2,"wal-append.queue_time_us":291874}
I20260709 12:26:28.435851 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:28.436949 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:28.439168 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:28.430728 18019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3618) took 1396 ms. Trace:
I20260709 12:26:28.442490 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:28.444850 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:28.446172 18096 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
W20260709 12:26:28.448523 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.377939 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:28.381624 18023 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:28.451751 18023 raft_consensus.cc:1217] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Deduplicated request from leader. Original: 26.16->[28.17-31.19]   Dedup: 30.18->[31.19-31.19]
I20260709 12:26:28.452535 18023 raft_consensus.cc:1240] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 31. Current term is 32. Ops: [28.17-31.19]
I20260709 12:26:28.452705 18020 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 23.
I20260709 12:26:28.384168 18017 rpcz_store.cc:276] 0709 12:26:27.011100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.343351 (+1332251us) service_pool.cc:224] Handling call
0709 12:26:28.383469 (+ 40118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.455385 18017 raft_consensus.cc:1240] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:28.456132 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3625) took 1419 ms. Trace:
I20260709 12:26:28.456485 18017 rpcz_store.cc:276] 0709 12:26:27.036520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.454381 (+1417861us) service_pool.cc:224] Handling call
0709 12:26:28.455064 (+   683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.455792 (+   728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.456027 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.059715 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.408279 18167 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:28.070888 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.058439 18169 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:28.409811 18164 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50510 (request call id 3723) took 2004 ms. Trace:
I20260709 12:26:28.411736 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:28.458947 18166 raft_consensus.cc:2393] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:28.468387 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.457736 18022 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
I20260709 12:26:28.056453 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.474202 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.475385 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:28.453547 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49528 (request call id 3634) took 1197 ms. Trace:
I20260709 12:26:28.475998 18020 rpcz_store.cc:276] 0709 12:26:27.256539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.988514 (+731975us) service_pool.cc:224] Handling call
0709 12:26:28.453444 (+464930us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.476835 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:28.415424 18171 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3714) took 2154 ms (client timeout 2448 ms). Trace:
W20260709 12:26:28.478070 18171 rpcz_store.cc:269] 0709 12:26:26.260876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.798640 (+537764us) service_pool.cc:224] Handling call
0709 12:26:28.414711 (+1616071us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b540029e0d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:28.415094 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":618499968}
I20260709 12:26:28.479773 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:28.481230 18171 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:28.470263 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:28.483043 18166 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:28.484642 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:28.466053 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.486064 18169 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:28.487936 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:28.488783 18169 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:28.491671 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.453500 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3619) took 1418 ms. Trace:
I20260709 12:26:28.495934 18023 rpcz_store.cc:276] 0709 12:26:27.034848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.450352 (+1415504us) service_pool.cc:224] Handling call
0709 12:26:28.451102 (+   750us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:28.453087 (+  1985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.453399 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.464433 18164 rpcz_store.cc:276] 0709 12:26:26.404746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.056841 (+652095us) service_pool.cc:224] Handling call
0709 12:26:27.057267 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:27.555115 (+497848us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:27.602279 (+ 47164us) log.cc:844] Serialized 14 byte log entry
0709 12:26:28.040191 (+437912us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:28.040276 (+    85us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:28.040475 (+   199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.408875 (+368400us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54001b3d54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:28.408987 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.409438 (+   451us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":49705,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":32025,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":21555,"spinlock_wait_cycles":1145163008,"thread_start_us":405425,"threads_started":1}
I20260709 12:26:28.499089 18164 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:28.412743 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:28.500923 18016 raft_consensus.cc:2468] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:26:28.466938 18024 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:28.467975 18017 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:28.464049 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:28.506757 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica cab4a2f5d3ca48a2a803ee8ec3906e1d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:28.494130 18172 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:28.528276 18168 raft_consensus.cc:2368] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31.
I20260709 12:26:28.528507 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3627) took 1491 ms. Trace:
I20260709 12:26:28.528977 18017 rpcz_store.cc:276] 0709 12:26:27.037048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.457617 (+1420569us) service_pool.cc:224] Handling call
0709 12:26:28.467550 (+  9933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.528102 (+ 60552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.528406 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.394274 18021 raft_consensus.cc:1275] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:26:28.521840 17915 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 280) took 112900 ms (client timeout 15000 ms). Trace:
I20260709 12:26:28.471696 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:28.534876 17915 rpcz_store.cc:269] 0709 12:24:35.577016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.594656 (+ 17640us) service_pool.cc:224] Handling call
0709 12:26:28.477330 (+112882674us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":158085,"reported_tablets":50}
W20260709 12:26:28.544236 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.071s	sys 0.083s
W20260709 12:26:28.545104 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.071s	sys 0.083s
W20260709 12:26:28.550130 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 645836us
I20260709 12:26:28.545049 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49528 (request call id 3616) took 1509 ms. Trace:
I20260709 12:26:28.551788 18780 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:28.551368 18021 rpcz_store.cc:276] 0709 12:26:27.033966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.380948 (+1346982us) service_pool.cc:224] Handling call
0709 12:26:28.381607 (+   659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.535530 (+153923us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.543612 (+  8082us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:28.055779 28576 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:28.442287 18019 rpcz_store.cc:276] 0709 12:26:27.034563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.395756 (+1361193us) service_pool.cc:224] Handling call
0709 12:26:28.397285 (+  1529us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:28.430256 (+ 32971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.430616 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:28.608570 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 204959us
I20260709 12:26:28.611531 28789 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.612744 28789 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.614650 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:28.615257 28750 consensus_queue.cc:1048] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:28.616451 28593 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.620056 18807 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:28.622637 18806 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:26:28.605511 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54007.1us
I20260709 12:26:28.652127 18813 raft_consensus.cc:2393] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:28.654302 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:28.656841 28722 raft_consensus.cc:2804] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:28.617434 28593 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.661368 18019 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:28.678354 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 232946us
W20260709 12:26:28.679486 18093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.056s	sys 0.065s
I20260709 12:26:28.689260 18808 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 33. Current term is 34. Ops: []
W20260709 12:26:28.690019 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 313306us
I20260709 12:26:28.657747 28722 raft_consensus.cc:697] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:28.704016 18093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.057s	sys 0.065s
I20260709 12:26:28.240665 28551 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:28.701323 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 531151us
I20260709 12:26:28.706712 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:28.710210 28806 consensus_queue.cc:1059] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:28.711488 28789 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.712677 28806 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:28.727283 18172 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33.
I20260709 12:26:28.709625 28551 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.729015 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:28.748399 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 44791.2us
I20260709 12:26:28.758562 18015 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
W20260709 12:26:28.765803 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.768111 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:28.771127 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:28.621405 28606 consensus_queue.cc:237] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.736599 28064 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.789197 18015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:28.789793 28593 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:28.793573 28770 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.893s	user 0.004s	sys 0.012s
I20260709 12:26:28.796039 28528 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.797384 28804 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.798007 28589 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.798470 28719 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.798729 28825 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.799450 28340 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.800143 28832 raft_consensus.cc:3055] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:28.805277 18021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:28.822583 18021 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:28.806015 28820 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.807260 28799 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.811658 18022 raft_consensus.cc:2393] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:28.787285 18018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4797) took 2625 ms. Trace:
I20260709 12:26:28.815313 18015 raft_consensus.cc:2393] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:28.825510 18022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:28.832825 18022 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:28.827549 28825 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.839100 28825 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.830119 28799 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.830778 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1588) took 3028 ms. Trace:
I20260709 12:26:28.848008 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:28.831378 28820 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.828465 28832 raft_consensus.cc:740] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:28.840106 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.844762 18812 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:28.846238 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:28.846938 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:28.848457 18165 rpcz_store.cc:276] 0709 12:26:25.802443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.803780 (+  1337us) service_pool.cc:224] Handling call
0709 12:26:25.804214 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:25.804723 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.804758 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:25.805350 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.234034 (+428684us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:26.234095 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.234198 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.792288 (+2558090us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:28.829943 (+ 37655us) raft_consensus.cc:1685] finished
0709 12:26:28.829981 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.830470 (+   489us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":41067,"spinlock_wait_cycles":79844608,"thread_start_us":411560,"threads_started":1,"wal-append.queue_time_us":948227}
W20260709 12:26:28.847939 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.851060 28804 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.852298 28064 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.854101 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:28.857054 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.823763 18018 rpcz_store.cc:276] 0709 12:26:26.161640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.164650 (+  3010us) service_pool.cc:224] Handling call
0709 12:26:26.165066 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.165367 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:26.896147 (+730780us) log.cc:844] Serialized 14 byte log entry
0709 12:26:28.365769 (+1469622us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b18004bf0a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c1625f4bc 00007f6c135ad48b 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c1625723a
0709 12:26:28.757953 (+392184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:28.769495 (+ 11542us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:28.786607 (+ 17112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.786759 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.787011 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1102816,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":250667,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":242,"spinlock_wait_cycles":50188672,"thread_start_us":1493323,"threads_started":3,"wal-append.queue_time_us":825564,"wal-append.run_cpu_time_us":2439,"wal-append.run_wall_time_us":1091710}
I20260709 12:26:28.846242 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:28.876551 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 465371us
I20260709 12:26:28.878209 18137 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:28.878443 28722 consensus_queue.cc:237] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:28.879364 28589 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.878470 18018 raft_consensus.cc:1275] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:26:28.886713 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:28.894668 18015 raft_consensus.cc:1240] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:26:28.899884 18021 raft_consensus.cc:2468] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
W20260709 12:26:28.904078 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.902431 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 288945us
I20260709 12:26:28.907246 18016 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:28.908895 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4159) took 1069 ms. Trace:
I20260709 12:26:28.947365 18807 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:26:28.914642 18093 rpcz_store.cc:276] 0709 12:26:27.839923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.840102 (+   179us) service_pool.cc:224] Handling call
0709 12:26:27.840578 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.841021 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.841054 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.856183 (+ 15129us) log.cc:844] Serialized 26 byte log entry
0709 12:26:28.706849 (+850666us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:28.706937 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.707062 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.898847 (+191785us) raft_consensus.cc:1685] finished
0709 12:26:28.898878 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.908820 (+  9942us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2233,"raft.queue_time_us":346,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":21839,"thread_start_us":771206,"threads_started":1,"wal-append.queue_time_us":948735}
I20260709 12:26:28.961928 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:28.964048 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 254383us
W20260709 12:26:28.964846 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.995118 18809 raft_consensus.cc:2393] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 26: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:28.970248 28832 consensus_queue.cc:260] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:28.971493 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.977941 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 213306us
I20260709 12:26:28.987171 28820 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:28.987843 28593 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.990126 28817 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:28.990274 28814 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:28.907498 28719 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:28.941205 28800 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:28.962805 18064 leader_election.cc:400] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30.
I20260709 12:26:29.012002 18064 leader_election.cc:403] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:28.963436 28608 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:29.005338 18022 raft_consensus.cc:2468] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
W20260709 12:26:29.007118 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 99716.4us
W20260709 12:26:29.021365 18169 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.532s	user 0.012s	sys 0.008s
I20260709 12:26:28.944352 28825 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:29.034418 18815 raft_consensus.cc:2393] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 25: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:29.036106 28593 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.022449 18169 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:29.016894 18019 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:29.041015 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:29.050493 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 172531us
I20260709 12:26:29.051707 18137 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:29.040279 28589 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.053076 28839 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:29.063498 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:29.054689 28817 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30.
I20260709 12:26:29.066732 28064 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:29.092288 28785 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 1.803s	user 0.005s	sys 0.009s
I20260709 12:26:29.095570 17987 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:29.098377 18020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1555) took 1118 ms. Trace:
I20260709 12:26:29.099893 28593 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.099083 18020 rpcz_store.cc:276] 0709 12:26:27.979957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.476951 (+496994us) service_pool.cc:224] Handling call
0709 12:26:28.477615 (+   664us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:28.630969 (+153354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:28.631021 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:28.659528 (+ 28507us) log.cc:844] Serialized 26 byte log entry
0709 12:26:28.925570 (+266042us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:28.925621 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.925740 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:29.097955 (+172215us) raft_consensus.cc:1685] finished
0709 12:26:29.097988 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:29.098268 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":137899,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"thread_start_us":395461,"threads_started":2,"wal-append.queue_time_us":392390}
I20260709 12:26:29.101970 28839 consensus_queue.cc:1059] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:29.102382 28751 consensus_queue.cc:1059] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:29.103766 28839 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:29.103711 28751 consensus_queue.cc:1059] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:29.104926 28562 raft_consensus.cc:3055] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:29.105167 28589 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:29.076282 28551 raft_consensus.cc:2764] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election decision vote started in defunct term 24: won
I20260709 12:26:29.115012 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:29.105585 18063 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:29.073539 28340 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:29.045670 28800 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.062836 28825 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.112548 28562 raft_consensus.cc:740] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:29.126129 18809 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
I20260709 12:26:29.131513 28839 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:29.130234 28832 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:29.147847 18171 raft_consensus.cc:1240] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:29.122211 28820 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
W20260709 12:26:29.157513 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 333948us
I20260709 12:26:29.155797 28771 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.178181 18813 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1822) took 1310 ms. Trace:
I20260709 12:26:29.162599 28633 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:29.187402 18064 leader_election.cc:400] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Vote denied by peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965) with higher term. Message: Invalid argument: T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29.
I20260709 12:26:29.217079 28771 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.207612 28521 consensus_queue.cc:1059] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260709 12:26:29.162812 28740 raft_consensus.cc:3055] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:29.219123 28740 raft_consensus.cc:740] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:29.219832 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:29.220232 28771 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.216976 18813 rpcz_store.cc:276] 0709 12:26:27.868202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.868415 (+   213us) service_pool.cc:224] Handling call
0709 12:26:29.177697 (+1309282us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b54004b46d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:29.178031 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1562865024}
W20260709 12:26:29.162806 28542 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.075s	user 0.020s	sys 0.001s
I20260709 12:26:29.171494 28800 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.222532 28542 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:29.210649 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 190411us
I20260709 12:26:29.223696 28800 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:26:29.224296 18780 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:29.224452 28804 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.177965 28817 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:29.225705 28806 raft_consensus.cc:2804] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:26:29.226226 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67204.9us
I20260709 12:26:29.226502 28806 raft_consensus.cc:697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:29.207242 28562 consensus_queue.cc:260] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.227924 28806 consensus_queue.cc:237] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.229102 28562 raft_consensus.cc:3060] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.211176 18815 raft_consensus.cc:1240] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:29.178740 18808 raft_consensus.cc:1240] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:29.225281 28804 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:29.232511 28804 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:26:29.232978 28804 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:29.233320 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4020) took 1173 ms. Trace:
I20260709 12:26:29.217510 28799 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.234297 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:29.237469 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 83836.9us
I20260709 12:26:29.159417 28846 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:29.235211 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:29.233803 18171 rpcz_store.cc:276] 0709 12:26:28.059692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.492713 (+433021us) service_pool.cc:224] Handling call
0709 12:26:28.493420 (+   707us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.153342 (+659922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:29.232982 (+ 79640us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b5400301bd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:29.233206 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1496184704}
I20260709 12:26:29.237771 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:29.222081 28849 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:26:29.234788 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.240053 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:29.181146 28785 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.087s	user 0.002s	sys 0.000s
W20260709 12:26:29.262739 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73584.6us
W20260709 12:26:29.264138 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.223156 28719 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.268221 18099 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:29.235834 28608 raft_consensus.cc:3055] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:26:29.273622 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47855.1us
I20260709 12:26:29.254562 28633 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:29.253528 28817 raft_consensus.cc:697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:29.157193 28340 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:29.278141 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 116199us
I20260709 12:26:29.283627 18093 raft_consensus.cc:3060] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:29.272405 28608 raft_consensus.cc:740] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:29.305308 28340 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.220338 28740 consensus_queue.cc:260] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.206687 28839 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:29.342164 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 43539.8us
I20260709 12:26:29.319913 28836 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:29.320562 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.320457 28740 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:29.321295 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 82831us
W20260709 12:26:29.357667 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 117722us
I20260709 12:26:29.322803 28608 consensus_queue.cc:260] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.326726 28633 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.328227 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4010) took 3386 ms. Trace:
I20260709 12:26:29.330999 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:29.365864 18099 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:29.368120 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:29.331338 28771 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:26:29.332217 28789 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:29.311489 28551 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:29.371991 28551 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.308928 28722 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:29.374361 28722 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:29.379750 28860 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:29.380756 28860 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.381584 28762 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:29.382736 28762 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:29.384276 28860 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:29.345660 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 123888us
I20260709 12:26:29.386992 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:29.345101 28825 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.392270 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:29.392840 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:29.393044 28860 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:29.396596 28799 raft_consensus.cc:3055] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:29.385330 28762 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:29.456425 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:29.460460 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.461458 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.467926 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:29.469201 18097 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.448050 28799 raft_consensus.cc:740] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:29.484963 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.485898 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.486073 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.485612 28799 consensus_queue.cc:260] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.487082 18172 raft_consensus.cc:2393] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:29.485247 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.487923 28799 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.488775 18165 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 30: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:29.316435 28804 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 32.25, Last appended by leader: 25, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.491385 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.492518 18090 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:29.458173 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1593) took 1394 ms. Trace:
I20260709 12:26:29.449851 28860 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:29.494266 28860 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:29.493978 18170 rpcz_store.cc:276] 0709 12:26:28.063834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.501717 (+437883us) service_pool.cc:224] Handling call
0709 12:26:28.502197 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.776436 (+274239us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:29.158590 (+382154us) log.cc:844] Serialized 14 byte log entry
0709 12:26:29.457417 (+298827us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:29.457544 (+   127us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:29.457680 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:29.457799 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:29.458062 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":402805,"prepare.queue_time_us":310430,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":66,"raft.queue_time_us":157156,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":52975,"spinlock_wait_cycles":21163264,"thread_start_us":583660,"threads_started":3}
I20260709 12:26:29.491638 28804 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:26:29.514232 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 121534us
I20260709 12:26:29.307989 28820 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.141s
I20260709 12:26:29.514566 28817 consensus_queue.cc:237] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.556656 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:29.525313 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:29.572351 18814 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:29.574222 28836 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:29.579835 28528 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:29.580209 28542 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:29.580466 28528 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:29.581208 28528 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:29.555830 28064 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:26:29.586004 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 63706.2us
W20260709 12:26:29.586246 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 194171us
I20260709 12:26:29.574451 28814 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.591519 28814 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.453071 28719 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.141s
I20260709 12:26:29.594115 28783 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.595147 28825 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:29.594918 28783 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.595309 28719 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:26:29.596009 28719 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:29.521605 28593 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:29.365334 18173 rpcz_store.cc:276] 0709 12:26:25.941752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.991445 (+ 49693us) service_pool.cc:224] Handling call
0709 12:26:26.021102 (+ 29657us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:26.100568 (+ 79466us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.100642 (+    74us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:26.101672 (+  1030us) log.cc:844] Serialized 52 byte log entry
0709 12:26:26.807840 (+706168us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:26.876646 (+ 68806us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.335081 (+458435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.335186 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.960960 (+1625774us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:29.180763 (+219803us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b24001bd808. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134509c4 00007f6c1344eacb 00007f6c13609dc9 00007f6c13609a55 00007f6c1360909d 00007f6c1623b10e 00007f6c1625288c 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221
0709 12:26:29.327742 (+146979us) raft_consensus.cc:1685] finished
0709 12:26:29.327771 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:29.328003 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":183579,"prepare.queue_time_us":68310,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":34804,"raft.queue_time_us":244202,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":13785,"spinlock_wait_cycles":110819200,"thread_start_us":1008141,"threads_started":2,"wal-append.queue_time_us":975831}
W20260709 12:26:29.605239 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 156864us
I20260709 12:26:29.610087 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:29.611214 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 155291us
I20260709 12:26:29.612313 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:29.615777 28601 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:29.616411 28601 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:29.616859 28601 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.610402 28551 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.619793 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:29.586402 28064 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:29.622396 28608 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:29.622303 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.623450 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.624425 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.624569 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.625424 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.626315 28783 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.626442 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.630091 28648 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:29.630748 28861 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:26:29.631202 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.631597 28719 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.602418 28593 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:29.621663 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.677412 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.683023 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
I20260709 12:26:29.672776 18090 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:29.646085 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:29.648710 18093 raft_consensus.cc:2410] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 30 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
W20260709 12:26:29.658068 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 86338.9us
I20260709 12:26:29.657461 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.375882 28722 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.634168 28789 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:29.686545 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:29.687983 28593 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.674955 28751 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:29.695248 18168 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:26:29.696470 18136 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:29.697997 28820 raft_consensus.cc:3060] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:29.698601 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 108857us
I20260709 12:26:29.674242 28771 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:29.670997 28542 consensus_queue.cc:1059] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:26:29.728907 18095 raft_consensus.cc:2410] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 31 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
W20260709 12:26:29.736698 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 122891us
I20260709 12:26:29.732420 28719 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:29.741302 28542 raft_consensus.cc:3055] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:29.742069 28542 raft_consensus.cc:740] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:29.749253 28844 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:29.750473 28844 raft_consensus.cc:697] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:29.752058 28844 consensus_queue.cc:237] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.759045 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:29.728052 28771 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:29.762434 28771 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.766070 28751 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.771625 28857 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:29.773725 28839 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.774814 28839 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:29.775713 28064 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.779628 28593 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.782960 28633 consensus_queue.cc:1059] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:29.786213 28789 raft_consensus.cc:3055] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:29.786943 28789 raft_consensus.cc:740] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:29.788471 28789 consensus_queue.cc:260] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.790309 28814 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:29.792433 18023 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:29.792778 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:29.793460 18020 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 23.
I20260709 12:26:29.743583 28542 consensus_queue.cc:260] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.796169 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
W20260709 12:26:29.800479 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.761646 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:29.808535 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.805469 18024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:29.809146 18017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:29.809129 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.808126 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:29.830731 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:29.833048 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 71584.5us
W20260709 12:26:29.833082 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 223338us
I20260709 12:26:29.843416 18097 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:29.845453 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:29.846812 18092 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:29.854559 18172 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:29.855337 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 151844us
I20260709 12:26:29.731088 28648 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:29.860975 18170 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:29.872545 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:29.839769 28857 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:29.882845 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:29.883963 18169 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.888378 18016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d"
I20260709 12:26:29.888805 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:29.889783 18171 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.871888 28542 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:29.892657 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46531.1us
I20260709 12:26:29.893602 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:29.901858 18806 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:29.893560 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 113519us
I20260709 12:26:29.880033 28839 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:29.907063 18167 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.879765 28857 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:29.901942 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66381.1us
I20260709 12:26:29.916473 18166 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 34.
I20260709 12:26:29.918913 18138 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:29.940979 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
I20260709 12:26:29.942648 18020 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:29.944193 18173 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:26:29.945438 18780 leader_election.cc:400] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31.
I20260709 12:26:29.911567 28857 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.916431 28789 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:29.957736 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.898741 28814 consensus_queue.cc:1048] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:26:29.970345 18020 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:29.974146 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.974097 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:29.976773 18813 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:26:29.986506 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:29.976158 28844 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:29.999079 28762 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:26:30.001724 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.944770 28817 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
W20260709 12:26:29.985158 28873 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.223s	user 0.000s	sys 0.002s
W20260709 12:26:30.016017 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46635.8us
W20260709 12:26:30.028781 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.016368 18172 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 34.
I20260709 12:26:30.021225 18169 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:30.021314 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:30.021597 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 110338us
I20260709 12:26:30.032909 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:30.047385 18808 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:26:30.045882 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:30.057282 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 50719.8us
I20260709 12:26:30.057021 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:30.076308 28762 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:30.083981 28762 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:30.084419 28762 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:30.062264 28563 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:30.090255 28563 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:30.093199 28563 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:30.096791 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.096297 18093 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:30.114239 28662 raft_consensus.cc:3055] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:30.114924 28662 raft_consensus.cc:740] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:30.116146 28751 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:30.117214 28878 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:26:30.124130 18171 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:26:30.079433 28589 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:30.124862 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:30.104539 28763 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:30.126417 28849 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:26:30.161432 18138 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:30.158025 18093 raft_consensus.cc:1240] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:30.107702 28648 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:30.162173 28858 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:26:30.166236 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 145083us
I20260709 12:26:30.167039 28839 consensus_queue.cc:1059] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:30.169544 28880 log.cc:927] Time spent T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
W20260709 12:26:30.170621 28877 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:26:30.174357 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:30.175408 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:30.176103 28887 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:30.177454 28876 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:30.179414 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:30.187119 18170 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33.
I20260709 12:26:30.080302 28879 consensus_queue.cc:1059] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:26:30.167071 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:30.178305 28608 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.506s	user 0.011s	sys 0.003s
I20260709 12:26:30.203715 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:30.204560 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:30.205178 28839 raft_consensus.cc:3055] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:26:30.247334 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 80377.8us
I20260709 12:26:30.198367 28763 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:30.207701 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:30.208324 28870 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.320s	user 0.005s	sys 0.000s
I20260709 12:26:30.208550 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:30.208889 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:30.167089 28762 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:30.199158 28858 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:30.211194 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.082191 28528 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:30.202665 18808 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 25.
W20260709 12:26:30.213423 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52123.6us
I20260709 12:26:30.144964 28832 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:30.251583 18807 raft_consensus.cc:2393] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:26:30.214186 28740 consensus_meta.cc:287] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.826s	user 0.008s	sys 0.014s
W20260709 12:26:30.257656 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48921.3us
I20260709 12:26:30.265439 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:30.268723 28465 raft_consensus.cc:2804] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:30.247025 28876 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:30.246956 28839 raft_consensus.cc:740] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:30.262729 28832 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:30.291859 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.278048 28606 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:30.279881 28871 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:30.368273 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.389629 18017 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:30.269550 28465 raft_consensus.cc:697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:30.368925 18167 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:26:30.262460 28884 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.273s	user 0.002s	sys 0.000s
I20260709 12:26:30.395046 18812 raft_consensus.cc:1275] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:26:30.398538 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 141475us
W20260709 12:26:30.373870 28799 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.885s	user 0.022s	sys 0.005s
I20260709 12:26:30.367224 28740 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:30.371822 28606 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:30.424324 18023 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:30.401607 18136 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:30.370509 28832 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:30.432788 18138 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:30.388516 28871 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:30.441347 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 263648us
I20260709 12:26:30.260210 28887 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:30.449668 18808 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260709 12:26:30.450181 18808 pending_rounds.cc:85] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 15
W20260709 12:26:30.451537 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50038.7us
I20260709 12:26:30.450435 18808 pending_rounds.cc:107] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:26:30.456182 18808 raft_consensus.cc:2889] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:30.432821 28606 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:30.467586 28601 consensus_meta.cc:287] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.850s	user 0.015s	sys 0.007s
I20260709 12:26:30.467407 28811 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:30.395468 28814 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:26:30.415349 28719 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:26:30.474699 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:30.449363 28871 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:30.460517 28648 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:30.494915 18093 raft_consensus.cc:1275] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:26:30.497046 28898 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:26:30.503242 28894 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:26:30.503396 28633 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:30.473642 28562 raft_consensus.cc:2804] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:26:30.514568 18137 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): Timed out: RequestConsensusVote RPC to 127.16.102.4:46307 timed out after 1.944s (SENT)
I20260709 12:26:30.515955 18137 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:30.515094 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 472677us
W20260709 12:26:30.518862 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.523866 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:30.536012 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:30.539420 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75241.4us
I20260709 12:26:30.493562 28811 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:30.578235 18018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" is_pre_election: true
W20260709 12:26:30.577909 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 66785.8us
I20260709 12:26:30.655864 18018 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 26.
I20260709 12:26:30.661434 18093 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:30.664285 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:30.666049 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 199458us
I20260709 12:26:30.483690 28601 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:30.557633 28763 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:30.668843 18096 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:26:30.580350 28820 consensus_meta.cc:287] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.844s	user 0.016s	sys 0.002s
I20260709 12:26:30.566157 28562 raft_consensus.cc:697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:30.573009 28648 raft_consensus.cc:697] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:30.479939 28839 consensus_queue.cc:260] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:30.469631 28858 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:30.798583 18096 raft_consensus.cc:1240] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
W20260709 12:26:30.805418 18780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.031s	sys 0.053s
W20260709 12:26:30.807060 18780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.031s	sys 0.053s
W20260709 12:26:30.778447 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 68037.7us
I20260709 12:26:30.512391 28817 consensus_queue.cc:1048] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.154s
I20260709 12:26:30.780565 18016 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:26:30.781076 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 111299us
I20260709 12:26:30.824168 18018 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:26:30.826757 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.814541 18810 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:30.830133 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:30.559288 28876 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 32.25, Last appended by leader: 25, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:30.574066 28465 consensus_queue.cc:237] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.115006 28528 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.000528 28919 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:30.797358 28608 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:31.119550 28909 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.116s	user 0.003s	sys 0.000s
W20260709 12:26:30.879221 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 54755.5us
I20260709 12:26:31.091326 28811 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:31.094141 28593 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.314s	user 0.003s	sys 0.014s
I20260709 12:26:31.123409 18090 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:31.124948 28844 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:31.013376 28897 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.621s	user 0.000s	sys 0.001s
I20260709 12:26:30.905067 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.123862 28811 consensus_queue.cc:1048] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:26:31.128410 28908 log.cc:927] Time spent T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 12:26:31.125692 28844 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.018920 28662 consensus_queue.cc:260] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.131049 28844 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:31.131407 28662 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:31.099406 28064 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:31.134116 28886 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:26:31.123737 28914 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 12:26:31.136770 18023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1559) took 1345 ms. Trace:
I20260709 12:26:31.138319 18165 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4030) took 1541 ms. Trace:
W20260709 12:26:30.814277 28859 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:26:31.138779 18165 rpcz_store.cc:276] 0709 12:26:29.597359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.606423 (+  9064us) service_pool.cc:224] Handling call
0709 12:26:29.608942 (+  2519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.901472 (+292530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.901521 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:29.910571 (+  9050us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:29.910634 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.138037 (+1227403us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54003025d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:31.138088 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.138242 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":647888640}
I20260709 12:26:31.011816 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5013) took 1406 ms. Trace:
I20260709 12:26:31.141359 28740 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.140349 18094 rpcz_store.cc:276] 0709 12:26:29.605194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.682276 (+ 77082us) service_pool.cc:224] Handling call
0709 12:26:29.791186 (+108910us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:29.791746 (+   560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.791779 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:29.792483 (+   704us) log.cc:844] Serialized 26 byte log entry
0709 12:26:29.853986 (+ 61503us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:29.854055 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:29.854145 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:30.480720 (+626575us) raft_consensus.cc:1685] finished
0709 12:26:30.480748 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.011059 (+530311us) spinlock_profiling.cc:243] Waited 7.02 ms on lock 0x7b5400214650. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b68ad 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:31.011586 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1491,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":110,"spinlock_wait_cycles":16142592,"thread_start_us":48619,"threads_started":1,"wal-append.queue_time_us":146596}
I20260709 12:26:31.142036 28740 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:30.813843 28762 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:30.813134 28542 consensus_meta.cc:287] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.902s	user 0.030s	sys 0.000s
I20260709 12:26:31.146565 28740 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:30.942299 28906 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.137261 18023 rpcz_store.cc:276] 0709 12:26:29.791847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.792071 (+   224us) service_pool.cc:224] Handling call
0709 12:26:31.136484 (+1344413us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b540024d454. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:31.136665 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1448993664}
I20260709 12:26:31.003775 28601 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:30.986229 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.172721 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.173995 28593 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:31.174706 28593 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.176646 28722 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
W20260709 12:26:31.108806 28789 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.147s	user 0.011s	sys 0.005s
I20260709 12:26:31.177450 28722 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.151218 28906 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:31.115792 28904 log.cc:927] Time spent T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.311s	user 0.002s	sys 0.000s
I20260709 12:26:30.750828 28820 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:30.913769 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:31.210857 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 297213us
I20260709 12:26:31.212956 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:30.836208 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 172092us
I20260709 12:26:31.151592 28886 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:31.091781 28890 log.cc:927] Time spent T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.914s	user 0.003s	sys 0.000s
I20260709 12:26:31.180523 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:31.185421 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:30.809132 28860 consensus_meta.cc:287] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 1.314s	user 0.018s	sys 0.014s
I20260709 12:26:30.792048 28751 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:30.941762 28857 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.974s	user 0.013s	sys 0.000s
W20260709 12:26:31.006067 28899 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.344s	user 0.001s	sys 0.000s
W20260709 12:26:31.006908 28896 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.497s	user 0.001s	sys 0.000s
W20260709 12:26:30.828732 28771 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.066s	user 0.013s	sys 0.011s
I20260709 12:26:31.222695 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:31.226578 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:30.935972 28839 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:30.937536 28858 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:31.236652 18024 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:31.184551 28722 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:31.177752 28593 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:31.282869 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 52240.7us
I20260709 12:26:31.236980 18809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1832) took 1072 ms. Trace:
I20260709 12:26:31.231424 18092 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
W20260709 12:26:31.279721 18095 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.079s	sys 0.147s
W20260709 12:26:31.292037 18095 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.079s	sys 0.148s
I20260709 12:26:31.119658 28919 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:30.937086 28887 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:31.277910 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 763325us
I20260709 12:26:30.935557 28817 consensus_queue.cc:1048] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.310s
I20260709 12:26:31.225301 28601 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:31.281672 18019 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:26:31.293880 18020 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.001s	sys 0.009s
I20260709 12:26:31.322803 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.278537 28799 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.000054 28562 consensus_queue.cc:237] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.279956 28789 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.346328 28789 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.321956 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:31.349803 18167 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:26:31.350401 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52100.2us
I20260709 12:26:31.327527 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5016) took 1519 ms. Trace:
I20260709 12:26:31.282541 28542 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.350405 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.360145 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.248523 28860 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:31.357329 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.968s (SENT)
I20260709 12:26:31.282756 28740 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:31.291031 18809 rpcz_store.cc:276] 0709 12:26:30.164810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.279474 (+114664us) service_pool.cc:224] Handling call
0709 12:26:30.365235 (+ 85761us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.406500 (+ 41265us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:30.722375 (+315875us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.236259 (+513884us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.236377 (+   118us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:31.236463 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.236634 (+   171us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.236871 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":79941,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":91,"raft.queue_time_us":10961,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":27979,"thread_start_us":639134,"threads_started":2}
I20260709 12:26:31.266214 28857 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:31.366127 18099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.013s	sys 0.029s
W20260709 12:26:31.377900 18099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.014s	sys 0.029s
I20260709 12:26:31.276118 28771 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.375650 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:31.378773 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4173) took 1364 ms. Trace:
I20260709 12:26:31.378974 18808 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:31.323499 28601 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:31.329033 28722 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.374940 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:31.382925 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:31.335666 28906 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.383665 18167 raft_consensus.cc:1275] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:26:31.326634 28887 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.395130 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2041) took 1577 ms. Trace:
I20260709 12:26:31.379175 18099 rpcz_store.cc:276] 0709 12:26:30.014050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.023737 (+  9687us) service_pool.cc:224] Handling call
0709 12:26:30.045779 (+ 22042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.046078 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:30.403917 (+357839us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.378342 (+974425us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.378414 (+    72us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:31.378480 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.378561 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.378704 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":268574,"prepare.queue_time_us":92766,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":66,"raft.queue_time_us":422262,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":155,"thread_start_us":1281969,"threads_started":3,"wal-append.queue_time_us":588849}
I20260709 12:26:31.397607 18806 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:31.395186 18167 pending_rounds.cc:85] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 17
I20260709 12:26:31.398942 18167 pending_rounds.cc:107] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:26:31.398874 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.399345 18167 raft_consensus.cc:2889] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:31.393067 17989 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:31.370282 28814 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:31.353009 18095 rpcz_store.cc:276] 0709 12:26:29.808598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.809407 (+   809us) service_pool.cc:224] Handling call
0709 12:26:29.848291 (+ 38884us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.882268 (+ 33977us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:30.014112 (+131844us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.117765 (+1103653us) spinlock_profiling.cc:243] Waited 63.4 ms on lock 0x7b1800148f00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135982a6 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db
0709 12:26:31.118633 (+   868us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:31.325121 (+206488us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.325201 (+    80us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:31.325559 (+   358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.327251 (+  1692us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.327455 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34029,"prepare.queue_time_us":169631,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":495904,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"spinlock_wait_cycles":174437504,"thread_start_us":1316516,"threads_started":3,"wal-append.queue_time_us":971557}
I20260709 12:26:31.336546 28804 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:31.336319 28913 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:31.420368 18137 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:31.403950 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:31.406585 18096 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33.
I20260709 12:26:31.422098 18170 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:31.423301 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.424152 18096 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:31.395802 18814 rpcz_store.cc:276] 0709 12:26:29.807100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.807266 (+   166us) service_pool.cc:224] Handling call
0709 12:26:29.830769 (+ 23503us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:29.845001 (+ 14232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.845053 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:29.877535 (+ 32482us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.348068 (+470533us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:30.348127 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.348223 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.384019 (+1035796us) raft_consensus.cc:1685] finished
0709 12:26:31.384052 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.384227 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6204,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":10580,"thread_start_us":431606,"threads_started":1,"wal-append.queue_time_us":592038}
I20260709 12:26:31.368248 28740 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.410378 18022 raft_consensus.cc:1240] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:26:31.411059 18017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4842) took 1516 ms. Trace:
W20260709 12:26:31.412533 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55271.2us
I20260709 12:26:31.458864 18017 rpcz_store.cc:276] 0709 12:26:29.894962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.913133 (+ 18171us) service_pool.cc:224] Handling call
0709 12:26:29.934089 (+ 20956us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.390344 (+456255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.410700 (+1020356us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b5400250654. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:31.410971 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1047129984}
W20260709 12:26:31.459692 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.247s (SENT)
I20260709 12:26:31.444502 28740 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:31.410507 28905 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:31.427665 28763 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:31.432214 28906 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.390595 28601 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:31.429730 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.028s	sys 0.050s
W20260709 12:26:31.457149 28783 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.278s	user 0.013s	sys 0.018s
I20260709 12:26:31.403231 28722 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.400965 28608 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.451536 18015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4840) took 1574 ms. Trace:
I20260709 12:26:31.410611 28814 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.425575 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.406870 28789 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.328224 28648 consensus_queue.cc:237] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:31.449123 28832 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.987s	user 0.012s	sys 0.003s
I20260709 12:26:31.461321 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:31.456664 18097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.950s	user 0.026s	sys 0.036s
W20260709 12:26:31.578102 18097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.026s	sys 0.036s
I20260709 12:26:31.418357 28913 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.382975 28542 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.413635 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.584055 18099 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:31.585872 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.586905 18099 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:31.573807 28930 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.588090 28930 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.470366 17913 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 23 to 25, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 25 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:31.455475 28871 consensus_meta.cc:287] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.947s	user 0.020s	sys 0.000s
I20260709 12:26:31.594347 28871 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.473960 18098 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34. Ops: []
I20260709 12:26:31.414656 28919 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.599174 28919 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:31.450700 28606 consensus_meta.cc:287] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.950s	user 0.008s	sys 0.008s
W20260709 12:26:31.479899 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.028s	sys 0.050s
W20260709 12:26:31.444837 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51877.5us
W20260709 12:26:31.601049 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 1.07719e+06us
I20260709 12:26:31.600864 28606 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.601644 17989 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:31.602344 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:31.603404 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:31.603824 28606 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:31.338171 28799 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.407264 28887 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.604635 18165 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:31.499595 18170 raft_consensus.cc:2468] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:26:31.606230 28799 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.502060 18096 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:31.607169 28887 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:31.481956 28783 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.307s	user 0.014s	sys 0.018s
I20260709 12:26:31.607686 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.609218 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.609150 28783 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:31.610026 28783 consensus_queue.cc:1243] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 28.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:26:31.504312 18095 raft_consensus.cc:2468] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
W20260709 12:26:31.576098 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 221463us
I20260709 12:26:31.611528 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:31.485186 18015 rpcz_store.cc:276] 0709 12:26:29.877277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.877467 (+   190us) service_pool.cc:224] Handling call
0709 12:26:29.892439 (+ 14972us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:30.407596 (+515157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.407635 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:30.414045 (+  6410us) log.cc:844] Serialized 52 byte log entry
0709 12:26:30.723683 (+309638us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:30.780018 (+ 56335us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.126204 (+346186us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b180028e820. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c13544c0a 00007f6c1cc84219 00007f6c135cca40 00007f6c135c74e4 00007f6c135c7455 00007f6c166af060 00007f6c166c1716 00007f6c166d2287 00007f6c166d2208
0709 12:26:31.428944 (+302740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.429033 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.429079 (+    46us) raft_consensus.cc:1685] finished
0709 12:26:31.450772 (+ 21693us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.451192 (+   420us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3750,"prepare.queue_time_us":57546,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":9546,"raft.queue_time_us":403069,"raft.run_cpu_time_us":410,"raft.run_wall_time_us":12519,"spinlock_wait_cycles":20010752,"thread_start_us":1040973,"threads_started":3,"wal-append.queue_time_us":554000}
I20260709 12:26:31.611603 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.613157 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:31.613585 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.610069 18096 raft_consensus.cc:2468] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:31.615208 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:31.618675 18780 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:31.621246 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 161578us
I20260709 12:26:31.571292 28814 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:31.622681 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.626570 18096 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:31.409032 28886 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.593060 17913 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:31.633510 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5022) took 1583 ms. Trace:
I20260709 12:26:31.637784 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:31.617326 28913 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:31.646689 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.647423 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.648126 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.636322 18778 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.953s (SENT) [suppressed 14 similar messages]
W20260709 12:26:31.628283 28900 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:26:31.484064 28906 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.570345 28608 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.629050 28905 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:31.628984 28903 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:31.666815 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 205623us
I20260709 12:26:31.667974 18137 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:31.616891 28832 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.652056 17913 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:31.625276 28799 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:31.659448 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.635841 28606 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:26:31.666206 28937 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:31.709411 17913 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 21 to 27. New cstate: current_term: 27 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:31.658099 28924 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:31.631886 28934 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:31.722551 17913 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 18 to 21, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 21 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:31.710822 28901 log.cc:927] Time spent T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.342s	user 0.001s	sys 0.001s
W20260709 12:26:31.743723 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 107491us
I20260709 12:26:31.712931 28783 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:31.759085 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
W20260709 12:26:31.764413 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 126792us
I20260709 12:26:31.716629 28521 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:31.780390 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.722977 28857 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.724277 28924 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:31.787184 17913 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 21 to 22, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 22 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:31.795867 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:31.785808 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 127290us
I20260709 12:26:31.737355 28771 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:31.793303 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.743723 28465 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:26:31.762780 28871 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.749991 28938 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:31.695047 28903 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.807875 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.805930 17987 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:31.777561 18099 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:31.731949 28937 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:31.808930 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:31.819490 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.815690 17987 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:31.820619 18091 raft_consensus.cc:2376] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27. Re-sending same reply.
I20260709 12:26:31.826635 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.827787 18099 raft_consensus.cc:2376] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33. Re-sending same reply.
I20260709 12:26:31.829298 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:31.831691 28064 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.697s	user 0.013s	sys 0.003s
W20260709 12:26:31.831789 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 229568us
I20260709 12:26:31.801954 28924 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:31.838951 18094 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 33 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:26:31.841279 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.841655 28935 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:26:31.844130 28722 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:31.845485 28936 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.192s	user 0.000s	sys 0.002s
W20260709 12:26:31.846602 17917 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 180) took 179404 ms (client timeout 15000 ms). Trace:
I20260709 12:26:31.847044 28906 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:31.847163 17917 rpcz_store.cc:269] 0709 12:23:32.390930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.391305 (+   375us) service_pool.cc:224] Handling call
0709 12:26:31.785588 (+179394283us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1294135,"reported_tablets":31,"spinlock_wait_cycles":3328}
I20260709 12:26:31.850395 28903 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:31.804737 28887 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:26:31.841447 17913 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 16 to 19, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 19 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:31.822628 28521 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.792622 28944 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 12:26:31.856038 28521 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.856832 17913 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 18 to 23, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:31.858139 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 190256us
I20260709 12:26:31.859195 17913 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 24 to 26, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 26 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:31.862543 28927 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.344s	user 0.002s	sys 0.000s
I20260709 12:26:31.863447 28771 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.863736 28930 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:31.864164 28771 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.865120 28937 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.831300 28879 raft_consensus.cc:2804] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:31.870450 28879 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.871083 28879 raft_consensus.cc:3060] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:31.874751 18021 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49542 (request call id 4838) took 2004 ms. Trace:
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:454: Failure
Failed
Bad status: Timed out: LeaderStepDown RPC to 127.16.102.2:40047 timed out after 10.000s (SENT)
I20260709 12:26:31.875556 18021 rpcz_store.cc:276] 0709 12:26:29.870803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.870996 (+   193us) service_pool.cc:224] Handling call
0709 12:26:29.871462 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:29.887844 (+ 16382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.887886 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:29.888683 (+   797us) log.cc:844] Serialized 52 byte log entry
0709 12:26:30.139520 (+250837us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b18004566a0. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:26:30.485426 (+345906us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:30.485476 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.485567 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.874170 (+1388603us) raft_consensus.cc:1685] finished
0709 12:26:31.874214 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.874634 (+   420us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3718,"raft.queue_time_us":40802,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":16567,"spinlock_wait_cycles":8936320,"thread_start_us":344020,"threads_started":1,"wal-append.queue_time_us":1353033}
W20260709 12:26:31.882694 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 49854.9us
I20260709 12:26:31.842248 18091 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
W20260709 12:26:31.884176 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.888386 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.889292 28064 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.890005 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5020) took 1916 ms. Trace:
I20260709 12:26:31.889947 28064 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:31.886298 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.892907 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.893196 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.854710 28832 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:31.889192 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73616.5us
I20260709 12:26:31.836234 18064 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:31.895277 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72069.7us
W20260709 12:26:31.901872 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.876405 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:26:31.904295 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:31.887792 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:31.890470 18098 rpcz_store.cc:276] 0709 12:26:29.973648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.984201 (+ 10553us) service_pool.cc:224] Handling call
0709 12:26:30.016002 (+ 31801us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.597680 (+1581678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.835809 (+238129us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001af754. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:31.889927 (+ 54118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3173185920}
I20260709 12:26:31.905735 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:31.870725 28608 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:31.923945 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.934168 28064 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:31.934417 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:31.935184 28064 consensus_queue.cc:1048] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:31.913635 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:31.936787 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.937693 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:31.906642 28936 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:26:31.884444 28938 raft_consensus.cc:697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:31.907087 28912 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.908s	user 0.002s	sys 0.006s
I20260709 12:26:31.941360 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.935187 18171 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:31.891598 28876 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.946871 18091 raft_consensus.cc:2410] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:26:31.947211 18170 pending_rounds.cc:85] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 20
I20260709 12:26:31.952606 18810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2042) took 2144 ms. Trace:
I20260709 12:26:31.955860 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.956990 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.960026 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1633) took 1031 ms. Trace:
I20260709 12:26:31.976368 28832 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.004s
I20260709 12:26:31.907349 28871 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.978071 28771 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.978545 28871 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:31.979732 28771 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.980438 28771 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:31.966781 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.898976 28836 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.976019 18094 rpcz_store.cc:276] 0709 12:26:31.157221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.191985 (+ 34764us) service_pool.cc:224] Handling call
0709 12:26:31.959584 (+767599us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b54002977d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:31.959877 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1315601664}
I20260709 12:26:31.976778 28521 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:31.960726 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.976246 18808 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 34. Current term is 35. Ops: []
I20260709 12:26:31.976099 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:31.984396 18170 pending_rounds.cc:107] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 28.21
I20260709 12:26:31.996358 18170 raft_consensus.cc:2889] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:31.911077 28931 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:31.988739 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.973500 18810 rpcz_store.cc:276] 0709 12:26:29.807924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.836287 (+ 28363us) service_pool.cc:224] Handling call
0709 12:26:29.848636 (+ 12349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.837945 (+989309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.952104 (+1114159us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b54004a60d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:31.952459 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2183004544}
I20260709 12:26:31.940338 28465 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:32.012543 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.997741 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:31.931818 28887 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:31.993563 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:32.003420 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:31.930047 28783 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:32.004612 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:31.947463 28722 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.939850 28542 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:31.946390 28903 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:32.016511 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.013407 18091 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:31.982323 28906 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:32.020805 18099 raft_consensus.cc:2393] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:31.965713 28934 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:32.027473 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 119611us
I20260709 12:26:32.035980 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.035681 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:32.038066 18780 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:32.057405 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.056706 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.040439 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.053192 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.012658 28465 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:32.070206 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.068217 18098 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:32.067302 18813 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:32.064414 18164 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:32.072139 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:32.076582 18806 raft_consensus.cc:2468] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 25.
W20260709 12:26:32.057757 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 252831us
I20260709 12:26:32.003687 28876 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:32.069181 28940 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.181s	user 0.002s	sys 0.000s
I20260709 12:26:32.055797 28906 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.088169 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.100261 18091 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:32.096122 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.007171 28836 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.055533 28871 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:32.094125 18811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1831) took 1929 ms. Trace:
I20260709 12:26:32.104357 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:32.028278 28832 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:26:32.083642 28465 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.035893 28887 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.059564 28903 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:32.087374 28930 raft_consensus.cc:2804] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:32.114555 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.048358 28722 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.111312 28935 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:32.123805 28740 consensus_meta.cc:287] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.654s	user 0.013s	sys 0.002s
I20260709 12:26:32.118708 18811 rpcz_store.cc:276] 0709 12:26:30.164424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.179145 (+ 14721us) service_pool.cc:224] Handling call
0709 12:26:30.209410 (+ 30265us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.209793 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:30.209859 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.210550 (+   691us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.523089 (+312539us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:30.523133 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.523227 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.866819 (+1343592us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:32.086939 (+220120us) raft_consensus.cc:1685] finished
0709 12:26:32.086977 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.094008 (+  7031us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":181293824,"thread_start_us":311669,"threads_started":1,"wal-append.queue_time_us":576082}
I20260709 12:26:32.130460 18812 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:32.088128 28926 raft_consensus.cc:2804] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:32.111634 28906 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.143787 18098 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:32.143208 28930 raft_consensus.cc:697] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:32.155196 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53614.5us
I20260709 12:26:32.155709 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:32.156370 28930 consensus_queue.cc:237] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:32.158671 28942 log.cc:927] Time spent T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:26:32.147166 28926 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.160526 28926 raft_consensus.cc:3060] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.160992 28804 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:32.161569 28804 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:32.163338 17989 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:32.164112 18019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1562) took 2368 ms (client timeout 1724 ms). Trace:
W20260709 12:26:32.164587 18019 rpcz_store.cc:269] 0709 12:26:29.795679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.795836 (+   157us) service_pool.cc:224] Handling call
0709 12:26:32.163871 (+2368035us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400268254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:32.164017 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3188967168}
I20260709 12:26:32.165069 17989 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.170182 28542 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:32.169971 28948 consensus_queue.cc:1048] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.121s
I20260709 12:26:32.173080 28887 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.140903 28935 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:32.174355 28935 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:32.162442 28903 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.157395 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.182662 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:32.185575 28945 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
I20260709 12:26:32.190583 28905 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:32.193121 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.193996 28934 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.177003 28804 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.195905 28876 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.196702 28926 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.197930 28804 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:32.201334 28858 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.919s	user 0.011s	sys 0.001s
I20260709 12:26:32.201431 28935 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.203876 28926 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.206849 28771 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.258960 28606 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.216190 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.133625 28938 consensus_queue.cc:237] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.148630 28783 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.148873 28740 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:32.150410 28662 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.018s	user 0.012s	sys 0.007s
W20260709 12:26:32.154606 28839 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.875s	user 0.017s	sys 0.000s
I20260709 12:26:32.303223 28528 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.303884 28528 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.225558 28836 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:32.227337 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.237442 28905 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:32.307130 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.238855 18094 raft_consensus.cc:1240] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 25. Current term is 26. Ops: []
W20260709 12:26:32.240206 18024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36156 (request call id 1561) took 2445 ms (client timeout 1686 ms). Trace:
W20260709 12:26:32.242803 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84976us
I20260709 12:26:32.244131 28956 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.140843 28928 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:32.247527 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5033) took 1066 ms. Trace:
I20260709 12:26:32.250182 18096 raft_consensus.cc:1275] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:32.255733 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:32.315912 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.258342 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.258687 28648 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.259055 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.316648 18098 rpcz_store.cc:276] 0709 12:26:31.181073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.151890 (+970817us) service_pool.cc:224] Handling call
0709 12:26:32.246894 (+ 95004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:32.247148 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.247178 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:32.247224 (+    46us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:32.247256 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.247314 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.247453 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:32.259780 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:32.330854 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:32.331652 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:32.263360 18095 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.572s	user 0.013s	sys 0.000s
I20260709 12:26:32.250192 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.333513 18095 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 34.
W20260709 12:26:32.201674 28913 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.000s
I20260709 12:26:31.634188 18097 rpcz_store.cc:276] 0709 12:26:30.050294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.211227 (+160933us) service_pool.cc:224] Handling call
0709 12:26:30.253723 (+ 42496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.254089 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:26:30.506313 (+252224us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.620022 (+1113709us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.632625 (+ 12603us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.632958 (+   333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.633035 (+    77us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:31.633110 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.633220 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.633414 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":249256,"prepare.queue_time_us":249893,"prepare.run_cpu_time_us":188,"prepare.run_wall_time_us":5404,"raft.queue_time_us":49489,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":17652,"spinlock_wait_cycles":106377088,"thread_start_us":1072110,"threads_started":1,"wal-append.queue_time_us":1110209}
W20260709 12:26:32.285300 28601 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.800s	user 0.016s	sys 0.000s
W20260709 12:26:32.288534 28814 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.628s	user 0.016s	sys 0.000s
W20260709 12:26:32.288826 28919 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.689s	user 0.012s	sys 0.001s
I20260709 12:26:32.292807 28806 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:32.294299 28935 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.264941 28804 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:32.302433 28740 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.224120 28722 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:32.309993 18024 rpcz_store.cc:269] 0709 12:26:29.795042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.795258 (+   216us) service_pool.cc:224] Handling call
0709 12:26:32.239853 (+2444595us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540024ca54. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:32.240073 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3069780736}
I20260709 12:26:32.313805 18094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5031) took 1133 ms. Trace:
I20260709 12:26:32.301096 28906 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.336620 28608 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:26:32.317464 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 61863.1us
I20260709 12:26:32.333961 18092 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:32.334144 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1634) took 1105 ms. Trace:
I20260709 12:26:32.334856 18091 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:26:32.335424 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 77200.5us
I20260709 12:26:32.333742 18164 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:32.306533 28606 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.338922 28662 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:32.355651 28814 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.414678 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.419632 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:32.361718 18094 rpcz_store.cc:276] 0709 12:26:31.180389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.150836 (+970447us) service_pool.cc:224] Handling call
0709 12:26:32.238514 (+ 87678us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:32.313107 (+ 74593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.313692 (+   585us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:32.324257 18171 raft_consensus.cc:2468] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 35.
I20260709 12:26:32.420868 18172 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 34. Current term is 35. Ops: []
I20260709 12:26:32.417784 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:32.413964 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1629) took 1832 ms. Trace:
I20260709 12:26:32.396667 18090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4172) took 2706 ms. Trace:
I20260709 12:26:32.420064 18095 rpcz_store.cc:276] 0709 12:26:31.229018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.644148 (+415130us) service_pool.cc:224] Handling call
0709 12:26:32.334073 (+689925us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:32.423095 18090 rpcz_store.cc:276] 0709 12:26:29.689957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.693127 (+  3170us) service_pool.cc:224] Handling call
0709 12:26:29.693745 (+   618us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.123903 (+1430158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.365899 (+1241996us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540035ded4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:32.396561 (+ 30662us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3286352256}
I20260709 12:26:32.401804 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.416354 18096 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:26:32.415644 18097 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:26:32.422694 18093 rpcz_store.cc:276] 0709 12:26:30.581888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.728844 (+146956us) service_pool.cc:224] Handling call
0709 12:26:30.787462 (+ 58618us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.826816 (+ 39354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.826868 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.915588 (+ 88720us) log.cc:844] Serialized 26 byte log entry
0709 12:26:31.394846 (+479258us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:26:31.421364 (+ 26518us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.421695 (+   331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.421784 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.882984 (+461200us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b1800102940. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:32.403397 (+520413us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b1800102940. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:32.413589 (+ 10192us) raft_consensus.cc:1685] finished
0709 12:26:32.413627 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.413869 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27085,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":3520,"raft.queue_time_us":280621,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":194,"spinlock_wait_cycles":20062848,"thread_start_us":294633,"threads_started":1,"wal-append.queue_time_us":372854}
I20260709 12:26:32.379760 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.430684 18165 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:32.387425 18022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36156 (request call id 1563) took 2591 ms. Trace:
I20260709 12:26:32.436367 18091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1635) took 1182 ms. Trace:
I20260709 12:26:32.434190 18022 rpcz_store.cc:276] 0709 12:26:29.797323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.801363 (+  4040us) service_pool.cc:224] Handling call
0709 12:26:29.801952 (+   589us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:31.458794 (+1656842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.387008 (+928214us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54001d0954. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:32.387319 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3698329344}
W20260709 12:26:32.440600 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.339401 28817 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:32.436993 18091 rpcz_store.cc:276] 0709 12:26:31.254320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.069717 (+815397us) service_pool.cc:224] Handling call
0709 12:26:32.436254 (+366537us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:32.341264 28928 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:32.444126 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.444353 18136 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:32.455542 18138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.026s	sys 0.077s
W20260709 12:26:32.456231 18138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.026s	sys 0.077s
I20260709 12:26:32.406940 28860 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.355923 28926 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:32.414213 28962 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:26:32.457800 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:32.488662 18166 raft_consensus.cc:1240] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
W20260709 12:26:32.481472 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:32.456604 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 515403us
W20260709 12:26:32.463599 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 44030.4us
I20260709 12:26:32.438814 28608 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:32.438104 28955 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:32.469313 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.469870 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.461987 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:32.469385 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.491874 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.396312 28740 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:32.463208 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:32.368064 28924 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.518s	user 0.014s	sys 0.000s
I20260709 12:26:32.471045 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.471572 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.339264 28648 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.470074 28906 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.365497 28804 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.500833 28814 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:32.491084 28926 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.507555 28886 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:32.562291 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 99226.7us
I20260709 12:26:32.526800 28608 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.510092 28956 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:32.410537 28919 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.080s
I20260709 12:26:32.459539 28916 consensus_queue.cc:1059] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.051s
I20260709 12:26:32.511260 28943 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:32.366700 28806 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.512701 28928 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.513994 28845 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:32.342094 28913 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.518030 28528 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.518741 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.497785 18165 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:32.528659 18173 raft_consensus.cc:1240] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:32.529309 18812 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:32.347694 28601 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.535583 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:32.500120 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 56461.7us
W20260709 12:26:32.546229 18092 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40710 (request call id 1638) took 1254 ms (client timeout 1560 ms). Trace:
I20260709 12:26:32.442747 28662 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
I20260709 12:26:32.488211 28811 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.557647 18171 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:32.563133 28949 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:32.491834 28832 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:32.508425 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.603899 18093 raft_consensus.cc:2468] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:32.509284 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.604374 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.605052 18090 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:32.509642 28606 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:32.465665 28817 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.408234 28905 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.128s
W20260709 12:26:32.577742 18815 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.008s	sys 0.016s
W20260709 12:26:32.638159 18815 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.008s	sys 0.017s
I20260709 12:26:32.586321 18095 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 15. (term mismatch)
I20260709 12:26:32.459411 28839 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.102s
I20260709 12:26:32.552237 28789 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:32.562127 28836 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:32.594744 18092 rpcz_store.cc:269] 0709 12:26:31.277659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.375289 (+ 97630us) service_pool.cc:224] Handling call
0709 12:26:32.531379 (+1156090us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b5400297cd4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:32.531610 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1995406208}
I20260709 12:26:32.604961 18097 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:32.563220 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:32.583086 28771 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:32.579200 28943 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.631480 18808 raft_consensus.cc:1240] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:32.551538 28924 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.556257 28740 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.639851 18807 raft_consensus.cc:1240] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:32.640290 18095 pending_rounds.cc:85] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 14
I20260709 12:26:32.665603 18095 pending_rounds.cc:107] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 28.15
I20260709 12:26:32.666052 18095 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:32.639664 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58850 (request call id 1837) took 1285 ms. Trace:
I20260709 12:26:32.580502 28930 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 30.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.051s
I20260709 12:26:32.541374 28955 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:32.655083 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 92010.8us
I20260709 12:26:32.657953 28943 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.652643 28771 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.662374 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:32.666850 18815 rpcz_store.cc:276] 0709 12:26:31.354216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.354366 (+   150us) service_pool.cc:224] Handling call
0709 12:26:31.375148 (+ 20782us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:31.674676 (+299528us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:31.674733 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:31.761752 (+ 87019us) log.cc:844] Serialized 26 byte log entry
0709 12:26:31.989044 (+227292us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:32.638781 (+649737us) log.cc:844] Serialized 14 byte log entry
0709 12:26:32.639139 (+   358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.639255 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:32.639325 (+    70us) raft_consensus.cc:1685] finished
0709 12:26:32.639356 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.639571 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":223626,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":9080,"thread_start_us":1071440,"threads_started":3,"wal-append.queue_time_us":359592}
I20260709 12:26:32.634673 28832 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.693113 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.665182 28563 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:32.668882 28935 raft_consensus.cc:2804] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:26:32.675258 28465 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.529s	user 0.011s	sys 0.003s
I20260709 12:26:32.697844 28648 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:32.641283 28949 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.660781 28836 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.702077 28858 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:32.703258 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.704113 28836 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.706275 18811 raft_consensus.cc:2410] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:26:32.656912 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.707260 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.707546 18093 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:32.707942 18095 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:32.714727 18778 leader_election.cc:336] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.560s (SENT)
I20260709 12:26:32.721625 18167 raft_consensus.cc:1240] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
W20260709 12:26:32.722193 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60006.3us
I20260709 12:26:32.653769 28789 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.694500 28771 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:32.705966 28839 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.706825 28935 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.705349 28563 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:32.744073 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.750824 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.751606 18092 raft_consensus.cc:2468] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
W20260709 12:26:32.753633 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:32.762732 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 263732us
W20260709 12:26:32.765473 18778 leader_election.cc:336] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.611s (SENT)
I20260709 12:26:32.765802 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:32.739123 28935 raft_consensus.cc:3060] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:32.784660 28960 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:26:32.748222 28606 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.785915 28606 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.791276 28926 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.794567 28563 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.796257 28563 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.711021 28465 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:32.769109 28879 consensus_meta.cc:287] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.897s	user 0.019s	sys 0.000s
I20260709 12:26:32.741704 28930 consensus_queue.cc:1059] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:26:32.822170 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:32.826597 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.797704 28924 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:32.843376 28806 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.844518 18809 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 34. Current term is 35. Ops: []
I20260709 12:26:32.844823 18096 raft_consensus.cc:1275] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:26:32.845037 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.845639 28740 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:32.846148 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.847805 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.849298 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:32.806016 18095 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
W20260709 12:26:32.832569 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 84503.5us
I20260709 12:26:32.835465 18093 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
W20260709 12:26:32.841583 18778 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.686s (SENT)
I20260709 12:26:32.853566 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.854429 18093 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:32.821010 28804 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.784693 28789 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:32.748651 28839 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.810483 28919 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:26:32.875034 18780 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:32.854449 28963 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:26:32.888518 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:32.891670 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:32.799710 28916 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:26:32.893103 18091 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 34 because replica has last-logged OpId of term: 32 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 24.
W20260709 12:26:32.900990 18780 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.901924 28938 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.832193 28648 consensus_queue.cc:1059] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:26:32.903806 28906 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:32.869359 28903 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.005s	sys 0.018s
I20260709 12:26:32.894798 28913 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.926582 28905 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:32.916545 18780 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:32.904937 28955 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:32.914044 28789 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.939059 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45040 (request call id 5035) took 1661 ms. Trace:
I20260709 12:26:32.941190 18090 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:32.952051 18090 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:32.829567 28879 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.919993 28919 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:32.947588 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 194131us
I20260709 12:26:32.952739 18097 raft_consensus.cc:1240] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:32.950933 18099 rpcz_store.cc:276] 0709 12:26:31.278150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.235944 (+957794us) service_pool.cc:224] Handling call
0709 12:26:32.236540 (+   596us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.266573 (+ 30033us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.266616 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:32.322978 (+ 56362us) log.cc:844] Serialized 26 byte log entry
0709 12:26:32.677440 (+354462us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:32.677509 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.677624 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:32.938799 (+261175us) raft_consensus.cc:1685] finished
0709 12:26:32.938824 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.938978 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23078,"raft.queue_time_us":30725,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":24501,"thread_start_us":297981,"threads_started":1,"wal-append.queue_time_us":496922}
W20260709 12:26:32.942394 28903 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.006s	sys 0.020s
W20260709 12:26:32.959362 28968 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.230s	user 0.002s	sys 0.000s
I20260709 12:26:32.968762 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:32.969727 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 82978us
I20260709 12:26:32.936861 28804 consensus_queue.cc:1059] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:26:32.975132 18780 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:32.942850 28906 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.982132 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.982417 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:32.984440 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.985018 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.986688 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:32.952675 28905 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:32.987583 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:32.987866 18172 raft_consensus.cc:2468] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:33.025782 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:33.031216 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:33.033941 28906 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:33.034804 28465 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.034935 28836 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.986313 28926 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:33.047427 28976 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:26:33.049192 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 69200.9us
I20260709 12:26:33.051213 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:33.053401 28064 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:33.054490 28972 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:26:33.056733 28970 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:26:33.058692 28966 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
W20260709 12:26:33.060199 28961 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.201s	user 0.001s	sys 0.001s
I20260709 12:26:33.060969 28817 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.061753 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4042) took 1077 ms. Trace:
I20260709 12:26:33.062211 18168 rpcz_store.cc:276] 0709 12:26:31.983901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.004510 (+ 20609us) service_pool.cc:224] Handling call
0709 12:26:32.007245 (+  2735us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.007808 (+   563us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.007863 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:32.043169 (+ 35306us) log.cc:844] Serialized 26 byte log entry
0709 12:26:32.572919 (+529750us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:32.572992 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.573114 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:33.061480 (+488366us) raft_consensus.cc:1685] finished
0709 12:26:33.061508 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:33.061683 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8188,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":6289,"thread_start_us":476744,"threads_started":1,"wal-append.queue_time_us":697675}
I20260709 12:26:33.063469 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.064832 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:32.980775 28814 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.066291 28814 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.067227 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.037740 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.068750 28814 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.030241 28919 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:32.986687 28662 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.038139 28839 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:33.070289 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.070926 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.072135 28967 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260709 12:26:33.038553 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:32.956660 28789 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.087008 28811 raft_consensus.cc:3055] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:26:33.087630 28811 raft_consensus.cc:740] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:33.088776 28789 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:32.978781 28903 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:33.068776 28980 consensus_queue.cc:1048] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:26:33.046506 18164 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:26:33.061033 18806 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:33.064985 28971 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:26:33.067140 28926 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:33.036403 28879 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:33.039412 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 60406.9us
W20260709 12:26:33.124944 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.035702 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 48199.9us
I20260709 12:26:32.980371 28955 raft_consensus.cc:2804] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:26:33.084183 28905 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.088650 28811 consensus_queue.cc:260] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.130904 28811 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:33.131645 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.040791 18173 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:33.064846 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 1.766s (SENT)
I20260709 12:26:33.078889 18136 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:32.985088 28740 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.113692 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.116189 18170 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:33.136082 28740 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.137580 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.118782 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.138325 18099 raft_consensus.cc:2468] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:33.138825 18093 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:33.139322 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:33.149595 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.135998 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:33.152269 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 87455.7us
W20260709 12:26:33.158324 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 89543.4us
W20260709 12:26:33.164664 28975 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.230s	user 0.002s	sys 0.000s
I20260709 12:26:33.165001 28955 raft_consensus.cc:697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:33.166195 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.166736 28955 consensus_queue.cc:237] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.167867 28465 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:33.173067 18814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58850 (request call id 1838) took 1270 ms. Trace:
I20260709 12:26:33.173633 18814 rpcz_store.cc:276] 0709 12:26:31.903146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.903855 (+   709us) service_pool.cc:224] Handling call
0709 12:26:33.172773 (+1268918us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b54004b46d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:33.172985 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2287357824}
I20260709 12:26:33.176910 28783 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:33.177666 28783 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:33.179422 28740 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.185230 28949 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:33.179121 28783 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.187711 28783 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:33.198263 17987 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.199180 28601 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:33.200376 28916 raft_consensus.cc:3055] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:33.200830 28916 raft_consensus.cc:740] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:33.201766 28916 consensus_queue.cc:260] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.203174 28916 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:33.204388 28608 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.002s
I20260709 12:26:33.204885 28608 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.209281 28845 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.211938 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.211493 28845 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.215220 28943 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.218190 18171 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 31: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:33.218377 28606 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.218986 28845 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.219620 28956 raft_consensus.cc:3055] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:33.154655 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:33.222873 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 127753us
I20260709 12:26:33.200253 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:33.232147 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 101207us
I20260709 12:26:33.220177 28832 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:33.227087 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 73814.8us
I20260709 12:26:33.235919 18166 raft_consensus.cc:1240] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:33.234457 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4190) took 1319 ms. Trace:
I20260709 12:26:33.231215 18094 raft_consensus.cc:2468] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
W20260709 12:26:33.235132 18138 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.255234 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:33.255870 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.256222 18780 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:33.254968 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 95492.6us
I20260709 12:26:33.253710 18138 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:33.257360 18780 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.262282 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1614) took 1329 ms. Trace:
I20260709 12:26:33.220494 28858 raft_consensus.cc:3055] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:26:33.263379 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.252238 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:33.249199 18098 rpcz_store.cc:276] 0709 12:26:31.914854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.330828 (+415974us) service_pool.cc:224] Handling call
0709 12:26:32.345699 (+ 14871us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.346567 (+   868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.346625 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:32.347451 (+   826us) log.cc:844] Serialized 26 byte log entry
0709 12:26:32.512326 (+164875us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b18003ba100. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ace07 00007f6c1c4079a9 00007f6c135c969b 00007f6c135c793f 00007f6c135c9c02 00007f6c1625120b 00007f6c16256d23 00007f6c162570d1 00007f6c16672a0f
0709 12:26:32.724308 (+211982us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:32.724370 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.724551 (+   181us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:33.163892 (+439341us) spinlock_profiling.cc:243] Waited 5.69 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:33.234136 (+ 70244us) raft_consensus.cc:1685] finished
0709 12:26:33.234168 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:33.234365 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":135,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":10842,"spinlock_wait_cycles":48868480,"thread_start_us":211777,"threads_started":1,"wal-append.queue_time_us":472460}
I20260709 12:26:33.258615 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:33.278450 28845 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:33.279050 28845 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.279450 28845 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:33.280447 18137 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.280761 18136 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:33.282219 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.283337 28465 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.283749 28978 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:33.211975 28789 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.282279 28648 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:33.282107 28751 consensus_queue.cc:1059] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:33.284806 28648 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.284785 28789 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.285516 28648 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.285694 28751 raft_consensus.cc:3055] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:33.286188 28751 raft_consensus.cc:740] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:33.217651 28943 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:33.287178 28751 consensus_queue.cc:260] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.283469 18091 raft_consensus.cc:1240] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:33.291721 28981 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:33.292110 18780 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.291042 28751 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:33.293078 18780 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.294624 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.295735 18810 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:33.301995 28905 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.302803 28905 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.230515 28956 raft_consensus.cc:740] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:33.305537 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.156167 28804 consensus_queue.cc:1059] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.150s
W20260709 12:26:33.306303 28971 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:26:33.306854 28906 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:33.310412 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.282618 18092 raft_consensus.cc:1240] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 33. Current term is 34. Ops: []
I20260709 12:26:33.262724 18167 rpcz_store.cc:276] 0709 12:26:31.932469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.933114 (+   645us) service_pool.cc:224] Handling call
0709 12:26:31.972947 (+ 39833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:32.722401 (+749454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.262057 (+539656us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540002a3d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:33.262222 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1721033088}
I20260709 12:26:33.332655 18173 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:33.339450 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4055) took 1180 ms. Trace:
W20260709 12:26:33.339790 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 87630.1us
W20260709 12:26:33.339883 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46841.5us
W20260709 12:26:33.340582 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 111651us
I20260709 12:26:33.346639 18166 rpcz_store.cc:276] 0709 12:26:32.159316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.493293 (+333977us) service_pool.cc:224] Handling call
0709 12:26:32.551248 (+ 57955us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:33.257749 (+706501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.339023 (+ 81274us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54001b4254. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b7e01 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2 00007f6c149f248c 00007f6c149f18c6
0709 12:26:33.339334 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1574000000}
W20260709 12:26:33.341233 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 111168us
W20260709 12:26:33.348026 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:33.342154 28563 consensus_meta.cc:287] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.004s
I20260709 12:26:33.309397 28858 raft_consensus.cc:740] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:33.358372 18780 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:33.361127 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:33.363240 18780 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.309075 28528 raft_consensus.cc:3055] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:26:33.343816 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 143685us
I20260709 12:26:33.348344 28662 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:33.364843 28064 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.308516 28879 consensus_queue.cc:1059] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.051s
I20260709 12:26:33.381000 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.385681 18097 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 34 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:26:33.378913 28064 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.352954 28804 consensus_queue.cc:1243] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer 910f8c8d787e4abbb092413c1f90ebf3 log is divergent from this leader: its last log entry 29.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:26:33.388794 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.390009 18095 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 34.
I20260709 12:26:33.390390 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.390861 18092 raft_consensus.cc:1275] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:26:33.351676 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 71285.6us
I20260709 12:26:33.395222 18173 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 30 index: 24. Preceding OpId from leader: term: 33 index: 26. (index mismatch)
I20260709 12:26:33.384236 28662 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.362241 28608 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:33.398909 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.403570 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:33.367498 28528 raft_consensus.cc:740] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:33.409340 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:33.410809 18090 raft_consensus.cc:2393] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 31: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:33.363584 28980 consensus_queue.cc:1059] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:26:33.406942 28662 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:33.383491 28978 consensus_queue.cc:1059] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
W20260709 12:26:33.367169 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:33.390537 28740 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
W20260709 12:26:33.416713 28935 consensus_meta.cc:287] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.006s
W20260709 12:26:33.416378 28771 consensus_meta.cc:287] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.670s	user 0.014s	sys 0.003s
W20260709 12:26:33.422947 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55838.3us
I20260709 12:26:33.416863 28804 raft_consensus.cc:2764] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20260709 12:26:33.428079 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:33.448633 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.463218 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Timed out: RequestConsensusVote RPC to 127.16.102.2:40047 timed out after 2.141s (SENT)
I20260709 12:26:33.483695 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.484164 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.426626 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:33.482148 18780 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:33.495436 18780 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:33.438700 28965 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.581s	user 0.000s	sys 0.004s
I20260709 12:26:33.430598 28905 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:33.497189 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 167755us
I20260709 12:26:33.500027 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:33.496515 18780 leader_election.cc:336] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.415211 28858 consensus_queue.cc:260] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.501516 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:33.495666 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69677.1us
I20260709 12:26:33.512208 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:33.445874 28722 raft_consensus.cc:3055] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:26:33.510133 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:33.450554 28789 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.509934 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 37.
W20260709 12:26:33.515005 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.512223 18167 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 26: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:33.483482 28817 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.518461 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:33.527981 18169 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:26:33.529871 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:33.534376 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.536958 18170 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:26:33.494927 28763 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:33.543711 18171 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:26:33.508963 28903 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:33.443150 28935 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.524809 28465 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:33.519338 28722 raft_consensus.cc:740] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:33.446615 28771 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.532758 28905 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:33.497104 28836 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:26:33.550727 18165 raft_consensus.cc:1275] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:33.441042 28528 consensus_queue.cc:260] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.513581 28928 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:26:33.514544 28804 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:26:33.550300 28858 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:33.577958 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 150025us
W20260709 12:26:33.595378 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 111365us
I20260709 12:26:33.567085 28465 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:33.601171 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 65117.8us
I20260709 12:26:33.538290 28817 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.549919 28763 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:33.613408 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.575567 28903 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:33.609081 18165 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:33.574028 28836 consensus_queue.cc:1243] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer 768723648a9444fda53659e6a8cc3091 log is divergent from this leader: its last log entry 26.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:26:33.622954 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:33.626001 17987 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:33.641229 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.643069 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.604928 28465 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:33.656231 18092 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:33.623958 28994 raft_consensus.cc:2804] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:33.665410 18090 raft_consensus.cc:1275] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:26:33.666167 18090 pending_rounds.cc:85] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 17
I20260709 12:26:33.666460 18090 pending_rounds.cc:107] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:26:33.666786 18090 raft_consensus.cc:2889] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:33.605337 28563 consensus_queue.cc:1048] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
W20260709 12:26:33.673372 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 131881us
W20260709 12:26:33.657856 28986 log.cc:927] Time spent T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:26:33.671965 28722 consensus_queue.cc:260] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.675078 17988 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:33.644659 28996 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:33.623183 28905 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:26:33.743427 28839 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:33.744271 28839 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:33.744705 28905 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:33.675113 28845 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.745352 28905 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.745579 28839 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.692131 28064 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.668488 28994 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:33.695142 28608 raft_consensus.cc:3055] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:33.671123 28978 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:26:33.771027 28994 raft_consensus.cc:3060] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.771039 28608 raft_consensus.cc:740] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:33.772547 28608 consensus_queue.cc:260] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:33.704969 18780 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.671594 28995 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.710897 28973 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:33.775157 28973 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:33.775832 28608 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:33.714620 28988 log.cc:927] Time spent T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:26:33.719527 28811 consensus_queue.cc:1048] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:26:33.720084 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.722012 28935 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.727604 18814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.728443 28817 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:33.692202 29002 log.cc:927] Time spent T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:26:33.742988 28789 raft_consensus.cc:2804] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:33.747560 28993 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.172s	user 0.001s	sys 0.001s
I20260709 12:26:33.747768 28771 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.747824 28845 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.736493 28763 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.748761 18173 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 32: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
W20260709 12:26:33.691498 28998 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:26:33.750155 28956 consensus_queue.cc:260] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.694216 28562 raft_consensus.cc:3055] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:33.825167 28562 raft_consensus.cc:740] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:33.825384 28956 raft_consensus.cc:3060] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:33.755312 28528 raft_consensus.cc:3060] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:33.756654 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:33.757257 18811 raft_consensus.cc:1240] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 34. Current term is 35. Ops: []
I20260709 12:26:33.757557 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:33.827669 18092 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:33.757939 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:33.829037 18809 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.758561 28905 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:33.764271 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.765373 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:33.767513 18807 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:33.770081 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 127237us
I20260709 12:26:33.714382 28722 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:33.792102 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.799065 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.811777 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 464626us
W20260709 12:26:33.817888 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 192049us
I20260709 12:26:33.838485 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:33.814328 29005 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:33.829802 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46700.2us
I20260709 12:26:33.777787 28934 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:33.811051 28789 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:33.832161 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.849110 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.854836 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:33.857234 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:33.865381 17987 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:33.869793 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:33.877146 18173 raft_consensus.cc:1275] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:26:33.810776 28601 consensus_queue.cc:1059] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.047s
W20260709 12:26:33.853950 28916 consensus_meta.cc:287] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.650s	user 0.012s	sys 0.005s
I20260709 12:26:33.810446 28662 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.867532 28789 raft_consensus.cc:3060] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:33.859689 28935 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:33.900033 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.866006 28978 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:33.893703 28928 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:26:33.891367 28926 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:33.914407 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.845212 28955 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.047s
I20260709 12:26:33.875155 29003 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:33.888216 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:33.919160 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.874938 28879 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:33.910943 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.903573 28783 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:26:33.929582 28783 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:33.929661 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:33.924120 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.930547 18172 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.931092 18169 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:33.894062 28836 raft_consensus.cc:2804] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:33.934423 28836 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:33.934957 28836 raft_consensus.cc:3060] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:33.935705 29012 consensus_queue.cc:1059] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 30.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:33.936089 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.936779 18167 raft_consensus.cc:2468] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:33.938309 18815 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58848 (request call id 2098) took 1178 ms. Trace:
I20260709 12:26:33.927999 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.931596 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:33.932250 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:33.938753 18815 rpcz_store.cc:276] 0709 12:26:32.759930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.760128 (+   198us) service_pool.cc:224] Handling call
0709 12:26:32.779209 (+ 19081us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.935015 (+155806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.935052 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:33.511825 (+576773us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b2c000fafd8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:26:33.534538 (+ 22713us) log.cc:844] Serialized 26 byte log entry
0709 12:26:33.662639 (+128101us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:33.662697 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.662804 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:33.937910 (+275106us) raft_consensus.cc:1685] finished
0709 12:26:33.937943 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:33.938237 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1934,"raft.queue_time_us":30562,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":11602,"spinlock_wait_cycles":7357696,"thread_start_us":242968,"threads_started":2,"wal-append.queue_time_us":259868}
I20260709 12:26:33.933768 28783 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:26:33.940689 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:33.944512 28995 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:33.831297 28562 consensus_queue.cc:260] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:33.897897 28804 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.859416 29005 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:33.957899 29005 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:33.869656 28934 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:33.922842 28751 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.008s
I20260709 12:26:33.969221 18172 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
W20260709 12:26:33.988898 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 139930us
I20260709 12:26:33.950977 28860 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:33.998271 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 178010us
I20260709 12:26:33.917368 28771 consensus_queue.cc:1059] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:26:33.916407 28935 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:34.017757 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 76007us
I20260709 12:26:34.017530 28601 consensus_queue.cc:1243] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 31.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:26:34.041694 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.005700 29004 raft_consensus.cc:2804] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:34.046414 29004 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.046938 29004 raft_consensus.cc:3060] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:34.048300 28771 raft_consensus.cc:3055] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:34.048794 28771 raft_consensus.cc:740] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:34.049168 29021 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:34.050719 29025 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:34.051491 29025 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.058558 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.071112 28973 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.072659 28973 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:34.071844 28806 raft_consensus.cc:3055] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:34.074409 28806 raft_consensus.cc:740] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:33.977257 28804 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.075489 28806 consensus_queue.cc:260] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.076112 29025 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:34.076555 28662 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:34.077268 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.079073 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.077556 28804 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.078032 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.076951 28806 raft_consensus.cc:3060] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:34.084649 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.085345 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.087257 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:34.044976 28987 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:26:34.090204 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.093710 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.006074 28817 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:34.113585 18809 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
W20260709 12:26:34.090643 28648 consensus_meta.cc:287] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.804s	user 0.019s	sys 0.013s
I20260709 12:26:34.120527 18164 raft_consensus.cc:2468] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:34.121459 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.079193 28935 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.143843 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:34.128665 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.156953 28064 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.158882 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.159499 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.155220 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.160845 18173 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:34.161760 18099 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 32 index: 25. Preceding OpId from leader: term: 33 index: 26. (index mismatch)
W20260709 12:26:34.164033 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 275917us
I20260709 12:26:34.164187 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.165908 18097 raft_consensus.cc:2410] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 33 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:26:34.141372 29025 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:34.167114 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.167634 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.165850 28064 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.169624 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.170873 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.173528 18099 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:26:34.137482 28817 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.146369 28648 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.182245 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.184547 18095 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:26:34.184799 28648 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:34.185675 29025 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.185712 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.187980 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.188432 29025 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:34.188975 18095 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:34.157488 28926 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.190071 28839 raft_consensus.cc:3055] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:34.190551 28839 raft_consensus.cc:740] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:34.191035 18095 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 21. (term mismatch)
I20260709 12:26:34.191465 18095 pending_rounds.cc:85] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 20
I20260709 12:26:34.191835 18095 pending_rounds.cc:107] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 32.21
I20260709 12:26:34.191511 28839 consensus_queue.cc:260] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.192158 28562 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:34.171522 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.192139 18095 raft_consensus.cc:2889] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:34.198426 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.217710 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:34.209591 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.207711 18173 raft_consensus.cc:2368] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30.
I20260709 12:26:34.198171 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:34.235765 18097 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 33 index: 26. Preceding OpId from leader: term: 34 index: 27. (index mismatch)
I20260709 12:26:34.164578 28879 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:34.239343 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 74273.1us
W20260709 12:26:34.240288 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.240536 17989 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.242137 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:34.240464 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70846.7us
I20260709 12:26:34.243971 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.169433 28465 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.246393 28817 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.176744 28836 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.246140 28879 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.247602 18165 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 34. Current term is 35. Ops: []
I20260709 12:26:34.247991 28465 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.237275 29003 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:34.248436 28836 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.249047 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.249778 18809 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:34.249915 29003 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.254343 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.155334 28934 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:34.246026 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.243000 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:34.150949 28955 consensus_queue.cc:1048] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.080s
I20260709 12:26:34.256785 18169 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: [29.15-29.15]
I20260709 12:26:34.255787 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.255290 18098 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:34.267802 28064 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.274344 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.270447 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.267606 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
W20260709 12:26:34.271643 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.267789 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.260783 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:34.279352 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.280923 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.152881 28601 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.108s
I20260709 12:26:34.284055 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.299929 28064 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:34.317951 28064 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:34.318471 28064 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:34.271579 28836 raft_consensus.cc:2804] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:34.326371 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.328195 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.329058 18098 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:34.304482 18097 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:34.254046 28740 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:34.336117 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.336298 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.254046 28903 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:34.189927 28926 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.320158 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:34.258153 29012 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:34.316358 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:34.358258 18064 leader_election.cc:400] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30.
I20260709 12:26:34.358889 18064 leader_election.cc:403] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:26:34.285105 28608 consensus_meta.cc:287] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.508s	user 0.004s	sys 0.012s
I20260709 12:26:34.360673 28956 raft_consensus.cc:2764] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
W20260709 12:26:34.361294 28858 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.756s	user 0.008s	sys 0.006s
I20260709 12:26:34.362228 18172 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:34.325695 28836 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:34.323145 18099 raft_consensus.cc:1275] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:26:34.273959 29025 raft_consensus.cc:2804] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:34.384351 29025 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.384886 29025 raft_consensus.cc:3060] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:34.356554 29024 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:34.386448 29024 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.389297 29024 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:34.324465 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.328477 18173 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:34.331007 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.333655 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:34.292834 18136 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.398043 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.176744 28783 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.397526 18097 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:26:34.400995 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.348330 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.402441 18090 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:26:34.188874 28771 consensus_queue.cc:260] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.305227 18164 raft_consensus.cc:2368] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 30. Current term is 31.
I20260709 12:26:34.363299 18093 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
W20260709 12:26:34.369930 28528 consensus_meta.cc:287] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.526s	user 0.005s	sys 0.009s
I20260709 12:26:34.335422 28905 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.378455 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:34.391744 28836 raft_consensus.cc:3060] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:34.324344 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.384377 18168 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
W20260709 12:26:34.394656 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 402645us
I20260709 12:26:34.403678 18166 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 32: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:34.431313 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:34.408632 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 138328us
W20260709 12:26:34.383306 29006 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.480s	user 0.007s	sys 0.000s
I20260709 12:26:34.422906 28064 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.426311 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.432816 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.440186 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.440958 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.444249 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:34.444670 18099 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:34.444181 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:34.445652 18090 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:34.441211 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.439126 18173 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 33.
I20260709 12:26:34.448493 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.452117 18064 leader_election.cc:400] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 30. Current term is 31.
I20260709 12:26:34.452495 18099 pending_rounds.cc:85] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 17
I20260709 12:26:34.452620 18064 leader_election.cc:403] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:26:34.452884 18099 pending_rounds.cc:107] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:26:34.453182 18099 raft_consensus.cc:2889] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:34.464422 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.465135 18166 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 31: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:34.466374 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.467212 18169 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 33.
W20260709 12:26:34.472067 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.474359 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:34.477048 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.479918 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:34.482262 18091 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:26:34.483978 18167 raft_consensus.cc:2468] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 34.
W20260709 12:26:34.484167 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51470.2us
I20260709 12:26:34.486092 18167 raft_consensus.cc:1240] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 30. Current term is 31. Ops: []
W20260709 12:26:34.486603 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 245109us
I20260709 12:26:34.488477 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:34.468708 29007 log.cc:927] Time spent T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:26:34.429780 29022 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:34.502019 18165 raft_consensus.cc:2393] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 27: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:26:34.508105 28722 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.657s	user 0.014s	sys 0.000s
I20260709 12:26:34.473584 28995 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:34.510803 18097 raft_consensus.cc:2468] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 35.
I20260709 12:26:34.481289 28804 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 30. Current term is 31.
I20260709 12:26:34.519797 28804 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:34.520292 28804 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.520676 28804 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:34.522527 29026 consensus_queue.cc:1059] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20260709 12:26:34.523960 29026 raft_consensus.cc:3055] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:34.524497 29026 raft_consensus.cc:740] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:34.488512 28806 consensus_queue.cc:1048] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:26:34.520691 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:34.531275 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:34.533144 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.487193 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:34.488679 29013 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:34.554266 28926 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:34.554798 29029 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:26:34.556083 18091 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:34.558804 28783 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:34.494666 28879 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:34.557688 18091 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Rejecting Update request from peer ae9641d5c3184572a102d51a75f670ee for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:34.510864 28771 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:34.509295 28722 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.040s
I20260709 12:26:34.478127 28955 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.071s
I20260709 12:26:34.515353 28995 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:34.561513 29004 consensus_meta.cc:287] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.514s	user 0.007s	sys 0.007s
I20260709 12:26:34.483069 28934 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.066s
W20260709 12:26:34.583985 28994 consensus_meta.cc:287] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.812s	user 0.009s	sys 0.010s
W20260709 12:26:34.610936 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 123826us
I20260709 12:26:34.613595 18168 raft_consensus.cc:1240] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 28. Current term is 29. Ops: [26.19-28.20]
I20260709 12:26:34.619815 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:34.614852 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:34.620342 18780 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:34.621150 18166 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 34: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:34.617130 18090 raft_consensus.cc:2468] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:34.617637 18172 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:34.624336 18168 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:34.582242 28601 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:34.638533 18096 raft_consensus.cc:1240] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: [29.15-29.15]
W20260709 12:26:34.641470 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.644171 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:34.655723 18136 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:34.659385 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:34.660600 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.595868 28955 consensus_queue.cc:1243] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer ae9641d5c3184572a102d51a75f670ee log is divergent from this leader: its last log entry 25.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:26:34.601344 28995 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:34.679649 28806 raft_consensus.cc:2804] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:34.712169 28806 raft_consensus.cc:697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:34.713585 28806 consensus_queue.cc:237] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.719069 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.727525 18806 raft_consensus.cc:1275] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 21. (term mismatch)
I20260709 12:26:34.728221 18806 pending_rounds.cc:85] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 20
I20260709 12:26:34.728561 18806 pending_rounds.cc:107] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 30.21
I20260709 12:26:34.728888 18806 raft_consensus.cc:2889] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:34.733279 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 242949us
W20260709 12:26:34.742954 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98948.3us
I20260709 12:26:34.746704 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:34.642306 29018 consensus_queue.cc:1048] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.108s
I20260709 12:26:34.738129 28905 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.688709 28983 raft_consensus.cc:2804] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:26:34.730476 28996 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:34.525666 29026 consensus_queue.cc:260] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:34.734054 28789 consensus_meta.cc:287] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.819s	user 0.016s	sys 0.000s
I20260709 12:26:34.758541 18809 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:34.760232 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:34.720839 28839 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:34.619364 29004 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.718645 29033 consensus_queue.cc:1059] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.078s
I20260709 12:26:34.730559 28879 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:34.786883 18137 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.750519 28926 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:34.771054 28648 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:34.770876 17908 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a6ac73d7138d4200bbfcbf2f84c2c775 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ae9641d5c3184572a102d51a75f670ee is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 30 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:34.771416 28722 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:34.779016 18098 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:34.775002 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:34.781054 29003 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:34.644382 28994 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.767314 28064 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.789592 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:34.812426 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:34.761194 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.782043 29018 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:34.813078 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:34.932452 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:34.929379 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 114763us
I20260709 12:26:34.952168 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:34.818933 28858 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905) with higher term. Message: Invalid argument: T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for earlier term 29. Current term is 30.
W20260709 12:26:34.955761 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 475886us
W20260709 12:26:34.869284 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:34.958014 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 196934us
I20260709 12:26:34.783298 28996 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.825683 28817 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:34.783231 28789 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:34.775597 28983 raft_consensus.cc:697] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:34.802923 29026 raft_consensus.cc:3060] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:34.794081 28879 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.000687 28804 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.001437 28860 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.837867 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:35.019297 18139 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:34.805320 18780 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:35.020170 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 214916us
W20260709 12:26:35.021147 18780 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.819283 28648 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.027329 28648 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:34.959342 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:35.029444 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70237us
I20260709 12:26:35.030431 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.939558 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:34.972647 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:34.975479 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.975389 28064 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.035318 28064 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:34.930159 28934 raft_consensus.cc:3055] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:26:34.974697 28903 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:34.944126 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 118160us
W20260709 12:26:35.035593 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 96223.4us
I20260709 12:26:34.943565 28528 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.040072 17988 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:35.041189 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:35.042476 18093 raft_consensus.cc:2410] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:26:35.039505 18136 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:35.006079 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 420445us
I20260709 12:26:35.054566 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:34.949368 29017 consensus_queue.cc:1048] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.085s
W20260709 12:26:35.037379 29016 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:26:35.059214 28905 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:35.059511 28814 raft_consensus.cc:2804] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:35.112339 28814 raft_consensus.cc:697] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:35.060177 28804 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.118857 28804 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:35.061491 28996 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:35.064350 29025 consensus_meta.cc:287] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.679s	user 0.012s	sys 0.000s
I20260709 12:26:35.042704 28934 raft_consensus.cc:740] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:35.065201 28903 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:35.051316 28528 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:35.067135 18138 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.072352 28771 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:35.074908 28994 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.031967 28860 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:35.078198 28973 consensus_meta.cc:287] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 1.005s	user 0.006s	sys 0.012s
I20260709 12:26:35.049348 29024 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.083s
I20260709 12:26:35.043042 29003 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:35.074936 29038 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:26:35.083154 28064 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.083792 28648 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.968475 29018 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.107005 29004 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.128481 29017 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:35.114166 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.116384 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.184484 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:35.117045 18096 raft_consensus.cc:2393] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 34: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:35.113548 28814 consensus_queue.cc:237] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.037269 28928 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:35.119357 28804 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:35.126937 18138 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.080508 28789 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:35.143234 18170 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.019s	sys 0.016s
W20260709 12:26:35.204226 18170 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.020s	sys 0.017s
W20260709 12:26:35.176440 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 430365us
I20260709 12:26:35.110249 28879 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.193274 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:35.126929 29025 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.137627 28771 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:35.195925 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 69050.5us
W20260709 12:26:35.232122 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.206537 28860 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.208904 28528 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:35.187767 28836 consensus_meta.cc:287] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.739s	user 0.014s	sys 0.003s
I20260709 12:26:35.216953 29018 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:35.221081 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4086) took 1016 ms. Trace:
I20260709 12:26:35.220362 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.179539 28648 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.222965 28934 consensus_queue.cc:260] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 34.27, Last appended by leader: 27, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.227224 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:35.241734 18809 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 24.
W20260709 12:26:35.238881 18138 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:35.248880 18138 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:35.229930 18173 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.016s	sys 0.021s
W20260709 12:26:35.250008 18173 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.017s	sys 0.021s
I20260709 12:26:35.235612 18170 rpcz_store.cc:276] 0709 12:26:34.204303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.204500 (+   197us) service_pool.cc:224] Handling call
0709 12:26:34.254865 (+ 50365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:34.255409 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.255455 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:34.274185 (+ 18730us) log.cc:844] Serialized 26 byte log entry
0709 12:26:34.590766 (+316581us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:35.215100 (+624334us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.215735 (+   635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.215828 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:35.220646 (+  4818us) raft_consensus.cc:1685] finished
0709 12:26:35.220684 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:35.220978 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5340,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":6340,"thread_start_us":839121,"threads_started":2,"wal-append.queue_time_us":514287}
I20260709 12:26:35.202293 28804 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:35.238250 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.255225 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:35.136816 28983 consensus_queue.cc:237] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.249490 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.275617 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:35.284910 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.288800 18171 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:35.290134 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:35.191787 29017 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.293123 29043 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:35.293762 29043 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.294246 29043 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:35.293021 28751 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:35.292163 18778 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.268923 29042 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:35.305753 18138 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.293202 28722 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.016s
W20260709 12:26:35.309168 29045 log.cc:927] Time spent T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:26:35.310578 28722 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:35.304312 28601 consensus_meta.cc:287] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.673s	user 0.016s	sys 0.000s
I20260709 12:26:35.313192 28601 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:35.283366 28955 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:35.315599 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:35.280922 28783 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.011s	sys 0.018s
I20260709 12:26:35.312336 28771 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.236125 28836 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:35.317873 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.303527 28934 raft_consensus.cc:3060] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:35.297780 28806 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:35.312318 28722 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:35.314522 28955 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:35.337419 28751 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.304925 29042 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:35.341858 18171 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50510 (request call id 3746) took 1144 ms. Trace:
I20260709 12:26:35.362042 18171 rpcz_store.cc:276] 0709 12:26:34.197460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.197740 (+   280us) service_pool.cc:224] Handling call
0709 12:26:35.341625 (+1143885us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:35.341756 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2104044288}
I20260709 12:26:35.341923 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:35.326339 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.330314 28601 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:35.368842 18778 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.360841 18812 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
W20260709 12:26:35.374154 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.321075 28817 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:35.375325 29033 consensus_queue.cc:1048] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.003s
I20260709 12:26:35.381214 28817 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.370400 29018 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:35.382838 28528 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.383494 28528 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.383965 29017 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.351466 28955 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:35.394276 28064 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.395195 28064 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.365136 29042 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:35.398573 28064 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:35.413479 29022 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.881s	user 0.010s	sys 0.006s
I20260709 12:26:35.415619 29022 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:35.416724 28528 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.400485 28836 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.363915 28751 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.419909 29025 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:35.428694 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59947us
W20260709 12:26:35.372139 28995 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.650s	user 0.016s	sys 0.000s
W20260709 12:26:35.430323 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.430570 28995 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.419625 28751 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:35.439643 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:35.440905 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:35.451645 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75450.7us
I20260709 12:26:35.433660 28722 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:35.457175 18138 leader_election.cc:336] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.433288 28995 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.459084 18138 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.471956 28783 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.475185 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 502648us
W20260709 12:26:35.476307 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 413993us
W20260709 12:26:35.475193 29005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.000s	sys 0.016s
W20260709 12:26:35.491915 18166 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.000s	sys 0.018s
W20260709 12:26:35.493031 18166 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.000s	sys 0.018s
I20260709 12:26:35.476836 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:35.472956 28783 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:35.490717 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:35.482818 29005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.000s	sys 0.017s
I20260709 12:26:35.483369 28973 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:35.515061 17989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.023s	sys 0.024s
W20260709 12:26:35.516048 17989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.023s	sys 0.025s
W20260709 12:26:35.489300 29021 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.028s	sys 0.019s
I20260709 12:26:35.504271 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:35.542456 17987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.029s	sys 0.028s
W20260709 12:26:35.543835 17987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.030s	sys 0.028s
W20260709 12:26:35.544787 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 506264us
I20260709 12:26:35.472908 28955 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.550781 28064 raft_consensus.cc:2764] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:26:35.496259 28562 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:35.543162 29021 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.029s	sys 0.020s
W20260709 12:26:35.554340 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48809us
I20260709 12:26:35.554622 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:35.570962 18138 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:35.574796 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.565371 28064 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:35.540589 29025 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.540345 29026 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.550671 28836 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.556082 29022 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.585705 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.576062 28064 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.550909 28528 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.580812 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.581651 18093 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:35.589754 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:35.582546 18094 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:35.557809 29066 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.599422 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:35.549794 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67808.3us
I20260709 12:26:35.588306 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:35.550911 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74176.2us
I20260709 12:26:35.593315 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:35.601049 18096 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:35.601464 18064 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:35.607105 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:35.609515 28064 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.585909 29004 raft_consensus.cc:3060] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:35.624325 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:35.626820 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:35.640782 18097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.004s	sys 0.012s
W20260709 12:26:35.641601 18097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.004s	sys 0.012s
W20260709 12:26:35.613657 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.008s	sys 0.005s
W20260709 12:26:35.627205 18099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.000s	sys 0.011s
W20260709 12:26:35.613205 17988 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.007s	sys 0.011s
I20260709 12:26:35.580624 29025 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:35.696573 17988 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.010s	sys 0.013s
I20260709 12:26:35.616827 28879 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:35.685248 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:35.658777 18099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.000s	sys 0.012s
I20260709 12:26:35.590396 28836 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:35.694384 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 92614.3us
W20260709 12:26:35.660109 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.009s	sys 0.005s
W20260709 12:26:35.664036 18781 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.697458 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 657449us
I20260709 12:26:35.587299 29026 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:35.703148 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55359.2us
I20260709 12:26:35.591888 29022 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.721453 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:35.721704 18138 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.639340 28858 raft_consensus.cc:2804] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:35.705986 18096 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:35.713943 18806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:35.599058 28528 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.655900 28995 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.733937 28064 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.724650 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:35.726861 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:35.730065 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:35.722311 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:35.735680 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:35.734742 28064 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.640707 28860 consensus_queue.cc:1059] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.076s
I20260709 12:26:35.737093 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:35.738759 18806 raft_consensus.cc:2393] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 32: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:35.607293 29066 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.740162 18811 raft_consensus.cc:2393] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 24: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:35.742326 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:35.743860 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:35.744064 18093 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:26:35.746917 28783 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.748868 18810 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:26:35.687639 29054 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.753911 18096 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:26:35.715598 29073 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:35.740895 29053 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.251s	user 0.001s	sys 0.001s
I20260709 12:26:35.711732 28789 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:26:35.741715 29041 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.801681 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:35.742715 28955 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:35.794924 28064 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.804433 18810 raft_consensus.cc:1240] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:35.720556 28926 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:26:35.716521 28879 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.826442 28064 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:35.831225 28064 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:35.812422 29003 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.662s	user 0.016s	sys 0.003s
I20260709 12:26:35.809748 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:35.741400 28995 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.741194 29043 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.859339 18096 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:35.809092 29005 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.833863 29056 log.cc:927] Time spent T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.280s	user 0.000s	sys 0.002s
I20260709 12:26:35.878707 18099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4218) took 1614 ms. Trace:
I20260709 12:26:35.765954 29054 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.825243 28836 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.834578 28783 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:35.883265 18099 rpcz_store.cc:276] 0709 12:26:34.264702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.451806 (+187104us) service_pool.cc:224] Handling call
0709 12:26:34.452231 (+   425us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:34.478573 (+ 26342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.478612 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:34.479226 (+   614us) log.cc:844] Serialized 52 byte log entry
0709 12:26:34.942611 (+463385us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:35.029374 (+ 86763us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.194484 (+165110us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b1800440860. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca557 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:35.705724 (+511240us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.854523 (+148799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.854771 (+   248us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:35.878342 (+ 23571us) raft_consensus.cc:1685] finished
0709 12:26:35.878378 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:35.878618 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62279,"prepare.queue_time_us":101023,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":128,"raft.queue_time_us":258121,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":13578,"spinlock_wait_cycles":3075584,"thread_start_us":1198718,"threads_started":3,"wal-append.queue_time_us":631946}
W20260709 12:26:35.899778 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.902985 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.887154 29053 log.cc:927] Time spent T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:26:35.879076 29003 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:35.912645 18807 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:35.825583 29073 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.922935 28064 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.926331 18097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4226) took 1605 ms. Trace:
I20260709 12:26:35.926915 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:35.930720 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 190664us
I20260709 12:26:35.828728 29041 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.910229 28994 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:35.922129 29050 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.314s	user 0.000s	sys 0.002s
I20260709 12:26:35.926901 18097 rpcz_store.cc:276] 0709 12:26:34.320752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.517341 (+196589us) service_pool.cc:224] Handling call
0709 12:26:34.517726 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:34.518016 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:34.946058 (+428042us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.925802 (+979744us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:35.925894 (+    92us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:35.925976 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.926082 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:35.926250 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96297,"prepare.queue_time_us":168160,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":248046,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":10029,"thread_start_us":1307952,"threads_started":3,"wal-append.queue_time_us":772837}
I20260709 12:26:35.895901 29005 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.924988 29003 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.937369 29058 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.328s	user 0.000s	sys 0.002s
I20260709 12:26:35.935074 29067 raft_consensus.cc:3055] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:35.911347 28783 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:35.941385 28858 raft_consensus.cc:697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:35.963564 17988 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:35.967283 29059 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.322s	user 0.002s	sys 0.000s
I20260709 12:26:35.984395 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:35.993867 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 81539.5us
I20260709 12:26:35.996094 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:35.973408 28839 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:35.984163 29074 log.cc:927] Time spent T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
W20260709 12:26:35.982331 29080 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.153s	user 0.002s	sys 0.000s
I20260709 12:26:35.972688 29085 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.029655 29085 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.030553 28995 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.032168 29026 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:35.984584 29057 log.cc:927] Time spent T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:26:36.034458 28860 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:36.035248 28860 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:35.988508 28528 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:36.000018 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 186745us
W20260709 12:26:36.021509 29072 log.cc:927] Time spent T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
W20260709 12:26:36.022235 29061 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:26:36.045053 29066 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:36.045331 18168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1637) took 1248 ms. Trace:
I20260709 12:26:36.045761 18168 rpcz_store.cc:276] 0709 12:26:34.797345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.797509 (+   164us) service_pool.cc:224] Handling call
0709 12:26:34.797993 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:34.811388 (+ 13395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.811442 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:34.812142 (+   700us) log.cc:844] Serialized 26 byte log entry
0709 12:26:35.710004 (+897862us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:35.710062 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.710152 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.045049 (+334897us) raft_consensus.cc:1685] finished
0709 12:26:36.045077 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.045259 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12771,"raft.queue_time_us":20019,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":24813,"thread_start_us":723393,"threads_started":1,"wal-append.queue_time_us":1040749}
I20260709 12:26:36.047138 29073 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:35.988340 28817 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:35.995100 29071 log.cc:927] Time spent T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.291s	user 0.001s	sys 0.001s
I20260709 12:26:36.032168 29085 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.047999 28817 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.049430 18098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1672) took 1088 ms. Trace:
I20260709 12:26:36.049937 18098 rpcz_store.cc:276] 0709 12:26:34.960905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.968487 (+  7582us) service_pool.cc:224] Handling call
0709 12:26:34.968909 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:35.002077 (+ 33168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:35.002115 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:35.002752 (+   637us) log.cc:844] Serialized 26 byte log entry
0709 12:26:35.468715 (+465963us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:35.468769 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.468853 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.049131 (+580278us) raft_consensus.cc:1685] finished
0709 12:26:36.049171 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.049353 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32709,"raft.queue_time_us":69443,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":8434,"thread_start_us":465502,"threads_started":1,"wal-append.queue_time_us":586936}
I20260709 12:26:35.982448 29067 raft_consensus.cc:740] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:36.052556 29067 consensus_queue.cc:260] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.055047 29054 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:36.056067 28934 consensus_meta.cc:287] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.705s	user 0.013s	sys 0.000s
W20260709 12:26:36.057322 29079 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:26:36.057366 28860 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:36.024637 29083 log.cc:927] Time spent T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
W20260709 12:26:36.025502 29081 log.cc:927] Time spent T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:26:36.045054 28858 consensus_queue.cc:237] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:36.025925 29062 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.300s	user 0.002s	sys 0.000s
I20260709 12:26:36.077857 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1638) took 1280 ms. Trace:
I20260709 12:26:36.078281 18166 rpcz_store.cc:276] 0709 12:26:34.797810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.798434 (+   624us) service_pool.cc:224] Handling call
0709 12:26:34.798875 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:34.812445 (+ 13570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.812499 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:34.813155 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 12:26:35.493730 (+680575us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:35.493805 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.494023 (+   218us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.077555 (+583532us) raft_consensus.cc:1685] finished
0709 12:26:36.077581 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.077753 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13050,"raft.queue_time_us":20096,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":25773,"thread_start_us":523209,"threads_started":1,"wal-append.queue_time_us":755530}
W20260709 12:26:36.018630 29070 log.cc:927] Time spent T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.272s	user 0.001s	sys 0.001s
I20260709 12:26:36.081780 18092 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40710 (request call id 1671) took 1121 ms. Trace:
I20260709 12:26:36.082248 18092 rpcz_store.cc:276] 0709 12:26:34.960542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.960738 (+   196us) service_pool.cc:224] Handling call
0709 12:26:34.961160 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:34.976122 (+ 14962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.976157 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:35.021002 (+ 44845us) log.cc:844] Serialized 26 byte log entry
0709 12:26:35.475869 (+454867us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:35.487714 (+ 11845us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.488471 (+   757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.488604 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.081519 (+592915us) raft_consensus.cc:1685] finished
0709 12:26:36.081549 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.081706 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14489,"prepare.queue_time_us":11523,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":46954,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":12862,"thread_start_us":421292,"threads_started":1,"wal-append.queue_time_us":587720}
I20260709 12:26:36.026984 28648 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:36.026631 28926 consensus_queue.cc:1059] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:36.027530 29041 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.085364 29098 raft_consensus.cc:3055] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:36.028453 29043 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.985064 29003 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.086166 18091 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:26:36.086638 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.022651 29086 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:36.047289 28804 consensus_meta.cc:287] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.758s	user 0.016s	sys 0.000s
I20260709 12:26:36.060392 28879 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.060705 18095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4217) took 1796 ms. Trace:
I20260709 12:26:36.062405 18167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1635) took 1281 ms. Trace:
I20260709 12:26:36.066318 29005 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:36.047330 18807 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:36.066465 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:36.067291 29022 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:35.963819 28994 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.067800 28836 raft_consensus.cc:3060] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:36.070792 28783 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.020356 28789 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:36.072224 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:36.072540 29042 consensus_meta.cc:287] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.675s	user 0.013s	sys 0.004s
W20260709 12:26:36.023826 29087 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:26:36.072917 29067 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.085899 29098 raft_consensus.cc:740] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:36.086716 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.087455 18097 raft_consensus.cc:2468] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:36.118714 18092 raft_consensus.cc:2410] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 32 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:26:36.119267 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.087519 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.119788 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:36.120318 18091 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:36.121538 29098 consensus_queue.cc:260] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.122313 29004 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:36.123358 29098 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.059693 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.100019 29043 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:36.126920 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 67333.6us
I20260709 12:26:36.128307 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:36.089725 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:36.096063 18167 rpcz_store.cc:276] 0709 12:26:34.780541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.780686 (+   145us) service_pool.cc:224] Handling call
0709 12:26:34.792263 (+ 11577us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:34.811210 (+ 18947us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.811257 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:34.954032 (+142775us) log.cc:844] Serialized 52 byte log entry
0709 12:26:35.475456 (+521424us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:26:35.491228 (+ 15772us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.491609 (+   381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.491742 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.062119 (+570377us) raft_consensus.cc:1685] finished
0709 12:26:36.062145 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.062316 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71791,"prepare.queue_time_us":13626,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":20018,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":22229,"spinlock_wait_cycles":7291648,"thread_start_us":479514,"threads_started":1,"wal-append.queue_time_us":582481}
I20260709 12:26:36.097954 18815 raft_consensus.cc:3060] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:36.053480 28817 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.103636 29025 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.120450 18092 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.095033 18095 rpcz_store.cc:276] 0709 12:26:34.264434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.441621 (+177187us) service_pool.cc:224] Handling call
0709 12:26:34.477544 (+ 35923us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:34.936545 (+459001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:34.936604 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:34.991489 (+ 54885us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b2c0005b308. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c14fb0a11 00007f6c14fc0372 00007f6c166f2a01 00007f6c166b743a 00007f6c166b683e 00007f6c1b67b419 00007f6c14bf2bd4 00007f6c14bf2b62 00007f6c14bf2a91 00007f6c14bf2a0c 00007f6c14bf1cc2
0709 12:26:34.992738 (+  1249us) log.cc:844] Serialized 52 byte log entry
0709 12:26:35.360248 (+367510us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:35.376536 (+ 16288us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.377011 (+   475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.377132 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:35.416470 (+ 39338us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b18003b9e00. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:36.060424 (+643954us) raft_consensus.cc:1685] finished
0709 12:26:36.060453 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.060632 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3949,"prepare.queue_time_us":16003,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":40114,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":215,"spinlock_wait_cycles":53967360,"thread_start_us":819475,"threads_started":2,"wal-append.queue_time_us":568109}
I20260709 12:26:36.121521 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.099912 28804 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.144479 29003 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.145569 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.149194 18091 raft_consensus.cc:2393] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:36.146350 18095 raft_consensus.cc:2468] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:36.160614 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.134418 28934 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:36.132187 28783 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:36.134425 28789 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.162289 18090 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 33. Current term is 34. Ops: [33.26-33.26]
I20260709 12:26:36.168282 28994 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.173779 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.150163 29055 consensus_queue.cc:1059] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:26:36.117532 29042 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.177681 28983 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.178416 28983 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:36.178952 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 457256us
W20260709 12:26:36.179553 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 179313us
W20260709 12:26:36.179895 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.180491 18138 leader_election.cc:336] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.180836 28983 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:36.181077 28814 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:36.182071 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 350331us
I20260709 12:26:36.182937 18137 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:36.150166 29054 consensus_queue.cc:1059] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
W20260709 12:26:36.183575 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 577631us
I20260709 12:26:36.182065 18812 raft_consensus.cc:1240] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:36.186595 18171 raft_consensus.cc:1240] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 LEADER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 34. Current term is 35. Ops: [34.27-34.27]
I20260709 12:26:36.182207 18173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50512 (request call id 4089) took 1963 ms. Trace:
I20260709 12:26:36.188858 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.183007 29024 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:36.181494 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.195645 18092 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:36.189076 18173 rpcz_store.cc:276] 0709 12:26:34.218741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.454538 (+235797us) service_pool.cc:224] Handling call
0709 12:26:34.455017 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:34.455271 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:26:35.251217 (+795946us) log.cc:844] Serialized 14 byte log entry
0709 12:26:36.181758 (+930541us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:36.181856 (+    98us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:26:36.181907 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:36.181991 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.182139 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":449307,"prepare.queue_time_us":959135,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":450209,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":1244050,"threads_started":2,"wal-append.queue_time_us":809602}
I20260709 12:26:36.181625 28814 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.201586 28751 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:36.205863 18168 pending_rounds.cc:85] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Aborting all ops after (but not including) 19
I20260709 12:26:36.206283 18168 pending_rounds.cc:107] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Aborting uncommitted NO_OP operation due to leader change: 25.20
I20260709 12:26:36.206561 18168 raft_consensus.cc:2889] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:36.207931 28814 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:36.208779 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.209363 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.210341 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.210598 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.210983 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.211263 18169 raft_consensus.cc:2410] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:26:36.136657 29043 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.212662 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.214318 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:36.194918 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.217738 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.218562 18173 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:36.220505 18091 raft_consensus.cc:2468] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 26.
I20260709 12:26:36.225220 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:36.220031 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.228384 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.227344 18091 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:36.239553 18172 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
W20260709 12:26:36.259325 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.181502 28983 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:36.261449 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.264791 18094 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:36.265158 18097 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:36.261478 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.284886 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:36.286811 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.195770 28789 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.290413 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.291925 18169 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 27.
I20260709 12:26:36.212407 29024 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.269268 28751 raft_consensus.cc:697] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:36.341553 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:36.346402 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 188461us
I20260709 12:26:36.362407 18810 raft_consensus.cc:1240] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 38 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 37. Current term is 38. Ops: []
I20260709 12:26:36.299523 28858 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:36.383752 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:36.356710 29104 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.242s	user 0.002s	sys 0.000s
I20260709 12:26:36.302393 28817 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:36.387228 18809 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:36.383090 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:36.390445 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:36.397030 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:36.394147 18808 raft_consensus.cc:2468] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:36.340994 29054 raft_consensus.cc:3055] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:26:36.396979 18778 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:36.393548 29107 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.405145 29054 raft_consensus.cc:740] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:36.456688 18808 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 24. Preceding OpId from leader: term: 31 index: 25. (index mismatch)
W20260709 12:26:36.461328 29102 log.cc:927] Time spent T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.342s	user 0.000s	sys 0.004s
I20260709 12:26:36.461607 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:36.464092 28783 raft_consensus.cc:3055] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:36.466737 29106 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:26:36.470705 29043 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.474690 29041 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:26:36.400496 28817 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.479947 29110 consensus_queue.cc:1059] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 32.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:26:36.395145 28858 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.501132 29089 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.508368 17987 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.510885 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:36.512274 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:36.532074 18810 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:36.514979 18097 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:36.533339 17987 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.450464 28601 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:36.554832 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.555658 18172 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:36.517047 28903 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:36.493976 28783 raft_consensus.cc:740] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
W20260709 12:26:36.561426 18064 leader_election.cc:336] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.565061 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.567771 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.571326 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.572342 18165 raft_consensus.cc:2393] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 31: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:36.579102 28783 consensus_queue.cc:260] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.579772 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.580497 28783 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.558907 28528 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:36.582345 28528 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.582806 28528 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.584306 28926 raft_consensus.cc:2804] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:36.584789 28926 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:36.585191 28926 raft_consensus.cc:3060] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:36.554243 29082 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:36.582003 29068 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:36.585915 29054 consensus_queue.cc:260] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.586433 29082 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.565006 28903 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.588685 28839 raft_consensus.cc:2804] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:36.588056 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.589154 28839 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:36.589608 28839 raft_consensus.cc:3060] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.590225 29068 raft_consensus.cc:2804] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:36.590710 29068 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:36.591133 29068 raft_consensus.cc:3060] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:36.593986 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.595332 28926 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.595911 17987 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.596745 28789 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:36.555145 29089 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.609280 28926 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.609511 29089 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.450734 29107 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.611910 29107 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.555778 29004 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:36.617074 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55694.1us
I20260709 12:26:36.617779 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:36.618057 18064 leader_election.cc:336] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.618516 18064 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.618939 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:36.619814 18064 leader_election.cc:336] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.620319 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.621376 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:36.624732 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 74573.1us
I20260709 12:26:36.625090 18809 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 34 index: 27. Preceding OpId from leader: term: 35 index: 27. (term mismatch)
I20260709 12:26:36.627017 18809 pending_rounds.cc:85] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting all ops after (but not including) 26
I20260709 12:26:36.627314 18809 pending_rounds.cc:107] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting uncommitted NO_OP operation due to leader change: 34.27
I20260709 12:26:36.627555 18809 raft_consensus.cc:2889] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:36.629037 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:36.633004 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.634902 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.635746 18172 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:36.636739 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.637424 18172 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:36.622174 28903 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.656836 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.642063 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.649783 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.681092 18167 raft_consensus.cc:2393] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 31: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:36.658207 18090 raft_consensus.cc:1275] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 33 index: 26. Preceding OpId from leader: term: 35 index: 27. (index mismatch)
I20260709 12:26:36.660959 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.690635 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.691459 18170 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
I20260709 12:26:36.694139 18168 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:36.681263 18098 raft_consensus.cc:2410] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee for term 31 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:36.650831 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:36.721655 18815 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.590s	user 0.013s	sys 0.001s
I20260709 12:26:36.722543 18815 raft_consensus.cc:2468] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 25.
I20260709 12:26:36.728652 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.729703 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.733668 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.621598 29089 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:36.737303 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.730832 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:36.740327 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 108840us
I20260709 12:26:36.655843 29005 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:36.641216 29017 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:36.630689 29099 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:36.741495 18063 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:36.630872 29004 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:36.749853 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.519634 28751 consensus_queue.cc:237] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.752679 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:36.741640 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 121841us
I20260709 12:26:36.757349 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.729857 18173 raft_consensus.cc:2376] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27. Re-sending same reply.
I20260709 12:26:36.758459 18169 raft_consensus.cc:2393] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 31: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:36.676214 29024 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.762146 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:36.690153 28858 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.768517 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.768597 28606 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:36.774307 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
W20260709 12:26:36.740120 28836 consensus_meta.cc:287] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.629s	user 0.014s	sys 0.000s
I20260709 12:26:36.607062 28817 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:36.745926 28860 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:36.790300 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.792309 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.792770 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.777724 29041 raft_consensus.cc:2804] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:36.815887 18169 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 24. Preceding OpId from leader: term: 31 index: 25. (index mismatch)
I20260709 12:26:36.817520 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.812597 28606 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.819509 18164 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:36.820413 28606 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:36.813171 28836 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:36.827337 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.827965 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 567743us
W20260709 12:26:36.828749 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.829380 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.795994 18064 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.760139 29005 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.806342 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.733217 28789 consensus_queue.cc:1059] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.052s
I20260709 12:26:36.812870 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:36.854030 29004 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:36.855612 29003 raft_consensus.cc:3055] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:26:36.856118 29003 raft_consensus.cc:740] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:36.770471 29017 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.859234 18165 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:36.838956 18064 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.860702 29017 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:36.812597 18172 raft_consensus.cc:1240] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
W20260709 12:26:36.839766 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96203.6us
I20260709 12:26:36.861912 29041 raft_consensus.cc:697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:36.866496 18136 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:36.846344 29086 consensus_meta.cc:287] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.741s	user 0.013s	sys 0.002s
I20260709 12:26:36.857070 29003 consensus_queue.cc:260] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:36.844696 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.868567 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72635.4us
I20260709 12:26:36.783473 29068 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:36.844628 29005 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:36.862601 29067 consensus_meta.cc:287] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.744s	user 0.013s	sys 0.002s
I20260709 12:26:36.871695 29041 consensus_queue.cc:237] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:36.875669 28934 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.710s	user 0.010s	sys 0.003s
W20260709 12:26:36.878605 29118 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:26:36.872045 28817 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:36.872423 18138 leader_election.cc:336] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:36.872997 18064 leader_election.cc:336] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:36.888952 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.873234 29086 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:36.903653 18138 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:36.909929 18138 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 111 similar messages]
I20260709 12:26:36.904541 18064 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.878239 29069 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:36.913777 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 282512us
I20260709 12:26:36.909574 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:36.916390 18064 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:36.917528 18064 leader_election.cc:336] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.920578 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.923282 18815 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:36.927408 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.933429 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:36.942998 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:36.948680 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 314805us
I20260709 12:26:36.853551 29082 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:36.952910 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:36.955901 18064 leader_election.cc:336] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.959208 18064 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:36.949870 17988 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:36.974116 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:36.898753 28934 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:36.985155 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:36.956023 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:36.991353 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:36.908385 28879 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:36.913117 28608 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:36.963565 29098 consensus_meta.cc:287] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.840s	user 0.017s	sys 0.000s
I20260709 12:26:36.922524 28860 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:37.011174 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55308us
I20260709 12:26:37.012230 28771 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:37.014739 29055 raft_consensus.cc:3055] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:26:37.019404 28905 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:37.025908 28562 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:26:37.035554 29025 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:37.038955 29042 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:37.039769 28804 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:37.043517 29054 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:37.044452 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.103624 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:37.104390 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 344073us
I20260709 12:26:37.105253 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:37.106113 18095 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:37.055301 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 313884us
I20260709 12:26:37.057072 18808 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:36.963992 28465 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:36.976468 28817 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.113245 29067 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.113986 29067 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:37.066946 18064 leader_election.cc:336] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.984844 28601 raft_consensus.cc:697] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:37.117638 18064 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:37.118006 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.118438 28601 consensus_queue.cc:237] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.119479 28817 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:37.092041 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.007318 29113 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:37.046995 28858 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.053651 29082 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:37.118909 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52044.7us
W20260709 12:26:37.121992 18064 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.120983 29113 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.123328 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 36 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:37.123801 29113 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:37.045475 29068 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:37.123368 18064 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.131004 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.132694 29043 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:37.135392 18064 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.054354 28879 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.142016 18090 raft_consensus.cc:1275] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:26:37.136839 29113 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.072679 29055 raft_consensus.cc:740] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:37.146992 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:37.084062 29025 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.072443 28528 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.117367 29067 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:37.087280 29004 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.071s
W20260709 12:26:37.240065 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 245056us
W20260709 12:26:37.175899 29125 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:26:37.199662 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:37.242169 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:37.253638 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:37.271903 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 126975us
W20260709 12:26:37.273722 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:37.246117 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 336234us
W20260709 12:26:37.247656 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 294837us
W20260709 12:26:37.174517 29124 log.cc:927] Time spent T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:26:37.167011 29099 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.159375 29025 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:37.278918 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79414.5us
I20260709 12:26:37.279500 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.158250 29082 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.194011 29021 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:37.201136 28879 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:37.274693 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.250682 29120 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:37.196087 28528 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:37.280184 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:37.278653 29121 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.073s	user 0.004s	sys 0.000s
W20260709 12:26:37.452143 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 177523us
W20260709 12:26:37.447692 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 497955us
W20260709 12:26:37.462929 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 183696us
I20260709 12:26:37.464413 17988 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:37.468138 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.289498 28814 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:37.473201 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 193072us
W20260709 12:26:37.470733 18064 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.477203 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.479964 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:37.479223 29125 log.cc:927] Time spent T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:26:37.461129 29021 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.484186 28528 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:37.484877 29021 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:37.485078 28465 consensus_queue.cc:1048] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:37.487131 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:37.480762 28879 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:37.489050 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.510999 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 36 candidate_status { last_received { term: 34 index: 27 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:37.496631 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:37.512552 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.521203 18808 raft_consensus.cc:2393] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 38: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:37.448460 29082 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:37.448016 29129 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:37.544077 18166 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60850 (request call id 1641) took 1359 ms. Trace:
I20260709 12:26:37.447800 29126 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.483908 29086 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:37.486979 29069 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.433125 29093 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.103s
I20260709 12:26:37.510747 28528 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:37.519451 29067 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:37.550110 18166 rpcz_store.cc:276] 0709 12:26:36.184406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:36.186743 (+  2337us) service_pool.cc:224] Handling call
0709 12:26:36.225939 (+ 39196us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:36.290442 (+ 64503us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:36.398957 (+108515us) log.cc:844] Serialized 14 byte log entry
0709 12:26:36.562755 (+163798us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:36.675467 (+112712us) log.cc:844] Serialized 78 byte log entry
0709 12:26:36.675700 (+   233us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:36.676016 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 12:26:36.725040 (+ 49024us) log.cc:844] Serialized 14 byte log entry
0709 12:26:36.725309 (+   269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:36.725426 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.849332 (+123906us) spinlock_profiling.cc:243] Waited 5.97 ms on lock 0x7b1800372c40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:37.202552 (+353220us) spinlock_profiling.cc:243] Waited 8.97 ms on lock 0x7b1800372c40. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca3f3 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
0709 12:26:37.543739 (+341187us) raft_consensus.cc:1685] finished
0709 12:26:37.543776 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:37.543991 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64013,"prepare.queue_time_us":192430,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":72273,"raft.run_cpu_time_us":612,"raft.run_wall_time_us":44061,"spinlock_wait_cycles":64502912,"thread_start_us":125381,"threads_started":2,"wal-append.queue_time_us":288457}
I20260709 12:26:37.512749 28608 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:37.521265 28905 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:37.519940 28926 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:37.535842 28814 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.429592 29055 consensus_queue.cc:260] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.538897 28648 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:37.558557 29104 log.cc:927] Time spent T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:26:37.466466 29099 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.560878 29129 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:37.565886 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61239.6us
I20260709 12:26:37.535172 29068 raft_consensus.cc:2804] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:26:37.605475 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 93029.1us
I20260709 12:26:37.472685 29120 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:37.645354 17990 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.043s	sys 0.065s
I20260709 12:26:37.637017 17989 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:37.637928 18064 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.549085 29109 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:37.596760 29024 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.026s	sys 0.073s
W20260709 12:26:37.621606 29139 log.cc:927] Time spent T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
W20260709 12:26:37.664215 17990 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.044s	sys 0.065s
W20260709 12:26:37.688412 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51514.6us
W20260709 12:26:37.665325 18064 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.604233 28858 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:37.702687 29146 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:26:37.704733 29137 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260709 12:26:37.631191 28608 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:37.686064 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.615572 29130 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:26:37.695258 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:37.696816 18064 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.699396 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:37.574949 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 54577.7us
I20260709 12:26:37.636893 29068 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:37.708823 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.568490 28562 consensus_queue.cc:1048] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.118s
I20260709 12:26:37.649341 29004 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
I20260709 12:26:37.694555 28771 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:37.701579 28858 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.735036 17989 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.702713 29129 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:37.738054 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:37.633656 29142 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.122s
I20260709 12:26:37.721771 29068 raft_consensus.cc:3060] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:26:37.726042 28839 consensus_meta.cc:287] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 1.131s	user 0.014s	sys 0.002s
I20260709 12:26:37.725555 28804 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:37.572551 29135 consensus_queue.cc:1059] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.168s
W20260709 12:26:37.714167 18064 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.664899 28814 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.654048 28648 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.575510 29126 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.720696 29113 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:37.714624 28608 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:37.589975 29069 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.775579 29130 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.778913 18812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35286 (request call id 2098) took 1274 ms. Trace:
I20260709 12:26:37.734409 28905 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:37.785391 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.779986 28562 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
W20260709 12:26:37.803995 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 89878.3us
W20260709 12:26:37.804836 18064 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.805433 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:37.781975 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 307745us
I20260709 12:26:37.788075 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:37.787704 28771 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.690466 29109 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:37.802227 18063 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.780992 29063 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:37.788540 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53611.5us
W20260709 12:26:37.809212 18064 leader_election.cc:336] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.791110 28926 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:37.800823 29024 raft_consensus.cc:3055] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:37.801718 18812 rpcz_store.cc:276] 0709 12:26:36.504622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:36.504841 (+   219us) service_pool.cc:224] Handling call
0709 12:26:36.505263 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:36.764105 (+258842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:36.764154 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:36.767999 (+  3845us) log.cc:844] Serialized 26 byte log entry
0709 12:26:37.246530 (+478531us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:26:37.246605 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:37.246734 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:37.778571 (+531837us) raft_consensus.cc:1685] finished
0709 12:26:37.778607 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:37.778809 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13730,"raft.queue_time_us":106920,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":501841,"threads_started":2,"wal-append.queue_time_us":606326}
I20260709 12:26:37.815363 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:37.819378 29152 log.cc:927] Time spent T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:26:37.791986 28528 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:37.827800 18064 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.829391 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.812156 29093 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.780539 28839 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.790616 28804 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.798000 29147 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.852957 29099 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:37.855567 28858 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:37.858033 29005 consensus_meta.cc:287] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.976s	user 0.018s	sys 0.000s
I20260709 12:26:37.832619 29063 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.820914 28562 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.859688 29069 leader_election.cc:290] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:37.865485 17989 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.844451 28928 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.842434 29024 raft_consensus.cc:740] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:37.882043 28905 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:37.911795 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:37.912792 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:37.913179 29166 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:37.931625 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.934388 29135 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:37.934478 29024 consensus_queue.cc:260] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.936218 29024 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:37.936565 17989 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:37.937561 29143 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:37.942955 28814 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:37.908420 28926 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:37.914155 18166 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:37.893734 29099 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.922760 29167 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:37.922799 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113623us
I20260709 12:26:37.925966 18173 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 32. Current term is 33. Ops: [32.18-32.18]
I20260709 12:26:37.846583 29004 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:26:37.825773 28771 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.934374 29164 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.915740 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:37.860893 28528 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:37.912794 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 70751.6us
W20260709 12:26:37.949975 18064 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.811621 29130 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.953125 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.955497 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:37.966444 18808 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:37.970443 29126 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:37.975004 29054 consensus_meta.cc:287] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.834s	user 0.011s	sys 0.004s
I20260709 12:26:37.975531 28839 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:37.976727 29042 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.977641 28528 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:37.977351 29042 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.897192 29005 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.924952 28928 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:37.990810 18064 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.996025 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.896261 29147 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:37.993186 28528 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.991811 28905 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.989338 28858 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:37.991645 29099 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.986274 29069 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:38.022117 18064 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.065775 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:37.957629 29167 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:38.066088 18138 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.070647 18138 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:38.071947 29161 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:26:38.072464 28562 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:38.074105 29130 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.074956 29021 consensus_meta.cc:287] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 0.590s	user 0.009s	sys 0.006s
I20260709 12:26:38.075454 29021 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:37.981722 29164 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.079488 18170 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:38.079883 29167 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.080675 29164 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:38.023278 28528 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.084210 28528 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.085991 29134 raft_consensus.cc:3055] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:38.086488 29134 raft_consensus.cc:740] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:38.087414 29134 consensus_queue.cc:260] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.088836 28928 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.092311 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.094677 28804 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.101415 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 115880us
I20260709 12:26:38.058919 28858 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.117264 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.011359 29109 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.124349 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:38.126988 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.027333 29089 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:38.129356 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:38.112601 28783 consensus_meta.cc:287] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: Time spent flushing consensus metadata: real 1.532s	user 0.015s	sys 0.005s
I20260709 12:26:38.131213 18095 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 34: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:26:38.134977 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 112910us
W20260709 12:26:38.135941 18064 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.024670 28905 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:38.153688 16792 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:38.160742 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.086257 29063 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.137300 18064 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:38.163129 18064 leader_election.cc:336] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.165066 18095 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:38.127295 28928 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:38.167663 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 101608us
W20260709 12:26:38.185252 18138 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.185726 18138 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:38.182065 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 59057us
W20260709 12:26:38.188618 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.189250 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:38.190732 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:38.190692 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:38.191201 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.193709 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.133394 29021 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.196005 18165 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:26:38.186522 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.198176 29156 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:38.201333 28858 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.219398 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.220568 29042 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.203451 28905 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.221381 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.204447 18136 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:38.223477 29113 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:38.157874 28804 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:38.120002 29069 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.217147 28836 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:38.229324 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.217805 29021 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.183043 29165 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:26:38.225737 29025 consensus_meta.cc:287] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.747s	user 0.012s	sys 0.000s
I20260709 12:26:38.225040 29164 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:38.226802 28924 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:38.228019 29113 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:38.266307 29113 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:38.266083 28924 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.267019 29147 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.268863 28924 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.259399 29021 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.271823 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.272773 29021 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.274631 29003 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:26:38.278549 29120 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.277940 29025 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:38.229166 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:38.285512 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.234288 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.287573 18171 raft_consensus.cc:2410] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
W20260709 12:26:38.223966 18138 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.238215 28836 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.196156 29130 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.067s
I20260709 12:26:38.197990 18095 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:38.296396 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:38.290961 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.196372 29004 consensus_queue.cc:1059] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:26:38.195812 28928 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:38.310665 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 113820us
W20260709 12:26:38.317052 18138 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.227936 18170 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:38.314199 29169 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:38.334450 28608 consensus_meta.cc:287] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.004s
I20260709 12:26:38.333043 28973 raft_consensus.cc:2764] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
W20260709 12:26:38.346884 28994 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.016s	sys 0.021s
W20260709 12:26:38.307894 29041 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.019s	sys 0.049s
W20260709 12:26:38.360750 28994 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.016s	sys 0.021s
W20260709 12:26:38.363803 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46808.2us
I20260709 12:26:38.385777 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.386865 18097 raft_consensus.cc:2393] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
W20260709 12:26:38.368983 29041 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.019s	sys 0.050s
I20260709 12:26:38.356077 28608 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.375969 28648 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.380880 29147 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.439571 29147 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.444214 18167 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:38.445677 28608 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:38.456194 18138 leader_election.cc:336] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:38.473536 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 309475us
I20260709 12:26:38.480288 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:38.481400 18169 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:38.471987 29069 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.474349 28608 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:38.479064 29165 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:38.496268 29068 consensus_meta.cc:287] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Time spent flushing consensus metadata: real 0.706s	user 0.016s	sys 0.007s
I20260709 12:26:38.511186 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.497298 28836 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:38.502799 29082 consensus_meta.cc:287] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: Time spent flushing consensus metadata: real 0.934s	user 0.016s	sys 0.008s
W20260709 12:26:38.504768 29178 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:26:38.520104 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.506131 28608 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.540665 28608 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.522468 29182 log.cc:927] Time spent T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 12:26:38.523438 29180 log.cc:927] Time spent T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:26:38.526722 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.535259 18169 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:38.566498 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.511085 29147 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.558559 18165 raft_consensus.cc:1275] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:26:38.536453 28836 consensus_queue.cc:1059] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:38.578960 28836 raft_consensus.cc:3055] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:38.579527 28836 raft_consensus.cc:740] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:38.579138 29113 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.580093 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 36 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.580716 18171 raft_consensus.cc:1240] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:38.515668 29025 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.563134 18064 connection.cc:256] RPC call timeout handler was delayed by -1.3907546410000577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
I20260709 12:26:38.587447 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.568183 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.589327 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.590785 18091 raft_consensus.cc:2410] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
W20260709 12:26:38.598786 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.601673 18064 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.613343 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:38.613710 18064 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.616379 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.616650 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:38.614159 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.617522 18091 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:38.619009 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.619406 18064 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.620118 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.618104 18099 raft_consensus.cc:2468] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:38.622728 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.626741 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.630507 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.569347 28858 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:38.645953 18815 raft_consensus.cc:1275] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:26:38.634557 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.640262 18096 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:26:38.615226 29099 raft_consensus.cc:2749] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:26:38.642479 29089 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:38.514452 29165 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.519423 29068 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.551156 29082 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.653340 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.611594 28817 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:38.665362 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.580581 28836 consensus_queue.cc:260] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.681844 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.670759 18809 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:38.633163 29162 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:38.660012 28860 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:38.689277 17989 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:38.692404 18169 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:38.693831 28860 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.696177 18173 raft_consensus.cc:2393] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 29: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:38.698539 29165 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:38.699177 29113 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:38.694804 29188 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:26:38.706537 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.708819 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:38.709725 18168 raft_consensus.cc:3060] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:26:38.705103 28789 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:38.711745 28789 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:38.712891 28789 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:38.714221 28836 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:38.715207 29068 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.717845 29082 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:38.718659 28860 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.657459 28858 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.723088 28858 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:38.734817 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 115510us
I20260709 12:26:38.735949 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.738447 29195 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:38.739148 29195 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:38.739678 29195 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:38.746472 28858 consensus_queue.cc:1059] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:38.748243 18064 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.751827 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.751889 28860 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:38.754966 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.755744 29162 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:38.762662 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:38.765038 18809 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 34.
I20260709 12:26:38.764914 18165 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:26:38.766567 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.768723 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.772923 29089 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:26:38.773932 28860 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:26:38.777119 18064 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:38.769833 18165 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:26:38.779143 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.780123 18165 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 30.
I20260709 12:26:38.784010 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.786480 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.788763 18091 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:38.789299 18064 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.790091 18096 raft_consensus.cc:1240] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:38.792055 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.790201 18099 raft_consensus.cc:2393] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 34: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:38.796366 18165 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 33.
I20260709 12:26:38.805088 18091 raft_consensus.cc:1240] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:26:38.765550 29162 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:38.806968 29166 raft_consensus.cc:2764] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:26:38.808692 28860 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:38.808089 29082 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:38.821442 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:38.820498 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:38.827363 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5240f7371cc74cb48dffa1dadcbeec49" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 36 candidate_status { last_received { term: 34 index: 27 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.832297 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:38.836149 18097 raft_consensus.cc:1240] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 33. Current term is 34. Ops: [33.26-33.26]
I20260709 12:26:38.854897 18064 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:38.860435 18097 raft_consensus.cc:1275] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:38.861129 29099 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:38.862118 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f78d7b0f7f422eacbfa07bbc248bfa" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.862854 29099 consensus_queue.cc:1048] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:26:38.863376 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.865330 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:38.866432 29195 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.867627 29099 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:38.868088 29099 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:38.868489 29099 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:38.869082 29195 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.823359 29082 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:38.871309 18780 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.871999 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.873368 29082 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:38.874114 18780 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.874624 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.845046 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.876585 18168 raft_consensus.cc:2468] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:26:38.878551 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.879475 18092 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:38.884240 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:38.886063 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:38.887065 18168 raft_consensus.cc:2393] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 33: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:26:38.888957 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 25 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.895247 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.895937 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:38.896416 18098 raft_consensus.cc:2410] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:26:38.896589 18092 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:26:38.897533 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.898854 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:38.896270 28771 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:38.907823 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:38.924196 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 50109.2us
W20260709 12:26:38.925673 18780 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.926777 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:38.927848 29136 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:38.928893 29136 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:26:38.929339 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:38.931334 29055 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:38.933928 29005 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:38.926208 28928 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.940559 29203 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:38.921511 29162 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:38.964080 29099 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:38.966965 29190 log.cc:927] Time spent T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.271s	user 0.002s	sys 0.001s
I20260709 12:26:38.967056 29099 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:38.974506 29134 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:38.976871 29184 log.cc:927] Time spent T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.539s	user 0.002s	sys 0.000s
I20260709 12:26:38.979409 29024 raft_consensus.cc:3055] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:26:38.982486 29189 log.cc:927] Time spent T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.287s	user 0.000s	sys 0.003s
I20260709 12:26:39.001775 29024 raft_consensus.cc:740] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:39.004791 29024 consensus_queue.cc:260] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.057751 17915 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:39.146412 29024 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:39.072556 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:38.900661 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:39.152621 18098 raft_consensus.cc:2410] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:39.083549 18095 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:39.099292 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:39.155395 16792 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:39.100915 28771 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.156955 28771 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.101150 29022 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.142550 28836 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.162425 28983 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:39.163384 28836 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.164063 28648 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:39.163883 28983 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.143731 28994 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.101780 18810 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 34.
W20260709 12:26:39.134640 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.165189 28648 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.168272 28994 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:39.184260 18779 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:39.198450 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.200778 28994 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.202435 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:39.206542 18092 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:39.193187 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.213346 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.214607 18167 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 28.
I20260709 12:26:39.215255 18091 raft_consensus.cc:1240] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:26:39.201661 28994 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.217885 28994 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.162573 29022 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.222553 17915 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 27, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 27 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.190022 28771 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:39.248370 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.258050 18809 raft_consensus.cc:1240] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:39.265211 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.273269 28648 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.284662 18168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.264298 28983 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.273401 29022 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:39.294526 18170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50512 (request call id 4127) took 1382 ms. Trace:
I20260709 12:26:39.291469 28836 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.296211 18170 rpcz_store.cc:276] 0709 12:26:37.911969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.915351 (+  3382us) service_pool.cc:224] Handling call
0709 12:26:39.294074 (+1378723us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540002b7d4. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166bb2fa 00007f6c1b67cb17 00007f6c14bf76d4 00007f6c14bf7662 00007f6c14bf7591 00007f6c14bf750c 00007f6c14bf67c2 00007f6c149f248c 00007f6c149f18c6 00007f6c149f126f
0709 12:26:39.294296 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":294220544}
W20260709 12:26:39.308218 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 374848us
I20260709 12:26:39.312773 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:39.313488 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.315034 18171 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:39.322916 29209 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:39.323737 29209 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.324179 29209 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.331821 28994 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:39.339233 18168 raft_consensus.cc:1240] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 32. Current term is 33. Ops: [32.18-32.18]
W20260709 12:26:39.353394 17906 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 85764dcebf974f7cbec322be36e3dd5f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 768723648a9444fda53659e6a8cc3091 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:39.357285 17915 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:39.367705 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:39.370489 29195 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:39.371129 29195 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.371510 29195 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:39.380986 18064 leader_election.cc:336] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:39.382107 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:39.388401 28994 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.389950 28994 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.392850 29022 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.393190 28994 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.388926 28860 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:39.330132 29169 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.025s	sys 0.030s
I20260709 12:26:39.404872 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.405881 18169 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:39.407310 18136 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:39.393777 29022 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:39.410857 18062 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.412218 29022 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.412958 29022 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.418536 17915 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 23. New cstate: current_term: 23 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:39.419585 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:39.422065 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:39.423014 18094 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.422302 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:39.427446 18780 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:39.428891 29120 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:39.428992 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:39.429617 29120 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:39.430737 29120 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:39.432782 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:39.434278 29147 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:39.434849 29147 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.435292 29147 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.436139 18778 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:39.437879 29136 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:26:39.441784 29169 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.999s	user 0.026s	sys 0.030s
W20260709 12:26:39.388500 18064 leader_election.cc:336] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:39.420708 17915 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 26, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:39.451022 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44534us
I20260709 12:26:39.458715 18064 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:39.413653 29022 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.465910 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:39.464613 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:39.476364 29067 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:39.477681 28751 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:39.483861 29195 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.478453 28751 raft_consensus.cc:697] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:39.462849 29136 consensus_queue.cc:1048] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260709 12:26:39.445972 28836 consensus_queue.cc:1059] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
W20260709 12:26:39.493018 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 88796.7us
I20260709 12:26:39.478816 17915 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 23, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 23 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:39.512441 18099 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 29: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:39.517457 29136 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:39.518163 29136 raft_consensus.cc:493] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.518573 29136 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:39.519455 28771 raft_consensus.cc:3055] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:39.519985 28771 raft_consensus.cc:740] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
W20260709 12:26:39.500895 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71059us
I20260709 12:26:39.524070 28751 consensus_queue.cc:237] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.521310 28771 consensus_queue.cc:260] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.537279 18094 raft_consensus.cc:2468] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:39.558429 17913 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 185) took 60812 ms (client timeout 15000 ms). Trace:
I20260709 12:26:39.560624 28771 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:39.561789 29120 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.562076 29205 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:39.562911 29207 log.cc:927] Time spent T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:26:39.571156 18780 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:39.579172 29022 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:39.594676 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:39.553001 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 91806.7us
I20260709 12:26:39.599694 18169 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 31.
I20260709 12:26:39.607234 18171 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
W20260709 12:26:39.559957 29134 consensus_meta.cc:287] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.016s
I20260709 12:26:39.620245 29120 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:39.603688 29195 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:39.626811 29205 raft_consensus.cc:2804] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:26:39.628432 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 240004us
W20260709 12:26:39.629935 18064 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:39.632113 18064 leader_election.cc:336] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:39.632795 18064 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:39.634099 17915 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 26, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 26 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.636108 17915 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 17 to 23. New cstate: current_term: 23 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.645949 17989 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:39.647385 29022 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:39.648123 29022 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:39.650385 28804 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.651136 28804 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.649602 29022 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.606715 28836 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:39.559351 17913 rpcz_store.cc:269] 0709 12:25:38.746451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.946517 (+6200066us) service_pool.cc:224] Handling call
0709 12:26:39.558064 (+54611547us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:32.146715 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:32.178713 (+ 31998us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:32.178773 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:26:32.178808 (+    35us) tablet.cc:662] Decoding operations
0709 12:26:32.185006 (+  6198us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:32.185138 (+   132us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:32.185169 (+    31us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:26:32.186275 (+  1106us) tablet.cc:701] Row locks acquired
0709 12:26:32.186336 (+    61us) write_op.cc:260] PREPARE: finished
0709 12:26:32.186573 (+   237us) write_op.cc:270] Start()
0709 12:26:32.186765 (+   192us) write_op.cc:276] Timestamp: P: 1783599992186486 usec, L: 0
0709 12:26:32.186806 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:26:32.187680 (+   874us) log.cc:844] Serialized 3632 byte log entry
0709 12:26:33.631422 (+1443742us) op_driver.cc:464] REPLICATION: finished
0709 12:26:34.050018 (+418596us) write_op.cc:301] APPLY: starting
0709 12:26:34.050151 (+   133us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:34.376572 (+326421us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:34.376746 (+   174us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:35.296048 (+919302us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:35.296511 (+   463us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:26:35.296588 (+    77us) write_op.cc:312] APPLY: finished
0709 12:26:35.297105 (+   517us) log.cc:844] Serialized 89 byte log entry
0709 12:26:35.708057 (+410952us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:35.727511 (+ 19454us) write_op.cc:454] Released schema lock
0709 12:26:35.749955 (+ 22444us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:35.875486 (+125531us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:35.880212 (+  4726us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2560649,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":356437,"apply.run_cpu_time_us":77219,"apply.run_wall_time_us":1762767,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41801,"num_ops":9,"prepare.queue_time_us":219459,"prepare.run_cpu_time_us":47472,"prepare.run_wall_time_us":553876,"raft.queue_time_us":498231,"raft.run_cpu_time_us":87802,"raft.run_wall_time_us":317176,"replication_time_us":1408890,"thread_start_us":1355818,"threads_started":5,"wal-append.queue_time_us":1319339,"wal-append.run_cpu_time_us":177465,"wal-append.run_wall_time_us":2682290}]]}
W20260709 12:26:39.647135 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92848.9us
I20260709 12:26:39.656534 17915 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 28, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 28 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.658696 28804 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:39.667522 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.672472 18809 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:39.659083 17915 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.659557 29205 raft_consensus.cc:697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:39.674227 29067 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:39.697075 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 52197.9us
I20260709 12:26:39.703867 29147 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.706043 29147 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:39.696028 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:39.700078 29136 raft_consensus.cc:515] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.713718 17989 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:39.713392 29205 consensus_queue.cc:237] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.715592 29136 leader_election.cc:290] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:39.716897 17989 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.718261 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:39.691627 28836 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.723480 28860 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:39.754097 28860 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.754618 28860 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:39.757697 28836 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:39.748921 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 116905us
I20260709 12:26:39.712044 17915 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 33, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 33 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:39.751902 29173 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:39.752467 28994 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:39.747766 29024 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:39.759586 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 51618.7us
I20260709 12:26:39.741662 29209 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.789450 29209 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:39.801424 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.806106 29173 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.810086 17915 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 27. New cstate: current_term: 27 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:39.814899 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 70487us
I20260709 12:26:39.817231 29209 consensus_queue.cc:1059] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:26:39.818465 29209 raft_consensus.cc:3055] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:39.818850 29209 raft_consensus.cc:740] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:39.819989 29209 consensus_queue.cc:260] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:39.821707 29209 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:39.812711 17915 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:39.838112 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60041.3us
I20260709 12:26:39.783224 28994 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:39.809562 29173 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:39.855504 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.855181 28994 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.857999 17915 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 26, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 26 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.861701 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.862834 17915 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 26 to 28, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 28 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:39.864084 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.861409 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.865768 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.871085 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.865193 17915 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 28, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 28 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:39.878311 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 97251.4us
I20260709 12:26:39.882928 29173 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:39.896153 29173 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:39.899848 29173 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:39.907630 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.919780 18809 raft_consensus.cc:1240] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:39.922665 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:39.927964 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.928076 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.930115 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.930262 18166 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 37.
I20260709 12:26:39.931154 18170 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:26:39.929198 28858 consensus_queue.cc:1059] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.003s
I20260709 12:26:39.933082 28858 raft_consensus.cc:3055] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:39.935101 28858 raft_consensus.cc:740] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:39.936045 18168 raft_consensus.cc:1240] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Rejecting Update request from peer 910f8c8d787e4abbb092413c1f90ebf3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:39.941041 18809 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:26:39.941217 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.942159 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.942529 18815 raft_consensus.cc:2468] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:39.943058 18167 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.943241 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.943823 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.944212 18173 raft_consensus.cc:2410] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:26:39.950004 17915 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:39.953047 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.955091 18166 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:26:39.955345 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:39.957098 18165 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.955925 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:39.952345 17915 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 25, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 25 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:39.958516 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:40.003070 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 120091us
W20260709 12:26:40.003948 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 65074.3us
I20260709 12:26:40.006812 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:26:40.006551 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 295936us
I20260709 12:26:40.013079 18809 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:26:40.008754 17915 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:40.019166 18064 leader_election.cc:336] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.014055 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:40.014207 18137 leader_election.cc:304] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:40.024698 18064 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.033243 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.033083 29134 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:40.038203 18136 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:40.042178 28983 raft_consensus.cc:2804] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:40.043843 29169 consensus_queue.cc:1048] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:40.046126 28751 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:40.046600 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:40.049073 29222 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:26:40.054324 29134 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:40.055119 29134 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.051826 17915 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 30, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:40.053421 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.066092 18095 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:40.065281 17915 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:40.029330 28860 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.094364 18170 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 35.
W20260709 12:26:40.096264 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53500us
I20260709 12:26:40.100322 18165 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:40.105962 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.102432 17915 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 25, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 25 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:40.108073 18167 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
W20260709 12:26:40.114141 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 99508.7us
W20260709 12:26:40.117002 18064 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:40.118673 18064 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.119189 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.123270 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.120769 18095 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 31.
I20260709 12:26:40.125378 18810 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 38.
I20260709 12:26:40.119448 28860 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.058157 28858 consensus_queue.cc:260] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 31.25, Last appended by leader: 25, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.134258 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:40.150686 18815 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:40.113039 28751 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.152280 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.135262 17915 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 27, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 27 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:40.155320 18137 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
W20260709 12:26:40.156250 16792 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:40.159727 17915 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:40.166455 29162 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:40.167742 29224 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.168778 29224 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.171669 29022 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.172418 29022 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.174610 18809 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:40.176558 29022 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.178967 29022 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.180060 29022 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.194336 29022 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.194418 29225 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.195315 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 177371us
I20260709 12:26:40.196725 18815 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:40.151499 28983 raft_consensus.cc:697] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
W20260709 12:26:40.161756 18780 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:40.202662 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84094.9us
I20260709 12:26:40.201633 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.204701 18808 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
I20260709 12:26:40.205134 18063 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:40.159231 28994 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:40.209667 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 79422.2us
I20260709 12:26:40.202090 28858 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:40.211930 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.213289 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.212625 29162 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:40.222254 29162 raft_consensus.cc:697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:40.209388 29233 raft_consensus.cc:2764] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:40.225783 29224 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.198648 28751 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.228845 18137 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:40.227911 17915 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 27, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 27 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:40.233040 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:40.234475 29229 log.cc:927] Time spent T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.088s	user 0.002s	sys 0.000s
I20260709 12:26:40.215624 29225 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.237809 29195 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:40.242359 29195 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.243127 28860 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:40.243932 28860 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.246145 29134 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.246472 28860 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.241202 29235 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:40.249524 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.250535 18809 raft_consensus.cc:2393] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 30: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
I20260709 12:26:40.252569 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.252336 29235 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.254092 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.251542 17915 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 22 to 26, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 26 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:40.256568 29235 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.251117 29233 raft_consensus.cc:2764] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:26:40.256451 17915 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:40.258327 29162 consensus_queue.cc:237] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.258550 17915 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 25, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:40.260637 17915 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 21, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 21 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:40.262303 17915 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 30, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 30 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:40.263118 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.266157 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:40.267140 29022 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:40.267622 29233 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:40.267959 29225 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.268075 29233 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:40.268765 29233 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:40.269644 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.269676 28983 consensus_queue.cc:237] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:40.274433 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 56900us
I20260709 12:26:40.280180 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.265571 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:40.282374 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 129103us
W20260709 12:26:40.285430 18780 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.282594 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:40.302454 29232 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:40.303083 29232 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:40.303503 29232 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:40.303682 28751 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.308352 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.311774 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:40.312124 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72069.8us
W20260709 12:26:40.317348 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 54040.2us
W20260709 12:26:40.322468 18778 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.247988 29195 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.328163 18808 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:26:40.327641 18090 raft_consensus.cc:2468] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:40.340186 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.341717 18137 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:40.346971 28836 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:40.359524 17990 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.359059 29195 raft_consensus.cc:493] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:40.366003 28994 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:40.380999 29195 raft_consensus.cc:515] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:40.386855 18138 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.806s (SENT)
W20260709 12:26:40.388314 18064 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:40.382179 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 53872.1us
I20260709 12:26:40.390167 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.390198 28562 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:40.388950 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:40.394336 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 72839.4us
I20260709 12:26:40.394373 17989 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:40.395475 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:40.397032 29225 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:26:40.383769 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 100236us
I20260709 12:26:40.397596 29225 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:40.398590 18780 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.399258 18780 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.399570 29134 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:40.400992 18780 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.401074 29209 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:40.398651 29225 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:26:40.404819 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 139444us
I20260709 12:26:40.405992 17988 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.406347 28994 raft_consensus.cc:2804] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:40.406914 28994 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.407181 28771 raft_consensus.cc:2804] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:40.407400 28994 raft_consensus.cc:3060] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:40.407733 28771 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.408222 28771 raft_consensus.cc:3060] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:40.408972 17988 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.410167 29224 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:40.412161 28836 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:26:40.397522 17987 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:40.401607 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:40.389618 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.601s (SENT)
I20260709 12:26:40.432538 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.389424 29195 leader_election.cc:290] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.449688 29209 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:40.454386 29223 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:26:40.458107 18065 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.405985 29233 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:40.481033 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 72142.5us
I20260709 12:26:40.482659 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:40.486220 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55628.8us
I20260709 12:26:40.488749 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:40.489702 18167 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 in current term 29: Already voted for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in this term.
I20260709 12:26:40.492738 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:40.493495 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 96162us
I20260709 12:26:40.493681 18170 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 26.
I20260709 12:26:40.494700 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.509395 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:40.515450 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 114502us
I20260709 12:26:40.519595 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.507858 29173 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:26:40.521157 18167 raft_consensus.cc:1240] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer 768723648a9444fda53659e6a8cc3091 for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:40.518714 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 105392us
W20260709 12:26:40.524678 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 135148us
I20260709 12:26:40.521442 18169 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:26:40.533366 29240 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.519855 18780 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.536121 18780 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.536113 18170 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:26:40.534162 29223 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:40.545446 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:40.549187 18169 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:40.545459 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 25 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:40.504896 29232 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.549686 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:40.550064 17989 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:40.520427 29209 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:40.541240 28751 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.558526 29245 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:40.550987 29136 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:40.548130 28860 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.562556 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:40.563211 18094 raft_consensus.cc:3060] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:26:40.570297 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 60664.1us
I20260709 12:26:40.572570 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.571230 17988 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:40.578459 17988 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.546638 29195 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:40.582005 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.570026 29240 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.587160 18091 raft_consensus.cc:3060] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:40.587046 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 70683.6us
I20260709 12:26:40.579828 18064 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:40.592255 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "666440ca642c4715bd2f211bb5f14d29" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.594117 18098 raft_consensus.cc:1275] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:40.594240 28858 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.571170 29209 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.596091 28858 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.565119 29244 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:40.562232 29245 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.605201 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 35 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.606307 18808 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 35 because replica has last-logged OpId of term: 33 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 25.
I20260709 12:26:40.608204 28858 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.561003 29243 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:40.610225 29242 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:40.562675 28751 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.563091 28860 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.613037 29209 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.604904 29245 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.615665 29245 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.594357 29195 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.591681 29223 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:40.624253 29223 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.624715 29223 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:40.626188 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 131676us
I20260709 12:26:40.629434 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:40.630457 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 48564.4us
I20260709 12:26:40.631000 29244 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:40.631551 29242 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:26:40.625424 29195 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.634066 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.634363 17988 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.639094 18169 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:40.655567 18808 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 34.
I20260709 12:26:40.662213 18808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:40.669433 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77148.1us
I20260709 12:26:40.632495 29240 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.670360 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.674544 18090 raft_consensus.cc:2468] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:26:40.658018 29134 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:40.674690 18091 raft_consensus.cc:2468] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:26:40.657382 29022 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:26:40.682243 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 114569us
I20260709 12:26:40.593572 28771 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.655330 28994 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.686146 29249 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.687057 29249 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.687497 28994 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.690586 29240 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:40.690151 29244 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.696091 28751 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:40.670280 29243 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:40.705206 29232 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:40.706284 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.652655 29225 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 39 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.711113 18815 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 27: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:40.714243 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:40.715708 18094 raft_consensus.cc:2468] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 35.
I20260709 12:26:40.709206 29233 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:40.717670 29067 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:40.677186 29245 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:26:40.718465 29067 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:40.721966 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 92650.4us
W20260709 12:26:40.725499 18064 leader_election.cc:336] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.726141 18064 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:40.727007 18138 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.807s (SENT)
I20260709 12:26:40.731667 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.709810 29134 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:40.737058 18810 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:40.739085 29249 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.733036 29162 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:40.721884 28858 consensus_queue.cc:1059] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:26:40.744135 29232 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:26:40.762365 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:40.764678 18064 leader_election.cc:336] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.766851 28860 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.753693 29195 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.015s
I20260709 12:26:40.769212 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.775104 29134 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:40.777676 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:26:40.778223 29242 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.778982 29242 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.785795 18064 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.789362 29235 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:40.790705 29225 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 39 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.791529 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:40.789397 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 78955.8us
I20260709 12:26:40.820799 29067 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.824311 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.815582 29162 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:40.826965 28771 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Requested vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.812418 29235 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting leader election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
I20260709 12:26:40.828040 29235 raft_consensus.cc:3060] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:40.831303 18138 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 1.852s (SENT)
I20260709 12:26:40.831954 18138 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:40.832427 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 54880.5us
I20260709 12:26:40.832186 29223 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:40.833438 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Timed out: RequestConsensusVote RPC to 127.16.102.1:45965 timed out after 2.107s (SENT)
W20260709 12:26:40.840517 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75871.9us
I20260709 12:26:40.834519 29242 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:40.835731 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 44354.2us
W20260709 12:26:40.850179 18064 leader_election.cc:336] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.850483 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.850833 18064 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.834108 28924 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:40.834054 29022 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:40.858940 29245 raft_consensus.cc:3060] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:40.848364 29223 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:40.842628 29232 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:40.869035 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 146620us
I20260709 12:26:40.891546 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.892931 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:40.893654 18091 raft_consensus.cc:2393] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in current term 29: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:40.890815 18780 leader_election.cc:304] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:40.896809 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 347014us
I20260709 12:26:40.892549 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.897965 18099 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 29.
I20260709 12:26:40.870534 29162 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:40.898294 18780 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:40.900107 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.900107 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.902770 18090 raft_consensus.cc:3055] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:40.902715 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.905759 18090 raft_consensus.cc:740] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:40.902151 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 171242us
I20260709 12:26:40.900372 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.909098 28836 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.909847 28836 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.910502 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.911429 18090 consensus_queue.cc:260] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:40.911804 29242 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:40.902817 29223 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:26:40.912806 18090 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:40.906949 18778 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.918705 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:40.920920 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.923142 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:40.924031 18815 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:26:40.921057 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:40.928717 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:40.930308 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.940802 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e793f93c4f740b78f31d2f0dd643095" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:40.943013 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 92877.9us
I20260709 12:26:40.950455 18167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:40.952728 29251 log.cc:927] Time spent T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.101s	user 0.005s	sys 0.000s
I20260709 12:26:40.953236 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:40.955502 29258 raft_consensus.cc:2804] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:40.956254 29258 raft_consensus.cc:697] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:40.956573 18807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:40.957278 18807 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:40.957671 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.958102 29264 raft_consensus.cc:2804] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:26:40.943995 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:40.959349 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.960027 18095 raft_consensus.cc:2468] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 30.
W20260709 12:26:40.960121 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:40.967100 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.968012 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.966677 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:40.973308 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:40.974175 18170 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:26:40.978555 18164 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:40.979172 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:40.967746 18064 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.943437 29260 raft_consensus.cc:2804] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:40.923261 28836 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:40.982750 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.987191 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 35 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.987907 18064 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:40.988061 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.991003 18171 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 33.
I20260709 12:26:40.993160 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:40.995373 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.993295 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.994845 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:40.999576 18099 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:26:40.997409 18093 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 33.
I20260709 12:26:41.000500 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:41.011420 18090 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 14. (term mismatch)
I20260709 12:26:40.982199 28751 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.012797 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:41.015141 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:40.986706 29242 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:41.016077 18815 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 39.
I20260709 12:26:40.997993 29244 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:41.013625 18090 pending_rounds.cc:85] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 13
I20260709 12:26:41.014535 17989 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:41.017916 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:41.018061 17988 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:41.021100 29223 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:26:41.021943 29223 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 39 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:41.018453 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:41.040421 18170 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 39.
I20260709 12:26:41.020089 18090 pending_rounds.cc:107] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260709 12:26:41.043033 18090 raft_consensus.cc:2889] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:41.043630 18093 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:41.045539 29256 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:41.014288 29224 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:26:41.016742 28771 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:40.957414 29258 consensus_queue.cc:237] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 31.25, Last appended by leader: 25, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.046656 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:41.007016 28562 consensus_queue.cc:1048] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:26:41.028198 29249 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:41.051527 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
W20260709 12:26:41.060876 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 93154.2us
I20260709 12:26:41.069666 18094 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:41.070204 18064 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.028541 29240 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:41.036140 28994 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:41.068310 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 50522.4us
I20260709 12:26:41.052088 29225 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:41.077652 18064 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:41.083269 29270 log.cc:927] Time spent T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260709 12:26:41.063653 28860 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260709 12:26:41.069774 29223 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.107507 18099 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:41.071441 29245 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:41.043936 28751 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.117205 29260 raft_consensus.cc:697] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:41.051548 29067 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:41.053912 29242 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:41.118779 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 220566us
I20260709 12:26:41.077399 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:41.122668 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 75304.5us
I20260709 12:26:41.071741 29244 consensus_queue.cc:1059] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
W20260709 12:26:41.132928 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55701.3us
I20260709 12:26:41.135808 17987 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
W20260709 12:26:41.138929 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:41.147509 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 77366.1us
W20260709 12:26:41.148656 18064 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:41.156782 16792 thread.cc:527] Waited for 4000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:41.139823 28994 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:41.162424 29260 consensus_queue.cc:237] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:41.140571 18780 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.167984 17987 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:41.171140 18094 raft_consensus.cc:2468] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 28.
I20260709 12:26:41.174661 18807 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
W20260709 12:26:41.177956 18064 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.177343 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.175727 28994 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.181761 28994 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.184211 28751 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:41.184728 28994 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:41.154211 28836 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:41.140049 29067 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.154247 29264 raft_consensus.cc:697] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:41.211071 29264 consensus_queue.cc:237] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.211699 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:41.213550 29235 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:41.182614 29254 log.cc:927] Time spent T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: Append to log took a long time: real 0.118s	user 0.003s	sys 0.000s
W20260709 12:26:41.216317 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48453.1us
I20260709 12:26:41.217278 17987 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:41.231698 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:41.251226 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:41.209619 29067 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:41.253892 18172 raft_consensus.cc:2410] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768723648a9444fda53659e6a8cc3091 for term 35 because replica has last-logged OpId of term: 33 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 25.
I20260709 12:26:41.249239 18093 raft_consensus.cc:2468] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:41.258003 18171 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:41.258419 29242 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:41.258718 18170 raft_consensus.cc:1275] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:26:41.259341 29277 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:41.254649 18136 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:41.261204 29257 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.334s	user 0.005s	sys 0.000s
I20260709 12:26:41.250869 28994 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.245124 29246 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:41.264900 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:41.275213 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 319663us
W20260709 12:26:41.277272 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 60088.6us
I20260709 12:26:41.278662 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:41.280359 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 139811us
W20260709 12:26:41.281396 18780 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.282015 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.262238 29195 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:41.266319 28983 raft_consensus.cc:2804] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:41.299752 18780 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:41.309849 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48081.4us
I20260709 12:26:41.285910 29224 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:41.297667 29258 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:41.326545 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 48042.6us
I20260709 12:26:41.297523 29235 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:41.332474 18136 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
W20260709 12:26:41.338045 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 170968us
I20260709 12:26:41.350124 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:41.319537 29224 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:41.349112 18138 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.355773 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:41.326480 28983 raft_consensus.cc:697] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:41.371791 18170 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:26:41.355492 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.338127 28836 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:41.360284 29224 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:41.324546 29249 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:26:41.358680 29173 raft_consensus.cc:2804] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:26:41.396296 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46328.9us
W20260709 12:26:41.385957 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 73810.4us
I20260709 12:26:41.397802 17987 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:41.348766 29240 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:26:41.422072 18809 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:26:41.368777 29272 raft_consensus.cc:2804] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:41.392421 28924 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.422662 29289 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:41.426268 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:41.427901 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:41.428568 18093 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:41.417053 29225 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.384578 28994 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.053s
I20260709 12:26:41.445259 29289 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:41.461750 29289 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.463055 28983 consensus_queue.cc:237] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:41.466945 29282 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
W20260709 12:26:41.469607 29279 log.cc:927] Time spent T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:26:41.434645 29272 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.437526 29173 raft_consensus.cc:697] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:41.493181 29173 consensus_queue.cc:237] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.476445 29255 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.479030 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:41.488193 29022 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.499303 29022 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.474905 29272 raft_consensus.cc:3060] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:41.502007 29277 consensus_queue.cc:1048] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:41.503463 29022 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:41.505357 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 177008us
W20260709 12:26:41.526530 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 137003us
I20260709 12:26:41.534336 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:41.548609 29296 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.016s
I20260709 12:26:41.573989 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:41.575268 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:41.594513 18780 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.595284 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.596980 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:41.606065 18138 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.606729 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:41.607434 18099 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 31.
I20260709 12:26:41.586289 29245 raft_consensus.cc:3055] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:41.594435 29244 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:41.601332 29264 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:41.629225 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:41.615006 28771 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.632076 18092 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
W20260709 12:26:41.636821 18138 leader_election.cc:336] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.638862 18138 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.612056 29245 raft_consensus.cc:740] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:41.604379 29225 consensus_queue.cc:1059] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
I20260709 12:26:41.641440 29245 consensus_queue.cc:260] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.647315 29224 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.651278 18099 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:41.651867 18092 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 19. (term mismatch)
I20260709 12:26:41.661068 18809 raft_consensus.cc:2468] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 34.
W20260709 12:26:41.665249 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 324471us
I20260709 12:26:41.652631 18092 pending_rounds.cc:85] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting all ops after (but not including) 18
I20260709 12:26:41.667717 18092 pending_rounds.cc:107] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Aborting uncommitted NO_OP operation due to leader change: 27.19
W20260709 12:26:41.666450 18064 leader_election.cc:336] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.668838 18064 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.669879 18092 raft_consensus.cc:2889] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:41.678254 18138 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.662851 29233 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:26:41.691094 18138 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.692420 29256 raft_consensus.cc:3055] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:41.692979 29256 raft_consensus.cc:740] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:41.699649 29256 consensus_queue.cc:260] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.700503 18137 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:41.701318 29256 raft_consensus.cc:3060] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:41.701623 29233 raft_consensus.cc:2804] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:41.702271 29233 raft_consensus.cc:697] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 34 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:41.703405 29233 consensus_queue.cc:237] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.705438 18172 raft_consensus.cc:3060] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:41.693612 18138 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.711018 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:41.714242 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 46898.6us
I20260709 12:26:41.650328 29224 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:41.728122 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 61722us
I20260709 12:26:41.728739 18173 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:26:41.729453 29280 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.729548 29272 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.730175 29280 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:41.732758 29272 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:41.733491 29280 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:41.737043 29255 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:41.737454 18064 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.738055 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.739005 29255 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:41.740623 18064 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.741149 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.742288 29255 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:26:41.748504 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 136263us
I20260709 12:26:41.749581 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
I20260709 12:26:41.761164 29286 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:41.749404 29277 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.016s
W20260709 12:26:41.766400 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72826.9us
W20260709 12:26:41.768051 18138 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.768638 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:41.777485 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7a707b110a44c8c97606fdb8d7dece0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:41.778630 18090 raft_consensus.cc:2393] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 34: Already voted for candidate 768723648a9444fda53659e6a8cc3091 in this term.
I20260709 12:26:41.771780 28751 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:41.781872 18138 leader_election.cc:336] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.788769 18809 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:26:41.792840 29245 raft_consensus.cc:3060] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:41.809139 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 51642.7us
I20260709 12:26:41.816809 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:41.817623 18170 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:41.818007 18172 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 31 index: 25. Preceding OpId from leader: term: 32 index: 26. (index mismatch)
I20260709 12:26:41.818681 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:41.828024 18169 raft_consensus.cc:1240] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:41.828295 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:41.842478 29209 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:26:41.809141 28994 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:41.849676 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:41.850853 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:41.855269 18090 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:41.869287 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 98539.7us
I20260709 12:26:41.860776 18166 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 32.
I20260709 12:26:41.854353 28994 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:41.884752 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 140975us
I20260709 12:26:41.889209 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:41.852624 29224 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:26:41.896221 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:41.897949 29274 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.074s
W20260709 12:26:41.900808 17988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.892067 28994 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:41.883042 29244 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.942315 29244 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:41.826841 29243 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.060s
I20260709 12:26:41.927532 18170 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 34.
I20260709 12:26:41.939126 29022 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.940209 18173 raft_consensus.cc:1240] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Rejecting Update request from peer cab4a2f5d3ca48a2a803ee8ec3906e1d for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:41.949179 18097 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:41.945349 18815 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:41.945744 18167 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:26:41.956347 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:41.945572 29244 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:41.972678 29022 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:41.975354 17989 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:41.961684 18780 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 96 similar messages]
I20260709 12:26:41.981707 18098 raft_consensus.cc:1275] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:26:41.985100 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e84e056b63c34d8da15d2fde3877279b" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:41.999926 18090 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 34.
I20260709 12:26:42.002642 18093 raft_consensus.cc:1275] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 31 index: 25. Preceding OpId from leader: term: 32 index: 26. (index mismatch)
I20260709 12:26:41.985903 18097 raft_consensus.cc:2376] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 35. Re-sending same reply.
I20260709 12:26:42.016918 29244 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:42.021740 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.007913 29225 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:26:42.035282 18064 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.043980 29272 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.044946 29272 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.048748 18064 leader_election.cc:304] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:42.049211 29272 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:41.986107 29223 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.048s
I20260709 12:26:42.017839 29244 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.058415 18064 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:42.073405 18064 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.074389 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.027691 29249 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.034s
W20260709 12:26:42.088699 18064 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.089250 18064 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.095116 18171 raft_consensus.cc:1275] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:42.103535 28994 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.108724 29280 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:42.109500 29280 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.112205 29280 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:42.113030 29280 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:42.114756 18064 leader_election.cc:336] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.115372 18064 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:42.116816 18064 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.117336 18064 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.118129 29296 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.058542 29225 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:42.059873 29292 raft_consensus.cc:2749] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:42.129750 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 97709us
I20260709 12:26:42.088207 29255 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:42.155689 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.122246 29162 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:42.163336 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 46571.2us
I20260709 12:26:42.122619 29067 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 910f8c8d787e4abbb092413c1f90ebf3)
W20260709 12:26:42.167490 16792 thread.cc:527] Waited for 5000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:42.113662 29280 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.170822 29225 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:42.168407 29319 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.069s	user 0.002s	sys 0.000s
W20260709 12:26:42.183456 18064 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.187016 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.191049 18166 raft_consensus.cc:1275] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:26:42.164204 29296 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:42.197114 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.199965 18090 raft_consensus.cc:1275] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:42.187158 29067 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:42.205673 18098 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:26:42.208003 29067 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:42.204841 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 194396us
I20260709 12:26:42.210062 29162 consensus_queue.cc:1048] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:42.190006 29292 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:42.220986 18138 leader_election.cc:336] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.228224 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.221669 18138 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.236452 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:42.238173 28994 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:42.244756 29280 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:42.250728 29296 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.250804 18098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:42.251477 29296 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:42.260538 29312 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.243s	user 0.003s	sys 0.000s
I20260709 12:26:42.261372 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.263926 29307 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.264400 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:42.264599 29307 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.265113 18095 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:26:42.270381 29244 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:42.270350 29296 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:42.292999 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:42.295792 29307 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:42.302526 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 81600us
I20260709 12:26:42.302246 29286 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:42.316805 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:42.285351 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:42.339519 18064 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.340080 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 54873.2us
I20260709 12:26:42.340509 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:42.343129 18064 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.343751 18064 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.346722 18099 raft_consensus.cc:1275] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:26:42.348503 18064 leader_election.cc:336] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.349025 18064 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.364233 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:42.366142 18092 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 32 index: 25. Preceding OpId from leader: term: 34 index: 27. (index mismatch)
I20260709 12:26:42.352059 29292 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:42.374377 18064 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.362104 29296 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.375218 18064 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.378006 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.351931 29324 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:42.362428 29289 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.378940 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.351840 29195 consensus_queue.cc:1048] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.022s
I20260709 12:26:42.391300 18097 raft_consensus.cc:1275] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:26:42.394951 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:42.404567 18095 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 33.
I20260709 12:26:42.407582 17987 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 
W20260709 12:26:42.415328 18064 leader_election.cc:336] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.415620 18166 raft_consensus.cc:1275] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:26:42.415928 18064 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.418030 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:42.421092 18164 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:42.382637 29280 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.422504 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.424857 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:42.425832 29324 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:42.424003 29264 consensus_queue.cc:1048] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.003s
I20260709 12:26:42.429162 29277 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:42.435698 29293 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:42.447165 18164 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab4a2f5d3ca48a2a803ee8ec3906e1d in term 33.
I20260709 12:26:42.418167 29022 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:42.494135 17989 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:42.438216 29249 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:42.407847 29317 consensus_queue.cc:1048] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.074s
I20260709 12:26:42.511605 17989 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:42.479411 29225 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:42.510057 28751 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.513196 29224 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:42.518766 29249 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 LEADER]: Becoming Leader. State: Replica: cab4a2f5d3ca48a2a803ee8ec3906e1d, State: Running, Role: LEADER
I20260709 12:26:42.571324 18807 raft_consensus.cc:1275] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 33 index: 26. Preceding OpId from leader: term: 34 index: 27. (index mismatch)
I20260709 12:26:42.540329 29243 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:42.545253 29067 raft_consensus.cc:493] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.549573 29232 consensus_queue.cc:1048] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:26:42.545504 29225 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:42.623420 29329 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
W20260709 12:26:42.624706 29327 log.cc:927] Time spent T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:26:42.513175 29246 consensus_queue.cc:1048] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
W20260709 12:26:42.658339 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 118962us
W20260709 12:26:42.661340 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 139361us
I20260709 12:26:42.678241 18172 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:26:42.556432 28751 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:42.680382 29296 raft_consensus.cc:493] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:42.747677 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 443098us
W20260709 12:26:42.741603 29329 log.cc:927] Time spent T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:26:42.661664 29067 raft_consensus.cc:515] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:42.701545 29286 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.051s
I20260709 12:26:42.733953 29337 consensus_queue.cc:1048] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.065s
I20260709 12:26:42.743067 29256 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
W20260709 12:26:42.801122 29321 log.cc:927] Time spent T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.278s	user 0.003s	sys 0.000s
W20260709 12:26:42.803557 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 47006.6us
W20260709 12:26:42.806668 29335 log.cc:927] Time spent T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:26:42.807345 29022 raft_consensus.cc:493] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.807992 29022 raft_consensus.cc:515] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.731689 29173 consensus_queue.cc:1048] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.027s
W20260709 12:26:42.754108 18138 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.813419 28751 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:42.822520 18138 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:42.826444 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72376.2us
W20260709 12:26:42.828292 18138 leader_election.cc:336] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.829366 18138 leader_election.cc:336] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.829782 18138 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:42.833069 18138 leader_election.cc:336] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.833611 18138 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.753449 29296 raft_consensus.cc:515] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:42.836964 29237 consensus_queue.cc:1048] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.076s
I20260709 12:26:42.740806 29249 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:42.859619 29234 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:42.860679 29341 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:42.862687 29067 leader_election.cc:290] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:42.864475 29022 leader_election.cc:290] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:42.889552 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:42.845623 18138 leader_election.cc:336] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.908869 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 84449.2us
W20260709 12:26:42.910470 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 64915.8us
I20260709 12:26:42.837774 29224 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.838676 28751 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:42.930033 18138 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.931129 29224 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:42.966236 29224 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:42.969885 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:42.972388 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:42.967909 18138 leader_election.cc:336] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.978665 18138 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:42.998723 28836 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.999426 28836 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.012802 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.017028 29286 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:43.016278 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48429.2us
W20260709 12:26:43.017534 18064 leader_election.cc:336] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:43.018620 18064 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:43.021441 18064 leader_election.cc:336] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:43.022066 29255 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:43.025029 29358 log.cc:927] Time spent T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:26:43.027923 18064 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:43.032578 18064 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:43.033161 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:43.029181 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.036634 28836 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.036821 29296 leader_election.cc:290] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:43.037622 29352 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.038378 29352 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.041399 29352 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:43.034288 18138 leader_election.cc:336] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:43.054718 18138 leader_election.cc:336] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:43.051992 29292 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:43.095328 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 31 index: 25 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:43.064497 18138 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:43.098342 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:43.100528 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.052661 29296 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:43.096076 29326 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.530s	user 0.000s	sys 0.008s
I20260709 12:26:43.156792 29225 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.159298 18093 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45032 (request call id 4322) took 1148 ms. Trace:
I20260709 12:26:43.159751 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f3da088a314a87a78898050dab39e9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.159896 18093 rpcz_store.cc:276] 0709 12:26:42.010904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.014120 (+  3216us) service_pool.cc:224] Handling call
0709 12:26:42.014651 (+   531us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:42.016386 (+  1735us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:42.016444 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:42.023528 (+  7084us) log.cc:844] Serialized 26 byte log entry
0709 12:26:42.434779 (+411251us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:42.434834 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:42.434933 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:43.027108 (+592175us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4400022b4c. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c134f0c85 00007f6c134f16a2 00007f6c162631db 00007f6c1625eaaf 00007f6c1625286e 00007f6c16251bb6 00007f6c162517d8 00007f6c1625b221 00007f6c1625b1d9 00007f6c1625b169
0709 12:26:43.158940 (+131832us) raft_consensus.cc:1685] finished
0709 12:26:43.158969 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:43.159198 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1175,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":12468,"spinlock_wait_cycles":30576512,"thread_start_us":373796,"threads_started":1,"wal-append.queue_time_us":453787}
W20260709 12:26:43.169513 16792 thread.cc:527] Waited for 6000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:43.177508 28994 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.178373 28994 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.187505 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.190701 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.196944 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e00b535474906948e66659110db3b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 31 index: 25 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.198627 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.199280 29315 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:43.197103 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.208316 18063 leader_election.cc:304] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:43.211663 29355 log.cc:927] Time spent T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.247s	user 0.002s	sys 0.000s
I20260709 12:26:43.228163 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.236088 18063 leader_election.cc:304] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.249271 29280 raft_consensus.cc:2749] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:43.257375 29346 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: Append to log took a long time: real 0.431s	user 0.003s	sys 0.000s
I20260709 12:26:43.234512 29296 raft_consensus.cc:2749] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:43.259042 28994 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:43.262254 29360 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:26:43.265633 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.269398 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.280603 18094 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.284763 18092 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:26:43.287468 17989 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.288553 18137 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:43.289601 29234 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:43.291710 29353 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:43.295925 28836 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.296694 29249 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.296545 28836 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.298089 29272 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.298689 29272 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.301030 29272 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.302877 28836 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.297279 29249 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.309898 29249 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.272656 29225 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.311509 29225 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.312201 28836 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:43.312947 28836 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.315327 28836 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.320465 17989 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:43.391384 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 71047.6us
I20260709 12:26:43.323895 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c743e221c2c477e9b998557d9b9749a" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.331489 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.335295 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.404142 18171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.406906 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.342496 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.411755 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:43.329396 29350 log.cc:927] Time spent T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.362s	user 0.001s	sys 0.002s
I20260709 12:26:43.431373 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.444448 18170 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:26:43.451704 17989 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.351347 29272 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.324352 28836 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.437000 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.316934 29225 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:43.473045 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.474267 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.411759 29351 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:43.472093 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:43.478495 18138 leader_election.cc:336] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:43.479079 18138 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:43.476459 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.471689 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.480618 29307 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.005s
I20260709 12:26:43.485960 29249 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.486663 29249 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.475582 28836 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.489121 29249 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.491156 28994 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.491967 29353 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:43.491925 28994 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.472637 29244 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:43.495404 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.533172 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:43.534351 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 98215.5us
I20260709 12:26:43.533172 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.462746 29272 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.535499 29246 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:43.550597 17987 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.566548 17987 leader_election.cc:304] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.568607 17987 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:43.579167 17915 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 300) took 26436 ms (client timeout 15000 ms). Trace:
I20260709 12:26:43.583900 28994 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:43.584674 28994 raft_consensus.cc:2749] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:43.585414 28994 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:43.516780 29351 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:43.590293 29351 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.592900 29351 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.549794 29244 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.594017 29244 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.516445 29353 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.081s
I20260709 12:26:43.572172 29225 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:43.607180 17987 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.613629 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:43.579774 17915 rpcz_store.cc:269] 0709 12:26:17.142775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.540768 (+11397993us) service_pool.cc:224] Handling call
0709 12:26:43.579012 (+15038244us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:40.610689 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:40.611520 (+   831us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:40.611580 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:26:40.611600 (+    20us) tablet.cc:662] Decoding operations
0709 12:26:40.615128 (+  3528us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:40.615251 (+   123us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:40.615286 (+    35us) tablet.cc:685] Acquiring locks for 30 operations
0709 12:26:40.718331 (+103045us) tablet.cc:701] Row locks acquired
0709 12:26:40.718396 (+    65us) write_op.cc:260] PREPARE: finished
0709 12:26:40.718616 (+   220us) write_op.cc:270] Start()
0709 12:26:40.718795 (+   179us) write_op.cc:276] Timestamp: P: 1783600000718533 usec, L: 0
0709 12:26:40.718845 (+    50us) op_driver.cc:348] REPLICATION: starting
0709 12:26:40.719803 (+   958us) log.cc:844] Serialized 11758 byte log entry
0709 12:26:41.525106 (+805303us) op_driver.cc:464] REPLICATION: finished
0709 12:26:41.682104 (+156998us) write_op.cc:301] APPLY: starting
0709 12:26:41.682246 (+   142us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:42.068529 (+386283us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:42.068700 (+   171us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:42.809607 (+740907us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:42.812409 (+  2802us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=60,key_file_lookups=60,delta_file_lookups=0,mrs_lookups=0
0709 12:26:42.812486 (+    77us) write_op.cc:312] APPLY: finished
0709 12:26:42.813198 (+   712us) log.cc:844] Serialized 260 byte log entry
0709 12:26:42.813626 (+   428us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:42.820086 (+  6460us) write_op.cc:454] Released schema lock
0709 12:26:42.826752 (+  6666us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:42.827321 (+   569us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:42.827353 (+    32us) write_op.cc:454] Released schema lock
0709 12:26:42.989647 (+162294us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b1800055a80. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1c492ab1 00007f6c1c49f281 00007f6c135ca557 00007f6c135cc7c1 00007f6c135cc779 00007f6c135cc709 00007f6c135cc6d1 00007f6c135cb9cd 0000000000388634 0000000000388479 00007f6c135ad83a
Metrics: {"mutex_wait_us":939147,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":152392,"apply.run_cpu_time_us":60082,"apply.run_wall_time_us":1145206,"cfile_cache_hit":122,"cfile_cache_hit_bytes":129707,"num_ops":30,"prepare.queue_time_us":162467,"prepare.run_cpu_time_us":31790,"prepare.run_wall_time_us":352116,"raft.queue_time_us":82679,"raft.run_cpu_time_us":49490,"raft.run_wall_time_us":72631,"replication_time_us":806089,"spinlock_wait_cycles":10164992,"thread_start_us":394724,"threads_started":4,"wal-append.queue_time_us":445421,"wal-append.run_cpu_time_us":24523,"wal-append.run_wall_time_us":1544521}]]}
I20260709 12:26:43.618083 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.599601 28836 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:43.549263 29352 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.611925 29294 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:43.631052 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.636240 29244 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.619544 29351 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:43.656186 28836 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:43.661545 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.668316 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.677595 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:43.680651 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 311794us
I20260709 12:26:43.691071 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.692000 18093 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.693477 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.694326 18096 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Refusing update from remote peer cab4a2f5d3ca48a2a803ee8ec3906e1d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:26:43.694799 18173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:43.642215 29352 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.702505 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.690104 29294 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:43.724383 29272 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:43.748646 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.756939 29352 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:43.767071 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.777200 29294 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:43.784555 28836 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:43.786406 17987 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.788638 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:43.793048 29272 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:43.793866 29352 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:43.800431 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.803339 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.809777 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.762568 29249 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.817900 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:43.815485 29326 log.cc:927] Time spent T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:26:43.812407 28836 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:26:43.838322 29272 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.843214 29272 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:43.850260 28836 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.822594 29249 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:43.853209 28836 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:43.854247 29249 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:43.854542 29369 log.cc:927] Time spent T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:26:43.860046 18092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.861697 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.863265 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:43.866138 29272 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.893249 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef30e67b67894670b0cf3980d6b0f624" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.897130 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:43.994418 29225 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:44.005774 29225 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:44.004781 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 199999us
I20260709 12:26:44.009310 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:44.013017 29225 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:43.954751 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.026480 28836 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:44.028115 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 73654.6us
I20260709 12:26:44.056272 17988 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:44.046798 28836 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:44.060295 17918 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 282) took 195015 ms (client timeout 15000 ms). Trace:
I20260709 12:26:44.080698 28836 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:44.080910 17918 rpcz_store.cc:269] 0709 12:23:29.045215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.045539 (+   324us) service_pool.cc:224] Handling call
0709 12:26:44.059960 (+195014421us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":612468,"reported_tablets":50}
I20260709 12:26:44.083636 28836 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:44.080768 17988 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.040412 17919 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 13 to 15, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 15 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.078871 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.079938 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.089329 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:44.081144 29225 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:44.099143 17988 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:44.097746 28836 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:44.101151 17988 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:44.100903 29225 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:44.110752 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.124145 29225 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:44.127991 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:44.170617 16792 thread.cc:527] Waited for 7000ms trying to join with heartbeat (tid 18049)
W20260709 12:26:44.195778 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.196918 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.197858 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.214030 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.235405 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.236817 29352 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:44.237012 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.238009 29249 raft_consensus.cc:2749] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:44.239552 17989 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:44.288405 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 61622.5us
I20260709 12:26:44.287544 29249 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:44.278043 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.320415 29224 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:44.358976 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 66447.1us
W20260709 12:26:44.385699 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.390908 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.418183 29353 raft_consensus.cc:493] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.418877 29353 raft_consensus.cc:515] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:44.421366 29353 leader_election.cc:290] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:44.429419 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.428793 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f144ec392b043d8b68e284ffecdee43" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:44.445467 17989 leader_election.cc:304] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.448354 29353 raft_consensus.cc:2749] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:44.494829 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.506093 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.511484 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.543015 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.584107 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.623698 17919 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 281) took 111221 ms (client timeout 15000 ms). Trace:
W20260709 12:26:44.624414 17919 rpcz_store.cc:269] 0709 12:24:53.398851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.399262 (+   411us) service_pool.cc:224] Handling call
0709 12:26:44.619733 (+111220471us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:44.256591 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:44.358511 (+101920us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:44.358553 (+    42us) write_op.cc:435] Acquired schema lock
0709 12:26:44.358572 (+    19us) tablet.cc:662] Decoding operations
0709 12:26:44.364474 (+  5902us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:44.364628 (+   154us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:44.368316 (+  3688us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:26:44.377030 (+  8714us) tablet.cc:701] Row locks acquired
0709 12:26:44.380258 (+  3228us) write_op.cc:260] PREPARE: finished
0709 12:26:44.389893 (+  9635us) write_op.cc:270] Start()
0709 12:26:44.397230 (+  7337us) write_op.cc:276] Timestamp: P: 1783600004389780 usec, L: 0
0709 12:26:44.398785 (+  1555us) op_driver.cc:348] REPLICATION: starting
0709 12:26:44.407532 (+  8747us) log.cc:844] Serialized 533 byte log entry
0709 12:26:44.498561 (+ 91029us) op_driver.cc:464] REPLICATION: finished
0709 12:26:44.527216 (+ 28655us) write_op.cc:301] APPLY: starting
0709 12:26:44.527324 (+   108us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:44.528652 (+  1328us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:44.528760 (+   108us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:44.529594 (+   834us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:44.529893 (+   299us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:26:44.529942 (+    49us) write_op.cc:312] APPLY: finished
0709 12:26:44.530197 (+   255us) log.cc:844] Serialized 25 byte log entry
0709 12:26:44.530408 (+   211us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:44.530510 (+   102us) write_op.cc:454] Released schema lock
0709 12:26:44.530554 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:44.531083 (+   529us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:44.531114 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":103181,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":28332,"apply.run_cpu_time_us":3849,"apply.run_wall_time_us":3855,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8313,"num_ops":1,"prepare.queue_time_us":91405,"prepare.run_cpu_time_us":18493,"prepare.run_wall_time_us":241095,"raft.queue_time_us":29601,"raft.run_cpu_time_us":1332,"raft.run_wall_time_us":1994,"replication_time_us":99572,"thread_start_us":153435,"threads_started":4,"wal-append.queue_time_us":58865}]]}
I20260709 12:26:44.658825 29272 raft_consensus.cc:493] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.659513 29272 raft_consensus.cc:515] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:44.664147 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:44.664393 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49a68a064bb44579af6ee5ad50121ffc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.665946 29272 leader_election.cc:290] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:44.666323 17987 leader_election.cc:304] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.667752 29353 raft_consensus.cc:2749] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:44.679672 17915 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 30, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 30 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.689157 17915 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 29, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 29 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.691094 17915 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 27, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 27 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.700917 17915 catalog_manager.cc:5697] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.702750 17915 catalog_manager.cc:5697] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 25, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.705621 17915 catalog_manager.cc:5697] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 26, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.711881 17915 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 26. New cstate: current_term: 26 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.716226 17915 catalog_manager.cc:5697] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 29, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 29 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.721233 17915 catalog_manager.cc:5697] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 30, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.723105 17915 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 29, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 29 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:44.710769 17921 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 283) took 164382 ms (client timeout 15000 ms). Trace:
W20260709 12:26:44.724527 17921 rpcz_store.cc:269] 0709 12:24:00.327864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.328140 (+   276us) service_pool.cc:224] Handling call
0709 12:26:44.710510 (+164382370us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14396,"reported_tablets":50}
I20260709 12:26:44.727890 17915 catalog_manager.cc:5697] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 26. New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.729534 17915 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 30, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 30 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.731400 17915 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 26 to 29. New cstate: current_term: 29 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.733031 17915 catalog_manager.cc:5697] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 29, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 29 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.734650 17915 catalog_manager.cc:5697] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 31, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 31 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.736454 17915 catalog_manager.cc:5697] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 28, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 28 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.738451 17915 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.745689 17915 catalog_manager.cc:5697] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 31, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 31 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.751549 17915 catalog_manager.cc:5697] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 31, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 31 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:44.768841 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.753252 17915 catalog_manager.cc:5697] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 15 to 23, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 23 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.778844 17915 catalog_manager.cc:5697] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 30. New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:44.780403 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.780804 17915 catalog_manager.cc:5697] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 20 to 27. New cstate: current_term: 27 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.782421 17915 catalog_manager.cc:5697] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 26, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.783862 17915 catalog_manager.cc:5697] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.792578 29272 raft_consensus.cc:493] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.793395 29272 raft_consensus.cc:515] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:44.797859 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.800343 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e0ff0e5252f4ace898a73a2268e5dc8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:44.810501 17987 leader_election.cc:304] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.812137 17915 catalog_manager.cc:5697] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 26 to 29, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 29 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.813414 29272 leader_election.cc:290] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:44.813947 17915 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 29 to 30, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 30 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.815810 17915 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:44.816516 29353 raft_consensus.cc:2749] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:44.817545 17915 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 21 to 24, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 24 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:44.843535 17915 catalog_manager.cc:5697] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 30, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:44.858073 17915 catalog_manager.cc:5697] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 22 to 27, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 27 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:44.867033 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.927620 29353 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.947359 29353 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:44.969305 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:44.997428 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.000952 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:44.986677 29353 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:45.004782 29224 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:45.067080 29353 raft_consensus.cc:493] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:45.067865 29353 raft_consensus.cc:515] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.072110 29294 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.072801 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.072862 29294 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.075124 29353 leader_election.cc:290] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:45.063164 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.086206 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.117230 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.138450 29294 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:45.119343 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.156682 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:45.171305 16792 thread.cc:527] Waited for 8000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:45.169663 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd93736a3314dbbb5c62b5af8fbed11" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.183091 17987 leader_election.cc:304] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.189167 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:45.187471 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.235879 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 77312.7us
I20260709 12:26:45.197322 29294 raft_consensus.cc:2749] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:45.283255 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 107648us
I20260709 12:26:45.291122 29272 raft_consensus.cc:493] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.291890 29272 raft_consensus.cc:515] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:45.324267 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.336534 29272 leader_election.cc:290] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:45.337224 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.350522 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d2c4b70e164856a205483966ca4963" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:45.360129 17988 leader_election.cc:304] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.364980 29272 raft_consensus.cc:2749] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:26:45.418548 17987 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 235544us
I20260709 12:26:45.447476 29406 raft_consensus.cc:493] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.448133 29406 raft_consensus.cc:515] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.450644 29406 leader_election.cc:290] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:45.451818 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.456104 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.457090 29406 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:45.490906 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c6e41c6a46d4fc99220183ed20b6921" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.494522 29406 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:26:45.494804 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.495592 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.495301 29406 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.499132 29406 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:45.501345 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.501791 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.508188 17987 leader_election.cc:304] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.509696 29272 raft_consensus.cc:2749] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:45.513792 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.514755 29272 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:45.515717 29272 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.516211 29272 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:45.534961 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.536916 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.539413 29272 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:45.539680 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:45.539712 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.540282 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c289d2c3e9fc40fbb7102b5788f74aab" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:45.542107 17989 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.543752 29272 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:45.545046 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.546713 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.548121 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.549005 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.630803 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.648998 29406 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.649664 29406 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.653957 29406 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:45.654033 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.654911 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.657606 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:45.659888 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.666844 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:45.672629 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.681229 29406 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:45.701388 29406 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.702371 29406 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.705844 29272 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.706569 29272 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:45.711577 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:45.709560 29272 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:45.726334 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.727586 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.729651 29406 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:45.736547 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df3d21a581ef40feafcce4ab8b817645" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.767089 17987 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.768244 17987 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:45.771221 29294 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:45.775528 29374 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:45.832469 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.863237 29272 raft_consensus.cc:493] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:45.864050 29272 raft_consensus.cc:515] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:45.869035 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.872625 29272 leader_election.cc:290] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:45.897982 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf56c3bba4a4641a72b648d02c669f6" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:45.931111 17987 leader_election.cc:304] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:45.932159 29272 raft_consensus.cc:2749] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:45.985605 29294 raft_consensus.cc:493] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.986341 29294 raft_consensus.cc:515] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:45.988894 29294 leader_election.cc:290] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:45.998857 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 301) took 9164 ms. Trace:
I20260709 12:26:45.999421 17915 rpcz_store.cc:276] 0709 12:26:36.834152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:43.653500 (+6819348us) service_pool.cc:224] Handling call
0709 12:26:45.998739 (+2345239us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:45.208422 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:45.209158 (+   736us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:45.209215 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:26:45.209239 (+    24us) tablet.cc:662] Decoding operations
0709 12:26:45.211799 (+  2560us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:45.211878 (+    79us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:45.211913 (+    35us) tablet.cc:685] Acquiring locks for 30 operations
0709 12:26:45.214508 (+  2595us) tablet.cc:701] Row locks acquired
0709 12:26:45.214532 (+    24us) write_op.cc:260] PREPARE: finished
0709 12:26:45.214715 (+   183us) write_op.cc:270] Start()
0709 12:26:45.214893 (+   178us) write_op.cc:276] Timestamp: P: 1783600005214655 usec, L: 0
0709 12:26:45.214915 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:26:45.215812 (+   897us) log.cc:844] Serialized 11707 byte log entry
0709 12:26:45.401595 (+185783us) op_driver.cc:464] REPLICATION: finished
0709 12:26:45.454321 (+ 52726us) write_op.cc:301] APPLY: starting
0709 12:26:45.454435 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:45.597631 (+143196us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:45.597804 (+   173us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:45.838029 (+240225us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:45.858770 (+ 20741us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=60,key_file_lookups=60,delta_file_lookups=0,mrs_lookups=0
0709 12:26:45.858839 (+    69us) write_op.cc:312] APPLY: finished
0709 12:26:45.859577 (+   738us) log.cc:844] Serialized 260 byte log entry
0709 12:26:45.859847 (+   270us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:45.860181 (+   334us) write_op.cc:454] Released schema lock
0709 12:26:45.860270 (+    89us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:45.902207 (+ 41937us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:45.902257 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":45480,"reported_tablets":49,"spinlock_wait_cycles":42240,"child_traces":[["op",{"apply.queue_time_us":52333,"apply.run_cpu_time_us":46376,"apply.run_wall_time_us":447817,"cfile_cache_hit":122,"cfile_cache_hit_bytes":129707,"num_ops":30,"prepare.queue_time_us":206487,"prepare.run_cpu_time_us":7896,"prepare.run_wall_time_us":18121,"replication_time_us":170737,"thread_start_us":210033,"threads_started":2}]]}
I20260709 12:26:46.001452 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:46.009967 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1236f612fda5419089c0c9aa244bb5bc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:46.026012 17989 leader_election.cc:304] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:46.028131 29272 raft_consensus.cc:2749] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:46.031759 29374 raft_consensus.cc:493] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.032389 29374 raft_consensus.cc:515] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:46.035472 29374 leader_election.cc:290] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:46.044993 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:45.989605 29294 raft_consensus.cc:493] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.046664 29294 raft_consensus.cc:515] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:46.049300 29294 leader_election.cc:290] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:46.055675 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:46.058674 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a83278d09c914df888a57be57eb08f8e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:46.066941 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07f730a06794e2cb530edca57871763" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:46.071735 17988 leader_election.cc:304] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:46.073809 29272 raft_consensus.cc:2749] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:46.095865 17987 leader_election.cc:304] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:46.097682 29294 raft_consensus.cc:2749] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:46.122428 17916 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 20 to 27. New cstate: current_term: 27 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:46.126299 17916 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 25 to 27, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 27 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.129395 17916 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 20 to 23, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 23 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.131349 17916 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 22 to 26, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:46.155140 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:46.159204 17916 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 24 to 26, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 26 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:46.162338 17916 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 26 to 31, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 31 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:46.171913 16792 thread.cc:527] Waited for 9000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:46.171353 17916 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 13 to 19, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 19 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:46.182369 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:46.200681 17916 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 23 to 25, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 25 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.203315 17916 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:46.248462 17922 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 279) took 164048 ms (client timeout 15000 ms). Trace:
W20260709 12:26:46.264707 17922 rpcz_store.cc:269] 0709 12:24:02.200275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.200582 (+   307us) service_pool.cc:224] Handling call
0709 12:26:46.248137 (+164047555us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8959,"reported_tablets":50}
W20260709 12:26:46.399874 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:46.400616 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:46.454579 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:46.461153 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:46.465263 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:46.526726 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:46.706868 17915 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 30 to 31, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 31 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:46.709141 17915 catalog_manager.cc:5697] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:46.712906 17915 catalog_manager.cc:5697] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 28, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 28 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.714584 17915 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.716049 17915 catalog_manager.cc:5697] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 23 to 29, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 29 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.717545 17915 catalog_manager.cc:5697] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 28, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 28 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.732060 17915 catalog_manager.cc:5697] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 26 to 29, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 29 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.764932 17915 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 30. New cstate: current_term: 30 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.766914 17915 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 34. New cstate: current_term: 34 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
W20260709 12:26:46.774309 17916 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 186) took 49482 ms (client timeout 15000 ms). Trace:
I20260709 12:26:46.775964 17915 catalog_manager.cc:5697] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 28. New cstate: current_term: 28 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:26:46.774868 17916 rpcz_store.cc:269] 0709 12:25:57.381354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.540046 (+8158692us) service_pool.cc:224] Handling call
0709 12:26:46.774188 (+41234142us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:46.274620 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:46.275446 (+   826us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:46.279616 (+  4170us) write_op.cc:435] Acquired schema lock
0709 12:26:46.279653 (+    37us) tablet.cc:662] Decoding operations
0709 12:26:46.280906 (+  1253us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:46.281014 (+   108us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:46.281050 (+    36us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:26:46.282002 (+   952us) tablet.cc:701] Row locks acquired
0709 12:26:46.282029 (+    27us) write_op.cc:260] PREPARE: finished
0709 12:26:46.282236 (+   207us) write_op.cc:270] Start()
0709 12:26:46.282421 (+   185us) write_op.cc:276] Timestamp: P: 1783600006282168 usec, L: 0
0709 12:26:46.282457 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 12:26:46.310472 (+ 28015us) log.cc:844] Serialized 3636 byte log entry
0709 12:26:46.312792 (+  2320us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16620571 00007f6c166b2e28 00007f6c1a0ed581 00007f6c1a0ecd58 00007f6c1a0f19f1 00007f6c1a0f19a9 00007f6c1a0f1939 00007f6c1a0f1901 00007f6c1a0f0b2d 0000000000388634 0000000000388479
0709 12:26:46.436485 (+123693us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5400040ca8. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c1ca739e1 00007f6c1ca7264c 00007f6c1a0ee3bb 00007f6c1a0f071a 00007f6c1a0f06a8 00007f6c1a0f0628 00007f6c1a0f05e3 00007f6c1a0ef7fc 00007f6c1c5222f6 00007f6c1c51caa8 00007f6c166c3310
0709 12:26:46.436617 (+   132us) op_driver.cc:464] REPLICATION: finished
0709 12:26:46.491449 (+ 54832us) write_op.cc:301] APPLY: starting
0709 12:26:46.491569 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:46.534875 (+ 43306us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:46.534907 (+    32us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:46.558726 (+ 23819us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:46.589884 (+ 31158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:26:46.589969 (+    85us) write_op.cc:312] APPLY: finished
0709 12:26:46.590359 (+   390us) log.cc:844] Serialized 89 byte log entry
0709 12:26:46.590586 (+   227us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:46.590748 (+   162us) write_op.cc:454] Released schema lock
0709 12:26:46.590809 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:46.600965 (+ 10156us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:46.601015 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":52477,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":54501,"apply.run_cpu_time_us":17212,"apply.run_wall_time_us":109462,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41801,"num_ops":9,"prepare.queue_time_us":40395,"prepare.run_cpu_time_us":7462,"prepare.run_wall_time_us":38430,"replication_time_us":140053,"spinlock_wait_cycles":62672128,"thread_start_us":93355,"threads_started":2}]]}
I20260709 12:26:46.800383 29294 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.791867 17915 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 28, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 28 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.801100 29294 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:46.805315 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:46.802351 17915 catalog_manager.cc:5697] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 33 to 35, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 35 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.806710 17915 catalog_manager.cc:5697] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 29 to 30, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.808204 17915 catalog_manager.cc:5697] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 18 to 30, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.809643 17915 catalog_manager.cc:5697] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 30 to 35, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 35 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.812022 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:46.815101 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:46.811326 17915 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.818142 17915 catalog_manager.cc:5697] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 19 to 25, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 25 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:46.817945 29294 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:46.819365 29272 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:46.820701 17915 catalog_manager.cc:5697] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 29 to 32, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 32 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.822624 17915 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 25 to 33, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 33 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.824254 17915 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 24 to 28, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 28 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.827464 17915 catalog_manager.cc:5697] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 30. New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.829166 17915 catalog_manager.cc:5697] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.830950 17915 catalog_manager.cc:5697] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 30 to 34, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 34 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:46.833073 17915 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 29, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 29 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } }
I20260709 12:26:46.834656 17915 catalog_manager.cc:5697] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 29 to 34. New cstate: current_term: 34 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:46.891480 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:46.919930 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:46.974471 18138 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 279 similar messages]
I20260709 12:26:46.997368 29272 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.998168 29272 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:47.000794 29272 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.007364 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.010680 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:47.017073 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.018651 29272 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:47.107088 29272 raft_consensus.cc:493] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:47.107765 29272 raft_consensus.cc:515] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:47.109074 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 51921.3us
I20260709 12:26:47.111397 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:47.112386 29272 leader_election.cc:290] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.112633 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "555e520f5efe4bafb5947749ee16e734" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.115414 17987 leader_election.cc:304] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.116490 29272 raft_consensus.cc:2749] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:47.164384 29272 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.165064 29272 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:47.168548 29272 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.169953 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.170876 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:47.172546 16792 thread.cc:527] Waited for 10000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:47.174356 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.186538 29374 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:47.220134 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:47.278184 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 96614.1us
W20260709 12:26:47.335148 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:47.371145 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 57371.9us
I20260709 12:26:47.406656 29294 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:47.407341 29294 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:47.410275 29294 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.411885 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.442149 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:47.458384 29374 raft_consensus.cc:493] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.485426 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.488400 29374 raft_consensus.cc:515] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:47.532498 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69553.6us
I20260709 12:26:47.517448 29374 leader_election.cc:290] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.547967 29374 raft_consensus.cc:493] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.548647 29374 raft_consensus.cc:515] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:47.573933 17915 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41324 (request call id 302) took 1439 ms. Trace:
I20260709 12:26:47.503007 29416 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:47.552387 29294 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.574461 17915 rpcz_store.cc:276] 0709 12:26:46.120365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.120572 (+   207us) service_pool.cc:224] Handling call
0709 12:26:47.559237 (+1438665us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:46.899035 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:46.899706 (+   671us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:46.899753 (+    47us) write_op.cc:435] Acquired schema lock
0709 12:26:46.899776 (+    23us) tablet.cc:662] Decoding operations
0709 12:26:46.901927 (+  2151us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:46.902042 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:46.902079 (+    37us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:26:46.934771 (+ 32692us) tablet.cc:701] Row locks acquired
0709 12:26:46.934796 (+    25us) write_op.cc:260] PREPARE: finished
0709 12:26:46.935426 (+   630us) write_op.cc:270] Start()
0709 12:26:46.935591 (+   165us) write_op.cc:276] Timestamp: P: 1783600006935352 usec, L: 0
0709 12:26:46.935619 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:26:46.936591 (+   972us) log.cc:844] Serialized 9834 byte log entry
0709 12:26:46.943138 (+  6547us) op_driver.cc:464] REPLICATION: finished
0709 12:26:46.943683 (+   545us) write_op.cc:301] APPLY: starting
0709 12:26:46.943787 (+   104us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:47.224179 (+280392us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:47.224321 (+   142us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:47.509939 (+285618us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:47.510297 (+   358us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0709 12:26:47.510357 (+    60us) write_op.cc:312] APPLY: finished
0709 12:26:47.510868 (+   511us) log.cc:844] Serialized 220 byte log entry
0709 12:26:47.511802 (+   934us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:47.512067 (+   265us) write_op.cc:454] Released schema lock
0709 12:26:47.512143 (+    76us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":39059,"apply.run_wall_time_us":609034,"cfile_cache_hit":100,"cfile_cache_hit_bytes":104650,"num_ops":25,"prepare.queue_time_us":53701,"prepare.run_cpu_time_us":7421,"prepare.run_wall_time_us":41730,"replication_time_us":7303,"thread_start_us":25795,"threads_started":1}]]}
I20260709 12:26:47.577610 29374 leader_election.cc:290] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.595571 29294 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:47.615788 29294 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:47.733615 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:47.741539 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:47.747043 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:47.759850 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bfc0ef6acd420b829a9a1df3376f48" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.768747 18172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe4c9950f4d4f4588d08c7cd436b1dc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.771837 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:47.745540 17914 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 21 to 26, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:47.790776 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:47.794895 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 98099us
I20260709 12:26:47.817288 17914 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 27 to 28, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 28 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:47.832846 17914 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 28 to 34, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 34 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:47.846427 17989 leader_election.cc:304] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.849478 17987 leader_election.cc:304] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.851612 29272 raft_consensus.cc:2749] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:47.863658 29294 raft_consensus.cc:2749] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:47.881232 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 69785.1us
I20260709 12:26:47.900030 29422 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.900864 29422 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:47.903680 29422 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:47.906139 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:47.966514 17989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 116833us
I20260709 12:26:47.967549 17989 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:47.969126 29272 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:47.992555 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 126988us
I20260709 12:26:48.023761 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da76d5c707544d29a2f74d6015ec1cfc" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:48.032886 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.033527 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.040721 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.042541 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.043737 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.045356 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.046835 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:48.051120 29423 raft_consensus.cc:493] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.051807 29423 raft_consensus.cc:515] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.054273 29423 leader_election.cc:290] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.057097 17987 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:48.083062 29423 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:48.090553 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.093611 29423 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.094410 29423 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.099391 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0dc8b13a3294835a091ca099ee49b29" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:48.103329 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:48.103672 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.113579 17987 leader_election.cc:304] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.115489 29423 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.118523 29272 raft_consensus.cc:2749] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:48.118795 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ba0c897fcb346488dfe124a973cb2d1" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:48.137998 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:48.146175 17989 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.147231 29441 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:48.156567 29441 raft_consensus.cc:493] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.157223 29441 raft_consensus.cc:515] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.162523 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.165403 29441 leader_election.cc:290] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.170059 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47bac54b53844de9814420f86103a6f8" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:48.173147 16792 thread.cc:527] Waited for 11000ms trying to join with heartbeat (tid 18049)
I20260709 12:26:48.179070 17989 leader_election.cc:304] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.180761 29423 raft_consensus.cc:2749] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:48.189402 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.191942 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.198774 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.217224 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.247542 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:48.271643 29441 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.272379 29441 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:48.276947 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:48.280601 29423 raft_consensus.cc:493] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.281168 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.281931 29441 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:48.282481 29423 raft_consensus.cc:515] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:48.283408 17987 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:48.288571 29374 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:48.290519 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:48.294898 29423 leader_election.cc:290] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:48.300680 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.320168 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.338697 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.357622 18049 heartbeater.cc:646] Failed to heartbeat to 127.16.102.62:42293 (7 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.102.62:42293 timed out after 15.000s (SENT)
I20260709 12:26:48.360517 29374 raft_consensus.cc:493] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.366844 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4843c80e9d9348a397ce35e196deb9bd" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.371421 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9731c63bc7f406db2c77f5ef0415a78" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.365334 29374 raft_consensus.cc:515] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.378751 29374 leader_election.cc:290] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.379546 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.386005 17989 leader_election.cc:304] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.388971 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52abd819c1c443e28c6c41a392a7ed6f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.394495 17989 leader_election.cc:304] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.393877 17913 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 34 to 37. New cstate: current_term: 37 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.406548 29272 raft_consensus.cc:2749] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:48.403045 17913 catalog_manager.cc:5697] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:48.396193 29423 raft_consensus.cc:2749] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:48.412645 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.418447 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:48.421077 17913 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 31 to 34, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 34 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.426942 17913 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 26 to 30, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1). New cstate: current_term: 30 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:48.401446 17914 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 288) took 34330 ms (client timeout 15000 ms). Trace:
W20260709 12:26:48.430094 17914 rpcz_store.cc:269] 0709 12:26:14.070597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.466012 (+10395415us) service_pool.cc:224] Handling call
0709 12:26:48.401293 (+23935281us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:47.974687 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:47.975325 (+   638us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:47.975367 (+    42us) write_op.cc:435] Acquired schema lock
0709 12:26:47.975386 (+    19us) tablet.cc:662] Decoding operations
0709 12:26:47.976060 (+   674us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:47.976150 (+    90us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:47.976181 (+    31us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:26:47.976558 (+   377us) tablet.cc:701] Row locks acquired
0709 12:26:47.976577 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:26:47.976785 (+   208us) write_op.cc:270] Start()
0709 12:26:47.987430 (+ 10645us) write_op.cc:276] Timestamp: P: 1783600007976712 usec, L: 0
0709 12:26:47.987466 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 12:26:47.988336 (+   870us) log.cc:844] Serialized 1293 byte log entry
0709 12:26:48.157520 (+169184us) op_driver.cc:464] REPLICATION: finished
0709 12:26:48.181065 (+ 23545us) write_op.cc:301] APPLY: starting
0709 12:26:48.181171 (+   106us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:48.186403 (+  5232us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:48.193630 (+  7227us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:48.195965 (+  2335us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:48.196210 (+   245us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 12:26:48.196258 (+    48us) write_op.cc:312] APPLY: finished
0709 12:26:48.196540 (+   282us) log.cc:844] Serialized 41 byte log entry
0709 12:26:48.196734 (+   194us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:48.196854 (+   120us) write_op.cc:454] Released schema lock
0709 12:26:48.196903 (+    49us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:48.197377 (+   474us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:48.197431 (+    54us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":23236,"apply.run_cpu_time_us":6768,"apply.run_wall_time_us":16295,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16685,"num_ops":3,"prepare.queue_time_us":117178,"prepare.run_cpu_time_us":3696,"prepare.run_wall_time_us":19398,"replication_time_us":169837,"thread_start_us":134097,"threads_started":2}]]}
I20260709 12:26:48.455899 29423 raft_consensus.cc:493] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:48.456689 29423 raft_consensus.cc:515] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:48.457652 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:48.465317 29423 leader_election.cc:290] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.473258 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.461097 17913 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d reported cstate change: term changed from 23 to 26. New cstate: current_term: 26 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:48.484850 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d440327a63754b889f7edd91e1373942" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.489408 17987 leader_election.cc:304] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.496420 29441 raft_consensus.cc:2749] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:48.533241 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:48.536765 16792 tablet_replica.cc:333] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.541339 16792 raft_consensus.cc:2243] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.542524 16792 raft_consensus.cc:2272] T 5f144ec392b043d8b68e284ffecdee43 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.557281 16792 tablet_replica.cc:333] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
W20260709 12:26:48.568573 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:48.591825 16792 raft_consensus.cc:2243] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:48.597640 16792 raft_consensus.cc:2272] T 23e4eb0b942f4c7e81dfe76f4e015618 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.615760 16792 tablet_replica.cc:333] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
W20260709 12:26:48.617452 17913 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 285) took 15208 ms (client timeout 15000 ms). Trace:
W20260709 12:26:48.632725 17913 rpcz_store.cc:269] 0709 12:26:33.438360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.746083 (+6307723us) service_pool.cc:224] Handling call
0709 12:26:48.617351 (+8871268us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:48.492803 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:48.493409 (+   606us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:48.493458 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:26:48.493482 (+    24us) tablet.cc:662] Decoding operations
0709 12:26:48.494224 (+   742us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:48.494288 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:48.494314 (+    26us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:26:48.494912 (+   598us) tablet.cc:701] Row locks acquired
0709 12:26:48.494927 (+    15us) write_op.cc:260] PREPARE: finished
0709 12:26:48.495122 (+   195us) write_op.cc:270] Start()
0709 12:26:48.495273 (+   151us) write_op.cc:276] Timestamp: P: 1783600008495056 usec, L: 0
0709 12:26:48.495295 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:26:48.496167 (+   872us) log.cc:844] Serialized 2054 byte log entry
0709 12:26:48.499283 (+  3116us) op_driver.cc:464] REPLICATION: finished
0709 12:26:48.499800 (+   517us) write_op.cc:301] APPLY: starting
0709 12:26:48.499901 (+   101us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:48.544045 (+ 44144us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:48.544147 (+   102us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:48.547801 (+  3654us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:48.548017 (+   216us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 12:26:48.548063 (+    46us) write_op.cc:312] APPLY: finished
0709 12:26:48.548325 (+   262us) log.cc:844] Serialized 57 byte log entry
0709 12:26:48.554540 (+  6215us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:48.554694 (+   154us) write_op.cc:454] Released schema lock
0709 12:26:48.554757 (+    63us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:48.567508 (+ 12751us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:48.567549 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13739,"reported_tablets":50,"spinlock_wait_cycles":3456,"child_traces":[["op",{"apply.queue_time_us":272,"apply.run_cpu_time_us":9197,"apply.run_wall_time_us":67654,"cfile_cache_hit":20,"cfile_cache_hit_bytes":20930,"num_ops":5,"prepare.queue_time_us":5396,"prepare.run_cpu_time_us":3804,"prepare.run_wall_time_us":7989,"replication_time_us":3795}]]}
I20260709 12:26:48.640326 16792 raft_consensus.cc:2243] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:48.642017 16792 raft_consensus.cc:2272] T 583d61c844f048e4b53a66acc88cc1d9 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.651897 16792 tablet_replica.cc:333] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.671085 16792 raft_consensus.cc:2243] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:48.672710 16792 raft_consensus.cc:2272] T 0420fe6ea0ab490e922b7c35c62b6a19 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.676625 16792 tablet_replica.cc:333] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.678233 16792 raft_consensus.cc:2243] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.686487 16792 raft_consensus.cc:2272] T 8e0ff0e5252f4ace898a73a2268e5dc8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.691174 29272 raft_consensus.cc:493] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.694474 29272 raft_consensus.cc:515] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } attrs { replace: true } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:48.698045 16792 tablet_replica.cc:333] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.698601 29272 leader_election.cc:290] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:48.700026 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.707844 17919 catalog_manager.cc:5697] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 28 to 31. New cstate: current_term: 31 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:48.699999 16792 raft_consensus.cc:2243] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.732298 16792 raft_consensus.cc:2272] T ef30e67b67894670b0cf3980d6b0f624 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.732919 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.746438 16792 tablet_replica.cc:333] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.717479 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85764dcebf974f7cbec322be36e3dd5f" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:48.747038 17919 catalog_manager.cc:5697] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 27 to 29, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 29 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.749305 17988 leader_election.cc:304] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:48.750437 29441 raft_consensus.cc:2749] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:48.751499 16792 raft_consensus.cc:2243] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.752381 16792 raft_consensus.cc:2272] T 86bfc0ef6acd420b829a9a1df3376f48 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.764901 16792 tablet_replica.cc:333] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.769690 29441 raft_consensus.cc:493] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.770435 29441 raft_consensus.cc:515] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:48.779392 16792 raft_consensus.cc:2243] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.780478 16792 raft_consensus.cc:2272] T 49a68a064bb44579af6ee5ad50121ffc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.794070 16792 tablet_replica.cc:333] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.795373 16792 raft_consensus.cc:2243] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.795994 29272 raft_consensus.cc:493] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.796582 29272 raft_consensus.cc:515] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.798895 29272 leader_election.cc:290] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.801653 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:48.804545 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:48.809976 17919 catalog_manager.cc:5697] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 26 to 27, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 27 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.813758 17919 catalog_manager.cc:5697] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 19 to 26. New cstate: current_term: 26 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.816845 17919 catalog_manager.cc:5697] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 26 to 30, leader changed from ae9641d5c3184572a102d51a75f670ee (127.16.102.3) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 30 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.820206 17919 catalog_manager.cc:5697] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 34 to 35, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 35 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:48.836596 29272 raft_consensus.cc:493] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.838528 29272 raft_consensus.cc:515] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.846164 29272 leader_election.cc:290] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.846609 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.852033 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.860246 29272 raft_consensus.cc:493] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.863279 29272 raft_consensus.cc:515] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:48.862224 16792 raft_consensus.cc:2272] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:48.864003 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53b6c1b05fff4ff4ab4260c2c231ed5e" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:48.862643 29441 leader_election.cc:290] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:48.873015 18097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:48.876535 16792 tablet_replica.cc:333] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:48.877763 16792 raft_consensus.cc:2243] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:48.882946 29272 leader_election.cc:290] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:48.896286 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d525bcd825eb49d9be935bc89de569c7" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.898460 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804b0137cb0b4850b912b6519bae7417" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.901712 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeba5cb4432947efbaa2fa537357e992" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:48.907232 17989 leader_election.cc:304] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
W20260709 12:26:48.908095 17989 raft_consensus.cc:2686] T 53b6c1b05fff4ff4ab4260c2c231ed5e P cab4a2f5d3ca48a2a803ee8ec3906e1d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:48.909953 17989 leader_election.cc:304] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.911875 29441 raft_consensus.cc:2749] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:48.911899 17989 leader_election.cc:304] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.915999 29423 raft_consensus.cc:2749] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:48.919123 17989 leader_election.cc:304] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:48.929865 29441 raft_consensus.cc:2749] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:48.955431 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:48.969964 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:48.985625 16792 raft_consensus.cc:2272] T df3d21a581ef40feafcce4ab8b817645 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:48.989971 18137 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 54857.8us
W20260709 12:26:48.996377 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:48.997592 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:49.046412 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 48878.2us
I20260709 12:26:48.989568 16792 tablet_replica.cc:333] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.050962 16792 raft_consensus.cc:2243] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:26:49.052520 16792 raft_consensus.cc:2272] T 3304247386d14b18a8bdd4f25ae8630c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:49.030923 17920 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41312 (request call id 284) took 40664 ms (client timeout 15000 ms). Trace:
W20260709 12:26:49.063172 17920 rpcz_store.cc:269] 0709 12:26:08.366285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.291602 (+10925317us) service_pool.cc:224] Handling call
0709 12:26:19.557942 (+266340us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b5400002490. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c13452db1 00007f6c13592b71 00007f6c135930b2 00007f6c1c50c759 00007f6c1c4a4a76 00007f6c1c475d58 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc
0709 12:26:49.030681 (+29472739us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30876,"reported_tablets":50,"spinlock_wait_cycles":123214208}
W20260709 12:26:49.125175 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:49.077776 16792 tablet_replica.cc:333] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.136518 16792 raft_consensus.cc:2243] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:49.139765 16792 raft_consensus.cc:2272] T fd8fe387b2164c30aa94afed5a7c218d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:49.159698 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:49.169780 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:49.181023 17919 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 188) took 8968 ms. Trace:
I20260709 12:26:49.203135 17919 rpcz_store.cc:276] 0709 12:26:40.230946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:44.641260 (+4410314us) service_pool.cc:224] Handling call
0709 12:26:49.180882 (+4539622us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:48.979578 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:48.980170 (+   592us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:48.980222 (+    52us) write_op.cc:435] Acquired schema lock
0709 12:26:48.980241 (+    19us) tablet.cc:662] Decoding operations
0709 12:26:48.981030 (+   789us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:48.981095 (+    65us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:48.981121 (+    26us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:26:48.981670 (+   549us) tablet.cc:701] Row locks acquired
0709 12:26:48.981689 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:26:48.981911 (+   222us) write_op.cc:270] Start()
0709 12:26:48.982042 (+   131us) write_op.cc:276] Timestamp: P: 1783600008981857 usec, L: 0
0709 12:26:48.982063 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:26:48.982883 (+   820us) log.cc:844] Serialized 2430 byte log entry
0709 12:26:48.986512 (+  3629us) op_driver.cc:464] REPLICATION: finished
0709 12:26:48.986935 (+   423us) write_op.cc:301] APPLY: starting
0709 12:26:48.987037 (+   102us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:48.991143 (+  4106us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:48.991267 (+   124us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:49.001437 (+ 10170us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:49.001705 (+   268us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=0,mrs_lookups=0
0709 12:26:49.001765 (+    60us) write_op.cc:312] APPLY: finished
0709 12:26:49.002141 (+   376us) log.cc:844] Serialized 65 byte log entry
0709 12:26:49.015109 (+ 12968us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:49.015277 (+   168us) write_op.cc:454] Released schema lock
0709 12:26:49.015333 (+    56us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:49.050176 (+ 34843us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:49.050226 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":212279,"reported_tablets":31,"spinlock_wait_cycles":2816,"child_traces":[["op",{"apply.queue_time_us":118,"apply.run_cpu_time_us":10074,"apply.run_wall_time_us":63158,"cfile_cache_hit":24,"cfile_cache_hit_bytes":25116,"num_ops":6,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":3632,"prepare.run_wall_time_us":5712,"replication_time_us":4293}]]}
W20260709 12:26:49.229481 17988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 49007.3us
I20260709 12:26:49.243980 16792 tablet_replica.cc:333] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.266486 16792 raft_consensus.cc:2243] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 LEADER]: Raft consensus shutting down.
I20260709 12:26:49.268221 16792 raft_consensus.cc:2272] T acbc4a634f954522904f2b865531f456 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:49.282079 17917 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38984 (request call id 187) took 29394 ms (client timeout 15000 ms). Trace:
W20260709 12:26:49.296892 17917 rpcz_store.cc:269] 0709 12:26:19.887500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.849069 (+11961569us) service_pool.cc:224] Handling call
0709 12:26:33.047598 (+1198529us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400041354. stack: 00007f6c13598343 00007f6c13598240 00007f6c12f271d4 00007f6c1cc65304 00007f6c1cc65299 00007f6c16621090 00007f6c166bff3a 00007f6c1c475e9c 00007f6c1c57404e 00007f6c1aa352a4 00007f6c1aa35232 00007f6c1aa35161 00007f6c1aa350dc 00007f6c1aa34392 00007f6c149f248c 00007f6c149f18c6
0709 12:26:49.281780 (+16234182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33368,"reported_tablets":31,"spinlock_wait_cycles":978313088}
W20260709 12:26:49.308535 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:49.370551 16792 tablet_replica.cc:333] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.377540 16792 raft_consensus.cc:2243] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:49.380084 16792 pending_rounds.cc:70] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Trying to abort 1 pending ops.
I20260709 12:26:49.380712 16792 pending_rounds.cc:77] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting op as it isn't in flight: id { term: 29 index: 15 } timestamp: 7305625566751076352 op_type: NO_OP noop_request { }
I20260709 12:26:49.381914 16792 raft_consensus.cc:2889] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:49.382735 16792 raft_consensus.cc:2272] T f0dc8b13a3294835a091ca099ee49b29 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.389137 16792 tablet_replica.cc:333] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.391149 16792 raft_consensus.cc:2243] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:26:49.395670 16792 raft_consensus.cc:2272] T c53769058ca3443f805983b61a69328e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.402202 16792 tablet_replica.cc:333] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.513695 16792 raft_consensus.cc:2243] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:49.523334 16792 raft_consensus.cc:2272] T da76d5c707544d29a2f74d6015ec1cfc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.596458 29455 raft_consensus.cc:493] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:49.597939 29455 raft_consensus.cc:515] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:49.613049 16792 tablet_replica.cc:333] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
W20260709 12:26:49.623862 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 55485.9us
W20260709 12:26:49.630676 17918 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41336 (request call id 289) took 11924 ms (client timeout 15000 ms). Trace:
I20260709 12:26:49.631835 16792 raft_consensus.cc:2243] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:49.631443 17918 rpcz_store.cc:269] 0709 12:26:37.705872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:44.084795 (+6378923us) service_pool.cc:224] Handling call
0709 12:26:49.630362 (+5545567us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21163,"reported_tablets":50,"spinlock_wait_cycles":32512}
W20260709 12:26:49.642167 29441 raft_consensus.cc:670] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:26:49.644682 17917 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:49.663307 29448 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:49.666742 29448 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:49.670423 16792 raft_consensus.cc:2272] T 5c6e41c6a46d4fc99220183ed20b6921 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.676007 17917 catalog_manager.cc:5697] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 28 to 32, leader changed from 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 32 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:49.671171 29448 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:49.683300 17917 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 37 to 39. New cstate: current_term: 39 leader_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
I20260709 12:26:49.695714 16792 tablet_replica.cc:333] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.677583 29455 leader_election.cc:290] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:49.755461 16792 raft_consensus.cc:2243] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:49.756830 16792 raft_consensus.cc:2272] T aeba5cb4432947efbaa2fa537357e992 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.787972 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:49.806574 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:49.823135 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:49.833485 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:49.834025 16792 tablet_replica.cc:333] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.837661 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578328e4e1fd478cbc58dd1048417480" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:49.848016 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 75114.3us
I20260709 12:26:49.858421 16792 raft_consensus.cc:2243] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:49.859673 16792 raft_consensus.cc:2272] T 9fe4c9950f4d4f4588d08c7cd436b1dc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.863039 29433 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:49.863884 29433 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:49.866734 29433 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:49.878005 29421 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:49.878974 29421 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:49.881536 29421 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:49.876613 18064 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:49.892733 17987 leader_election.cc:304] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:26:49.894068 29455 raft_consensus.cc:2749] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.899719 16792 tablet_replica.cc:333] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.901675 16792 raft_consensus.cc:2243] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:49.919392 29451 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:49.920542 29451 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:49.924006 29451 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:49.932341 16792 raft_consensus.cc:2272] T 2217b6329873485ead60419dc5acbff0 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.952975 16792 tablet_replica.cc:333] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:49.961843 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:49.973631 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:49.979964 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:49.986825 16792 raft_consensus.cc:2243] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:49.988363 16792 raft_consensus.cc:2272] T a07f730a06794e2cb530edca57871763 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:49.989064 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:49.989949 18169 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 34.
W20260709 12:26:49.998003 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:49.998730 18138 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.001019 29421 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:50.010488 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.012480 29446 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:50.013020 29446 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.013406 29446 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:50.023284 16792 tablet_replica.cc:333] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.024909 16792 raft_consensus.cc:2243] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.025982 16792 pending_rounds.cc:70] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Trying to abort 1 pending ops.
I20260709 12:26:50.026302 16792 pending_rounds.cc:77] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting op as it isn't in flight: id { term: 30 index: 15 } timestamp: 7305625535255101440 op_type: NO_OP noop_request { }
I20260709 12:26:50.026786 16792 raft_consensus.cc:2889] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:50.027273 16792 raft_consensus.cc:2272] T 555e520f5efe4bafb5947749ee16e734 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:50.031224 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.048914 16792 tablet_replica.cc:333] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.052502 16792 raft_consensus.cc:2243] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 LEADER]: Raft consensus shutting down.
W20260709 12:26:50.052778 18780 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.062763 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.058081 16792 raft_consensus.cc:2272] T b5eedbd3c223402caf9a71dac2fcaa98 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.063453 29433 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.064605 29433 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.064178 29454 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:50.075376 16792 tablet_replica.cc:333] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.076758 16792 raft_consensus.cc:2243] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:26:50.076156 29433 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:50.078902 18064 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.080960 16792 raft_consensus.cc:2272] T af9dff14df364074b9d5e7e769f198b5 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.086393 16792 tablet_replica.cc:333] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.091662 16792 raft_consensus.cc:2243] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.094501 16792 raft_consensus.cc:2272] T 5c743e221c2c477e9b998557d9b9749a P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.089603 29446 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.110488 29446 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:50.107146 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:50.120388 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.112244 18169 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:50.131973 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "583d61c844f048e4b53a66acc88cc1d9" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:50.132932 18169 raft_consensus.cc:3060] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:26:50.133024 18064 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.134807 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.139231 29417 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:50.139875 29417 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.140319 29417 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:50.154363 29447 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.155458 29447 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.158788 29447 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
W20260709 12:26:50.166167 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52789.7us
I20260709 12:26:50.183957 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:50.185037 18809 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 32.
I20260709 12:26:50.206620 29433 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.207444 29433 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:50.215319 18138 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.216295 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.195232 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:50.224375 18169 raft_consensus.cc:2468] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 35.
I20260709 12:26:50.226404 18064 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.227470 29421 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:50.228084 29421 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.228574 29421 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:50.222313 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:50.237053 29417 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.216989 29433 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:50.241321 29447 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:50.269042 18064 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.270573 18064 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.273321 29433 raft_consensus.cc:2749] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:50.267886 29446 raft_consensus.cc:2804] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:26:50.286600 29446 raft_consensus.cc:697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 35 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:50.287977 29446 consensus_queue.cc:237] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.271716 29432 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.293149 29432 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.295866 29432 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:50.300803 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:50.319860 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52257.7us
I20260709 12:26:50.325489 16792 tablet_replica.cc:333] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.335480 16792 raft_consensus.cc:2243] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:50.337003 29417 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:50.350003 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23e4eb0b942f4c7e81dfe76f4e015618" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:50.350976 18165 raft_consensus.cc:3060] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:50.356374 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.358281 29421 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:50.367563 18064 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.369014 16792 raft_consensus.cc:2272] T 7c3aa170318f4e1f95796426dc71d60b P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.398183 18165 raft_consensus.cc:2468] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 29.
I20260709 12:26:50.400038 16792 tablet_replica.cc:333] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.403216 16792 raft_consensus.cc:2243] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 LEADER]: Raft consensus shutting down.
W20260709 12:26:50.389087 18138 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.408800 16792 raft_consensus.cc:2272] T a6ac73d7138d4200bbfcbf2f84c2c775 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.414011 16792 tablet_replica.cc:333] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.415189 16792 raft_consensus.cc:2243] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.416219 16792 raft_consensus.cc:2272] T d440327a63754b889f7edd91e1373942 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:50.418085 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.424002 18064 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.425086 29446 raft_consensus.cc:2804] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:50.425755 29446 raft_consensus.cc:697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:50.427198 29446 consensus_queue.cc:237] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.437060 29421 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:50.438613 29454 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.439453 29454 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.443444 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:50.445758 29454 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:50.460441 16792 tablet_replica.cc:333] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.461760 17919 catalog_manager.cc:5697] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 34 to 35, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 35 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:50.465673 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:50.467622 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:50.468427 18138 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.471581 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0420fe6ea0ab490e922b7c35c62b6a19" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:50.472190 18138 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.471452 29446 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.473414 29446 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.475956 29432 raft_consensus.cc:2749] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:26:50.461794 16792 raft_consensus.cc:2243] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.478219 16792 raft_consensus.cc:2272] T 9ba0c897fcb346488dfe124a973cb2d1 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.472438 18809 raft_consensus.cc:3060] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:50.479856 29441 raft_consensus.cc:493] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:50.480531 29441 raft_consensus.cc:515] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.483714 29441 leader_election.cc:290] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:50.486901 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:50.502622 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3c91f3d9da3442f89360b0c316ee411" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:50.505507 16792 tablet_replica.cc:333] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.507431 16792 raft_consensus.cc:2243] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:50.507527 17989 leader_election.cc:304] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:50.509096 16792 raft_consensus.cc:2272] T 4cc94b1c76be48d6813e7d1c89a4d66c P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:50.511193 18064 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.512176 29441 raft_consensus.cc:2749] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:50.518151 18138 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.522866 16792 tablet_replica.cc:333] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.524132 16792 raft_consensus.cc:2243] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.517326 18809 raft_consensus.cc:2468] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 33.
I20260709 12:26:50.550349 29446 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:50.551679 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:50.552567 29454 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.553241 29454 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.553546 18063 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.556344 29446 raft_consensus.cc:2749] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:50.556435 29454 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:50.557597 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.562599 17919 catalog_manager.cc:5697] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 28 to 29, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 29 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:50.576835 16792 raft_consensus.cc:2272] T 1236f612fda5419089c0c9aa244bb5bc P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.584393 18137 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.579697 29421 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.585470 29421 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.588158 29421 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:50.597244 29419 raft_consensus.cc:2804] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:50.597886 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
W20260709 12:26:50.602487 18780 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.602696 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 39.
I20260709 12:26:50.603088 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.607012 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:50.607822 18090 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
I20260709 12:26:50.615550 29451 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:50.617792 16792 tablet_replica.cc:333] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.634850 16792 raft_consensus.cc:2243] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.623215 29419 raft_consensus.cc:697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:50.626511 29441 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:50.621263 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:50.643273 29441 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.643321 29419 consensus_queue.cc:237] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.650846 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:50.659102 29441 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:50.659574 29465 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:50.660087 29465 raft_consensus.cc:493] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.660488 29465 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:50.662618 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aef65d70106f4f458174493864b4ba3d" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:50.665383 17988 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:50.667956 29441 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:50.689373 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:50.691483 29463 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.692337 29463 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.694074 29451 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:26:50.694613 29451 raft_consensus.cc:493] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.695006 29451 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:26:50.697391 29463 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:50.698625 18811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:50.699685 18811 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 27.
I20260709 12:26:50.702883 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:50.704924 29446 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:50.705493 29446 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.706147 29446 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:50.706359 18064 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.708141 16792 raft_consensus.cc:2272] T 14d2c4b70e164856a205483966ca4963 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:50.686708 18138 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.723333 29446 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.730242 29441 raft_consensus.cc:493] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:50.731067 29441 raft_consensus.cc:515] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:50.734176 18064 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.735066 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:50.735343 29446 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:50.736701 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c53769058ca3443f805983b61a69328e" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
I20260709 12:26:50.737478 18809 raft_consensus.cc:3060] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:50.743214 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff26d2026f1c4ff79de028b875b8ba96" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:50.745905 29441 leader_election.cc:290] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:50.776693 17987 leader_election.cc:304] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:50.783588 29465 raft_consensus.cc:515] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:50.784787 16792 tablet_replica.cc:333] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.777766 29455 raft_consensus.cc:2749] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:50.824643 16792 raft_consensus.cc:2243] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.831007 29465 leader_election.cc:290] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:50.834988 29451 raft_consensus.cc:515] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:50.839192 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:50.839781 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 72420.9us
I20260709 12:26:50.840780 17919 catalog_manager.cc:5697] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 32 to 33, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 33 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:50.849087 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd8fe387b2164c30aa94afed5a7c218d" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:50.849967 18095 raft_consensus.cc:3060] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:50.850453 16792 raft_consensus.cc:2272] T 7cd93736a3314dbbb5c62b5af8fbed11 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.852135 18809 raft_consensus.cc:2468] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 28.
I20260709 12:26:50.856992 18063 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.858946 16792 tablet_replica.cc:333] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.860132 16792 raft_consensus.cc:2243] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.861099 16792 raft_consensus.cc:2272] T c289d2c3e9fc40fbb7102b5788f74aab P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:50.864665 18780 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:50.867700 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.837595 29451 leader_election.cc:290] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 40 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:50.885805 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.888500 29433 raft_consensus.cc:2804] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:50.892015 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:50.895380 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.892403 29433 raft_consensus.cc:697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:50.890856 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3304247386d14b18a8bdd4f25ae8630c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:50.898229 18165 raft_consensus.cc:3060] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:26:50.900573 29451 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.901439 29451 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.897339 29433 consensus_queue.cc:237] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.904280 29451 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:50.908927 18138 leader_election.cc:336] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.890983 18095 raft_consensus.cc:2468] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:50.917259 18166 raft_consensus.cc:1275] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:26:50.925899 18165 raft_consensus.cc:2468] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 40.
I20260709 12:26:50.931564 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:50.938300 18780 leader_election.cc:336] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.959568 18099 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 24.
W20260709 12:26:50.960054 18780 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:50.945072 16792 tablet_replica.cc:333] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.963863 16792 raft_consensus.cc:2243] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:50.965010 16792 raft_consensus.cc:2272] T 578328e4e1fd478cbc58dd1048417480 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:50.973248 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.974438 29451 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:50.974931 29451 raft_consensus.cc:493] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:50.975355 29451 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:50.980414 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:50.981283 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.983647 18780 leader_election.cc:304] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.951722 18136 leader_election.cc:304] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:50.951915 17919 catalog_manager.cc:5697] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 27 to 28, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 28 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:50.985378 16792 tablet_replica.cc:333] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:50.991098 29421 raft_consensus.cc:2804] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:50.992800 29470 raft_consensus.cc:2804] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:26:50.993896 29470 raft_consensus.cc:697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 40 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:50.995139 29470 consensus_queue.cc:237] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:50.997396 29433 consensus_queue.cc:1048] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:26:51.006534 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:51.008097 16792 raft_consensus.cc:2243] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.009086 16792 raft_consensus.cc:2272] T 804b0137cb0b4850b912b6519bae7417 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.010921 29421 raft_consensus.cc:697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:51.012157 29421 consensus_queue.cc:237] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:51.024408 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:51.037756 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.043569 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.058084 29451 raft_consensus.cc:515] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.060637 29451 leader_election.cc:290] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.069478 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acbc4a634f954522904f2b865531f456" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:51.070245 18095 raft_consensus.cc:3060] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:51.074080 17918 catalog_manager.cc:5697] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 39 to 40, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 40 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:51.069274 17919 catalog_manager.cc:5697] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 29 to 30, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:51.086406 29455 raft_consensus.cc:493] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:26:51.087285 29455 raft_consensus.cc:515] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:51.088681 18780 leader_election.cc:336] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.092128 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:51.095538 29455 leader_election.cc:290] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:51.096515 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3412b19684e44e3d89ca8150a50a9d14" candidate_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.098518 17987 leader_election.cc:304] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab4a2f5d3ca48a2a803ee8ec3906e1d; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:26:51.099819 18166 raft_consensus.cc:1275] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:26:51.099947 29455 raft_consensus.cc:2749] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:51.106379 29433 consensus_queue.cc:1048] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:51.107569 16792 tablet_replica.cc:333] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.108644 16792 raft_consensus.cc:2243] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.109601 16792 raft_consensus.cc:2272] T 47bac54b53844de9814420f86103a6f8 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.130853 29448 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.131536 29448 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.135073 29448 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.139736 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.143787 18095 raft_consensus.cc:2468] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 25.
I20260709 12:26:51.145082 18778 leader_election.cc:304] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:51.145501 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:51.146564 29470 raft_consensus.cc:2804] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:51.147418 29470 raft_consensus.cc:697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:51.148437 29470 consensus_queue.cc:237] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:51.156468 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.157358 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:51.165310 18809 raft_consensus.cc:1275] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:26:51.171669 29446 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.172755 29446 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.171100 29432 consensus_queue.cc:1048] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.003s
I20260709 12:26:51.175757 29446 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:51.180341 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.198275 18138 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.198853 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:51.197499 18064 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.207612 17918 catalog_manager.cc:5697] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 24 to 25, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 25 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:51.210376 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:51.202306 29465 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:51.211972 18164 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 33.
I20260709 12:26:51.213985 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.214957 29463 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:51.216244 29463 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.216676 29463 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:51.214269 16792 tablet_replica.cc:333] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.223887 16792 raft_consensus.cc:2243] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.239455 29463 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:51.253168 18064 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.270675 16792 raft_consensus.cc:2272] T d525bcd825eb49d9be935bc89de569c7 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 27 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.254695 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:51.291711 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2217b6329873485ead60419dc5acbff0" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:51.292559 18164 raft_consensus.cc:3060] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:51.288810 29463 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:51.323678 18810 raft_consensus.cc:1275] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:51.325646 29417 consensus_queue.cc:1048] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:51.340582 16792 tablet_replica.cc:333] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.343037 29463 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.343693 29463 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.341964 16792 raft_consensus.cc:2243] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.349587 16792 raft_consensus.cc:2272] T 7e793f93c4f740b78f31d2f0dd643095 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.350168 18064 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.357807 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:51.360162 18063 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.361972 29433 raft_consensus.cc:2749] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:51.376888 16792 tablet_replica.cc:333] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
W20260709 12:26:51.377197 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:51.378010 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.391547 29463 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:51.394054 16792 raft_consensus.cc:2243] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.395321 16792 raft_consensus.cc:2272] T f9731c63bc7f406db2c77f5ef0415a78 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.399796 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:51.389045 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:51.418776 29454 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.419538 29454 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.423704 29454 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.425415 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.426421 18099 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
W20260709 12:26:51.426635 18780 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.428289 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.435391 16792 tablet_replica.cc:333] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.436939 16792 raft_consensus.cc:2243] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.437974 16792 raft_consensus.cc:2272] T a83278d09c914df888a57be57eb08f8e P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.437739 29454 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:51.438616 29454 raft_consensus.cc:493] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.439157 29454 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:51.439589 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.464447 16792 tablet_replica.cc:333] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.465704 16792 raft_consensus.cc:2243] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:51.481087 29417 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.481933 29417 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.482581 18164 raft_consensus.cc:2468] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 34.
I20260709 12:26:51.485206 29417 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:51.487370 16792 raft_consensus.cc:2272] T 17018f4f3f0d4fd5a11cac014a5abd36 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.491343 18064 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.494927 29454 raft_consensus.cc:515] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.491547 16792 tablet_replica.cc:333] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.501029 16792 raft_consensus.cc:2243] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.502345 16792 raft_consensus.cc:2272] T ff26d2026f1c4ff79de028b875b8ba96 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.505570 18064 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.506898 18166 raft_consensus.cc:1275] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 40 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:26:51.512210 16792 tablet_replica.cc:333] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.512009 29480 raft_consensus.cc:2804] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:51.512925 29480 raft_consensus.cc:697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 34 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
I20260709 12:26:51.513456 29468 consensus_queue.cc:1048] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:26:51.514079 29480 consensus_queue.cc:237] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.522347 29454 leader_election.cc:290] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.523587 18095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5eedbd3c223402caf9a71dac2fcaa98" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:51.524814 18095 raft_consensus.cc:3060] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:51.534422 29480 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.535279 29480 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.540133 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:51.540618 18169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:51.542747 29480 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:51.542887 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:26:51.543064 18064 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.544760 18063 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.545117 16792 raft_consensus.cc:2243] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.545989 16792 raft_consensus.cc:2272] T 3412b19684e44e3d89ca8150a50a9d14 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.554666 29480 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:51.557512 16792 tablet_replica.cc:333] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.558660 16792 raft_consensus.cc:2243] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.559429 16792 pending_rounds.cc:70] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Trying to abort 2 pending ops.
W20260709 12:26:51.559538 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:51.559686 16792 pending_rounds.cc:77] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting op as it isn't in flight: id { term: 30 index: 15 } timestamp: 7305625531297128448 op_type: NO_OP noop_request { }
I20260709 12:26:51.567277 16792 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:51.567741 16792 pending_rounds.cc:77] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting op as it isn't in flight: id { term: 28 index: 14 } timestamp: 7305625478366273536 op_type: NO_OP noop_request { }
I20260709 12:26:51.569397 16792 raft_consensus.cc:2889] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:51.570079 29432 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.570957 29432 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.575789 29432 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:51.577370 18064 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.582257 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.583074 29480 raft_consensus.cc:2749] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:51.583192 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:51.587185 18090 raft_consensus.cc:1275] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:26:51.566463 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.601699 29465 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.602375 29465 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.603118 16792 raft_consensus.cc:2272] T 9cf56c3bba4a4641a72b648d02c669f6 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.605329 29465 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.604954 29432 consensus_queue.cc:1048] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:51.610539 16792 tablet_replica.cc:333] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.611646 16792 raft_consensus.cc:2243] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.611521 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:51.628877 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
W20260709 12:26:51.636822 18780 leader_election.cc:336] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.646559 18095 raft_consensus.cc:2468] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:26:51.652396 29448 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.670970 29448 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.671214 17917 catalog_manager.cc:5697] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 33 to 34, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 34 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:51.655933 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:51.661710 29480 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.675681 29480 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.658250 16792 raft_consensus.cc:2272] T 4843c80e9d9348a397ce35e196deb9bd P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.677873 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.678686 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 26.
I20260709 12:26:51.680267 16792 tablet_replica.cc:333] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.680632 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:51.681238 16792 raft_consensus.cc:2243] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.686131 29480 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:51.687500 29432 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:51.687987 29432 raft_consensus.cc:493] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.688386 29432 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:51.688772 18064 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.673659 29448 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.694375 18778 leader_election.cc:304] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.695392 29468 raft_consensus.cc:2804] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:51.695941 29468 raft_consensus.cc:697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:51.696893 29468 consensus_queue.cc:237] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.700548 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:51.701427 18164 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 32.
I20260709 12:26:51.702955 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.703872 29417 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:51.704330 29417 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.704737 29417 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:51.711781 16792 raft_consensus.cc:2272] T 52abd819c1c443e28c6c41a392a7ed6f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.730480 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.735253 29433 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.736084 29433 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.738794 29433 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:26:51.740957 18809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:26:51.747239 29470 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.751406 29470 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.754166 16792 tablet_replica.cc:333] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
W20260709 12:26:51.754995 18064 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.750353 18095 raft_consensus.cc:1275] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:26:51.760687 18780 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.762130 29470 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:51.745844 29432 raft_consensus.cc:515] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.764078 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:51.764880 18164 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
W20260709 12:26:51.747782 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:51.748739 17919 catalog_manager.cc:5697] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 31 to 32, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 32 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:51.760957 18064 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:26:51.770474 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.786502 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.789670 29432 leader_election.cc:290] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.808904 29470 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.794265 29433 raft_consensus.cc:2749] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:51.809695 29470 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.810760 16792 raft_consensus.cc:2243] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.769385 29417 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.808280 29468 consensus_queue.cc:1048] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:26:51.812462 29470 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.800040 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.814016 29417 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:26:51.814668 18090 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 30.
I20260709 12:26:51.817869 18778 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.818004 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c3aa170318f4e1f95796426dc71d60b" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:51.818776 18164 raft_consensus.cc:3060] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:51.819780 29470 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:51.820214 29470 raft_consensus.cc:493] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.820621 29470 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:51.821168 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.822196 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.822856 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:51.826953 18138 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.827492 18138 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 910f8c8d787e4abbb092413c1f90ebf3, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.828423 18099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:51.829221 18099 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 28.
I20260709 12:26:51.811722 16792 raft_consensus.cc:2272] T 85764dcebf974f7cbec322be36e3dd5f P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:51.831179 18164 raft_consensus.cc:2468] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768723648a9444fda53659e6a8cc3091 in term 33.
I20260709 12:26:51.796581 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9dff14df364074b9d5e7e769f198b5" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:51.833695 18096 raft_consensus.cc:3060] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:51.836504 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:51.837106 29421 raft_consensus.cc:2749] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:51.837517 29468 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:51.838003 29468 raft_consensus.cc:493] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:51.838388 29468 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:51.840910 18780 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.834405 16792 tablet_replica.cc:333] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
W20260709 12:26:51.841727 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.842342 16792 raft_consensus.cc:2243] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.842339 18138 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:51.845356 29465 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:51.847251 18064 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:51.851509 29433 raft_consensus.cc:2804] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:51.863833 29433 raft_consensus.cc:697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 33 LEADER]: Becoming Leader. State: Replica: 768723648a9444fda53659e6a8cc3091, State: Running, Role: LEADER
W20260709 12:26:51.856144 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:51.865363 29433 consensus_queue.cc:237] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.870350 16792 pending_rounds.cc:70] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Trying to abort 1 pending ops.
I20260709 12:26:51.870771 16792 pending_rounds.cc:77] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d: Aborting op as it isn't in flight: id { term: 28 index: 15 } timestamp: 7305625536354422784 op_type: NO_OP noop_request { }
I20260709 12:26:51.871179 16792 raft_consensus.cc:2889] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:51.871596 16792 raft_consensus.cc:2272] T a3c91f3d9da3442f89360b0c316ee411 P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.878991 18064 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:51.903195 18138 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:51.930563 18096 raft_consensus.cc:2468] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
I20260709 12:26:51.937157 18136 leader_election.cc:304] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:51.938629 29421 raft_consensus.cc:2804] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:51.939281 29421 raft_consensus.cc:697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:51.940414 29421 consensus_queue.cc:237] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:51.942880 29468 raft_consensus.cc:515] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.957696 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cc94b1c76be48d6813e7d1c89a4d66c" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:51.958523 18090 raft_consensus.cc:3060] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:51.974928 18064 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 365 similar messages]
I20260709 12:26:51.979255 18090 raft_consensus.cc:2468] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 29.
I20260709 12:26:51.983608 16792 tablet_replica.cc:333] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d: stopping tablet replica
I20260709 12:26:51.984961 16792 raft_consensus.cc:2243] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:51.985724 16792 raft_consensus.cc:2272] T aef65d70106f4f458174493864b4ba3d P cab4a2f5d3ca48a2a803ee8ec3906e1d [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:51.992017 18138 leader_election.cc:336] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:51.993400 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:52.015539 18778 leader_election.cc:304] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3; no voters: 
I20260709 12:26:52.017748 29492 raft_consensus.cc:2804] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:52.017108 17918 catalog_manager.cc:5697] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 26 to 27, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 27 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:52.018471 29492 raft_consensus.cc:697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:52.019845 29492 consensus_queue.cc:237] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:51.997895 29468 leader_election.cc:290] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:51.937081 29470 raft_consensus.cc:515] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:26:52.064201 29470 leader_election.cc:290] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:52.065152 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:52.079731 17918 catalog_manager.cc:5697] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 reported cstate change: term changed from 32 to 33, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 768723648a9444fda53659e6a8cc3091 (127.16.102.2). New cstate: current_term: 33 leader_uuid: "768723648a9444fda53659e6a8cc3091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:52.095175 18166 raft_consensus.cc:1275] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:26:52.105777 17919 catalog_manager.cc:5697] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 28 to 29, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 29 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:52.110030 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 93414.1us
I20260709 12:26:52.119246 18096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:52.119395 29433 consensus_queue.cc:1048] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:52.120003 18096 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:52.139161 18164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6ac73d7138d4200bbfcbf2f84c2c775" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:26:52.139964 18164 raft_consensus.cc:3060] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:52.153214 18164 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
I20260709 12:26:52.164726 18099 raft_consensus.cc:1275] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:26:52.167320 29454 consensus_queue.cc:1048] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:26:52.212316 18096 raft_consensus.cc:2468] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
W20260709 12:26:52.263895 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:52.303702 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:52.294946 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 58128.5us
I20260709 12:26:52.313057 18780 leader_election.cc:304] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:52.314399 29492 raft_consensus.cc:2804] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:52.315512 29492 raft_consensus.cc:697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:26:52.316946 29492 consensus_queue.cc:237] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:26:52.406759 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 62107.7us
W20260709 12:26:52.415234 18780 leader_election.cc:336] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:52.417914 17919 catalog_manager.cc:5697] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 30 to 31, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 31 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:52.431650 18166 raft_consensus.cc:1275] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Refusing update from remote peer 768723648a9444fda53659e6a8cc3091: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:26:52.455287 29480 consensus_queue.cc:1048] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:52.470383 18090 raft_consensus.cc:1275] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:52.473528 29480 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:52.474350 29480 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:52.475082 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 58351.5us
W20260709 12:26:52.479120 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:52.482709 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "768723648a9444fda53659e6a8cc3091" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:26:52.484794 29432 consensus_queue.cc:1048] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:52.477108 29480 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:26:52.506953 18064 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:52.507524 18064 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 768723648a9444fda53659e6a8cc3091; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:26:52.508764 29480 raft_consensus.cc:2749] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:52.558930 18780 leader_election.cc:336] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:52.602165 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:52.646799 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:52.651737 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:52.690941 18091 raft_consensus.cc:1275] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:26:52.699575 29454 consensus_queue.cc:1048] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:26:52.734685 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:52.747232 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:52.756210 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:52.832288 18169 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:52.834263 29492 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:52.914898 29498 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:52.915724 29498 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:52.932523 29498 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:26:52.936338 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 49008.6us
I20260709 12:26:52.937947 18091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091" is_pre_election: true
I20260709 12:26:52.951989 18090 raft_consensus.cc:1275] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Refusing update from remote peer 910f8c8d787e4abbb092413c1f90ebf3: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:52.953987 18091 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 29.
W20260709 12:26:52.956010 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:52.959786 29493 consensus_queue.cc:1048] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.005s
W20260709 12:26:52.995016 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:52.996493 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:52.999366 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:52.972390 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:53.002987 29497 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:53.003508 29497 raft_consensus.cc:493] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting leader election (detected failure of leader cab4a2f5d3ca48a2a803ee8ec3906e1d)
I20260709 12:26:53.003948 29497 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:53.026254 18138 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:26:53.033303 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:53.041412 29497 raft_consensus.cc:515] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:26:53.051730 29497 leader_election.cc:290] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Requested vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:26:53.064116 18090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17018f4f3f0d4fd5a11cac014a5abd36" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "768723648a9444fda53659e6a8cc3091"
I20260709 12:26:53.066378 18090 raft_consensus.cc:3060] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:53.138402 18090 raft_consensus.cc:2468] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 30.
I20260709 12:26:53.145629 18136 leader_election.cc:304] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:26:53.147024 29497 raft_consensus.cc:2804] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:53.147747 29497 raft_consensus.cc:697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Becoming Leader. State: Replica: ae9641d5c3184572a102d51a75f670ee, State: Running, Role: LEADER
I20260709 12:26:53.149596 29497 consensus_queue.cc:237] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
W20260709 12:26:53.155694 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.156579 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.162484 18138 leader_election.cc:336] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:26:53.195835 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:53.200783 17919 catalog_manager.cc:5697] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee reported cstate change: term changed from 29 to 30, leader changed from cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) to ae9641d5c3184572a102d51a75f670ee (127.16.102.3). New cstate: current_term: 30 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:53.235226 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.251814 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.329128 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.357038 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:53.482121 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.508358 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:53.549211 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.555464 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.589685 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.591434 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:53.675515 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:53.683271 18091 raft_consensus.cc:1275] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:53.686327 29497 consensus_queue.cc:1048] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:26:53.795591 16792 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:53.809087 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.838739 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.841593 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:53.850055 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.905103 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:53.932106 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.941493 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.942971 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:53.983024 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:54.003499 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.034705 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:54.052143 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.083665 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.115129 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.228891 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.230197 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.318547 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.343741 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.344520 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:54.370576 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.395076 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:54.492825 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.703913 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.796414 16792 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:55.026407 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:55.034456 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.095391 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.122100 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:55.122823 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:55.195823 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:55.208531 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:55.273699 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.274539 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.324751 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.392349 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.490867 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.513805 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.633766 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.641259 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.654770 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.679531 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.797052 16792 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:55.852878 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.908653 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:55.955003 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080000b5e0 after lost signal to thread 16795
W20260709 12:26:55.956725 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080041e7e0 after lost signal to thread 17956
W20260709 12:26:55.959102 17953 debug-util.cc:398] Leaking SignalData structure 0x7b0800355f60 after lost signal to thread 18123
W20260709 12:26:55.967242 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080035d860 after lost signal to thread 18197
W20260709 12:26:55.969084 17953 debug-util.cc:398] Leaking SignalData structure 0x7b080027f240 after lost signal to thread 18839
W20260709 12:26:55.994856 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.002826 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.122061 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.158313 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.183094 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.250689 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.277884 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.320647 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.384109 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.389302 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:56.396723 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.408102 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.438607 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.474799 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.493590 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.514991 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.540867 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.567785 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.624179 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.655535 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.681190 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.694204 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.737082 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.797916 16792 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:56.799767 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.808900 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.821342 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.932340 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:56.944828 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.975584 18064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111) [suppressed 492 similar messages]
W20260709 12:26:56.996412 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.187206 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:57.210352 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 53903.4us
W20260709 12:26:57.277751 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.393208 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:57.488267 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.644995 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:57.675758 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.719466 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:57.761126 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.763562 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.798530 16792 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:57.809201 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:57.881855 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.907765 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.987063 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:57.996232 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.030439 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.057461 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:58.148185 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:58.152897 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.238579 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.320487 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.370675 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.434016 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.486831 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.502817 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:58.583958 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:58.779170 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.780057 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 68520.5us
W20260709 12:26:58.799154 16792 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:58.830228 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:58.870077 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.870978 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.879395 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.884097 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.933581 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.949530 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:58.999819 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.008404 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:59.009258 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.028098 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:59.027468 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:59.096660 18138 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.136709 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.187741 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.198930 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:59.225384 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.259450 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.284149 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.421402 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 73053.5us
W20260709 12:26:59.458448 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:59.491926 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.663043 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 78396.5us
W20260709 12:26:59.791409 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.806550 16792 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 18045)
W20260709 12:26:59.806766 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.831411 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.960404 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:59.961263 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.010589 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:00.015007 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.026703 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:00.093086 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:00.111061 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.214157 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:00.307603 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:00.400079 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.418550 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.449156 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.497186 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.517587 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.579595 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.627651 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:00.681301 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.720685 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.807365 16792 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:00.940832 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:00.976030 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.013393 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:01.024742 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.046134 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.111812 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.131696 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:01.235055 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 45199.7us
W20260709 12:27:01.273715 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 55451us
W20260709 12:27:01.279707 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:01.331125 18164 consensus_queue.cc:237] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:01.401444 18811 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:27:01.404451 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:27:01.414757 29545 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:27:01.437011 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:01.448418 18090 raft_consensus.cc:1275] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Refusing update from remote peer ae9641d5c3184572a102d51a75f670ee: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:27:01.458875 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:01.458875 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:01.466745 29547 consensus_queue.cc:1048] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:27:01.482766 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:27:01.493893 29546 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Committing config change with OpId 32.20: config changed from index 9 to 20, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:27:01.483349 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:01.526041 18811 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Committing config change with OpId 32.20: config changed from index 9 to 20, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:27:01.550138 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:01.606042 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:01.606875 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.607813 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:01.617739 17908 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f9731c63bc7f406db2c77f5ef0415a78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 13)
W20260709 12:27:01.673448 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:27:01.677495 17919 catalog_manager.cc:5697] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee reported cstate change: config changed from index 9 to 20, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New cstate: current_term: 32 leader_uuid: "ae9641d5c3184572a102d51a75f670ee" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:27:01.723181 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:01.732966 18095 raft_consensus.cc:2955] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Committing config change with OpId 32.20: config changed from index 9 to 20, VOTER cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } } }
W20260709 12:27:01.750048 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:01.775221 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:01.792060 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:01.808475 16792 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:01.847257 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:01.915589 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:01.936990 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.939309 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:01.954964 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.970177 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.975740 18138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111) [suppressed 485 similar messages]
W20260709 12:27:02.132623 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 156891us
W20260709 12:27:02.226258 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:02.278381 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 52267.5us
W20260709 12:27:02.340340 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:02.395524 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:02.395885 17908 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55418.1us
W20260709 12:27:02.405642 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:02.414588 17908 catalog_manager.cc:4755] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f9731c63bc7f406db2c77f5ef0415a78: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:02.567409 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:02.778416 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 45351.5us
W20260709 12:27:02.809072 16792 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:02.819491 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:02.862334 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:02.867993 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:02.871001 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:02.907354 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:02.919822 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:02.959095 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:02.984328 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.003477 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.037135 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.083460 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.215576 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:03.257012 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.276769 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:03.334301 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.371611 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:03.544653 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.628432 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:03.688320 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.719092 17908 catalog_manager.cc:4755] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f9731c63bc7f406db2c77f5ef0415a78: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:03.726155 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.756934 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08001e8e20 after lost signal to thread 16795
W20260709 12:27:03.758587 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002923e0 after lost signal to thread 17956
W20260709 12:27:03.760434 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08002e38c0 after lost signal to thread 18123
W20260709 12:27:03.766695 18836 debug-util.cc:398] Leaking SignalData structure 0x7b080028a3c0 after lost signal to thread 18197
W20260709 12:27:03.787534 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08003a8ee0 after lost signal to thread 18839
W20260709 12:27:03.782317 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:03.809128 18836 debug-util.cc:398] Leaking SignalData structure 0x7b080011ac80 after lost signal to thread 29560
W20260709 12:27:03.810905 16792 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:03.812676 18836 debug-util.cc:398] Leaking SignalData structure 0x7b080008f860 after lost signal to thread 29563
W20260709 12:27:03.825178 18836 debug-util.cc:398] Leaking SignalData structure 0x7b08003b21e0 after lost signal to thread 29568
W20260709 12:27:03.825680 18836 debug-util.cc:398] Leaking SignalData structure 0x7b0800109de0 after lost signal to thread 29572
W20260709 12:27:03.828696 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.853374 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.908118 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:03.943440 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:04.230288 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:04.247339 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.250290 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.264648 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.265977 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.300436 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:04.347529 18063 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:46307, user_credentials={real_user=slave}} blocked reactor thread for 76305.4us
W20260709 12:27:04.356549 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.365350 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.369306 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 92954.5us
W20260709 12:27:04.430917 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.440236 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.451119 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:04.452530 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:04.453094 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:04.453752 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:04.458876 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:04.487891 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:04.711051 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:04.729013 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:04.738600 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.813043 16792 thread.cc:527] Waited for 12000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:04.818872 17908 catalog_manager.cc:4755] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f9731c63bc7f406db2c77f5ef0415a78: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:04.837224 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:04.879844 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:04.936337 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.023530 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08001acfc0 after lost signal to thread 16795
W20260709 12:27:05.025228 18120 debug-util.cc:398] Leaking SignalData structure 0x7b080026e220 after lost signal to thread 17956
W20260709 12:27:05.027688 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08002a3c00 after lost signal to thread 18123
W20260709 12:27:05.029639 18120 debug-util.cc:398] Leaking SignalData structure 0x7b0800381400 after lost signal to thread 18197
W20260709 12:27:05.031787 18120 debug-util.cc:398] Leaking SignalData structure 0x7b08003f6d40 after lost signal to thread 18839
W20260709 12:27:05.049793 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.104441 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.211529 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:05.273518 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:05.315877 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:05.384089 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:05.396847 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.441290 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:05.492661 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.550639 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.567622 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.576090 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:05.628752 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:05.703358 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.743275 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.777686 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.813668 16792 thread.cc:527] Waited for 13000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:05.880715 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:05.912082 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:06.086432 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:06.099833 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.195252 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.200099 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.236745 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.404740 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:06.447678 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.488654 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.496119 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:06.519289 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.575484 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.704434 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.708094 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.747941 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:06.785681 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:06.790203 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:06.801200 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:06.814615 16792 thread.cc:527] Waited for 14000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:06.889199 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.890216 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.903398 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:06.928440 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:06.946533 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:06.947505 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:06.988974 17908 catalog_manager.cc:4755] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f9731c63bc7f406db2c77f5ef0415a78: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:07.021540 18064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111) [suppressed 489 similar messages]
W20260709 12:27:07.025239 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:07.035616 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.141362 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:07.182102 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:07.288573 18138 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 55787.5us
W20260709 12:27:07.332911 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:07.337023 18778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 64554us
W20260709 12:27:07.361169 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.421810 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.472062 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.588640 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:07.604871 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.747470 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:07.815229 16792 thread.cc:527] Waited for 15000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:07.832031 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:07.878870 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.914222 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:07.948504 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:07.967706 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:08.003839 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:08.084210 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:08.163295 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.185180 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.193291 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:08.247430 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.246999 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:08.252063 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.271122 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.360338 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:08.541415 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033335
W20260709 12:27:08.549688 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:08.574743 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.617435 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:08.707185 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.165s	user 0.046s	sys 0.000s Metrics: {"bytes_written":6269,"delete_count":0,"lbm_write_time_us":1137,"lbm_writes_lt_1ms":6,"reinsert_count":0,"thread_start_us":17482,"threads_started":1,"update_count":245,"wal-append.queue_time_us":17073}
I20260709 12:27:08.716080 17911 maintenance_manager.cc:419] P 7b17aa4109194c3c9f5ca41c3594e03c: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.274246
W20260709 12:27:08.724433 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.745345 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:08.787390 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:08.815864 16792 thread.cc:527] Waited for 16000ms trying to join with diag-logger (tid 18045)
W20260709 12:27:08.913375 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:08.980667 17910 maintenance_manager.cc:643] P 7b17aa4109194c3c9f5ca41c3594e03c: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.263s	user 0.097s	sys 0.018s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":26570,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":12111,"dirs.run_cpu_time_us":658,"dirs.run_wall_time_us":5522,"lbm_read_time_us":2473,"lbm_reads_lt_1ms":27,"lbm_write_time_us":1884,"lbm_writes_lt_1ms":11,"peak_mem_usage":286563,"reinsert_count":0,"thread_start_us":22852,"threads_started":2,"update_count":245}
W20260709 12:27:08.983708 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.084479 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.088761 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.145736 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.181878 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.189076 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:09.211859 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.220091 18064 consensus_peers.cc:597] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.275368 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.391343 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.434412 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.440877 18064 consensus_peers.cc:597] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:09.468600 18064 consensus_peers.cc:597] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.482141 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.504784 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:09.585809 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:09.644027 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.785742 18064 consensus_peers.cc:597] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.886773 18064 consensus_peers.cc:597] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:09.891826 18064 consensus_peers.cc:597] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:09.974272 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:10.007380 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:10.025138 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.103436 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.236943 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:10.413767 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:10.418201 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.436561 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.488289 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.493407 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:10.500005 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.525590 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:10.544711 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.570964 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:10.598553 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.615097 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:10.684752 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:10.731297 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:10.800340 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:10.897953 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
W20260709 12:27:10.963133 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:10.968040 18064 consensus_peers.cc:597] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:10.986387 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.067833 18064 consensus_peers.cc:597] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:11.129711 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.193260 17908 catalog_manager.cc:4755] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f9731c63bc7f406db2c77f5ef0415a78: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:11.355784 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.398742 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:11.414736 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:11.427709 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.493212 18064 consensus_peers.cc:597] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:11.538841 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:11.557071 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
W20260709 12:27:11.577906 18136 consensus_peers.cc:597] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.592021 18064 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 84729.2us
W20260709 12:27:11.592021 18136 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.603232 18136 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.606668 18136 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.607892 18778 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.608889 18778 consensus_peers.cc:597] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.619546 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.641561 18778 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.648876 18778 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.662415 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.678390 18136 consensus_peers.cc:597] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.714535 18136 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.713085 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.716322 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.718040 18136 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.723760 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:11.726975 18136 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.727864 18136 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.728523 18136 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.728840 18064 consensus_peers.cc:597] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:11.729105 18136 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.729741 18136 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.729914 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:11.734588 16792 tablet_replica.cc:333] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.735942 16792 raft_consensus.cc:2243] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:11.738459 16792 raft_consensus.cc:2272] T 8e0ff0e5252f4ace898a73a2268e5dc8 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:11.739859 18136 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.743070 16792 tablet_replica.cc:333] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.744540 16792 raft_consensus.cc:2243] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:11.746503 16792 raft_consensus.cc:2272] T df3d21a581ef40feafcce4ab8b817645 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:11.761730 18778 consensus_peers.cc:597] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.772023 18778 consensus_peers.cc:597] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.775676 16792 tablet_replica.cc:333] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.777031 16792 raft_consensus.cc:2243] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:11.779021 16792 raft_consensus.cc:2272] T 23e4eb0b942f4c7e81dfe76f4e015618 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:11.782332 18136 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.783088 16792 tablet_replica.cc:333] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.784287 16792 raft_consensus.cc:2243] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 35 LEADER]: Raft consensus shutting down.
W20260709 12:27:11.788575 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:11.789575 16792 raft_consensus.cc:2272] T 583d61c844f048e4b53a66acc88cc1d9 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:11.791627 18136 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.793370 16792 tablet_replica.cc:333] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.794610 16792 raft_consensus.cc:2243] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:11.795585 16792 raft_consensus.cc:2272] T 86bfc0ef6acd420b829a9a1df3376f48 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.801635 16792 tablet_replica.cc:333] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.802887 16792 raft_consensus.cc:2243] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:11.805979 18136 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.809669 18778 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.811161 16792 raft_consensus.cc:2272] T 49a68a064bb44579af6ee5ad50121ffc P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:11.817154 18136 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.821343 16792 tablet_replica.cc:333] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.822731 16792 raft_consensus.cc:2243] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 LEADER]: Raft consensus shutting down.
I20260709 12:27:11.824342 16792 raft_consensus.cc:2272] T ef30e67b67894670b0cf3980d6b0f624 P 768723648a9444fda53659e6a8cc3091 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:11.826417 18136 consensus_peers.cc:597] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.828059 16792 tablet_replica.cc:333] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
W20260709 12:27:11.829690 18136 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.837225 18778 consensus_peers.cc:597] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.841149 18136 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.837997 16792 raft_consensus.cc:2243] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:11.846457 16792 raft_consensus.cc:2272] T f1f3da088a314a87a78898050dab39e9 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.851910 16792 tablet_replica.cc:333] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.853240 16792 raft_consensus.cc:2243] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:11.855202 16792 raft_consensus.cc:2272] T fd8fe387b2164c30aa94afed5a7c218d P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.866329 16792 tablet_replica.cc:333] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.867473 16792 raft_consensus.cc:2243] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:11.869207 16792 raft_consensus.cc:2272] T acbc4a634f954522904f2b865531f456 P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.876016 16792 tablet_replica.cc:333] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.877225 16792 raft_consensus.cc:2243] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 25 LEADER]: Raft consensus shutting down.
W20260709 12:27:11.886363 18778 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.889076 16792 raft_consensus.cc:2272] T da76d5c707544d29a2f74d6015ec1cfc P 768723648a9444fda53659e6a8cc3091 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.911520 16792 tablet_replica.cc:333] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.912719 16792 raft_consensus.cc:2243] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:11.923557 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:11.924314 16792 raft_consensus.cc:2272] T 5c6e41c6a46d4fc99220183ed20b6921 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.940513 16792 tablet_replica.cc:333] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.941665 16792 raft_consensus.cc:2243] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:11.943001 16792 raft_consensus.cc:2272] T f0dc8b13a3294835a091ca099ee49b29 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.946959 16792 tablet_replica.cc:333] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:11.948235 16792 raft_consensus.cc:2243] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:11.950112 16792 raft_consensus.cc:2272] T c53769058ca3443f805983b61a69328e P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:11.960290 16792 tablet_replica.cc:333] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
W20260709 12:27:11.985139 18136 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.993479 16792 raft_consensus.cc:2243] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:12.004370 18778 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:12.014271 16792 raft_consensus.cc:2272] T e84e056b63c34d8da15d2fde3877279b P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.024317 18136 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:12.025957 18136 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 509 similar messages]
I20260709 12:27:12.069784 16792 tablet_replica.cc:333] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.079773 16792 raft_consensus.cc:2243] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.081990 16792 raft_consensus.cc:2272] T 9fe4c9950f4d4f4588d08c7cd436b1dc P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.083288 18138 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:12.106012 16792 tablet_replica.cc:333] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.107326 16792 raft_consensus.cc:2243] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:27:12.108681 16792 raft_consensus.cc:2272] T 2217b6329873485ead60419dc5acbff0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.113121 16792 tablet_replica.cc:333] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.114403 16792 raft_consensus.cc:2243] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.115433 16792 raft_consensus.cc:2272] T 555e520f5efe4bafb5947749ee16e734 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.120563 16792 tablet_replica.cc:333] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.121716 16792 raft_consensus.cc:2243] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.123049 16792 raft_consensus.cc:2272] T a07f730a06794e2cb530edca57871763 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.126746 16792 tablet_replica.cc:333] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.128093 16792 raft_consensus.cc:2243] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.130401 16792 raft_consensus.cc:2272] T f7a707b110a44c8c97606fdb8d7dece0 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.131836 18136 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:12.132658 18136 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:12.133765 18136 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:12.134471 18136 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:12.134480 16792 tablet_replica.cc:333] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.152051 16792 raft_consensus.cc:2243] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.156481 16792 raft_consensus.cc:2272] T af9dff14df364074b9d5e7e769f198b5 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.160334 16792 tablet_replica.cc:333] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.161923 16792 raft_consensus.cc:2243] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.163029 16792 raft_consensus.cc:2272] T b5eedbd3c223402caf9a71dac2fcaa98 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.166798 16792 tablet_replica.cc:333] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.167824 16792 raft_consensus.cc:2243] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.168723 16792 raft_consensus.cc:2272] T 5c743e221c2c477e9b998557d9b9749a P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.174010 16792 tablet_replica.cc:333] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.175129 16792 raft_consensus.cc:2243] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.176308 16792 raft_consensus.cc:2272] T aeba5cb4432947efbaa2fa537357e992 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.181160 16792 tablet_replica.cc:333] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.182329 16792 raft_consensus.cc:2243] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:27:12.184178 16792 raft_consensus.cc:2272] T 7c3aa170318f4e1f95796426dc71d60b P 768723648a9444fda53659e6a8cc3091 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.188052 16792 tablet_replica.cc:333] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.189249 16792 raft_consensus.cc:2243] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:12.190277 18136 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:12.190348 16792 raft_consensus.cc:2272] T a6ac73d7138d4200bbfcbf2f84c2c775 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.194751 16792 tablet_replica.cc:333] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.195856 16792 raft_consensus.cc:2243] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.197021 16792 raft_consensus.cc:2272] T 666440ca642c4715bd2f211bb5f14d29 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.200507 16792 tablet_replica.cc:333] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.202150 16792 raft_consensus.cc:2243] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.203532 16792 raft_consensus.cc:2272] T a29e00b535474906948e66659110db3b P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.206251 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:12.206928 16792 tablet_replica.cc:333] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.208158 16792 raft_consensus.cc:2243] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.209098 16792 raft_consensus.cc:2272] T 4cc94b1c76be48d6813e7d1c89a4d66c P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.213680 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:12.218942 16792 tablet_replica.cc:333] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.224727 16792 raft_consensus.cc:2243] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.225967 16792 raft_consensus.cc:2272] T 7cd93736a3314dbbb5c62b5af8fbed11 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.229321 16792 tablet_replica.cc:333] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.230443 16792 raft_consensus.cc:2243] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.231276 16792 raft_consensus.cc:2272] T 804b0137cb0b4850b912b6519bae7417 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.234861 16792 tablet_replica.cc:333] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.235984 16792 raft_consensus.cc:2243] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.249245 16792 raft_consensus.cc:2272] T 47bac54b53844de9814420f86103a6f8 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.255914 16792 tablet_replica.cc:333] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.257279 16792 raft_consensus.cc:2243] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.262375 16792 raft_consensus.cc:2272] T d525bcd825eb49d9be935bc89de569c7 P 768723648a9444fda53659e6a8cc3091 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.265949 16792 tablet_replica.cc:333] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.268843 16792 raft_consensus.cc:2243] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:12.270633 16792 raft_consensus.cc:2272] T 7e793f93c4f740b78f31d2f0dd643095 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.274226 16792 tablet_replica.cc:333] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.275398 16792 raft_consensus.cc:2243] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.276213 16792 raft_consensus.cc:2272] T 95f78d7b0f7f422eacbfa07bbc248bfa P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.279850 16792 tablet_replica.cc:333] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
W20260709 12:27:12.281587 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:12.280851 16792 raft_consensus.cc:2243] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:12.287546 18138 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:12.290411 16792 raft_consensus.cc:2272] T 578328e4e1fd478cbc58dd1048417480 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.294623 16792 tablet_replica.cc:333] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.296219 16792 raft_consensus.cc:2243] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.297130 16792 raft_consensus.cc:2272] T d440327a63754b889f7edd91e1373942 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.300498 16792 tablet_replica.cc:333] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.301924 16792 raft_consensus.cc:2243] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:27:12.303409 16792 raft_consensus.cc:2272] T 9ba0c897fcb346488dfe124a973cb2d1 P 768723648a9444fda53659e6a8cc3091 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.308337 16792 tablet_replica.cc:333] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.309382 16792 raft_consensus.cc:2243] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.310185 16792 raft_consensus.cc:2272] T 52abd819c1c443e28c6c41a392a7ed6f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.313416 16792 tablet_replica.cc:333] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.314428 16792 raft_consensus.cc:2243] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:12.315989 16792 raft_consensus.cc:2272] T b7fd19d9bf374e6da56b7ead11c666e9 P 768723648a9444fda53659e6a8cc3091 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.322312 18138 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:12.325842 16792 tablet_replica.cc:333] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.326956 16792 raft_consensus.cc:2243] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.329244 16792 raft_consensus.cc:2272] T f9731c63bc7f406db2c77f5ef0415a78 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.343207 16792 tablet_replica.cc:333] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.344172 16792 raft_consensus.cc:2243] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.345628 16792 raft_consensus.cc:2272] T 17018f4f3f0d4fd5a11cac014a5abd36 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.360064 16792 tablet_replica.cc:333] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.361050 16792 raft_consensus.cc:2243] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.363112 16792 raft_consensus.cc:2272] T ff26d2026f1c4ff79de028b875b8ba96 P 768723648a9444fda53659e6a8cc3091 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.366431 16792 tablet_replica.cc:333] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.367375 16792 raft_consensus.cc:2243] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.368462 16792 raft_consensus.cc:2272] T 3412b19684e44e3d89ca8150a50a9d14 P 768723648a9444fda53659e6a8cc3091 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.377658 16792 tablet_replica.cc:333] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.378575 16792 raft_consensus.cc:2243] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.386894 16792 raft_consensus.cc:2272] T 9cf56c3bba4a4641a72b648d02c669f6 P 768723648a9444fda53659e6a8cc3091 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.387859 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:12.391120 16792 tablet_replica.cc:333] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.397025 16792 raft_consensus.cc:2243] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.397855 16792 raft_consensus.cc:2272] T 5240f7371cc74cb48dffa1dadcbeec49 P 768723648a9444fda53659e6a8cc3091 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.401270 16792 tablet_replica.cc:333] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.402172 16792 raft_consensus.cc:2243] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:12.403662 16792 raft_consensus.cc:2272] T 18093241e4df42869e3d96cdf66a53e6 P 768723648a9444fda53659e6a8cc3091 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.407349 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:27:12.418104 16792 tablet_replica.cc:333] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.419106 16792 raft_consensus.cc:2243] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.419929 16792 raft_consensus.cc:2272] T 85764dcebf974f7cbec322be36e3dd5f P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:12.432852 16792 tablet_replica.cc:333] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091: stopping tablet replica
I20260709 12:27:12.433717 16792 raft_consensus.cc:2243] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:12.434583 16792 raft_consensus.cc:2272] T a3c91f3d9da3442f89360b0c316ee411 P 768723648a9444fda53659e6a8cc3091 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:12.509434 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:12.619463 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:12.859481 18138 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:12.971240 18138 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:12.978012 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:27:12.986405 29629 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:12.987246 29629 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:12.991755 29629 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:12.997465 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:13.028470 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:13.031432 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:13.034614 18138 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:13.044453 29629 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.045164 29629 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:13.051590 29629 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:13.057657 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:13.063432 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:13.078632 18780 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.081142 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.091069 29629 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:13.101295 29640 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.102190 29640 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:27:13.113799 29640 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:13.126179 18138 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:13.176447 29629 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.177214 29629 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:13.179963 29629 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:13.188094 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:13.199664 18138 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:13.206902 29633 raft_consensus.cc:493] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.208776 29633 raft_consensus.cc:515] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:27:13.211465 29633 leader_election.cc:290] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:13.218700 18780 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.219432 18780 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:27:13.206084 18136 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:13.221434 18138 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.223724 18138 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.224529 29629 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:13.225481 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:13.235409 29633 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:27:13.242194 29629 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.243080 29629 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:13.245759 29629 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:27:13.250919 29633 raft_consensus.cc:493] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.256189 29633 raft_consensus.cc:515] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:27:13.260443 29640 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.261265 29640 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:13.264974 29640 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:27:13.267263 29633 leader_election.cc:290] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:13.288905 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:13.303789 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
W20260709 12:27:13.312498 18136 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:13.319129 18780 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.328267 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.351599 29629 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:13.388448 18138 consensus_peers.cc:597] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:13.389755 18138 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:13.390301 18138 leader_election.cc:336] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.390739 18138 leader_election.cc:304] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.393862 29640 raft_consensus.cc:2749] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:27:13.395562 18138 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:13.396087 18138 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:13.385524 18780 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.398319 29634 raft_consensus.cc:493] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.399086 29634 raft_consensus.cc:515] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:27:13.402281 29634 leader_election.cc:290] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:27:13.410691 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.411844 29629 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:27:13.411952 18136 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:13.426859 18136 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:27:13.433185 29633 raft_consensus.cc:2749] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:13.432847 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:13.435786 18136 leader_election.cc:336] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:13.436512 18136 leader_election.cc:304] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.446856 29640 raft_consensus.cc:2749] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:27:13.459017 29642 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.459867 29642 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:27:13.465276 18138 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:13.469950 18138 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:13.472851 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:13.473987 29642 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:13.483633 18136 leader_election.cc:336] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:13.484340 18136 leader_election.cc:304] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:27:13.545894 18780 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:13.546730 18780 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:13.547596 29629 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:27:13.591526 29633 raft_consensus.cc:2749] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:27:13.591749 18136 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:40047, user_credentials={real_user=slave}} blocked reactor thread for 108149us
I20260709 12:27:13.909287 29642 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.910197 29642 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:13.933960 29650 raft_consensus.cc:493] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.935027 29650 raft_consensus.cc:515] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:27:13.939582 29650 leader_election.cc:290] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:27:13.945796 29654 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:13.953307 29654 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:13.956773 29654 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:27:13.946480 29642 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:13.999945 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.045226 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:45965, user_credentials={real_user=slave}} blocked reactor thread for 45392.8us
W20260709 12:27:14.047698 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:14.061918 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:14.064468 29648 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:14.065312 29648 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:14.068094 29648 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:27:14.126796 18166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:27:14.127717 18166 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 31.
W20260709 12:27:14.130357 18136 consensus_peers.cc:597] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.131047 18136 consensus_peers.cc:597] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.131662 18136 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:14.133378 18165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee" is_pre_election: true
I20260709 12:27:14.137231 18165 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 27.
I20260709 12:27:14.152554 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3" is_pre_election: true
I20260709 12:27:14.153478 18810 raft_consensus.cc:2468] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9641d5c3184572a102d51a75f670ee in term 27.
W20260709 12:27:14.163956 18136 consensus_peers.cc:597] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.164629 18138 consensus_peers.cc:597] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.164813 18136 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:14.166291 18138 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:14.172793 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:27:14.173882 29654 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:27:14.174306 29654 raft_consensus.cc:493] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:14.174700 29654 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:27:14.175231 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 
I20260709 12:27:14.176285 29645 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:27:14.176668 29645 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:14.177000 29645 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:27:14.180641 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:14.181962 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:14.182472 18136 consensus_peers.cc:597] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.182917 18778 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.191479 18137 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:27:14.192463 29634 raft_consensus.cc:2804] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:27:14.192994 29634 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Starting leader election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:14.193418 29634 raft_consensus.cc:3060] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:27:14.198100 29654 raft_consensus.cc:515] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:14.200887 29654 leader_election.cc:290] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:27:14.205560 29634 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:14.208137 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7fd19d9bf374e6da56b7ead11c666e9" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:27:14.208864 18170 raft_consensus.cc:3060] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:27:14.209863 29634 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307)
I20260709 12:27:14.212086 18810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "ae9641d5c3184572a102d51a75f670ee" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "910f8c8d787e4abbb092413c1f90ebf3"
W20260709 12:27:14.212347 18138 leader_election.cc:336] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:14.212879 18138 leader_election.cc:304] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
W20260709 12:27:14.206696 18778 leader_election.cc:336] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:14.214075 29634 raft_consensus.cc:2749] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:27:14.216449 18136 consensus_peers.cc:597] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.217218 18136 consensus_peers.cc:597] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.219935 18170 raft_consensus.cc:2468] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910f8c8d787e4abbb092413c1f90ebf3 in term 32.
I20260709 12:27:14.220748 29645 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:14.221843 18780 leader_election.cc:304] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3, ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091
I20260709 12:27:14.222131 18810 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ae9641d5c3184572a102d51a75f670ee in current term 28: Already voted for candidate 910f8c8d787e4abbb092413c1f90ebf3 in this term.
W20260709 12:27:14.223057 18138 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:14.223213 29654 raft_consensus.cc:2804] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:27:14.224298 29654 raft_consensus.cc:697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Becoming Leader. State: Replica: 910f8c8d787e4abbb092413c1f90ebf3, State: Running, Role: LEADER
I20260709 12:27:14.225418 29654 consensus_queue.cc:237] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:27:14.228399 18138 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.231087 18136 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.240547 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:14.243530 29645 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Requested vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
I20260709 12:27:14.245574 18170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18093241e4df42869e3d96cdf66a53e6" candidate_uuid: "910f8c8d787e4abbb092413c1f90ebf3" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ae9641d5c3184572a102d51a75f670ee"
I20260709 12:27:14.246568 18170 raft_consensus.cc:2393] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 910f8c8d787e4abbb092413c1f90ebf3 in current term 28: Already voted for candidate ae9641d5c3184572a102d51a75f670ee in this term.
I20260709 12:27:14.248271 18780 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
W20260709 12:27:14.250578 18136 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:14.252028 18136 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, 910f8c8d787e4abbb092413c1f90ebf3
I20260709 12:27:14.254101 29634 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:27:14.255013 18778 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.255424 17917 catalog_manager.cc:5697] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 reported cstate change: term changed from 31 to 32, leader changed from 768723648a9444fda53659e6a8cc3091 (127.16.102.2) to 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4). New cstate: current_term: 32 leader_uuid: "910f8c8d787e4abbb092413c1f90ebf3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:14.259123 29645 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:27:14.259698 18138 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.260560 18136 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.348315 18778 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.355347 18136 consensus_peers.cc:597] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.448181 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
W20260709 12:27:14.486434 18778 consensus_peers.cc:597] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.510466 18778 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.512953 18136 consensus_peers.cc:597] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.538128 18138 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.548388 18138 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.549317 18136 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.549999 18136 consensus_peers.cc:597] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.557021 18136 consensus_peers.cc:597] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.559351 18138 consensus_peers.cc:597] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.565078 18778 consensus_peers.cc:597] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.573393 18138 consensus_peers.cc:597] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.579492 18778 consensus_peers.cc:597] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.580194 18778 consensus_peers.cc:597] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.591423 18778 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.600260 18136 consensus_peers.cc:597] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.607370 18136 consensus_peers.cc:597] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.614776 18136 consensus_peers.cc:597] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.615805 18194 debug-util.cc:398] Leaking SignalData structure 0x7b08003a9e40 after lost signal to thread 16795
W20260709 12:27:14.617508 18194 debug-util.cc:398] Leaking SignalData structure 0x7b0800358160 after lost signal to thread 17956
W20260709 12:27:14.619602 18194 debug-util.cc:398] Leaking SignalData structure 0x7b080016ece0 after lost signal to thread 18197
W20260709 12:27:14.624150 18194 debug-util.cc:398] Leaking SignalData structure 0x7b0800357c60 after lost signal to thread 18839
W20260709 12:27:14.628263 18136 consensus_peers.cc:597] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.652910 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
W20260709 12:27:14.661548 18136 consensus_peers.cc:597] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.694707 18780 consensus_peers.cc:597] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:14.711472 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:14.712409 18136 consensus_peers.cc:597] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.714304 18136 consensus_peers.cc:597] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.719951 18136 consensus_peers.cc:597] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:14.735132 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:14.756385 18138 consensus_peers.cc:597] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:14.757567 18780 consensus_peers.cc:597] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:14.796211 18136 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.816779 29634 raft_consensus.cc:493] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:14.817792 29634 raft_consensus.cc:515] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } }
I20260709 12:27:14.826489 29634 leader_election.cc:290] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:14.828351 18138 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:14.838447 18138 consensus_peers.cc:597] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:14.841161 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:27:14.846693 18136 leader_election.cc:336] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:14.847383 18136 leader_election.cc:304] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae9641d5c3184572a102d51a75f670ee; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:14.848305 29634 raft_consensus.cc:2749] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:27:14.851456 18778 consensus_peers.cc:597] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:14.852413 16792 tablet_replica.cc:333] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.853526 16792 raft_consensus.cc:2243] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.854905 16792 raft_consensus.cc:2272] T 8e0ff0e5252f4ace898a73a2268e5dc8 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:14.857170 18136 consensus_peers.cc:597] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.859417 16792 tablet_replica.cc:333] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.860347 16792 raft_consensus.cc:2243] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.860982 16792 raft_consensus.cc:2272] T 5f144ec392b043d8b68e284ffecdee43 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.864068 16792 tablet_replica.cc:333] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.865242 16792 raft_consensus.cc:2243] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.866106 16792 raft_consensus.cc:2272] T 23e4eb0b942f4c7e81dfe76f4e015618 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.872916 16792 tablet_replica.cc:333] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.874425 16792 raft_consensus.cc:2243] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.876080 16792 raft_consensus.cc:2272] T 0420fe6ea0ab490e922b7c35c62b6a19 P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.877207 29645 raft_consensus.cc:493] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:14.878026 29645 raft_consensus.cc:515] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:14.880765 29645 leader_election.cc:290] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:14.885910 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:14.886656 16792 tablet_replica.cc:333] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.888067 16792 raft_consensus.cc:2243] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.889010 16792 raft_consensus.cc:2272] T 53b6c1b05fff4ff4ab4260c2c231ed5e P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.892961 16792 tablet_replica.cc:333] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.895011 16792 raft_consensus.cc:2243] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.897289 16792 raft_consensus.cc:2272] T 86bfc0ef6acd420b829a9a1df3376f48 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.902347 16792 tablet_replica.cc:333] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.903630 16792 raft_consensus.cc:2243] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Raft consensus shutting down.
W20260709 12:27:14.904240 18138 consensus_peers.cc:597] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:14.905320 16792 raft_consensus.cc:2272] T 49a68a064bb44579af6ee5ad50121ffc P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:14.909153 18778 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:14.909538 16792 tablet_replica.cc:333] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
W20260709 12:27:14.910454 18780 leader_election.cc:336] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:14.910826 16792 raft_consensus.cc:2243] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.911027 18780 leader_election.cc:304] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:14.913722 16792 raft_consensus.cc:2272] T c289d2c3e9fc40fbb7102b5788f74aab P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.915426 29654 raft_consensus.cc:2749] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:27:14.917546 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:14.917863 16792 tablet_replica.cc:333] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.919361 16792 raft_consensus.cc:2243] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.920347 16792 raft_consensus.cc:2272] T f1f3da088a314a87a78898050dab39e9 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.926775 16792 tablet_replica.cc:333] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.928018 16792 raft_consensus.cc:2243] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.929966 16792 raft_consensus.cc:2272] T 5c6e41c6a46d4fc99220183ed20b6921 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.934809 16792 tablet_replica.cc:333] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.935899 16792 raft_consensus.cc:2243] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.936794 16792 raft_consensus.cc:2272] T 3304247386d14b18a8bdd4f25ae8630c P ae9641d5c3184572a102d51a75f670ee [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.940974 16792 tablet_replica.cc:333] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.942162 16792 raft_consensus.cc:2243] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
W20260709 12:27:14.942020 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:14.945011 16792 raft_consensus.cc:2272] T fd8fe387b2164c30aa94afed5a7c218d P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:14.948419 18778 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.948669 16792 tablet_replica.cc:333] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.950309 16792 raft_consensus.cc:2243] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.951933 16792 raft_consensus.cc:2272] T 9fe4c9950f4d4f4588d08c7cd436b1dc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:14.954741 18780 consensus_peers.cc:597] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:14.955498 16792 tablet_replica.cc:333] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.956564 16792 raft_consensus.cc:2243] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.957335 16792 raft_consensus.cc:2272] T 2217b6329873485ead60419dc5acbff0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.960966 16792 tablet_replica.cc:333] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.962101 16792 raft_consensus.cc:2243] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.963600 16792 raft_consensus.cc:2272] T f0dc8b13a3294835a091ca099ee49b29 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.967202 16792 tablet_replica.cc:333] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.975637 16792 raft_consensus.cc:2243] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:14.977151 16792 raft_consensus.cc:2272] T e84e056b63c34d8da15d2fde3877279b P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:14.977337 18138 consensus_peers.cc:597] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:14.980542 16792 tablet_replica.cc:333] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.981593 16792 raft_consensus.cc:2243] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.983019 16792 raft_consensus.cc:2272] T 555e520f5efe4bafb5947749ee16e734 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.986130 16792 tablet_replica.cc:333] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.986999 16792 raft_consensus.cc:2243] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:14.988233 16792 raft_consensus.cc:2272] T a07f730a06794e2cb530edca57871763 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.991541 16792 tablet_replica.cc:333] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:14.992393 16792 raft_consensus.cc:2243] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 34 LEADER]: Raft consensus shutting down.
W20260709 12:27:14.993304 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:14.995026 18136 consensus_peers.cc:597] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:14.993721 16792 raft_consensus.cc:2272] T f7a707b110a44c8c97606fdb8d7dece0 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:14.999356 16792 tablet_replica.cc:333] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.000483 16792 raft_consensus.cc:2243] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.002036 16792 raft_consensus.cc:2272] T af9dff14df364074b9d5e7e769f198b5 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.005539 16792 tablet_replica.cc:333] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.006680 16792 raft_consensus.cc:2243] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.007791 16792 raft_consensus.cc:2272] T 7c3aa170318f4e1f95796426dc71d60b P ae9641d5c3184572a102d51a75f670ee [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.011866 16792 tablet_replica.cc:333] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.013140 16792 raft_consensus.cc:2243] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.014243 16792 raft_consensus.cc:2272] T a6ac73d7138d4200bbfcbf2f84c2c775 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.017795 16792 tablet_replica.cc:333] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.018934 16792 raft_consensus.cc:2243] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.020172 16792 raft_consensus.cc:2272] T 5c743e221c2c477e9b998557d9b9749a P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.023799 16792 tablet_replica.cc:333] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.024829 16792 raft_consensus.cc:2243] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.026424 16792 raft_consensus.cc:2272] T aeba5cb4432947efbaa2fa537357e992 P ae9641d5c3184572a102d51a75f670ee [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.033511 16792 tablet_replica.cc:333] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.034752 16792 raft_consensus.cc:2243] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.036305 16792 raft_consensus.cc:2272] T d440327a63754b889f7edd91e1373942 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.040134 16792 tablet_replica.cc:333] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.041316 16792 raft_consensus.cc:2243] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.042228 16792 raft_consensus.cc:2272] T 9ba0c897fcb346488dfe124a973cb2d1 P ae9641d5c3184572a102d51a75f670ee [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:15.044795 18138 consensus_peers.cc:597] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:15.046226 16792 tablet_replica.cc:333] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.052824 16792 raft_consensus.cc:2243] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:15.054178 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:15.054714 18780 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39905, user_credentials={real_user=slave}} blocked reactor thread for 46909.8us
I20260709 12:27:15.055256 16792 raft_consensus.cc:2272] T 666440ca642c4715bd2f211bb5f14d29 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.059221 16792 tablet_replica.cc:333] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.060252 16792 raft_consensus.cc:2243] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.061353 16792 raft_consensus.cc:2272] T a29e00b535474906948e66659110db3b P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.066556 16792 tablet_replica.cc:333] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.067775 16792 raft_consensus.cc:2243] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.068701 16792 raft_consensus.cc:2272] T 1236f612fda5419089c0c9aa244bb5bc P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.072137 16792 tablet_replica.cc:333] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.073222 16792 raft_consensus.cc:2243] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.074349 16792 raft_consensus.cc:2272] T 14d2c4b70e164856a205483966ca4963 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.077747 16792 tablet_replica.cc:333] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.079687 16792 raft_consensus.cc:2243] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.081300 16792 raft_consensus.cc:2272] T 95f78d7b0f7f422eacbfa07bbc248bfa P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:15.084399 18136 consensus_peers.cc:597] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:15.084708 16792 tablet_replica.cc:333] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.086790 16792 raft_consensus.cc:2243] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.088892 16792 raft_consensus.cc:2272] T 804b0137cb0b4850b912b6519bae7417 P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.093094 16792 tablet_replica.cc:333] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.094282 16792 raft_consensus.cc:2243] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.095845 16792 raft_consensus.cc:2272] T 47bac54b53844de9814420f86103a6f8 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.099354 16792 tablet_replica.cc:333] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.101802 16792 raft_consensus.cc:2243] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.103649 16792 raft_consensus.cc:2272] T d525bcd825eb49d9be935bc89de569c7 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.107074 16792 tablet_replica.cc:333] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.108052 16792 raft_consensus.cc:2243] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.119415 29654 raft_consensus.cc:493] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:15.120290 29654 raft_consensus.cc:515] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:15.123729 29654 leader_election.cc:290] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:27:15.126215 16792 raft_consensus.cc:2272] T 7cd93736a3314dbbb5c62b5af8fbed11 P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.130307 16792 tablet_replica.cc:333] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.131325 16792 raft_consensus.cc:2243] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.132807 16792 raft_consensus.cc:2272] T f9731c63bc7f406db2c77f5ef0415a78 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.136330 16792 tablet_replica.cc:333] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.137310 16792 raft_consensus.cc:2243] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:15.137974 18780 consensus_peers.cc:597] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:15.138208 16792 raft_consensus.cc:2272] T a83278d09c914df888a57be57eb08f8e P ae9641d5c3184572a102d51a75f670ee [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.153303 16792 tablet_replica.cc:333] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.154451 16792 raft_consensus.cc:2243] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
W20260709 12:27:15.155438 18780 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:15.155962 16792 raft_consensus.cc:2272] T 17018f4f3f0d4fd5a11cac014a5abd36 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:15.160308 18778 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:15.160714 18780 consensus_peers.cc:597] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:15.161621 18780 consensus_peers.cc:597] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:15.163547 16792 tablet_replica.cc:333] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
W20260709 12:27:15.166677 18780 leader_election.cc:336] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:15.167348 18780 leader_election.cc:304] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:15.168262 29645 raft_consensus.cc:2749] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:15.168315 16792 raft_consensus.cc:2243] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 LEADER]: Raft consensus shutting down.
W20260709 12:27:15.168571 18780 consensus_peers.cc:597] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:15.170069 16792 raft_consensus.cc:2272] T ff26d2026f1c4ff79de028b875b8ba96 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:15.170929 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:15.173419 16792 tablet_replica.cc:333] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.174499 16792 raft_consensus.cc:2243] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.175971 16792 raft_consensus.cc:2272] T 3412b19684e44e3d89ca8150a50a9d14 P ae9641d5c3184572a102d51a75f670ee [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:15.177752 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:15.179248 16792 tablet_replica.cc:333] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.180284 16792 raft_consensus.cc:2243] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 30 LEADER]: Raft consensus shutting down.
W20260709 12:27:15.180840 18780 consensus_peers.cc:597] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:15.181756 16792 raft_consensus.cc:2272] T 9cf56c3bba4a4641a72b648d02c669f6 P ae9641d5c3184572a102d51a75f670ee [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.185465 16792 tablet_replica.cc:333] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.186522 16792 raft_consensus.cc:2243] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.187988 16792 raft_consensus.cc:2272] T 5240f7371cc74cb48dffa1dadcbeec49 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.191447 16792 tablet_replica.cc:333] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.192477 16792 raft_consensus.cc:2243] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.193423 16792 raft_consensus.cc:2272] T b7fd19d9bf374e6da56b7ead11c666e9 P ae9641d5c3184572a102d51a75f670ee [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.196959 16792 tablet_replica.cc:333] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.198038 16792 raft_consensus.cc:2243] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.199792 16792 raft_consensus.cc:2272] T 52abd819c1c443e28c6c41a392a7ed6f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.203491 16792 tablet_replica.cc:333] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.204491 16792 raft_consensus.cc:2243] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.205854 16792 raft_consensus.cc:2272] T 4843c80e9d9348a397ce35e196deb9bd P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.209410 16792 tablet_replica.cc:333] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.210424 16792 raft_consensus.cc:2243] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.211246 16792 raft_consensus.cc:2272] T 583d61c844f048e4b53a66acc88cc1d9 P ae9641d5c3184572a102d51a75f670ee [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.215054 16792 tablet_replica.cc:333] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.216058 16792 raft_consensus.cc:2243] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.217345 16792 raft_consensus.cc:2272] T 85764dcebf974f7cbec322be36e3dd5f P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.220908 16792 tablet_replica.cc:333] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.221873 16792 raft_consensus.cc:2243] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.223490 16792 raft_consensus.cc:2272] T a3c91f3d9da3442f89360b0c316ee411 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.227196 16792 tablet_replica.cc:333] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.228098 16792 raft_consensus.cc:2243] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.231014 16792 raft_consensus.cc:2272] T 18093241e4df42869e3d96cdf66a53e6 P ae9641d5c3184572a102d51a75f670ee [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.234751 16792 tablet_replica.cc:333] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee: stopping tablet replica
I20260709 12:27:15.235545 16792 raft_consensus.cc:2243] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.237206 16792 raft_consensus.cc:2272] T aef65d70106f4f458174493864b4ba3d P ae9641d5c3184572a102d51a75f670ee [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.313066 29658 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:15.313931 29658 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:15.316763 29658 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:15.328588 18780 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:15.329756 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:15.330283 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:15.332314 29654 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:15.439385 29645 raft_consensus.cc:493] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:15.440155 29645 raft_consensus.cc:515] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:15.443156 29645 leader_election.cc:290] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:15.448745 18780 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:15.451454 18778 leader_election.cc:336] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:15.452132 18778 leader_election.cc:304] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:15.453074 29645 raft_consensus.cc:2749] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:27:15.584771 18780 consensus_peers.cc:597] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:15.642366 18780 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:15.662576 29645 raft_consensus.cc:493] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:15.663228 29645 raft_consensus.cc:515] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:15.667968 29645 leader_election.cc:290] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
W20260709 12:27:15.672569 18778 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:15.678093 18780 leader_election.cc:336] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:15.678675 18780 leader_election.cc:304] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:15.679608 29645 raft_consensus.cc:2749] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:27:15.679904 18780 consensus_peers.cc:597] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:15.690387 18780 consensus_peers.cc:597] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:15.774214 18780 consensus_peers.cc:597] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:15.885968 18780 consensus_peers.cc:597] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:15.903393 29645 raft_consensus.cc:493] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.904078 29645 raft_consensus.cc:515] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:15.906863 29645 leader_election.cc:290] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:15.914410 18780 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
W20260709 12:27:15.915441 18778 leader_election.cc:336] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:15.915930 18778 leader_election.cc:304] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:27:15.916850 29645 raft_consensus.cc:2749] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:27:16.079674 29658 raft_consensus.cc:493] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:27:16.080372 29658 raft_consensus.cc:515] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.083032 29658 leader_election.cc:290] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.089661 18780 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:16.090811 18780 leader_election.cc:336] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.091264 18780 leader_election.cc:304] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:16.092159 29658 raft_consensus.cc:2749] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:27:16.125538 29658 raft_consensus.cc:493] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:27:16.126293 29658 raft_consensus.cc:515] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.128846 29658 leader_election.cc:290] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.135875 18780 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:16.136961 18780 leader_election.cc:336] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.137465 18780 leader_election.cc:304] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:16.138438 29658 raft_consensus.cc:2749] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:27:16.240012 29658 raft_consensus.cc:493] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:27:16.240743 29658 raft_consensus.cc:515] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.243741 29658 leader_election.cc:290] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.250784 18780 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
I20260709 12:27:16.253361 29658 raft_consensus.cc:493] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
W20260709 12:27:16.253708 18780 leader_election.cc:336] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.254381 18780 leader_election.cc:304] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: ae9641d5c3184572a102d51a75f670ee, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:16.254387 29658 raft_consensus.cc:515] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.255301 29645 raft_consensus.cc:2749] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:16.261453 29658 leader_election.cc:290] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.264609 18778 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:16.270545 18780 leader_election.cc:336] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.271117 18780 leader_election.cc:304] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:27:16.272014 29658 raft_consensus.cc:2749] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:27:16.313531 18780 consensus_peers.cc:597] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:16.377911 18780 consensus_peers.cc:597] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:16.398351 18778 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:16.555164 18780 consensus_peers.cc:597] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Couldn't send request to peer cab4a2f5d3ca48a2a803ee8ec3906e1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:16.575877 29658 raft_consensus.cc:493] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:27:16.576683 29658 raft_consensus.cc:515] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.579663 29658 leader_election.cc:290] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.590103 18780 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.591164 29658 raft_consensus.cc:493] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:27:16.591773 29658 raft_consensus.cc:515] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
W20260709 12:27:16.592164 18778 consensus_peers.cc:597] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:16.592859 18778 leader_election.cc:336] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:16.593315 18778 leader_election.cc:304] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:27:16.594204 29645 raft_consensus.cc:2749] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:27:16.594205 29658 leader_election.cc:290] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.604568 18778 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:16.611577 18780 leader_election.cc:336] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.612128 18780 leader_election.cc:304] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:27:16.615099 29645 raft_consensus.cc:2749] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:27:16.648161 29669 raft_consensus.cc:493] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader ae9641d5c3184572a102d51a75f670ee)
I20260709 12:27:16.648824 29669 raft_consensus.cc:515] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "ae9641d5c3184572a102d51a75f670ee" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39905 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.651290 29669 leader_election.cc:290] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047), ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905)
W20260709 12:27:16.659029 18778 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
W20260709 12:27:16.662364 18780 leader_election.cc:336] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111)
I20260709 12:27:16.662887 18780 leader_election.cc:304] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, ae9641d5c3184572a102d51a75f670ee
I20260709 12:27:16.663718 29669 raft_consensus.cc:2749] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:27:16.673689 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
W20260709 12:27:16.791193 18778 consensus_peers.cc:597] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:16.814429 16792 tablet_server.cc:179] TabletServer@127.16.102.4:0 shutting down...
W20260709 12:27:16.839365 18778 consensus_peers.cc:597] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:16.873741 18778 consensus_peers.cc:597] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Couldn't send request to peer 768723648a9444fda53659e6a8cc3091. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:16.900871 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:16.901944 16792 tablet_replica.cc:333] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.903049 16792 raft_consensus.cc:2243] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.903899 16792 raft_consensus.cc:2272] T aef65d70106f4f458174493864b4ba3d P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.907630 16792 tablet_replica.cc:333] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.908468 16792 raft_consensus.cc:2243] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.910002 16792 raft_consensus.cc:2272] T a6ac73d7138d4200bbfcbf2f84c2c775 P 910f8c8d787e4abbb092413c1f90ebf3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.912719 16792 tablet_replica.cc:333] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.913483 16792 raft_consensus.cc:2243] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.914875 16792 raft_consensus.cc:2272] T 85764dcebf974f7cbec322be36e3dd5f P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.917500 16792 tablet_replica.cc:333] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.918412 16792 raft_consensus.cc:2243] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.919329 16792 raft_consensus.cc:2272] T 0420fe6ea0ab490e922b7c35c62b6a19 P 910f8c8d787e4abbb092413c1f90ebf3 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.923005 16792 tablet_replica.cc:333] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.923941 16792 raft_consensus.cc:2243] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.925320 16792 raft_consensus.cc:2272] T b5eedbd3c223402caf9a71dac2fcaa98 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.928555 16792 tablet_replica.cc:333] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.929255 16792 raft_consensus.cc:2243] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.930465 16792 raft_consensus.cc:2272] T acbc4a634f954522904f2b865531f456 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.933277 16792 tablet_replica.cc:333] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.934131 16792 raft_consensus.cc:2243] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.935292 16792 raft_consensus.cc:2272] T 3304247386d14b18a8bdd4f25ae8630c P 910f8c8d787e4abbb092413c1f90ebf3 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.938777 16792 tablet_replica.cc:333] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.939743 16792 raft_consensus.cc:2243] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.941082 16792 raft_consensus.cc:2272] T 4843c80e9d9348a397ce35e196deb9bd P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.945009 16792 tablet_replica.cc:333] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
W20260709 12:27:16.944839 18780 consensus_peers.cc:597] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 -> Peer ae9641d5c3184572a102d51a75f670ee (127.16.102.3:39905): Couldn't send request to peer ae9641d5c3184572a102d51a75f670ee. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:16.946030 16792 raft_consensus.cc:2243] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.947278 16792 raft_consensus.cc:2272] T c53769058ca3443f805983b61a69328e P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.950176 16792 tablet_replica.cc:333] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.950896 16792 raft_consensus.cc:2243] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.951629 16792 raft_consensus.cc:2272] T 5240f7371cc74cb48dffa1dadcbeec49 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.954265 16792 tablet_replica.cc:333] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.954929 16792 raft_consensus.cc:2243] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.955452 16792 raft_consensus.cc:2272] T 95f78d7b0f7f422eacbfa07bbc248bfa P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.958123 16792 tablet_replica.cc:333] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.958855 16792 raft_consensus.cc:2243] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.959930 16792 pending_rounds.cc:70] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Trying to abort 1 pending ops.
I20260709 12:27:16.960223 16792 pending_rounds.cc:77] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3: Aborting op as it isn't in flight: id { term: 32 index: 24 } timestamp: 7305625740203085824 op_type: NO_OP noop_request { }
I20260709 12:27:16.960611 16792 raft_consensus.cc:2889] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:27:16.960938 16792 raft_consensus.cc:2272] T b7fd19d9bf374e6da56b7ead11c666e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.961189 29658 raft_consensus.cc:493] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 768723648a9444fda53659e6a8cc3091)
I20260709 12:27:16.961843 29658 raft_consensus.cc:515] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cab4a2f5d3ca48a2a803ee8ec3906e1d" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 45965 } } peers { permanent_uuid: "768723648a9444fda53659e6a8cc3091" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 40047 } } peers { permanent_uuid: "910f8c8d787e4abbb092413c1f90ebf3" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 46307 } attrs { promote: false } }
I20260709 12:27:16.964402 29658 leader_election.cc:290] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965), 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047)
I20260709 12:27:16.964649 16792 tablet_replica.cc:333] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.965691 16792 raft_consensus.cc:2243] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.966449 16792 raft_consensus.cc:2272] T c289d2c3e9fc40fbb7102b5788f74aab P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.970062 16792 tablet_replica.cc:333] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
W20260709 12:27:16.970137 18780 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:16.970733 18778 leader_election.cc:336] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 768723648a9444fda53659e6a8cc3091 (127.16.102.2:40047): Network error: Client connection negotiation failed: client connection to 127.16.102.2:40047: connect: Connection refused (error 111)
I20260709 12:27:16.971202 16792 raft_consensus.cc:2243] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.971201 18778 leader_election.cc:304] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 910f8c8d787e4abbb092413c1f90ebf3; no voters: 768723648a9444fda53659e6a8cc3091, cab4a2f5d3ca48a2a803ee8ec3906e1d
I20260709 12:27:16.972719 16792 raft_consensus.cc:2272] T 4cc94b1c76be48d6813e7d1c89a4d66c P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.972704 29658 raft_consensus.cc:2749] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:16.976254 16792 tablet_replica.cc:333] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.977062 16792 raft_consensus.cc:2243] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.978341 16792 raft_consensus.cc:2272] T 14d2c4b70e164856a205483966ca4963 P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.981355 16792 tablet_replica.cc:333] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.982107 16792 raft_consensus.cc:2243] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.983320 16792 raft_consensus.cc:2272] T 1236f612fda5419089c0c9aa244bb5bc P 910f8c8d787e4abbb092413c1f90ebf3 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.986415 16792 tablet_replica.cc:333] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.987174 16792 raft_consensus.cc:2243] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.987821 16792 raft_consensus.cc:2272] T ef30e67b67894670b0cf3980d6b0f624 P 910f8c8d787e4abbb092413c1f90ebf3 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.990660 16792 tablet_replica.cc:333] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.991361 16792 raft_consensus.cc:2243] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.992511 16792 raft_consensus.cc:2272] T f1f3da088a314a87a78898050dab39e9 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.995551 16792 tablet_replica.cc:333] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:16.996295 16792 raft_consensus.cc:2243] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.997507 16792 raft_consensus.cc:2272] T 578328e4e1fd478cbc58dd1048417480 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.000437 16792 tablet_replica.cc:333] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.001188 16792 raft_consensus.cc:2243] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:17.001852 16792 raft_consensus.cc:2272] T 7e793f93c4f740b78f31d2f0dd643095 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.004492 16792 tablet_replica.cc:333] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.005204 16792 raft_consensus.cc:2243] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:27:17.006383 16792 raft_consensus.cc:2272] T e84e056b63c34d8da15d2fde3877279b P 910f8c8d787e4abbb092413c1f90ebf3 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.009060 16792 tablet_replica.cc:333] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.010061 16792 raft_consensus.cc:2243] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:17.011279 16792 raft_consensus.cc:2272] T a29e00b535474906948e66659110db3b P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.014127 16792 tablet_replica.cc:333] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.014767 16792 raft_consensus.cc:2243] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:17.015930 16792 raft_consensus.cc:2272] T 666440ca642c4715bd2f211bb5f14d29 P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.018785 16792 tablet_replica.cc:333] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.019629 16792 raft_consensus.cc:2243] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:17.020434 16792 raft_consensus.cc:2272] T f7a707b110a44c8c97606fdb8d7dece0 P 910f8c8d787e4abbb092413c1f90ebf3 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.023990 16792 tablet_replica.cc:333] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.024752 16792 raft_consensus.cc:2243] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:17.026211 16792 raft_consensus.cc:2272] T a83278d09c914df888a57be57eb08f8e P 910f8c8d787e4abbb092413c1f90ebf3 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.029332 16792 tablet_replica.cc:333] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.030061 16792 raft_consensus.cc:2243] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:17.030818 16792 raft_consensus.cc:2272] T 18093241e4df42869e3d96cdf66a53e6 P 910f8c8d787e4abbb092413c1f90ebf3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.033420 16792 tablet_replica.cc:333] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.034188 16792 raft_consensus.cc:2243] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:17.035388 16792 raft_consensus.cc:2272] T 53b6c1b05fff4ff4ab4260c2c231ed5e P 910f8c8d787e4abbb092413c1f90ebf3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.038472 16792 tablet_replica.cc:333] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.039214 16792 raft_consensus.cc:2243] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:17.040045 18780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39905: connect: Connection refused (error 111) [suppressed 641 similar messages]
I20260709 12:27:17.040446 16792 raft_consensus.cc:2272] T f9731c63bc7f406db2c77f5ef0415a78 P 910f8c8d787e4abbb092413c1f90ebf3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.043743 16792 tablet_replica.cc:333] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.044581 16792 raft_consensus.cc:2243] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:17.045521 16792 raft_consensus.cc:2272] T da76d5c707544d29a2f74d6015ec1cfc P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.048461 16792 tablet_replica.cc:333] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.049132 16792 raft_consensus.cc:2243] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:17.050437 16792 raft_consensus.cc:2272] T 5f144ec392b043d8b68e284ffecdee43 P 910f8c8d787e4abbb092413c1f90ebf3 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.053201 16792 tablet_replica.cc:333] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3: stopping tablet replica
I20260709 12:27:17.053793 16792 raft_consensus.cc:2243] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:17.054486 16792 raft_consensus.cc:2272] T df3d21a581ef40feafcce4ab8b817645 P 910f8c8d787e4abbb092413c1f90ebf3 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:17.653064 16792 tablet_server.cc:196] TabletServer@127.16.102.4:0 shutdown complete.
I20260709 12:27:17.731331 16792 master.cc:562] Master@127.16.102.62:42293 shutting down...
W20260709 12:27:17.771847 17970 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:27:17.772321 17970 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:27:17.772544 17970 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:27:19.437227 17908 catalog_manager.cc:4755] TS cab4a2f5d3ca48a2a803ee8ec3906e1d (127.16.102.1:45965): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f9731c63bc7f406db2c77f5ef0415a78: Network error: Client connection negotiation failed: client connection to 127.16.102.1:45965: connect: Connection refused (error 111)
W20260709 12:27:34.832013 17907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.4:46307: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260709 12:27:34.836532 17907 catalog_manager.cc:4755] TS 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): ChangeConfig:REMOVE_PEER RPC failed for tablet a6ac73d7138d4200bbfcbf2f84c2c775: Network error: Client connection negotiation failed: client connection to 127.16.102.4:46307: connect: Connection refused (error 111)
W20260709 12:27:39.398471 17907 catalog_manager.cc:4755] TS 910f8c8d787e4abbb092413c1f90ebf3 (127.16.102.4:46307): ChangeConfig:REMOVE_PEER RPC failed for tablet 85764dcebf974f7cbec322be36e3dd5f: Network error: Client connection negotiation failed: client connection to 127.16.102.4:46307: connect: Connection refused (error 111)
I20260709 12:27:44.328550 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:44.330083 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:44.330615 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7b17aa4109194c3c9f5ca41c3594e03c: stopping tablet replica
I20260709 12:27:44.673954 16792 master.cc:584] Master@127.16.102.62:42293 shutdown complete.
I20260709 12:27:44.739147 16792 test_util.cc:182] -----------------------------------------------
I20260709 12:27:44.739468 16792 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599579151043-16792-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (421078 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:27:44.744300 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:33633
I20260709 12:27:44.745810 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:44.753058 29679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:44.753572 29680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:44.755069 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:44.756412 16792 server_base.cc:1061] running on GCE node
I20260709 12:27:44.762882 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:44.763175 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:44.763288 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600064763277 us; error 0 us; skew 500 ppm
I20260709 12:27:44.763805 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:44.766855 16792 webserver.cc:533] Webserver started at http://127.16.102.62:43481/ using document root <none> and password file <none>
I20260709 12:27:44.767406 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:44.767568 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:44.767822 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:44.769411 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "9599a6d4076543719333b3d5fc20475a"
format_stamp: "Formatted at 2026-07-09 12:27:44 on dist-test-slave-7xt3"
I20260709 12:27:44.774744 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:27:44.778820 29687 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:44.779786 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
I20260709 12:27:44.780201 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "9599a6d4076543719333b3d5fc20475a"
format_stamp: "Formatted at 2026-07-09 12:27:44 on dist-test-slave-7xt3"
I20260709 12:27:44.780499 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:44.799098 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:44.800783 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:44.849675 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:33633
I20260709 12:27:44.849797 29738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:33633 every 8 connection(s)
I20260709 12:27:44.854779 29739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:44.869428 29739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a: Bootstrap starting.
I20260709 12:27:44.875578 29739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.880869 29739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a: No bootstrap required, opened a new log
I20260709 12:27:44.883929 29739 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9599a6d4076543719333b3d5fc20475a" member_type: VOTER }
I20260709 12:27:44.884495 29739 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.884778 29739 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9599a6d4076543719333b3d5fc20475a, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.885458 29739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9599a6d4076543719333b3d5fc20475a" member_type: VOTER }
I20260709 12:27:44.886050 29739 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:44.886322 29739 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:44.886636 29739 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.892997 29739 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9599a6d4076543719333b3d5fc20475a" member_type: VOTER }
I20260709 12:27:44.893698 29739 leader_election.cc:304] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9599a6d4076543719333b3d5fc20475a; no voters: 
I20260709 12:27:44.895143 29739 leader_election.cc:290] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:44.895656 29742 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:44.897258 29742 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 1 LEADER]: Becoming Leader. State: Replica: 9599a6d4076543719333b3d5fc20475a, State: Running, Role: LEADER
I20260709 12:27:44.898175 29742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9599a6d4076543719333b3d5fc20475a" member_type: VOTER }
I20260709 12:27:44.898713 29739 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:44.905686 29743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9599a6d4076543719333b3d5fc20475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9599a6d4076543719333b3d5fc20475a" member_type: VOTER } }
I20260709 12:27:44.905686 29744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9599a6d4076543719333b3d5fc20475a. Latest consensus state: current_term: 1 leader_uuid: "9599a6d4076543719333b3d5fc20475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9599a6d4076543719333b3d5fc20475a" member_type: VOTER } }
I20260709 12:27:44.906705 29743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:44.906841 29744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:44.912950 29750 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:44.921885 29750 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:44.923348 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:44.934387 29750 catalog_manager.cc:1383] Generated new cluster ID: 6a2d20e03a434d778babf577440a6309
I20260709 12:27:44.934794 29750 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:44.952029 29750 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:44.953619 29750 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:44.970762 29750 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a: Generated new TSK 0
I20260709 12:27:44.971567 29750 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:44.991598 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:44.999405 29760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:45.001130 29761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:45.004264 29763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:45.004964 16792 server_base.cc:1061] running on GCE node
I20260709 12:27:45.009078 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:45.009333 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:45.009469 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600065009456 us; error 0 us; skew 500 ppm
I20260709 12:27:45.010182 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:45.013139 16792 webserver.cc:533] Webserver started at http://127.16.102.1:35873/ using document root <none> and password file <none>
I20260709 12:27:45.013720 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:45.013999 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:45.014330 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:45.015794 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "1ae5481d76bf4a90a0ee824a14223421"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.021257 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:27:45.025369 29768 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.026438 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:27:45.026873 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "1ae5481d76bf4a90a0ee824a14223421"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.027309 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:45.050329 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:45.051746 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:45.053370 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:45.056156 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:45.056350 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.056689 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:45.056874 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.107883 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:42003
I20260709 12:27:45.108006 29830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:42003 every 8 connection(s)
I20260709 12:27:45.114152 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:45.124826 29835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:45.127756 29836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:45.130440 29838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:45.131014 16792 server_base.cc:1061] running on GCE node
I20260709 12:27:45.133785 29831 heartbeater.cc:344] Connected to a master server at 127.16.102.62:33633
I20260709 12:27:45.134362 29831 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:45.135720 29831 heartbeater.cc:507] Master 127.16.102.62:33633 requested a full tablet report, sending...
I20260709 12:27:45.137030 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:45.137329 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:45.137604 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600065137571 us; error 0 us; skew 500 ppm
I20260709 12:27:45.138397 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:45.139132 29704 ts_manager.cc:194] Registered new tserver with Master: 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:45.141584 16792 webserver.cc:533] Webserver started at http://127.16.102.2:41567/ using document root <none> and password file <none>
I20260709 12:27:45.142124 29704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56132
I20260709 12:27:45.142287 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:45.142649 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:45.142977 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:45.144910 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.151190 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:27:45.155948 29843 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.156965 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:27:45.157289 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.157580 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:45.186007 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:45.187508 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:45.189258 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:45.192030 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:45.192308 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.192603 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:45.192749 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.244891 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:35595
I20260709 12:27:45.245015 29905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:35595 every 8 connection(s)
I20260709 12:27:45.251390 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:45.261482 29910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:45.261615 29909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:45.265401 16792 server_base.cc:1061] running on GCE node
I20260709 12:27:45.266160 29906 heartbeater.cc:344] Connected to a master server at 127.16.102.62:33633
I20260709 12:27:45.266683 29906 heartbeater.cc:461] Registering TS with master...
W20260709 12:27:45.266943 29913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:45.267839 29906 heartbeater.cc:507] Master 127.16.102.62:33633 requested a full tablet report, sending...
I20260709 12:27:45.268286 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:45.268622 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:45.268851 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600065268828 us; error 0 us; skew 500 ppm
I20260709 12:27:45.269618 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:45.270562 29704 ts_manager.cc:194] Registered new tserver with Master: b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:45.272464 29704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56134
I20260709 12:27:45.272676 16792 webserver.cc:533] Webserver started at http://127.16.102.3:46437/ using document root <none> and password file <none>
I20260709 12:27:45.273347 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:45.273595 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:45.273972 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:45.276091 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.281311 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:27:45.285429 29917 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.286445 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:45.286828 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.287175 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:45.297714 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:45.299355 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:45.301635 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:45.304459 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:45.304690 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.304893 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:45.305024 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.364926 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:39029
I20260709 12:27:45.365047 29979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:39029 every 8 connection(s)
I20260709 12:27:45.371315 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:45.380589 29983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:45.382373 29984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:45.386252 29980 heartbeater.cc:344] Connected to a master server at 127.16.102.62:33633
W20260709 12:27:45.386399 29986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:45.387096 29980 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:45.387178 16792 server_base.cc:1061] running on GCE node
I20260709 12:27:45.388350 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:27:45.388312 29980 heartbeater.cc:507] Master 127.16.102.62:33633 requested a full tablet report, sending...
W20260709 12:27:45.388741 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:45.389003 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600065388984 us; error 0 us; skew 500 ppm
I20260709 12:27:45.389925 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:45.391278 29704 ts_manager.cc:194] Registered new tserver with Master: 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:45.393239 16792 webserver.cc:533] Webserver started at http://127.16.102.4:41943/ using document root <none> and password file <none>
I20260709 12:27:45.393339 29704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56142
I20260709 12:27:45.394177 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:45.394472 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:45.394774 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:45.396831 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-3-root/instance:
uuid: "1e2ceabfe50c47b49529b663fa133b86"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.402756 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:27:45.407518 29991 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.408658 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.001s
I20260709 12:27:45.409173 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-3-root
uuid: "1e2ceabfe50c47b49529b663fa133b86"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:27:45.409541 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:45.422479 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:45.424211 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:45.426260 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:45.430040 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:45.430356 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.430629 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:45.430814 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:45.482604 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.4:40301
I20260709 12:27:45.482771 30053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.4:40301 every 8 connection(s)
I20260709 12:27:45.499577 30054 heartbeater.cc:344] Connected to a master server at 127.16.102.62:33633
I20260709 12:27:45.500149 30054 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:45.501237 30054 heartbeater.cc:507] Master 127.16.102.62:33633 requested a full tablet report, sending...
I20260709 12:27:45.503718 29704 ts_manager.cc:194] Registered new tserver with Master: 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:45.504791 16792 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01818226s
I20260709 12:27:45.505400 29704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56150
I20260709 12:27:45.537647 29704 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56160:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:27:45.827953 29794 tablet_service.cc:1511] Processing CreateTablet for tablet 8a7202a2bfb344cdbb49a8cb58ac6b52 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:27:45.828542 29796 tablet_service.cc:1511] Processing CreateTablet for tablet d4a3b54cd9af44d69698e15cc4bccafa (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:27:45.830886 29795 tablet_service.cc:1511] Processing CreateTablet for tablet 80eac7cb175d4f0d84a00bd792dfd476 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:27:45.831648 29793 tablet_service.cc:1511] Processing CreateTablet for tablet 1f944367bb734866841edc565647828b (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:27:45.834214 29790 tablet_service.cc:1511] Processing CreateTablet for tablet f8c4c5d911ee4bcbbfb014114223cc07 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:27:45.834939 29792 tablet_service.cc:1511] Processing CreateTablet for tablet adaa2559c49e4cba8e82a850b5004d98 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:27:45.838164 29791 tablet_service.cc:1511] Processing CreateTablet for tablet 6872c9bba3c34283ac8c1a5a72884f7d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:27:45.839073 29789 tablet_service.cc:1511] Processing CreateTablet for tablet abceb9531803436f8fd8d02595d73cf4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:27:45.840451 29794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a7202a2bfb344cdbb49a8cb58ac6b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.841379 29795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80eac7cb175d4f0d84a00bd792dfd476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.846345 29789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abceb9531803436f8fd8d02595d73cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.850615 29793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f944367bb734866841edc565647828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.854599 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c4c5d911ee4bcbbfb014114223cc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.858618 29792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adaa2559c49e4cba8e82a850b5004d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.860555 29791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6872c9bba3c34283ac8c1a5a72884f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.861339 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4a3b54cd9af44d69698e15cc4bccafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.872134 29788 tablet_service.cc:1511] Processing CreateTablet for tablet ad89089259844673bbd8eeef685bb362 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:27:45.874060 29788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad89089259844673bbd8eeef685bb362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.899094 29865 tablet_service.cc:1511] Processing CreateTablet for tablet f8c4c5d911ee4bcbbfb014114223cc07 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:27:45.900985 29865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c4c5d911ee4bcbbfb014114223cc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.903873 29870 tablet_service.cc:1511] Processing CreateTablet for tablet 8a7202a2bfb344cdbb49a8cb58ac6b52 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:27:45.907330 29869 tablet_service.cc:1511] Processing CreateTablet for tablet 325254299d324d0b950855283ec2fcf0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:27:45.910837 29871 tablet_service.cc:1511] Processing CreateTablet for tablet 80eac7cb175d4f0d84a00bd792dfd476 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:27:45.911347 29866 tablet_service.cc:1511] Processing CreateTablet for tablet 7a554edb2449436ea89ed389c4aed651 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:27:45.914207 29867 tablet_service.cc:1511] Processing CreateTablet for tablet adaa2559c49e4cba8e82a850b5004d98 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
W20260709 12:27:45.917511 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:45.917768 29787 tablet_service.cc:1511] Processing CreateTablet for tablet 01f4a9dd66ad4df0a8f5a598a7b432fd (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:27:45.920365 29787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f4a9dd66ad4df0a8f5a598a7b432fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.905682 29870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a7202a2bfb344cdbb49a8cb58ac6b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.924930 29868 tablet_service.cc:1511] Processing CreateTablet for tablet 1f944367bb734866841edc565647828b (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:27:45.931062 30019 tablet_service.cc:1511] Processing CreateTablet for tablet d4a3b54cd9af44d69698e15cc4bccafa (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:27:45.932862 30019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4a3b54cd9af44d69698e15cc4bccafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.935497 30018 tablet_service.cc:1511] Processing CreateTablet for tablet 8a7202a2bfb344cdbb49a8cb58ac6b52 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:27:45.937372 30018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a7202a2bfb344cdbb49a8cb58ac6b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.939443 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 325254299d324d0b950855283ec2fcf0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:27:45.942695 29942 tablet_service.cc:1511] Processing CreateTablet for tablet 1f944367bb734866841edc565647828b (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:27:45.943295 29945 tablet_service.cc:1511] Processing CreateTablet for tablet d4a3b54cd9af44d69698e15cc4bccafa (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:27:45.944542 29942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f944367bb734866841edc565647828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.947322 30016 tablet_service.cc:1511] Processing CreateTablet for tablet adaa2559c49e4cba8e82a850b5004d98 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:27:45.950430 29944 tablet_service.cc:1511] Processing CreateTablet for tablet 80eac7cb175d4f0d84a00bd792dfd476 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:27:45.953534 29943 tablet_service.cc:1511] Processing CreateTablet for tablet 325254299d324d0b950855283ec2fcf0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:27:45.956540 29941 tablet_service.cc:1511] Processing CreateTablet for tablet 6872c9bba3c34283ac8c1a5a72884f7d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:27:45.960954 29794 tablet_service.cc:1511] Processing CreateTablet for tablet d49dfe78f7164655b82bfc664fbd11dd (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:27:45.986760 29794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d49dfe78f7164655b82bfc664fbd11dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.987298 29863 tablet_service.cc:1511] Processing CreateTablet for tablet 15ead15e87d345f48f74ca65fb97d5c5 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:27:45.961493 29940 tablet_service.cc:1511] Processing CreateTablet for tablet 7a554edb2449436ea89ed389c4aed651 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:27:45.992380 29864 tablet_service.cc:1511] Processing CreateTablet for tablet abceb9531803436f8fd8d02595d73cf4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:27:45.990435 29862 tablet_service.cc:1511] Processing CreateTablet for tablet 01f4a9dd66ad4df0a8f5a598a7b432fd (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:27:45.964707 29938 tablet_service.cc:1511] Processing CreateTablet for tablet ad89089259844673bbd8eeef685bb362 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:27:45.966216 29937 tablet_service.cc:1511] Processing CreateTablet for tablet 15ead15e87d345f48f74ca65fb97d5c5 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:27:45.967665 29936 tablet_service.cc:1511] Processing CreateTablet for tablet 01f4a9dd66ad4df0a8f5a598a7b432fd (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:27:45.969182 30015 tablet_service.cc:1511] Processing CreateTablet for tablet 6872c9bba3c34283ac8c1a5a72884f7d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:27:45.963080 29939 tablet_service.cc:1511] Processing CreateTablet for tablet f8c4c5d911ee4bcbbfb014114223cc07 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:27:45.997953 29793 tablet_service.cc:1511] Processing CreateTablet for tablet ffc3696cf98a402bb9d717b584cb229c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:27:45.973421 29788 tablet_service.cc:1511] Processing CreateTablet for tablet 80a7001c5f324ba1955a1eef4ea15e93 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:27:45.981388 30010 tablet_service.cc:1511] Processing CreateTablet for tablet 80a7001c5f324ba1955a1eef4ea15e93 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:27:45.975622 30013 tablet_service.cc:1511] Processing CreateTablet for tablet abceb9531803436f8fd8d02595d73cf4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:27:45.979434 30011 tablet_service.cc:1511] Processing CreateTablet for tablet 15ead15e87d345f48f74ca65fb97d5c5 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:27:45.984736 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325254299d324d0b950855283ec2fcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.977294 30012 tablet_service.cc:1511] Processing CreateTablet for tablet ad89089259844673bbd8eeef685bb362 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:27:46.007797 30010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a7001c5f324ba1955a1eef4ea15e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.008044 29789 tablet_service.cc:1511] Processing CreateTablet for tablet ba9fb933bf47466999dfd7892e6a6638 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:27:46.015424 30011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15ead15e87d345f48f74ca65fb97d5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.016108 29795 tablet_service.cc:1511] Processing CreateTablet for tablet 1b5584cc51f14a9a8dcee35a9998b380 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:27:45.950484 29945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4a3b54cd9af44d69698e15cc4bccafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.018929 30015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6872c9bba3c34283ac8c1a5a72884f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.973737 30014 tablet_service.cc:1511] Processing CreateTablet for tablet 7a554edb2449436ea89ed389c4aed651 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:27:46.002074 29788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a7001c5f324ba1955a1eef4ea15e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.024298 30014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a554edb2449436ea89ed389c4aed651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.026270 29944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80eac7cb175d4f0d84a00bd792dfd476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:45.969853 29869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325254299d324d0b950855283ec2fcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.032199 29937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15ead15e87d345f48f74ca65fb97d5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.040599 29939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c4c5d911ee4bcbbfb014114223cc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.050447 29867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adaa2559c49e4cba8e82a850b5004d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.054404 29940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a554edb2449436ea89ed389c4aed651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.055491 29796 tablet_service.cc:1511] Processing CreateTablet for tablet a4bd785dff644d0a8e536827c0304894 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:27:46.057291 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4bd785dff644d0a8e536827c0304894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.058379 29941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6872c9bba3c34283ac8c1a5a72884f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.062371 29866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a554edb2449436ea89ed389c4aed651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.066371 29936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f4a9dd66ad4df0a8f5a598a7b432fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.068171 29862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f4a9dd66ad4df0a8f5a598a7b432fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.078501 29871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80eac7cb175d4f0d84a00bd792dfd476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.082414 29938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad89089259844673bbd8eeef685bb362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.085696 29794 tablet_service.cc:1511] Processing CreateTablet for tablet dd88542c59c449a0904ad8b002efc5a0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:27:46.087545 29794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd88542c59c449a0904ad8b002efc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.090440 29790 tablet_service.cc:1511] Processing CreateTablet for tablet b820ed22868a4fe98bd2c74221e3e300 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:27:46.092496 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b820ed22868a4fe98bd2c74221e3e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.094213 29793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc3696cf98a402bb9d717b584cb229c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.098435 29868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f944367bb734866841edc565647828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.101512 30012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad89089259844673bbd8eeef685bb362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.110419 29863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15ead15e87d345f48f74ca65fb97d5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.113600 29789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9fb933bf47466999dfd7892e6a6638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.126483 29864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abceb9531803436f8fd8d02595d73cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.130241 30074 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:46.131814 30011 tablet_service.cc:1511] Processing CreateTablet for tablet ba9fb933bf47466999dfd7892e6a6638 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:27:46.126483 30016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adaa2559c49e4cba8e82a850b5004d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.137655 29795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5584cc51f14a9a8dcee35a9998b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.138665 30013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abceb9531803436f8fd8d02595d73cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.153983 30075 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:46.147001 29831 heartbeater.cc:499] Master 127.16.102.62:33633 was elected leader, sending a full tablet report...
I20260709 12:27:46.159246 30018 tablet_service.cc:1511] Processing CreateTablet for tablet 34b3b0e1b5d14def8b90e20a7a38ee24 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:27:46.164460 29794 tablet_service.cc:1511] Processing CreateTablet for tablet ff934fd9fe834c2ab8737af23cdc9505 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:27:46.152434 30011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9fb933bf47466999dfd7892e6a6638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.166240 29794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff934fd9fe834c2ab8737af23cdc9505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.168320 30074 tablet_bootstrap.cc:654] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.169989 29943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325254299d324d0b950855283ec2fcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.178365 30018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b3b0e1b5d14def8b90e20a7a38ee24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.231637 30076 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:46.239902 29865 tablet_service.cc:1511] Processing CreateTablet for tablet a4bd785dff644d0a8e536827c0304894 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:27:46.242475 29865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4bd785dff644d0a8e536827c0304894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.281174 29939 tablet_service.cc:1511] Processing CreateTablet for tablet 34b3b0e1b5d14def8b90e20a7a38ee24 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:27:46.285727 29906 heartbeater.cc:499] Master 127.16.102.62:33633 was elected leader, sending a full tablet report...
I20260709 12:27:46.286999 29939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b3b0e1b5d14def8b90e20a7a38ee24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.289961 30075 tablet_bootstrap.cc:654] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.306855 29791 tablet_service.cc:1511] Processing CreateTablet for tablet 4a33c77154df4be8aee967d4d7df39b0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:27:46.308560 29791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a33c77154df4be8aee967d4d7df39b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.310546 29868 tablet_service.cc:1511] Processing CreateTablet for tablet 34b3b0e1b5d14def8b90e20a7a38ee24 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:27:46.318758 30076 tablet_bootstrap.cc:654] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.336315 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 611c12107cde4b0e81c03ffd67cb64e2 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:27:46.340018 30016 tablet_service.cc:1511] Processing CreateTablet for tablet ffc3696cf98a402bb9d717b584cb229c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:27:46.345546 29870 tablet_service.cc:1511] Processing CreateTablet for tablet d49dfe78f7164655b82bfc664fbd11dd (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:27:46.346662 29936 tablet_service.cc:1511] Processing CreateTablet for tablet d49dfe78f7164655b82bfc664fbd11dd (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:27:46.367100 30077 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:46.367156 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611c12107cde4b0e81c03ffd67cb64e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.375078 29936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d49dfe78f7164655b82bfc664fbd11dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.379206 30016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc3696cf98a402bb9d717b584cb229c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.380642 30015 tablet_service.cc:1511] Processing CreateTablet for tablet 1b5584cc51f14a9a8dcee35a9998b380 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:27:46.382479 30015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5584cc51f14a9a8dcee35a9998b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.317499 29866 tablet_service.cc:1511] Processing CreateTablet for tablet 80a7001c5f324ba1955a1eef4ea15e93 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:27:46.397431 29980 heartbeater.cc:499] Master 127.16.102.62:33633 was elected leader, sending a full tablet report...
I20260709 12:27:46.404621 29940 tablet_service.cc:1511] Processing CreateTablet for tablet b820ed22868a4fe98bd2c74221e3e300 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:27:46.406606 29940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b820ed22868a4fe98bd2c74221e3e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.410084 30074 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:46.420964 29939 tablet_service.cc:1511] Processing CreateTablet for tablet 1b5584cc51f14a9a8dcee35a9998b380 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:27:46.422811 29939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5584cc51f14a9a8dcee35a9998b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.423002 29941 tablet_service.cc:1511] Processing CreateTablet for tablet 611c12107cde4b0e81c03ffd67cb64e2 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:27:46.423676 30076 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:46.424307 30076 ts_tablet_manager.cc:1403] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.193s	user 0.000s	sys 0.017s
I20260709 12:27:46.424732 29941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611c12107cde4b0e81c03ffd67cb64e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.435112 29942 tablet_service.cc:1511] Processing CreateTablet for tablet dd88542c59c449a0904ad8b002efc5a0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:27:46.427455 30074 ts_tablet_manager.cc:1403] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.298s	user 0.014s	sys 0.016s
I20260709 12:27:46.450937 29870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d49dfe78f7164655b82bfc664fbd11dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.456983 30019 tablet_service.cc:1511] Processing CreateTablet for tablet a4bd785dff644d0a8e536827c0304894 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:27:46.458707 29937 tablet_service.cc:1511] Processing CreateTablet for tablet ff934fd9fe834c2ab8737af23cdc9505 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:27:46.459507 29871 tablet_service.cc:1511] Processing CreateTablet for tablet 611c12107cde4b0e81c03ffd67cb64e2 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:27:46.460712 29869 tablet_service.cc:1511] Processing CreateTablet for tablet ffc3696cf98a402bb9d717b584cb229c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:27:46.463268 30013 tablet_service.cc:1511] Processing CreateTablet for tablet dd88542c59c449a0904ad8b002efc5a0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:27:46.463904 29942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd88542c59c449a0904ad8b002efc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.484388 29937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff934fd9fe834c2ab8737af23cdc9505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.487659 30019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4bd785dff644d0a8e536827c0304894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.488229 30077 tablet_bootstrap.cc:654] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.464572 29945 tablet_service.cc:1511] Processing CreateTablet for tablet ba9fb933bf47466999dfd7892e6a6638 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:27:46.489773 29945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9fb933bf47466999dfd7892e6a6638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.466257 30011 tablet_service.cc:1511] Processing CreateTablet for tablet 742ffdad627c4e71865fe5b85509976d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:27:46.467793 29867 tablet_service.cc:1511] Processing CreateTablet for tablet b820ed22868a4fe98bd2c74221e3e300 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:27:46.470501 29864 tablet_service.cc:1511] Processing CreateTablet for tablet 742ffdad627c4e71865fe5b85509976d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:27:46.472128 29866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a7001c5f324ba1955a1eef4ea15e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.495429 29944 tablet_service.cc:1511] Processing CreateTablet for tablet 742ffdad627c4e71865fe5b85509976d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:27:46.497315 29944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742ffdad627c4e71865fe5b85509976d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.498255 29867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b820ed22868a4fe98bd2c74221e3e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.499122 29869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc3696cf98a402bb9d717b584cb229c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.502326 29871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611c12107cde4b0e81c03ffd67cb64e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.501995 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 645003ea84b04f4a9e6d95a0bca63384 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:27:46.475888 29862 tablet_service.cc:1511] Processing CreateTablet for tablet ff934fd9fe834c2ab8737af23cdc9505 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:27:46.479079 29938 tablet_service.cc:1511] Processing CreateTablet for tablet 645003ea84b04f4a9e6d95a0bca63384 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:27:46.494756 30075 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:46.495088 30011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742ffdad627c4e71865fe5b85509976d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.538314 30075 ts_tablet_manager.cc:1403] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.385s	user 0.012s	sys 0.016s
I20260709 12:27:46.497699 30018 tablet_service.cc:1511] Processing CreateTablet for tablet 4a33c77154df4be8aee967d4d7df39b0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:27:46.540699 30018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a33c77154df4be8aee967d4d7df39b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.541532 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645003ea84b04f4a9e6d95a0bca63384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.478204 29791 tablet_service.cc:1511] Processing CreateTablet for tablet 645003ea84b04f4a9e6d95a0bca63384 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:27:46.509334 30054 heartbeater.cc:499] Master 127.16.102.62:33633 was elected leader, sending a full tablet report...
I20260709 12:27:46.508447 30076 raft_consensus.cc:359] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.544480 30076 raft_consensus.cc:385] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.544860 30076 raft_consensus.cc:740] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.518579 29868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b3b0e1b5d14def8b90e20a7a38ee24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.546128 30076 consensus_queue.cc:260] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.480907 29863 tablet_service.cc:1511] Processing CreateTablet for tablet 4a33c77154df4be8aee967d4d7df39b0 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:27:46.550400 30013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd88542c59c449a0904ad8b002efc5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.551659 29864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742ffdad627c4e71865fe5b85509976d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.556572 29862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff934fd9fe834c2ab8737af23cdc9505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.560461 29791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645003ea84b04f4a9e6d95a0bca63384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.564692 29863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a33c77154df4be8aee967d4d7df39b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.571293 30074 raft_consensus.cc:359] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.572321 30074 raft_consensus.cc:385] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.572661 30074 raft_consensus.cc:740] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.573668 29938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645003ea84b04f4a9e6d95a0bca63384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.573683 30074 consensus_queue.cc:260] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.559939 30075 raft_consensus.cc:359] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:46.592972 30075 raft_consensus.cc:385] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.593433 30075 raft_consensus.cc:740] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.593600 29790 tablet_service.cc:1511] Processing CreateTablet for tablet b914c950a335404b8369b7d5868ad408 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:27:46.595384 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b914c950a335404b8369b7d5868ad408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.608455 30010 tablet_service.cc:1511] Processing CreateTablet for tablet cefd124c3a724d06897bd37636aeb5b6 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:27:46.608959 30076 ts_tablet_manager.cc:1434] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.184s	user 0.016s	sys 0.000s
I20260709 12:27:46.611799 30076 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:46.626657 30082 raft_consensus.cc:493] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.630155 30082 raft_consensus.cc:515] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.632682 30075 consensus_queue.cc:260] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:46.631649 30010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefd124c3a724d06897bd37636aeb5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.634780 29940 tablet_service.cc:1511] Processing CreateTablet for tablet b914c950a335404b8369b7d5868ad408 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:27:46.636518 29940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b914c950a335404b8369b7d5868ad408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.638007 29866 tablet_service.cc:1511] Processing CreateTablet for tablet b914c950a335404b8369b7d5868ad408 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:27:46.650664 30082 leader_election.cc:290] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:46.605197 29865 tablet_service.cc:1511] Processing CreateTablet for tablet cefd124c3a724d06897bd37636aeb5b6 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:27:46.654949 29865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefd124c3a724d06897bd37636aeb5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.656668 30014 tablet_service.cc:1511] Processing CreateTablet for tablet b2c89d9015e54780811a19203939e9a5 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:27:46.658435 30014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c89d9015e54780811a19203939e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.662662 30076 tablet_bootstrap.cc:654] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.625420 29941 tablet_service.cc:1511] Processing CreateTablet for tablet cefd124c3a724d06897bd37636aeb5b6 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:27:46.685345 29870 tablet_service.cc:1511] Processing CreateTablet for tablet b2c89d9015e54780811a19203939e9a5 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:27:46.699316 30075 ts_tablet_manager.cc:1434] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.160s	user 0.004s	sys 0.013s
I20260709 12:27:46.700896 30075 tablet_bootstrap.cc:492] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:46.708899 30074 ts_tablet_manager.cc:1434] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.266s	user 0.015s	sys 0.015s
I20260709 12:27:46.710471 30074 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:46.711894 30085 raft_consensus.cc:493] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.712533 30085 raft_consensus.cc:515] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:46.697328 30077 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:46.702505 29866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b914c950a335404b8369b7d5868ad408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.730113 30077 ts_tablet_manager.cc:1403] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.363s	user 0.018s	sys 0.015s
I20260709 12:27:46.728211 30085 leader_election.cc:290] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:46.742122 30076 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
W20260709 12:27:46.749284 30055 tablet.cc:2406] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:27:46.753324 30076 ts_tablet_manager.cc:1403] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.142s	user 0.016s	sys 0.004s
I20260709 12:27:46.757421 30076 raft_consensus.cc:359] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.759394 30076 raft_consensus.cc:385] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.759948 30076 raft_consensus.cc:740] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.760957 30076 consensus_queue.cc:260] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.763396 29942 tablet_service.cc:1511] Processing CreateTablet for tablet 0d30c5d36c6e4ad2a38bb757a7773f78 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:27:46.748059 29870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c89d9015e54780811a19203939e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.775795 29942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d30c5d36c6e4ad2a38bb757a7773f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.777324 29869 tablet_service.cc:1511] Processing CreateTablet for tablet 57229c57228146d6ab456a83f82ef167 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:27:46.763893 29936 tablet_service.cc:1511] Processing CreateTablet for tablet 57229c57228146d6ab456a83f82ef167 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:27:46.780131 29936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57229c57228146d6ab456a83f82ef167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.779908 29796 tablet_service.cc:1511] Processing CreateTablet for tablet 0d30c5d36c6e4ad2a38bb757a7773f78 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:27:46.781410 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d30c5d36c6e4ad2a38bb757a7773f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.785948 29941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefd124c3a724d06897bd37636aeb5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.786002 29788 tablet_service.cc:1511] Processing CreateTablet for tablet 3dafcab3681f4882a86829813cbc7772 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:27:46.786600 29790 tablet_service.cc:1511] Processing CreateTablet for tablet b2c89d9015e54780811a19203939e9a5 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:27:46.788131 29788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dafcab3681f4882a86829813cbc7772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.792380 29943 tablet_service.cc:1511] Processing CreateTablet for tablet 3dafcab3681f4882a86829813cbc7772 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:27:46.794231 29943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dafcab3681f4882a86829813cbc7772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.814644 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c89d9015e54780811a19203939e9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.819376 29793 tablet_service.cc:1511] Processing CreateTablet for tablet 268205acac824554b94b834e990e1ed4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260709 12:27:46.921500 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:46.935582 29868 tablet_service.cc:1511] Processing CreateTablet for tablet 3dafcab3681f4882a86829813cbc7772 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:27:46.939769 30077 raft_consensus.cc:359] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.940725 30077 raft_consensus.cc:385] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.941030 30077 raft_consensus.cc:740] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.945741 29793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268205acac824554b94b834e990e1ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.949194 30082 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.947377 30075 tablet_bootstrap.cc:700] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: Time spent opening tablet: real 0.116s	user 0.004s	sys 0.000s
I20260709 12:27:46.960148 30082 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:46.963757 30082 leader_election.cc:290] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:46.966763 29871 tablet_service.cc:1511] Processing CreateTablet for tablet 268205acac824554b94b834e990e1ed4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:27:46.967312 29939 tablet_service.cc:1511] Processing CreateTablet for tablet e047383bb70448908508e00810ebff84 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:27:46.954521 30074 tablet_bootstrap.cc:654] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.968891 29939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e047383bb70448908508e00810ebff84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.960759 29945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57152 (request call id 10) took 1012 ms. Trace:
I20260709 12:27:46.968429 29795 tablet_service.cc:1511] Processing CreateTablet for tablet e047383bb70448908508e00810ebff84 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:27:46.961231 29869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57229c57228146d6ab456a83f82ef167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.971201 29795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e047383bb70448908508e00810ebff84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.968487 30075 tablet_bootstrap.cc:654] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.979676 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 268205acac824554b94b834e990e1ed4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:27:46.979956 30013 tablet_service.cc:1511] Processing CreateTablet for tablet e047383bb70448908508e00810ebff84 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:27:46.981860 30013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e047383bb70448908508e00810ebff84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.986671 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268205acac824554b94b834e990e1ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.977942 29945 rpcz_store.cc:276] 0709 12:27:45.948533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:46.385876 (+437343us) service_pool.cc:224] Handling call
0709 12:27:46.489128 (+103252us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:27:46.489261 (+   133us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:27:46.651608 (+162347us) tablet_metadata.cc:665] Metadata flushed
0709 12:27:46.960119 (+308511us) tablet_replica.cc:206] Creating consensus instance
0709 12:27:46.960682 (+   563us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:46.995360 30011 tablet_service.cc:1511] Processing CreateTablet for tablet 0d30c5d36c6e4ad2a38bb757a7773f78 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:27:46.997093 30011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d30c5d36c6e4ad2a38bb757a7773f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:46.959393 30076 ts_tablet_manager.cc:1434] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.205s	user 0.010s	sys 0.000s
I20260709 12:27:46.994930 30074 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:47.005358 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:47.004546 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:47.008651 29847 leader_election.cc:343] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.005354 30074 ts_tablet_manager.cc:1403] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.295s	user 0.013s	sys 0.004s
W20260709 12:27:47.009675 29847 leader_election.cc:343] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:47.010638 30061 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.16.102.62:33633, user_credentials={real_user=slave}} blocked reactor thread for 58251.8us
I20260709 12:27:47.012571 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:47.013351 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:47.016914 29846 leader_election.cc:343] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.017450 29846 leader_election.cc:304] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:47.018397 30082 raft_consensus.cc:2749] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:47.018922 29846 leader_election.cc:343] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.019471 29846 leader_election.cc:304] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:47.020346 30082 raft_consensus.cc:2749] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.028932 30084 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.029541 30074 raft_consensus.cc:359] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.029691 30084 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.030479 30074 raft_consensus.cc:385] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.031023 30074 raft_consensus.cc:740] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.034469 29867 tablet_service.cc:1511] Processing CreateTablet for tablet 71ae82107a97483ba4fa78a788adb334 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:27:47.035734 30084 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:47.031951 30074 consensus_queue.cc:260] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.047474 30074 ts_tablet_manager.cc:1434] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.002s
I20260709 12:27:47.048853 30074 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:47.054126 29938 tablet_service.cc:1511] Processing CreateTablet for tablet 71ae82107a97483ba4fa78a788adb334 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:27:47.056068 30018 tablet_service.cc:1511] Processing CreateTablet for tablet 71ae82107a97483ba4fa78a788adb334 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:27:47.057411 29790 tablet_service.cc:1511] Processing CreateTablet for tablet 66a7253beff14dc2a82c02db69def01c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:27:47.058940 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a7253beff14dc2a82c02db69def01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.060161 29944 tablet_service.cc:1511] Processing CreateTablet for tablet 66a7253beff14dc2a82c02db69def01c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:27:47.061414 30075 tablet_bootstrap.cc:492] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:47.062062 30075 ts_tablet_manager.cc:1403] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.362s	user 0.019s	sys 0.000s
I20260709 12:27:47.064533 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:47.067070 29992 leader_election.cc:343] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.068857 29871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268205acac824554b94b834e990e1ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.072149 29868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dafcab3681f4882a86829813cbc7772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.073120 30074 tablet_bootstrap.cc:654] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.073629 29862 tablet_service.cc:1511] Processing CreateTablet for tablet 66a7253beff14dc2a82c02db69def01c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:27:47.075307 29862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a7253beff14dc2a82c02db69def01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.087112 30015 tablet_service.cc:1511] Processing CreateTablet for tablet 4c799e9531994c7cba86a172713405e8 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:27:47.088996 30084 raft_consensus.cc:493] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.089661 30084 raft_consensus.cc:515] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.093777 30075 raft_consensus.cc:359] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.095718 30075 raft_consensus.cc:385] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.096087 30075 raft_consensus.cc:740] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.098583 29867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ae82107a97483ba4fa78a788adb334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.100428 30075 consensus_queue.cc:260] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.101245 29864 tablet_service.cc:1511] Processing CreateTablet for tablet 4c799e9531994c7cba86a172713405e8 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:27:47.103279 30075 ts_tablet_manager.cc:1434] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.041s	user 0.004s	sys 0.004s
I20260709 12:27:47.103647 29864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c799e9531994c7cba86a172713405e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.111219 29940 tablet_service.cc:1511] Processing CreateTablet for tablet 97428fa3f86d463fa4d309cffb192127 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:27:47.117681 30076 tablet_bootstrap.cc:492] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:47.120941 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 97428fa3f86d463fa4d309cffb192127 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:27:47.125036 30010 tablet_service.cc:1511] Processing CreateTablet for tablet 57229c57228146d6ab456a83f82ef167 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:27:47.114615 30084 leader_election.cc:290] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:47.128733 29869 tablet_service.cc:1511] Processing CreateTablet for tablet 42b6f3a05b40452e9c171932ca603b19 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:27:47.158668 29869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42b6f3a05b40452e9c171932ca603b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.131573 29793 tablet_service.cc:1511] Processing CreateTablet for tablet 97428fa3f86d463fa4d309cffb192127 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:27:47.165665 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:47.166551 29805 raft_consensus.cc:2468] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:47.166859 29793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97428fa3f86d463fa4d309cffb192127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.168591 29994 leader_election.cc:304] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:47.127851 30075 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:47.056084 29938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ae82107a97483ba4fa78a788adb334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.170133 30085 raft_consensus.cc:2804] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.170599 30085 raft_consensus.cc:493] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.171000 30085 raft_consensus.cc:3060] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.172655 30074 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:47.172951 29940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97428fa3f86d463fa4d309cffb192127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.175761 29944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a7253beff14dc2a82c02db69def01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.140488 30077 consensus_queue.cc:260] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.151386 30013 tablet_service.cc:1511] Processing CreateTablet for tablet 42b6f3a05b40452e9c171932ca603b19 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:27:47.176645 29796 tablet_service.cc:1511] Processing CreateTablet for tablet 4c799e9531994c7cba86a172713405e8 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:27:47.151477 30018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ae82107a97483ba4fa78a788adb334. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.178833 30013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42b6f3a05b40452e9c171932ca603b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.153796 29942 tablet_service.cc:1511] Processing CreateTablet for tablet 42b6f3a05b40452e9c171932ca603b19 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:27:47.173439 30074 ts_tablet_manager.cc:1403] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.125s	user 0.016s	sys 0.016s
I20260709 12:27:47.180382 29942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42b6f3a05b40452e9c171932ca603b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.186305 30097 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.186527 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97428fa3f86d463fa4d309cffb192127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.186057 30074 raft_consensus.cc:359] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.187222 30074 raft_consensus.cc:385] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.187072 30097 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.188771 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c799e9531994c7cba86a172713405e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.190395 30015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c799e9531994c7cba86a172713405e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.187677 30074 raft_consensus.cc:740] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.199612 29790 tablet_service.cc:1511] Processing CreateTablet for tablet 259a887eac9c4ed0b816a4ac0bc2c536 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:27:47.202541 29870 tablet_service.cc:1511] Processing CreateTablet for tablet 259a887eac9c4ed0b816a4ac0bc2c536 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:27:47.203862 30075 tablet_bootstrap.cc:654] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.202541 29941 tablet_service.cc:1511] Processing CreateTablet for tablet 259a887eac9c4ed0b816a4ac0bc2c536 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:27:47.209899 29870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259a887eac9c4ed0b816a4ac0bc2c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.207278 30097 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:47.210696 29941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259a887eac9c4ed0b816a4ac0bc2c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.208570 30010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57229c57228146d6ab456a83f82ef167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.209169 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259a887eac9c4ed0b816a4ac0bc2c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.208829 30074 consensus_queue.cc:260] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.226684 30074 ts_tablet_manager.cc:1434] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.002s
I20260709 12:27:47.228156 30074 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:47.240007 30074 tablet_bootstrap.cc:654] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.251856 30085 raft_consensus.cc:515] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.263782 30085 leader_election.cc:290] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:47.265313 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:47.266304 29805 raft_consensus.cc:3060] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.272656 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
W20260709 12:27:47.284274 29992 leader_election.cc:343] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.284291 30077 ts_tablet_manager.cc:1434] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.544s	user 0.009s	sys 0.007s
W20260709 12:27:47.285573 30077 ts_tablet_manager.cc:1467] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: Tablet startup took 545ms
W20260709 12:27:47.285971 30077 ts_tablet_manager.cc:1469] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: Trace:
0709 12:27:45.939628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:45.941936 (+  2308us) service_pool.cc:224] Handling call
0709 12:27:45.944380 (+  2444us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:27:45.944504 (+   124us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:27:46.035767 (+ 91263us) tablet_metadata.cc:665] Metadata flushed
0709 12:27:46.215256 (+179489us) tablet_replica.cc:206] Creating consensus instance
0709 12:27:46.363903 (+148647us) inbound_call.cc:177] Queueing success response
0709 12:27:46.366569 (+  2666us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:27:46.740261 (+373692us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:27:46.936889 (+196628us) tablet_replica.cc:261] Starting instrumentation
0709 12:27:46.937644 (+   755us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":4915,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":27822,"tablet-open.queue_time_us":150794,"thread_start_us":211012,"threads_started":2}
I20260709 12:27:47.288705 30016 tablet_service.cc:1511] Processing CreateTablet for tablet 889b2ffc0b554cbea97dc55e0aa95094 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:27:47.304586 29805 raft_consensus.cc:2468] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:47.315135 30076 tablet_bootstrap.cc:654] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.321376 30016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889b2ffc0b554cbea97dc55e0aa95094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.334995 29994 leader_election.cc:304] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:47.336839 30085 raft_consensus.cc:2804] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:47.348637 29869 tablet_service.cc:1511] Processing CreateTablet for tablet 889b2ffc0b554cbea97dc55e0aa95094 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:27:47.349920 29943 tablet_service.cc:1511] Processing CreateTablet for tablet 6a71487088564b8489c63edd9d182c5a (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:27:47.351591 29943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a71487088564b8489c63edd9d182c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.355159 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:47.358724 29769 leader_election.cc:343] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.350406 29869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889b2ffc0b554cbea97dc55e0aa95094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.361042 30014 tablet_service.cc:1511] Processing CreateTablet for tablet 6a71487088564b8489c63edd9d182c5a (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:27:47.364192 30014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a71487088564b8489c63edd9d182c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.368723 29795 tablet_service.cc:1511] Processing CreateTablet for tablet 6a71487088564b8489c63edd9d182c5a (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:27:47.392283 29868 tablet_service.cc:1511] Processing CreateTablet for tablet 35000da0d1054109927271f881d5b697 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:27:47.393142 29795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a71487088564b8489c63edd9d182c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.384294 30085 raft_consensus.cc:697] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:47.397578 29937 tablet_service.cc:1511] Processing CreateTablet for tablet 35000da0d1054109927271f881d5b697 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:27:47.399849 29937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35000da0d1054109927271f881d5b697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.397274 30085 consensus_queue.cc:237] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.403144 29868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35000da0d1054109927271f881d5b697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.406868 29790 tablet_service.cc:1511] Processing CreateTablet for tablet 889b2ffc0b554cbea97dc55e0aa95094 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:27:47.409194 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889b2ffc0b554cbea97dc55e0aa95094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.416613 30075 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:47.417433 30075 ts_tablet_manager.cc:1403] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.297s	user 0.019s	sys 0.008s
I20260709 12:27:47.433991 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:47.439962 29772 leader_election.cc:343] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.440055 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:47.442221 29772 leader_election.cc:304] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:47.443426 30084 raft_consensus.cc:2749] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.444067 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:47.444919 29955 raft_consensus.cc:2468] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:47.446451 30108 raft_consensus.cc:493] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.446633 29772 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:47.447187 30108 raft_consensus.cc:515] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.447603 30084 raft_consensus.cc:2804] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.448184 30084 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.448570 30084 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.451396 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:47.452387 30108 leader_election.cc:290] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:27:47.453306 29992 leader_election.cc:343] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.461069 30075 raft_consensus.cc:359] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:47.462066 30075 raft_consensus.cc:385] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.387962 30077 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:47.462385 30075 raft_consensus.cc:740] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.484700 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:47.487473 29921 leader_election.cc:343] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.507323 30074 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:47.508134 30074 ts_tablet_manager.cc:1403] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.280s	user 0.026s	sys 0.010s
I20260709 12:27:47.510185 29867 tablet_service.cc:1511] Processing CreateTablet for tablet d8ada01658a746029b8fd58389f608a4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:27:47.510617 29796 tablet_service.cc:1511] Processing CreateTablet for tablet d8ada01658a746029b8fd58389f608a4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:27:47.511759 29867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ada01658a746029b8fd58389f608a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.512214 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ada01658a746029b8fd58389f608a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.515446 30075 consensus_queue.cc:260] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:47.516813 29703 catalog_manager.cc:5697] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.518322 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:47.478063 30010 tablet_service.cc:1511] Processing CreateTablet for tablet 35000da0d1054109927271f881d5b697 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:27:47.512265 30074 raft_consensus.cc:359] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.531734 30076 tablet_bootstrap.cc:492] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:47.534654 30010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35000da0d1054109927271f881d5b697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.535307 30076 ts_tablet_manager.cc:1403] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.422s	user 0.018s	sys 0.021s
I20260709 12:27:47.538048 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:47.495569 30084 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.539527 29805 raft_consensus.cc:2393] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
W20260709 12:27:47.540060 29995 leader_election.cc:343] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.540715 30112 raft_consensus.cc:493] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.540938 30084 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:47.534644 30074 raft_consensus.cc:385] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.541432 30112 raft_consensus.cc:515] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:47.542804 30074 raft_consensus.cc:740] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.530296 29942 tablet_service.cc:1511] Processing CreateTablet for tablet d8ada01658a746029b8fd58389f608a4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:27:47.544390 30074 consensus_queue.cc:260] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.545404 29942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ada01658a746029b8fd58389f608a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.547191 30074 ts_tablet_manager.cc:1434] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.039s	user 0.005s	sys 0.004s
I20260709 12:27:47.547291 29920 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:47.548666 30074 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:47.549196 30107 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.554528 30112 leader_election.cc:290] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:47.557874 30076 raft_consensus.cc:359] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.558214 29793 tablet_service.cc:1511] Processing CreateTablet for tablet 444cae907f1d4bdab292e8a3f59f7f11 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:27:47.558889 30076 raft_consensus.cc:385] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.559255 30076 raft_consensus.cc:740] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.559860 29793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 444cae907f1d4bdab292e8a3f59f7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.516865 30077 tablet_bootstrap.cc:654] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.540735 29995 leader_election.cc:304] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:47.567133 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:47.568625 30085 raft_consensus.cc:2749] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.568871 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:47.570497 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:47.570995 30076 consensus_queue.cc:260] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.565698 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:47.573176 30107 raft_consensus.cc:2749] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.577757 29869 tablet_service.cc:1511] Processing CreateTablet for tablet 444cae907f1d4bdab292e8a3f59f7f11 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:27:47.579478 29869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 444cae907f1d4bdab292e8a3f59f7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.575668 30075 ts_tablet_manager.cc:1434] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.158s	user 0.012s	sys 0.000s
I20260709 12:27:47.581702 30075 tablet_bootstrap.cc:492] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
W20260709 12:27:47.573676 29772 leader_election.cc:343] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:47.585019 29772 leader_election.cc:343] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:47.588608 29772 leader_election.cc:343] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.589293 29772 leader_election.cc:304] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:47.593425 30112 raft_consensus.cc:2749] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.608523 29941 tablet_service.cc:1511] Processing CreateTablet for tablet 3a4b038ac5e344bf96cf8c672adb2dbb (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:27:47.610371 29941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4b038ac5e344bf96cf8c672adb2dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.612468 30085 raft_consensus.cc:493] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.612723 29955 raft_consensus.cc:2468] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:47.613204 30085 raft_consensus.cc:515] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:47.613634 29870 tablet_service.cc:1511] Processing CreateTablet for tablet 811ccdd2daa94877be1d910bd3745085 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:27:47.616019 29870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811ccdd2daa94877be1d910bd3745085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.618023 29790 tablet_service.cc:1511] Processing CreateTablet for tablet 3a4b038ac5e344bf96cf8c672adb2dbb (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:27:47.619690 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4b038ac5e344bf96cf8c672adb2dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.620064 30015 tablet_service.cc:1511] Processing CreateTablet for tablet 3a4b038ac5e344bf96cf8c672adb2dbb (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:27:47.621563 30015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4b038ac5e344bf96cf8c672adb2dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.624717 30074 tablet_bootstrap.cc:654] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.631225 29940 tablet_service.cc:1511] Processing CreateTablet for tablet 811ccdd2daa94877be1d910bd3745085 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:27:47.622138 30011 tablet_service.cc:1511] Processing CreateTablet for tablet 444cae907f1d4bdab292e8a3f59f7f11 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:27:47.623818 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:47.634166 29871 tablet_service.cc:1511] Processing CreateTablet for tablet 9ad7f32688184e219c1d203375c23682 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:27:47.634778 30011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 444cae907f1d4bdab292e8a3f59f7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.622272 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:47.625975 30085 leader_election.cc:290] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:47.638273 29871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad7f32688184e219c1d203375c23682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.634166 29788 tablet_service.cc:1511] Processing CreateTablet for tablet 9ad7f32688184e219c1d203375c23682 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:27:47.646206 29788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad7f32688184e219c1d203375c23682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.655587 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 811ccdd2daa94877be1d910bd3745085 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:27:47.657285 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811ccdd2daa94877be1d910bd3745085. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:47.650357 29992 leader_election.cc:343] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.653861 29940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811ccdd2daa94877be1d910bd3745085. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:47.666926 29994 leader_election.cc:343] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.666919 30075 tablet_bootstrap.cc:654] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.673898 29994 leader_election.cc:304] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:47.679388 29772 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:47.680779 30085 raft_consensus.cc:2749] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.680841 30112 raft_consensus.cc:2804] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:47.681964 30112 raft_consensus.cc:697] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:47.683009 30112 consensus_queue.cc:237] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.686292 30077 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:47.687027 30077 ts_tablet_manager.cc:1403] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.308s	user 0.008s	sys 0.009s
I20260709 12:27:47.691241 29867 tablet_service.cc:1511] Processing CreateTablet for tablet b9fd2b0fbb1d49da957faacba2d1b38f (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:27:47.692873 29867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fd2b0fbb1d49da957faacba2d1b38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.696008 29791 tablet_service.cc:1511] Processing CreateTablet for tablet 9ca5a52b63dd4c46bb69db5b73b48b3d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:27:47.697417 29862 tablet_service.cc:1511] Processing CreateTablet for tablet 4c3b379d58464fa6b60b8a33472252d4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:27:47.699102 29862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3b379d58464fa6b60b8a33472252d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.697695 29791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca5a52b63dd4c46bb69db5b73b48b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.703133 29937 tablet_service.cc:1511] Processing CreateTablet for tablet 9ca5a52b63dd4c46bb69db5b73b48b3d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:27:47.704882 29937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca5a52b63dd4c46bb69db5b73b48b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.709194 30016 tablet_service.cc:1511] Processing CreateTablet for tablet 9ca5a52b63dd4c46bb69db5b73b48b3d (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:27:47.699502 30077 raft_consensus.cc:359] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.709751 29790 tablet_service.cc:1511] Processing CreateTablet for tablet b9fd2b0fbb1d49da957faacba2d1b38f (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:27:47.710518 30077 raft_consensus.cc:385] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.711004 30016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca5a52b63dd4c46bb69db5b73b48b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.711354 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fd2b0fbb1d49da957faacba2d1b38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.697490 29943 tablet_service.cc:1511] Processing CreateTablet for tablet 9ad7f32688184e219c1d203375c23682 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:27:47.713843 30014 tablet_service.cc:1511] Processing CreateTablet for tablet b9fd2b0fbb1d49da957faacba2d1b38f (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:27:47.714581 29943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad7f32688184e219c1d203375c23682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.715371 30014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fd2b0fbb1d49da957faacba2d1b38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.715912 30084 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.716478 30084 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.719058 30084 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:47.720839 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:47.721693 29881 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:47.722895 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:47.723064 29769 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:47.724283 30084 raft_consensus.cc:2804] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.724747 30084 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.725113 30084 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.710973 30077 raft_consensus.cc:740] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.730268 29939 tablet_service.cc:1511] Processing CreateTablet for tablet 4c3b379d58464fa6b60b8a33472252d4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:27:47.730692 30077 consensus_queue.cc:260] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.740929 30107 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.741542 30107 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.743592 30076 ts_tablet_manager.cc:1434] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.208s	user 0.029s	sys 0.026s
I20260709 12:27:47.745036 30076 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:47.746053 29955 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:47.746423 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:47.741051 29939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3b379d58464fa6b60b8a33472252d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.760663 30013 tablet_service.cc:1511] Processing CreateTablet for tablet 4c3b379d58464fa6b60b8a33472252d4 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:27:47.762435 30013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3b379d58464fa6b60b8a33472252d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.768678 29796 tablet_service.cc:1511] Processing CreateTablet for tablet e6fa078c85764bb8b88bc6afff3c5bda (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:27:47.770826 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fa078c85764bb8b88bc6afff3c5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.773159 30084 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.754323 30107 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:47.801415 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:47.820920 29805 raft_consensus.cc:2393] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:47.809748 29881 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:47.837661 29944 tablet_service.cc:1511] Processing CreateTablet for tablet e6fa078c85764bb8b88bc6afff3c5bda (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:27:47.839403 29944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fa078c85764bb8b88bc6afff3c5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.837663 30076 tablet_bootstrap.cc:654] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.848274 29868 tablet_service.cc:1511] Processing CreateTablet for tablet e6fa078c85764bb8b88bc6afff3c5bda (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:27:47.850227 29918 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:27:47.851393 30107 raft_consensus.cc:2804] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.851851 30107 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.853955 29868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fa078c85764bb8b88bc6afff3c5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.853519 30107 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:47.859488 29992 consensus_peers.cc:597] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:47.863119 30015 tablet_service.cc:1511] Processing CreateTablet for tablet aaee6416f659496fb39a11ee1ae00ca3 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:27:47.864871 30015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaee6416f659496fb39a11ee1ae00ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.870832 30077 ts_tablet_manager.cc:1434] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.183s	user 0.006s	sys 0.009s
I20260709 12:27:47.872334 30077 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:47.880592 29795 tablet_service.cc:1511] Processing CreateTablet for tablet aaee6416f659496fb39a11ee1ae00ca3 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:27:47.883304 29862 tablet_service.cc:1511] Processing CreateTablet for tablet aaee6416f659496fb39a11ee1ae00ca3 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:27:47.891817 29703 catalog_manager.cc:5697] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.893669 29942 tablet_service.cc:1511] Processing CreateTablet for tablet 997e1531077444f3b82718c8b75b357e (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:27:47.895843 29942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 997e1531077444f3b82718c8b75b357e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.898181 29795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaee6416f659496fb39a11ee1ae00ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.913389 29862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaee6416f659496fb39a11ee1ae00ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.916617 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:47.917598 29881 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.924327 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
W20260709 12:27:47.925966 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:47.927903 30084 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:47.928443 30075 tablet_bootstrap.cc:492] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:47.929320 30075 ts_tablet_manager.cc:1403] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.348s	user 0.009s	sys 0.050s
I20260709 12:27:47.935792 29791 tablet_service.cc:1511] Processing CreateTablet for tablet 997e1531077444f3b82718c8b75b357e (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:27:47.951193 30017 tablet_service.cc:1511] Processing CreateTablet for tablet d5988ddbc0f84d47b9b94c1375502692 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:27:47.940313 30107 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:47.943639 29871 tablet_service.cc:1511] Processing CreateTablet for tablet d5988ddbc0f84d47b9b94c1375502692 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:27:47.951947 29791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 997e1531077444f3b82718c8b75b357e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.937937 30013 tablet_service.cc:1511] Processing CreateTablet for tablet 997e1531077444f3b82718c8b75b357e (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:27:47.952661 29790 tablet_service.cc:1511] Processing CreateTablet for tablet c08a657511124ab1ad80a227eeec2736 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:27:47.956214 29871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5988ddbc0f84d47b9b94c1375502692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.957129 30074 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:47.957942 30074 ts_tablet_manager.cc:1403] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.409s	user 0.014s	sys 0.014s
I20260709 12:27:47.959728 29790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08a657511124ab1ad80a227eeec2736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.964092 30013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 997e1531077444f3b82718c8b75b357e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.966656 30075 raft_consensus.cc:359] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.967590 30075 raft_consensus.cc:385] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.967907 30075 raft_consensus.cc:740] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.968901 30075 consensus_queue.cc:260] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:47.971320 29955 raft_consensus.cc:2393] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:47.952782 29939 tablet_service.cc:1511] Processing CreateTablet for tablet d5988ddbc0f84d47b9b94c1375502692 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:27:47.981226 30076 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:47.981988 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5988ddbc0f84d47b9b94c1375502692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.998886 30011 tablet_service.cc:1511] Processing CreateTablet for tablet 2f31657dfccf4d45ab006ef5aadcfb4c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:27:48.000605 30011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f31657dfccf4d45ab006ef5aadcfb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.000223 29795 tablet_service.cc:1511] Processing CreateTablet for tablet 2f31657dfccf4d45ab006ef5aadcfb4c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:27:47.995709 30076 ts_tablet_manager.cc:1403] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.251s	user 0.018s	sys 0.025s
I20260709 12:27:48.001713 29795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f31657dfccf4d45ab006ef5aadcfb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.996157 29939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5988ddbc0f84d47b9b94c1375502692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.998958 29937 tablet_service.cc:1511] Processing CreateTablet for tablet c08a657511124ab1ad80a227eeec2736 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:27:48.009532 29867 tablet_service.cc:1511] Processing CreateTablet for tablet c08a657511124ab1ad80a227eeec2736 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:27:48.010639 30077 tablet_bootstrap.cc:654] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.010936 30075 ts_tablet_manager.cc:1434] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.081s	user 0.008s	sys 0.000s
I20260709 12:27:47.981585 30074 raft_consensus.cc:359] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.038499 29937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08a657511124ab1ad80a227eeec2736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.042716 30114 raft_consensus.cc:493] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.043258 30075 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.043432 30114 raft_consensus.cc:515] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.046728 30114 leader_election.cc:290] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.047544 29868 tablet_service.cc:1511] Processing CreateTablet for tablet 2f31657dfccf4d45ab006ef5aadcfb4c (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:27:48.053170 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a554edb2449436ea89ed389c4aed651" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.057161 29940 tablet_service.cc:1511] Processing CreateTablet for tablet 4f1befeab89b48729c8ba035e0ef71a8 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:27:48.059876 29940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1befeab89b48729c8ba035e0ef71a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.061149 30085 consensus_queue.cc:1048] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.063232 29867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08a657511124ab1ad80a227eeec2736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.075098 30017 tablet_service.cc:1511] Processing CreateTablet for tablet 4f1befeab89b48729c8ba035e0ef71a8 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:27:48.075501 29796 tablet_service.cc:1511] Processing CreateTablet for tablet 4f1befeab89b48729c8ba035e0ef71a8 (DEFAULT_TABLE table=test-workload [id=0656bb84addc4940a450e6cf6542d58b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260709 12:27:48.076288 29772 consensus_peers.cc:597] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 -> Peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Couldn't send request to peer 1e2ceabfe50c47b49529b663fa133b86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.076802 30017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1befeab89b48729c8ba035e0ef71a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.079165 29796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1befeab89b48729c8ba035e0ef71a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.082687 30084 consensus_queue.cc:1048] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.037024 30074 raft_consensus.cc:385] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.005367 30076 raft_consensus.cc:359] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.106789 30076 raft_consensus.cc:385] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.108289 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:48.110186 29806 raft_consensus.cc:2393] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:48.109619 30076 raft_consensus.cc:740] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.070840 29881 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:48.114589 29868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f31657dfccf4d45ab006ef5aadcfb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.122910 29769 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.123593 30107 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:48.101459 30075 tablet_bootstrap.cc:654] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.101037 30074 raft_consensus.cc:740] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.127169 30084 raft_consensus.cc:2804] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.143132 30084 raft_consensus.cc:697] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:48.144315 30084 consensus_queue.cc:237] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.117326 30076 consensus_queue.cc:260] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.159238 30077 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:48.160104 30077 ts_tablet_manager.cc:1403] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.288s	user 0.031s	sys 0.049s
W20260709 12:27:48.166445 29847 leader_election.cc:343] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.170411 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:48.171587 29881 raft_consensus.cc:2393] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:48.175768 30074 consensus_queue.cc:260] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.204689 29918 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.227808 30075 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:48.228551 30075 ts_tablet_manager.cc:1403] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.186s	user 0.008s	sys 0.020s
I20260709 12:27:48.218961 30084 raft_consensus.cc:493] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.236012 30084 raft_consensus.cc:515] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.238602 30084 leader_election.cc:290] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.216497 30107 raft_consensus.cc:2749] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.241729 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.243345 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a554edb2449436ea89ed389c4aed651" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:48.244470 29772 leader_election.cc:343] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:48.244791 29847 leader_election.cc:343] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.245368 29847 leader_election.cc:304] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.251070 30114 raft_consensus.cc:2749] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.251307 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:48.258086 29769 leader_election.cc:343] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.258365 30077 raft_consensus.cc:359] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.258920 29769 leader_election.cc:304] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.259366 30077 raft_consensus.cc:385] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.259797 30077 raft_consensus.cc:740] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.264328 30075 raft_consensus.cc:359] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.266319 30074 ts_tablet_manager.cc:1434] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.308s	user 0.008s	sys 0.023s
I20260709 12:27:48.265632 29703 catalog_manager.cc:5697] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.267802 30074 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.271229 30133 raft_consensus.cc:2749] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.276284 30136 raft_consensus.cc:493] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.279857 30136 raft_consensus.cc:515] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.265602 30076 ts_tablet_manager.cc:1434] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.263s	user 0.006s	sys 0.005s
I20260709 12:27:48.283602 30076 tablet_bootstrap.cc:492] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:48.265594 30075 raft_consensus.cc:385] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.286145 30075 raft_consensus.cc:740] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.286964 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:48.288280 30136 leader_election.cc:290] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:27:48.288755 29992 leader_election.cc:343] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.289064 30075 consensus_queue.cc:260] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.292014 30074 tablet_bootstrap.cc:654] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.292529 30075 ts_tablet_manager.cc:1434] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.064s	user 0.010s	sys 0.002s
I20260709 12:27:48.298337 30077 consensus_queue.cc:260] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.305174 30074 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:48.306001 30074 ts_tablet_manager.cc:1403] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.038s	user 0.017s	sys 0.005s
I20260709 12:27:48.308724 30077 ts_tablet_manager.cc:1434] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.148s	user 0.007s	sys 0.003s
I20260709 12:27:48.310024 30077 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:48.310907 30074 raft_consensus.cc:359] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.311932 30074 raft_consensus.cc:385] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.312350 30074 raft_consensus.cc:740] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.312798 30075 tablet_bootstrap.cc:492] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.313551 30074 consensus_queue.cc:260] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.316787 30074 ts_tablet_manager.cc:1434] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:27:48.318221 30074 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.322062 30085 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.323029 30085 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.326776 30074 tablet_bootstrap.cc:654] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.327353 30076 tablet_bootstrap.cc:654] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.330358 29881 raft_consensus.cc:3060] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.330694 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:48.333288 29994 leader_election.cc:343] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.334342 30085 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:48.337266 30077 tablet_bootstrap.cc:654] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.337718 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.341162 30075 tablet_bootstrap.cc:654] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.352437 30076 tablet_bootstrap.cc:492] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:48.353327 30076 ts_tablet_manager.cc:1403] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.070s	user 0.005s	sys 0.007s
I20260709 12:27:48.338135 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:48.363391 29995 leader_election.cc:343] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.364296 29995 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.365840 30085 raft_consensus.cc:2749] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.367412 30074 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:48.368112 30074 ts_tablet_manager.cc:1403] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.050s	user 0.018s	sys 0.001s
W20260709 12:27:48.368549 29995 leader_election.cc:343] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.369942 29995 leader_election.cc:304] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.371764 30135 raft_consensus.cc:2749] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.367470 30075 tablet_bootstrap.cc:492] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:48.373235 30075 ts_tablet_manager.cc:1403] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.080s	user 0.014s	sys 0.001s
I20260709 12:27:48.369287 30076 raft_consensus.cc:359] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.374840 30076 raft_consensus.cc:385] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.375201 30076 raft_consensus.cc:740] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.376269 30076 consensus_queue.cc:260] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.372079 30074 raft_consensus.cc:359] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.378199 30074 raft_consensus.cc:385] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.378612 30074 raft_consensus.cc:740] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.379596 30074 consensus_queue.cc:260] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.380679 30077 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:48.381579 30077 ts_tablet_manager.cc:1403] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.072s	user 0.014s	sys 0.003s
I20260709 12:27:48.383245 30074 ts_tablet_manager.cc:1434] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.003s
I20260709 12:27:48.384999 30074 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.384657 30075 raft_consensus.cc:359] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.385583 30075 raft_consensus.cc:385] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.385982 30075 raft_consensus.cc:740] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.385720 30076 ts_tablet_manager.cc:1434] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.004s
I20260709 12:27:48.387785 30075 consensus_queue.cc:260] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.387979 30077 raft_consensus.cc:359] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.390156 30077 raft_consensus.cc:385] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.390515 30077 raft_consensus.cc:740] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.392068 30076 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:48.395119 30074 tablet_bootstrap.cc:654] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.394222 30075 ts_tablet_manager.cc:1434] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 12:27:48.401113 30075 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.402839 30077 consensus_queue.cc:260] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.406364 30077 ts_tablet_manager.cc:1434] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.003s
I20260709 12:27:48.408120 30077 tablet_bootstrap.cc:492] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:48.409392 30075 tablet_bootstrap.cc:654] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.413057 30076 tablet_bootstrap.cc:654] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.418429 30074 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:48.419111 30074 ts_tablet_manager.cc:1403] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.034s	user 0.026s	sys 0.003s
I20260709 12:27:48.423131 30074 raft_consensus.cc:359] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.424188 30074 raft_consensus.cc:385] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.424623 30074 raft_consensus.cc:740] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.425750 30074 consensus_queue.cc:260] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.428936 30074 ts_tablet_manager.cc:1434] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:48.431193 30076 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:48.431346 30074 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.431859 30076 ts_tablet_manager.cc:1403] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.040s	user 0.012s	sys 0.004s
I20260709 12:27:48.435117 30076 raft_consensus.cc:359] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.435832 30076 raft_consensus.cc:385] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.436189 30076 raft_consensus.cc:740] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.437089 30076 consensus_queue.cc:260] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.439078 30077 tablet_bootstrap.cc:654] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.440249 30076 ts_tablet_manager.cc:1434] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:48.442164 30074 tablet_bootstrap.cc:654] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.445649 30076 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:48.446556 30077 tablet_bootstrap.cc:492] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:48.447199 30077 ts_tablet_manager.cc:1403] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.039s	user 0.010s	sys 0.005s
I20260709 12:27:48.448951 30075 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:48.450704 30075 ts_tablet_manager.cc:1403] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.050s	user 0.016s	sys 0.006s
I20260709 12:27:48.451120 30077 raft_consensus.cc:359] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.452060 30077 raft_consensus.cc:385] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.452378 30077 raft_consensus.cc:740] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.453374 30077 consensus_queue.cc:260] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.454854 30075 raft_consensus.cc:359] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.455659 30075 raft_consensus.cc:385] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.455966 30077 ts_tablet_manager.cc:1434] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:48.455991 30075 raft_consensus.cc:740] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.458451 30077 tablet_bootstrap.cc:492] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:48.457118 30075 consensus_queue.cc:260] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.464191 30075 ts_tablet_manager.cc:1434] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.004s
I20260709 12:27:48.465727 30075 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.468325 30077 tablet_bootstrap.cc:654] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.475222 30076 tablet_bootstrap.cc:654] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.478894 30114 raft_consensus.cc:493] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.479578 30114 raft_consensus.cc:515] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.480685 30075 tablet_bootstrap.cc:654] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.482172 30114 leader_election.cc:290] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.484422 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.485297 30028 raft_consensus.cc:2468] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:48.487010 29847 leader_election.cc:304] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:48.488343 30114 raft_consensus.cc:2804] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.488840 30114 raft_consensus.cc:493] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.489217 30114 raft_consensus.cc:3060] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.484498 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:48.491943 29847 leader_election.cc:343] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.496946 30075 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:48.497720 30075 ts_tablet_manager.cc:1403] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.011s
I20260709 12:27:48.498237 30077 tablet_bootstrap.cc:492] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:48.498853 30077 ts_tablet_manager.cc:1403] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.041s	user 0.013s	sys 0.024s
I20260709 12:27:48.501180 30075 raft_consensus.cc:359] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.501869 30075 raft_consensus.cc:385] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.502127 30075 raft_consensus.cc:740] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.502378 30077 raft_consensus.cc:359] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.502853 30075 consensus_queue.cc:260] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.503233 30077 raft_consensus.cc:385] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.503577 30077 raft_consensus.cc:740] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.504791 30075 ts_tablet_manager.cc:1434] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:27:48.504365 30077 consensus_queue.cc:260] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.506139 30075 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.507156 30077 ts_tablet_manager.cc:1434] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:48.507331 30114 raft_consensus.cc:515] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.508466 30077 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:48.510795 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
W20260709 12:27:48.512633 29847 leader_election.cc:343] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.515247 30075 tablet_bootstrap.cc:654] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.516899 30074 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:48.517520 30074 ts_tablet_manager.cc:1403] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.086s	user 0.014s	sys 0.016s
I20260709 12:27:48.519124 30135 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.519721 30135 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.520700 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:48.521334 30028 raft_consensus.cc:3060] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.521512 30074 raft_consensus.cc:359] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.522282 30074 raft_consensus.cc:385] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.522650 30074 raft_consensus.cc:740] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.523386 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:48.523466 30074 consensus_queue.cc:260] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.524741 30135 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.524883 30133 raft_consensus.cc:493] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.525403 30133 raft_consensus.cc:515] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.528088 30133 leader_election.cc:290] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.528869 30074 ts_tablet_manager.cc:1434] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.003s
I20260709 12:27:48.530124 30074 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.530962 30076 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:48.531607 30076 ts_tablet_manager.cc:1403] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.086s	user 0.014s	sys 0.002s
I20260709 12:27:48.533259 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:48.533995 30114 leader_election.cc:290] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.534096 30028 raft_consensus.cc:2468] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:48.535785 29847 leader_election.cc:304] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.536231 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.536718 30114 raft_consensus.cc:2804] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.537438 30074 tablet_bootstrap.cc:654] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.536974 29955 raft_consensus.cc:2468] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:48.539219 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.540223 29772 leader_election.cc:304] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:48.534145 29881 raft_consensus.cc:2468] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:48.541759 29995 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.542377 30133 raft_consensus.cc:493] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.544498 30135 raft_consensus.cc:2749] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.545452 30129 raft_consensus.cc:2804] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.547832 30129 raft_consensus.cc:493] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.547443 30133 raft_consensus.cc:515] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.548244 30129 raft_consensus.cc:3060] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.551409 30133 leader_election.cc:290] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:48.536542 30076 raft_consensus.cc:359] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.553105 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.553529 30076 raft_consensus.cc:385] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.554013 30028 raft_consensus.cc:2468] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:48.554091 30076 raft_consensus.cc:740] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.554270 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:48.555500 29772 leader_election.cc:304] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:48.555394 30076 consensus_queue.cc:260] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:48.557139 29769 leader_election.cc:343] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.557116 30133 raft_consensus.cc:2804] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.558845 30129 raft_consensus.cc:515] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.559482 30133 raft_consensus.cc:493] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.560081 30133 raft_consensus.cc:3060] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.563472 30077 tablet_bootstrap.cc:654] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.564594 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:48.565452 29881 raft_consensus.cc:3060] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.566167 30114 raft_consensus.cc:697] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:48.567524 30114 consensus_queue.cc:237] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.571417 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:48.572220 29954 raft_consensus.cc:3060] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.574923 30075 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:48.575529 30075 ts_tablet_manager.cc:1403] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.070s	user 0.020s	sys 0.000s
I20260709 12:27:48.574174 30129 leader_election.cc:290] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.577195 30074 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:48.577840 30074 ts_tablet_manager.cc:1403] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.048s	user 0.013s	sys 0.003s
I20260709 12:27:48.582872 30074 raft_consensus.cc:359] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.583699 30074 raft_consensus.cc:385] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.584167 30074 raft_consensus.cc:740] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.584997 30074 consensus_queue.cc:260] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.587424 30074 ts_tablet_manager.cc:1434] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:48.588958 30074 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.589982 30076 ts_tablet_manager.cc:1434] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.058s	user 0.018s	sys 0.002s
I20260709 12:27:48.591282 30076 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:48.587424 29954 raft_consensus.cc:2468] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:48.593753 29772 leader_election.cc:304] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:48.595417 30129 raft_consensus.cc:2804] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.596669 30129 raft_consensus.cc:697] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:48.598441 30129 consensus_queue.cc:237] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.596787 29881 raft_consensus.cc:2468] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:48.603245 30075 raft_consensus.cc:359] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.604151 30075 raft_consensus.cc:385] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.604506 30075 raft_consensus.cc:740] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.605396 30075 consensus_queue.cc:260] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.607924 30075 ts_tablet_manager.cc:1434] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 12:27:48.612444 30074 tablet_bootstrap.cc:654] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.608350 30112 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.615054 30112 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.616593 30075 tablet_bootstrap.cc:492] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.617888 30133 raft_consensus.cc:515] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.618578 30112 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.624011 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.624931 30028 raft_consensus.cc:2468] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:48.626075 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:48.626739 30076 tablet_bootstrap.cc:654] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.626757 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.624042 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:48.620910 30133 leader_election.cc:290] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:48.626849 30029 raft_consensus.cc:3060] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:48.629099 29769 leader_election.cc:343] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.631552 29772 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
W20260709 12:27:48.632664 29772 leader_election.cc:343] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.637979 30077 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:48.639026 30077 ts_tablet_manager.cc:1403] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.131s	user 0.024s	sys 0.008s
I20260709 12:27:48.637938 29702 catalog_manager.cc:5697] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.644085 30077 raft_consensus.cc:359] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.645381 30077 raft_consensus.cc:385] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.646569 29703 catalog_manager.cc:5697] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.651015 30114 raft_consensus.cc:493] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.651177 30077 raft_consensus.cc:740] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.651850 30114 raft_consensus.cc:515] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.652940 30077 consensus_queue.cc:260] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.657042 30114 leader_election.cc:290] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:48.665139 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:48.666079 29805 raft_consensus.cc:2468] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:48.667937 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.670967 30028 raft_consensus.cc:2468] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:48.669777 29846 leader_election.cc:304] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:48.672832 30027 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.678570 30114 raft_consensus.cc:2804] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.679044 30114 raft_consensus.cc:493] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.679414 30114 raft_consensus.cc:3060] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.674149 30107 raft_consensus.cc:493] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.679602 30077 ts_tablet_manager.cc:1434] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.040s	user 0.013s	sys 0.000s
I20260709 12:27:48.681485 30077 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:48.682999 30135 raft_consensus.cc:493] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.683920 30074 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:48.683784 30135 raft_consensus.cc:515] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.684804 30074 ts_tablet_manager.cc:1403] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.096s	user 0.021s	sys 0.007s
I20260709 12:27:48.663401 30133 raft_consensus.cc:2804] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.691022 30133 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.691568 30133 raft_consensus.cc:3060] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.736217 30112 consensus_queue.cc:1048] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:48.748929 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 71168.3us
I20260709 12:27:48.680100 30107 raft_consensus.cc:515] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.751914 30075 tablet_bootstrap.cc:654] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.753705 30107 leader_election.cc:290] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.754578 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a554edb2449436ea89ed389c4aed651" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:48.756367 29921 leader_election.cc:343] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.758510 30029 raft_consensus.cc:2468] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:48.760980 29772 leader_election.cc:304] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.762642 30112 raft_consensus.cc:2804] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.763317 30112 raft_consensus.cc:697] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:48.764379 30112 consensus_queue.cc:237] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.766456 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:48.769181 29992 leader_election.cc:343] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.754774 30107 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.772615 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a554edb2449436ea89ed389c4aed651" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:48.760984 30074 raft_consensus.cc:359] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.778442 30074 raft_consensus.cc:385] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.779012 30074 raft_consensus.cc:740] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.780341 30074 consensus_queue.cc:260] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.785277 30077 tablet_bootstrap.cc:654] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.787428 30074 ts_tablet_manager.cc:1434] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.102s	user 0.000s	sys 0.027s
I20260709 12:27:48.801044 30133 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.792016 29881 raft_consensus.cc:2468] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:48.783000 30107 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.806902 30135 leader_election.cc:290] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.816793 30165 raft_consensus.cc:493] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.816952 29918 leader_election.cc:304] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:48.817523 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.817561 30165 raft_consensus.cc:515] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.819439 29955 raft_consensus.cc:2468] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:48.819911 30076 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:48.820493 30163 raft_consensus.cc:2804] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.820637 30076 ts_tablet_manager.cc:1403] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.230s	user 0.044s	sys 0.108s
I20260709 12:27:48.820904 30165 leader_election.cc:290] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.821005 30163 raft_consensus.cc:493] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.821887 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:48.821652 30163 raft_consensus.cc:3060] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.822427 30129 consensus_queue.cc:1048] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.822751 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.823431 30029 raft_consensus.cc:2468] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:48.826515 30107 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:48.826864 30076 raft_consensus.cc:359] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:48.823629 29918 leader_election.cc:343] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.828426 30136 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.828802 30133 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.829015 30136 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.830101 30076 raft_consensus.cc:385] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.829806 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:48.831467 30076 raft_consensus.cc:740] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.832186 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:48.833639 30076 consensus_queue.cc:260] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.834463 29953 raft_consensus.cc:3060] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.836313 30136 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:48.837874 29805 raft_consensus.cc:2393] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:48.837884 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:48.830114 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:48.839241 30029 raft_consensus.cc:3060] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.839663 30075 tablet_bootstrap.cc:492] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:48.838325 30077 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
W20260709 12:27:48.839789 29994 leader_election.cc:343] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.840265 30075 ts_tablet_manager.cc:1403] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.224s	user 0.010s	sys 0.009s
I20260709 12:27:48.840529 30077 ts_tablet_manager.cc:1403] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.159s	user 0.037s	sys 0.004s
I20260709 12:27:48.837908 30076 ts_tablet_manager.cc:1434] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:27:48.842053 30076 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:48.844906 30114 raft_consensus.cc:515] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:48.843492 30107 raft_consensus.cc:493] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.845353 30077 raft_consensus.cc:359] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.846923 30077 raft_consensus.cc:385] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.846652 30107 raft_consensus.cc:515] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.847240 30077 raft_consensus.cc:740] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.848107 30077 consensus_queue.cc:260] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.858633 30077 ts_tablet_manager.cc:1434] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 12:27:48.860091 30077 tablet_bootstrap.cc:492] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:48.857656 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:48.857726 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:48.866900 30075 raft_consensus.cc:359] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.856487 30114 leader_election.cc:290] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:48.869416 30075 raft_consensus.cc:385] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.871094 30075 raft_consensus.cc:740] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.872561 30074 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:48.871313 30133 raft_consensus.cc:493] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.874125 30133 raft_consensus.cc:515] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.872466 29805 raft_consensus.cc:3060] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.871537 30028 raft_consensus.cc:3060] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.877341 29921 leader_election.cc:304] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.880091 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:48.882942 29921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 58495.2us
I20260709 12:27:48.881352 30129 raft_consensus.cc:493] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.884135 30129 raft_consensus.cc:515] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.882942 30165 raft_consensus.cc:2804] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.885224 30165 raft_consensus.cc:493] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.885607 30165 raft_consensus.cc:3060] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.886768 30129 leader_election.cc:290] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.887096 30163 raft_consensus.cc:515] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.873108 30075 consensus_queue.cc:260] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:48.894079 29703 catalog_manager.cc:5697] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.899729 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.900969 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:48.903728 29772 leader_election.cc:343] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:48.896369 29769 leader_election.cc:343] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.903774 29995 leader_election.cc:304] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
W20260709 12:27:48.907575 29995 leader_election.cc:343] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.908128 29995 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.909081 30085 raft_consensus.cc:2804] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.910008 30135 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.910239 30133 leader_election.cc:290] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.909569 30085 raft_consensus.cc:493] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.917018 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.918885 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a554edb2449436ea89ed389c4aed651" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:48.919632 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:48.920624 30029 raft_consensus.cc:2468] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:48.920756 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.930565 30025 raft_consensus.cc:2393] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:48.919657 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:48.931501 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:48.922022 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:48.932618 29921 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:48.919689 29880 raft_consensus.cc:3060] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.925408 30075 ts_tablet_manager.cc:1434] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.084s	user 0.011s	sys 0.000s
W20260709 12:27:48.928957 29918 leader_election.cc:343] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.913391 30085 raft_consensus.cc:3060] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.932930 30114 raft_consensus.cc:493] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:48.922063 29772 leader_election.cc:343] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.939714 30075 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:48.925212 30107 leader_election.cc:290] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:27:48.944195 29920 leader_election.cc:343] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.957170 29920 leader_election.cc:304] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.962266 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a554edb2449436ea89ed389c4aed651" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:48.967572 30028 raft_consensus.cc:2468] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:48.949790 30163 leader_election.cc:290] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:48.948282 29772 leader_election.cc:304] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:48.969753 29847 leader_election.cc:304] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:48.972708 30163 raft_consensus.cc:2749] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.973237 30112 raft_consensus.cc:2749] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:48.973459 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 51457.8us
I20260709 12:27:48.973656 30157 raft_consensus.cc:2804] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.974434 30157 raft_consensus.cc:697] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:48.974932 29772 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:48.975471 30157 consensus_queue.cc:237] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:48.976065 29772 leader_election.cc:343] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.976615 29772 leader_election.cc:304] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:48.946628 30114 raft_consensus.cc:515] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:48.977674 30129 raft_consensus.cc:2804] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.978247 30129 raft_consensus.cc:697] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:48.979379 30112 raft_consensus.cc:2749] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.979233 30129 consensus_queue.cc:237] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:48.979864 30114 leader_election.cc:290] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:48.981143 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:48.982221 29806 raft_consensus.cc:2468] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:48.984349 30076 tablet_bootstrap.cc:654] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.986621 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:48.987711 29954 raft_consensus.cc:2468] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:48.984300 29846 leader_election.cc:304] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:49.011581 29805 raft_consensus.cc:2468] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
W20260709 12:27:48.998667 29921 leader_election.cc:343] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.003211 30114 raft_consensus.cc:2804] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.006775 30165 raft_consensus.cc:515] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.002310 29953 raft_consensus.cc:2468] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:49.032718 30107 raft_consensus.cc:2749] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.023119 30114 raft_consensus.cc:493] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.037024 30114 raft_consensus.cc:3060] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.043083 29880 raft_consensus.cc:2468] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:49.044685 29918 leader_election.cc:304] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:49.045725 30107 raft_consensus.cc:2804] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.046273 30107 raft_consensus.cc:697] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:49.026167 30085 raft_consensus.cc:515] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.056154 30074 tablet_bootstrap.cc:654] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.057150 30077 tablet_bootstrap.cc:654] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.047185 30107 consensus_queue.cc:237] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.060935 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:49.061334 30165 leader_election.cc:290] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
W20260709 12:27:49.064177 29992 leader_election.cc:343] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.065325 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:49.066743 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:49.067836 30027 raft_consensus.cc:2393] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
W20260709 12:27:49.067914 29918 leader_election.cc:343] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.069391 29921 leader_election.cc:304] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:49.070267 30075 tablet_bootstrap.cc:654] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.074376 30165 raft_consensus.cc:2749] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.083186 29704 catalog_manager.cc:5697] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:49.080744 30085 leader_election.cc:290] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.094499 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.095726 29954 raft_consensus.cc:2393] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:49.098459 29995 leader_election.cc:304] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:49.100257 30076 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:49.100723 30077 tablet_bootstrap.cc:492] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:49.101061 30076 ts_tablet_manager.cc:1403] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.259s	user 0.024s	sys 0.012s
I20260709 12:27:49.101264 30077 ts_tablet_manager.cc:1403] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.241s	user 0.016s	sys 0.003s
I20260709 12:27:49.109419 30076 raft_consensus.cc:359] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.110335 30076 raft_consensus.cc:385] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.110643 30076 raft_consensus.cc:740] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.111579 30076 consensus_queue.cc:260] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.111971 30077 raft_consensus.cc:359] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.112973 30077 raft_consensus.cc:385] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.116324 30074 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:49.116520 30077 raft_consensus.cc:740] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.117049 30074 ts_tablet_manager.cc:1403] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.245s	user 0.020s	sys 0.011s
I20260709 12:27:49.118131 30077 consensus_queue.cc:260] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.105261 30085 raft_consensus.cc:2749] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.122054 30077 ts_tablet_manager.cc:1434] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.004s
I20260709 12:27:49.125600 30077 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:49.122222 29704 catalog_manager.cc:5697] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:49.114182 30076 ts_tablet_manager.cc:1434] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 12:27:49.127632 30076 tablet_bootstrap.cc:492] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:49.128358 30085 raft_consensus.cc:493] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.115684 30114 raft_consensus.cc:515] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.129357 30085 raft_consensus.cc:515] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.133137 30114 leader_election.cc:290] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.125190 30074 raft_consensus.cc:359] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.134012 30077 tablet_bootstrap.cc:654] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.133497 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:49.134922 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:49.135692 29805 raft_consensus.cc:3060] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:49.136495 29992 leader_election.cc:343] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.137909 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.138741 29954 raft_consensus.cc:3060] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.147781 30085 leader_election.cc:290] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.134095 30074 raft_consensus.cc:385] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.149000 30074 raft_consensus.cc:740] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.149348 30075 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:49.149174 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.150071 30075 ts_tablet_manager.cc:1403] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.211s	user 0.017s	sys 0.006s
I20260709 12:27:49.151017 30084 consensus_queue.cc:1048] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:49.151827 29995 leader_election.cc:343] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.152400 29995 leader_election.cc:304] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:49.152594 29805 raft_consensus.cc:2468] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:49.153342 30085 raft_consensus.cc:2749] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.154274 30074 consensus_queue.cc:260] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.154714 29703 catalog_manager.cc:5697] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:49.159749 30075 raft_consensus.cc:359] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.160617 30075 raft_consensus.cc:385] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.160903 30075 raft_consensus.cc:740] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.161127 30074 ts_tablet_manager.cc:1434] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.000s
I20260709 12:27:49.162483 30074 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:49.164103 29954 raft_consensus.cc:2468] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:49.165349 30075 consensus_queue.cc:260] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.165802 29847 leader_election.cc:304] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:49.166839 30114 raft_consensus.cc:2804] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.167605 30114 raft_consensus.cc:697] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:49.169162 30084 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.169749 30084 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.172322 30084 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.173151 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:49.168582 30114 consensus_queue.cc:237] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:49.175827 29769 leader_election.cc:343] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.176673 30077 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:49.177372 30077 ts_tablet_manager.cc:1403] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.052s	user 0.007s	sys 0.009s
I20260709 12:27:49.178009 30155 raft_consensus.cc:493] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.178638 30155 raft_consensus.cc:515] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.179708 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.181746 30155 leader_election.cc:290] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:49.181183 30075 ts_tablet_manager.cc:1434] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.001s
I20260709 12:27:49.182818 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:49.183022 29880 raft_consensus.cc:3060] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.183765 30027 raft_consensus.cc:2393] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
W20260709 12:27:49.184801 29772 leader_election.cc:343] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.185338 29772 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:49.186264 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.188308 29847 leader_election.cc:304] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:49.189558 30084 raft_consensus.cc:2749] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.197682 30076 tablet_bootstrap.cc:654] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.188263 30077 raft_consensus.cc:359] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.198311 30077 raft_consensus.cc:385] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.198638 30077 raft_consensus.cc:740] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.199503 30077 consensus_queue.cc:260] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.200227 30155 consensus_queue.cc:1048] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.201031 30157 raft_consensus.cc:2749] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.202997 30077 ts_tablet_manager.cc:1434] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.025s	user 0.010s	sys 0.000s
I20260709 12:27:49.204281 30077 tablet_bootstrap.cc:492] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
W20260709 12:27:49.211640 29847 consensus_peers.cc:597] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:49.212136 30077 tablet_bootstrap.cc:654] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.213165 30075 tablet_bootstrap.cc:492] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:49.220257 30133 consensus_queue.cc:1048] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.218758 30084 consensus_queue.cc:1048] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:27:49.231278 30074 tablet_bootstrap.cc:654] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.233861 30076 tablet_bootstrap.cc:492] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:49.233796 30077 tablet_bootstrap.cc:492] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:49.234787 30076 ts_tablet_manager.cc:1403] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.107s	user 0.013s	sys 0.008s
I20260709 12:27:49.234827 30077 ts_tablet_manager.cc:1403] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.000s
I20260709 12:27:49.236891 30075 tablet_bootstrap.cc:654] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.255543 30077 raft_consensus.cc:359] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.256453 30077 raft_consensus.cc:385] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.262765 30077 raft_consensus.cc:740] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.268388 30076 raft_consensus.cc:359] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.269323 30076 raft_consensus.cc:385] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.269636 30076 raft_consensus.cc:740] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.274595 30076 consensus_queue.cc:260] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.280411 30077 consensus_queue.cc:260] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.283157 30129 raft_consensus.cc:493] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.284150 30129 raft_consensus.cc:515] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.291184 29702 catalog_manager.cc:5697] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:49.298651 30076 ts_tablet_manager.cc:1434] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.063s	user 0.005s	sys 0.004s
I20260709 12:27:49.300014 30076 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:49.301525 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.302917 29953 raft_consensus.cc:2468] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:49.305732 30129 leader_election.cc:290] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:49.307960 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:49.308421 29772 leader_election.cc:304] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:49.308892 30027 raft_consensus.cc:2468] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:49.311215 30077 ts_tablet_manager.cc:1434] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.076s	user 0.009s	sys 0.000s
I20260709 12:27:49.311610 30076 tablet_bootstrap.cc:654] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.313608 30077 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:49.314823 30129 raft_consensus.cc:2804] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.315335 30129 raft_consensus.cc:493] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.315747 30129 raft_consensus.cc:3060] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.320705 30155 raft_consensus.cc:493] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.321444 30155 raft_consensus.cc:515] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.322410 30074 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:49.325062 30155 leader_election.cc:290] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.327128 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:49.328883 29846 leader_election.cc:343] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.329617 30074 ts_tablet_manager.cc:1403] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.167s	user 0.026s	sys 0.029s
I20260709 12:27:49.331827 30075 tablet_bootstrap.cc:492] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:49.332430 30075 ts_tablet_manager.cc:1403] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.120s	user 0.020s	sys 0.012s
I20260709 12:27:49.337157 30074 raft_consensus.cc:359] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.337018 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.342609 29954 raft_consensus.cc:2468] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:49.353004 29847 leader_election.cc:304] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:27:49.352521 30075 raft_consensus.cc:359] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.363039 30075 raft_consensus.cc:385] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.363281 30155 raft_consensus.cc:2804] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.363557 30075 raft_consensus.cc:740] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.363873 30155 raft_consensus.cc:493] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.364337 30155 raft_consensus.cc:3060] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.355990 30112 consensus_queue.cc:1048] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:27:49.366264 30077 tablet_bootstrap.cc:654] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.368281 30074 raft_consensus.cc:385] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.368785 30074 raft_consensus.cc:740] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.369928 30075 consensus_queue.cc:260] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.372627 30075 ts_tablet_manager.cc:1434] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.040s	user 0.008s	sys 0.000s
I20260709 12:27:49.373924 30075 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:49.375118 30157 consensus_queue.cc:1048] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.369995 30074 consensus_queue.cc:260] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.379944 30074 ts_tablet_manager.cc:1434] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.000s
I20260709 12:27:49.382128 30074 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:49.386360 30155 raft_consensus.cc:515] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.395361 30076 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:49.396481 30076 ts_tablet_manager.cc:1403] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.097s	user 0.013s	sys 0.001s
I20260709 12:27:49.398461 30074 tablet_bootstrap.cc:654] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.395540 30075 tablet_bootstrap.cc:654] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.405761 30129 raft_consensus.cc:515] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.409583 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.410501 29954 raft_consensus.cc:3060] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.412990 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:49.425186 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.426330 29953 raft_consensus.cc:3060] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:49.430073 29846 leader_election.cc:343] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.440510 29953 raft_consensus.cc:2468] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:49.440788 30084 consensus_queue.cc:1048] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.442375 30076 raft_consensus.cc:359] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.442667 29847 leader_election.cc:304] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:27:49.443552 30076 raft_consensus.cc:385] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.443984 30076 raft_consensus.cc:740] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.444234 30157 raft_consensus.cc:2804] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.444933 30157 raft_consensus.cc:697] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:49.445145 30076 consensus_queue.cc:260] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.414469 30155 leader_election.cc:290] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.447579 30157 consensus_queue.cc:237] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.459652 30129 leader_election.cc:290] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:49.463572 29954 raft_consensus.cc:2468] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:49.466084 29772 leader_election.cc:304] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:49.469043 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:49.469993 30027 raft_consensus.cc:3060] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.474381 30133 raft_consensus.cc:2804] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.475142 30133 raft_consensus.cc:697] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:49.476202 30133 consensus_queue.cc:237] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.497618 30076 ts_tablet_manager.cc:1434] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.101s	user 0.026s	sys 0.011s
I20260709 12:27:49.494618 30077 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:49.551702 30074 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:49.552275 30076 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:49.552501 30074 ts_tablet_manager.cc:1403] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.171s	user 0.041s	sys 0.045s
I20260709 12:27:49.497618 30112 consensus_queue.cc:1048] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20260709 12:27:49.552340 30077 ts_tablet_manager.cc:1403] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.239s	user 0.029s	sys 0.018s
I20260709 12:27:49.556955 30074 raft_consensus.cc:359] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.530282 30136 raft_consensus.cc:493] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.566776 30136 raft_consensus.cc:515] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.569687 30136 leader_election.cc:290] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.568780 29704 catalog_manager.cc:5697] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:49.570756 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:49.571187 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.571600 29881 raft_consensus.cc:2468] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:49.571970 29953 raft_consensus.cc:2468] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:49.525964 30028 raft_consensus.cc:3060] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.573865 29995 leader_election.cc:304] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:49.574397 30076 tablet_bootstrap.cc:654] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.521629 30027 raft_consensus.cc:2468] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:49.533977 30165 raft_consensus.cc:493] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.552802 30075 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:49.578063 30165 raft_consensus.cc:515] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.579182 30075 ts_tablet_manager.cc:1403] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.206s	user 0.017s	sys 0.023s
I20260709 12:27:49.580899 30107 consensus_queue.cc:1048] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.565310 30074 raft_consensus.cc:385] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.564851 30157 consensus_queue.cc:1048] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.582863 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:49.611919 30026 raft_consensus.cc:2468] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:49.612195 30157 raft_consensus.cc:493] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.588706 30075 raft_consensus.cc:359] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.613198 30157 raft_consensus.cc:515] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.613971 29921 leader_election.cc:304] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:49.614332 30075 raft_consensus.cc:385] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.614715 30075 raft_consensus.cc:740] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.615670 30075 consensus_queue.cc:260] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.584409 30136 raft_consensus.cc:2804] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.588965 30077 raft_consensus.cc:359] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.620929 30077 raft_consensus.cc:385] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.621420 30077 raft_consensus.cc:740] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.584662 30165 leader_election.cc:290] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:49.597880 29701 catalog_manager.cc:5697] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:49.619671 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.605788 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:49.620144 30075 ts_tablet_manager.cc:1434] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.041s	user 0.009s	sys 0.001s
I20260709 12:27:49.582496 30074 raft_consensus.cc:740] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.623353 30077 consensus_queue.cc:260] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.625938 30136 raft_consensus.cc:493] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.632534 30075 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:49.636221 30076 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:49.636840 30076 ts_tablet_manager.cc:1403] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.085s	user 0.018s	sys 0.044s
I20260709 12:27:49.638051 30136 raft_consensus.cc:3060] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.646457 30074 consensus_queue.cc:260] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.647190 30076 raft_consensus.cc:359] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.648013 30076 raft_consensus.cc:385] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.648342 30076 raft_consensus.cc:740] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.648667 30074 ts_tablet_manager.cc:1434] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.096s	user 0.008s	sys 0.000s
I20260709 12:27:49.649227 30076 consensus_queue.cc:260] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.630908 30203 raft_consensus.cc:2804] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.650264 30203 raft_consensus.cc:493] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.650641 30203 raft_consensus.cc:3060] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.621109 30157 leader_election.cc:290] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:27:49.651906 29846 leader_election.cc:343] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.652536 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:49.653283 30076 ts_tablet_manager.cc:1434] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.004s
I20260709 12:27:49.654490 30076 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:49.654981 30077 ts_tablet_manager.cc:1434] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.098s	user 0.005s	sys 0.005s
W20260709 12:27:49.656173 29918 leader_election.cc:343] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.599778 30204 consensus_queue.cc:1048] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
W20260709 12:27:49.660079 29847 leader_election.cc:343] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.660640 29847 leader_election.cc:304] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:49.662034 30076 tablet_bootstrap.cc:654] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.664238 30157 raft_consensus.cc:2749] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.656188 30077 tablet_bootstrap.cc:492] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:49.669656 30074 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:49.697767 30203 raft_consensus.cc:515] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.707640 30077 tablet_bootstrap.cc:654] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.708981 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:49.709079 30203 leader_election.cc:290] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:49.710362 30025 raft_consensus.cc:3060] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.713486 30136 raft_consensus.cc:515] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.720322 30075 tablet_bootstrap.cc:654] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.726819 30157 consensus_queue.cc:1048] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:49.738026 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:49.740022 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:49.740990 29879 raft_consensus.cc:3060] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.742033 30085 raft_consensus.cc:493] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.742707 30085 raft_consensus.cc:515] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.745152 30136 leader_election.cc:290] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.746081 30076 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:49.746821 30076 ts_tablet_manager.cc:1403] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.093s	user 0.023s	sys 0.003s
I20260709 12:27:49.751073 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a7001c5f324ba1955a1eef4ea15e93" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.754580 30085 leader_election.cc:290] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:49.756808 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a7001c5f324ba1955a1eef4ea15e93" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:49.757928 29994 leader_election.cc:343] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.758778 30076 raft_consensus.cc:359] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.759662 30076 raft_consensus.cc:385] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.760425 30076 raft_consensus.cc:740] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.761289 30076 consensus_queue.cc:260] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.765802 30157 raft_consensus.cc:493] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.766772 30157 raft_consensus.cc:515] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.763901 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.770385 29954 raft_consensus.cc:3060] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:49.759680 29992 leader_election.cc:343] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.774220 29992 leader_election.cc:304] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:49.776364 30085 raft_consensus.cc:2749] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.778465 30076 ts_tablet_manager.cc:1434] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.003s
I20260709 12:27:49.782133 30084 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:49.782388 29918 leader_election.cc:343] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.783104 30084 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.785084 30074 tablet_bootstrap.cc:654] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.785271 30076 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:49.788620 30157 leader_election.cc:290] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.789274 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.790192 29805 raft_consensus.cc:2468] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:49.791850 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:49.792110 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.792752 30028 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:49.794363 29846 leader_election.cc:304] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:49.796079 29772 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:49.796530 30075 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:49.797084 30075 ts_tablet_manager.cc:1403] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.165s	user 0.028s	sys 0.024s
I20260709 12:27:49.798069 30133 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.798491 30133 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.798816 30133 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.799777 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:49.802822 30075 raft_consensus.cc:359] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.803655 30075 raft_consensus.cc:385] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.803936 30075 raft_consensus.cc:740] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
W20260709 12:27:49.804278 29847 leader_election.cc:343] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.804700 30075 consensus_queue.cc:260] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:49.797087 29772 leader_election.cc:343] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.807345 30200 raft_consensus.cc:2804] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.808007 30200 raft_consensus.cc:493] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.808488 30200 raft_consensus.cc:3060] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.817279 30084 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:49.817866 30075 ts_tablet_manager.cc:1434] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.006s
I20260709 12:27:49.820415 30075 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:49.793671 29879 raft_consensus.cc:2468] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:49.822265 29992 leader_election.cc:304] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:49.823261 30136 raft_consensus.cc:2804] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.825925 30076 tablet_bootstrap.cc:654] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.819494 30157 consensus_queue.cc:1048] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:27:49.847424 30075 tablet_bootstrap.cc:654] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.862376 30136 raft_consensus.cc:697] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:49.872824 30077 tablet_bootstrap.cc:492] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:49.873680 30077 ts_tablet_manager.cc:1403] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.218s	user 0.029s	sys 0.014s
I20260709 12:27:49.876094 30114 raft_consensus.cc:493] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.876713 30114 raft_consensus.cc:515] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.879298 30114 leader_election.cc:290] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:49.870067 30200 raft_consensus.cc:515] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.883175 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a7001c5f324ba1955a1eef4ea15e93" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:49.882965 30077 raft_consensus.cc:359] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.883795 30077 raft_consensus.cc:385] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.883900 30026 raft_consensus.cc:2468] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:49.884119 30077 raft_consensus.cc:740] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.885105 30077 consensus_queue.cc:260] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.885543 30200 leader_election.cc:290] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:49.886847 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a7001c5f324ba1955a1eef4ea15e93" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.887111 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:49.887823 29806 raft_consensus.cc:3060] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.890113 30076 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:49.890316 30025 raft_consensus.cc:2468] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:49.890733 30076 ts_tablet_manager.cc:1403] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.111s	user 0.021s	sys 0.000s
I20260709 12:27:49.887734 30077 ts_tablet_manager.cc:1434] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:27:49.892642 30077 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:49.892937 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.893004 29921 leader_election.cc:304] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:49.891161 29847 leader_election.cc:304] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:49.871134 30136 consensus_queue.cc:237] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.897161 30200 consensus_queue.cc:1048] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260709 12:27:49.902149 29846 leader_election.cc:343] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.903244 30076 raft_consensus.cc:359] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.904137 30076 raft_consensus.cc:385] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.904418 30076 raft_consensus.cc:740] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.897024 30107 raft_consensus.cc:2804] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.906862 30107 raft_consensus.cc:697] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:49.908116 30107 consensus_queue.cc:237] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:49.914968 30074 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:49.915666 30074 ts_tablet_manager.cc:1403] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.266s	user 0.019s	sys 0.022s
I20260709 12:27:49.905279 30076 consensus_queue.cc:260] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:49.921542 30200 raft_consensus.cc:2804] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.922115 30200 raft_consensus.cc:493] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.922524 30200 raft_consensus.cc:3060] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:49.923195 29847 leader_election.cc:343] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.925933 30204 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.926651 30204 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.930682 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.932608 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:49.933051 29920 leader_election.cc:343] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.930733 30085 raft_consensus.cc:493] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.935230 30085 raft_consensus.cc:515] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:49.936494 29921 leader_election.cc:343] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.937041 29954 raft_consensus.cc:2468] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:49.937723 30085 leader_election.cc:290] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:49.930737 30204 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:49.895238 30133 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:49.940300 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc3696cf98a402bb9d717b584cb229c" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:49.940579 30133 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:49.940495 30074 raft_consensus.cc:359] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.941448 30074 raft_consensus.cc:385] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:49.942006 29992 leader_election.cc:343] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.937091 29921 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:49.942904 30200 raft_consensus.cc:515] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.944039 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:49.944741 30025 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.946647 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a7001c5f324ba1955a1eef4ea15e93" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:49.946902 30076 ts_tablet_manager.cc:1434] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 12:27:49.947355 30026 raft_consensus.cc:3060] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.948984 30076 tablet_bootstrap.cc:492] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:49.948765 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:49.941774 30074 raft_consensus.cc:740] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.956032 30074 consensus_queue.cc:260] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:49.958102 30200 leader_election.cc:290] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:27:49.954381 29772 leader_election.cc:343] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.960301 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc3696cf98a402bb9d717b584cb229c" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:49.960541 30204 raft_consensus.cc:2749] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:49.962237 29805 raft_consensus.cc:3060] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.965001 30074 ts_tablet_manager.cc:1434] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.002s
I20260709 12:27:49.966356 30074 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:49.966382 29803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a7001c5f324ba1955a1eef4ea15e93" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:49.968065 30133 consensus_queue.cc:1048] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:49.968380 29846 leader_election.cc:343] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:49.975576 30076 tablet_bootstrap.cc:654] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.995453 30075 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:49.996187 30075 ts_tablet_manager.cc:1403] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.176s	user 0.009s	sys 0.020s
I20260709 12:27:49.973266 29806 raft_consensus.cc:2468] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:49.973879 29804 raft_consensus.cc:2468] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:49.996655 30077 tablet_bootstrap.cc:654] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.997941 30074 tablet_bootstrap.cc:654] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.995920 30026 raft_consensus.cc:2468] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:50.003033 30075 raft_consensus.cc:359] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.003629 29846 leader_election.cc:304] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:50.003985 30075 raft_consensus.cc:385] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.004657 30200 raft_consensus.cc:2804] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.005262 30200 raft_consensus.cc:697] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:50.003757 29847 leader_election.cc:304] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:27:50.006332 30200 consensus_queue.cc:237] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.008395 30157 raft_consensus.cc:2804] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.009184 30157 raft_consensus.cc:697] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:50.009783 30025 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:50.010424 30157 consensus_queue.cc:237] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.004304 30075 raft_consensus.cc:740] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.025938 30076 tablet_bootstrap.cc:492] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:50.026834 30076 ts_tablet_manager.cc:1403] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.078s	user 0.026s	sys 0.000s
I20260709 12:27:50.028038 30075 consensus_queue.cc:260] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.032603 30074 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:50.032734 30075 ts_tablet_manager.cc:1434] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.036s	user 0.009s	sys 0.001s
I20260709 12:27:50.033438 30074 ts_tablet_manager.cc:1403] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.067s	user 0.018s	sys 0.008s
I20260709 12:27:50.035876 30075 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:50.093874 29772 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:50.045674 30076 raft_consensus.cc:359] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.097610 30076 raft_consensus.cc:385] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.098237 30076 raft_consensus.cc:740] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.041975 30074 raft_consensus.cc:359] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.100001 30074 raft_consensus.cc:385] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.100596 30074 raft_consensus.cc:740] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.101068 29994 leader_election.cc:304] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
W20260709 12:27:50.103140 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:50.107035 30112 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.108310 30112 raft_consensus.cc:697] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:50.111449 30112 consensus_queue.cc:237] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.112409 30074 consensus_queue.cc:260] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.126256 30136 raft_consensus.cc:2804] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:50.126680 30133 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.127007 30136 raft_consensus.cc:493] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.128850 30204 raft_consensus.cc:493] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.127493 29701 catalog_manager.cc:5697] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:50.130052 30204 raft_consensus.cc:515] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.134893 29953 raft_consensus.cc:3060] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.140117 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:50.127610 30136 raft_consensus.cc:3060] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.128027 29703 catalog_manager.cc:5697] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:50.127650 30133 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.155972 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:50.159368 29921 leader_election.cc:304] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:50.099900 30076 consensus_queue.cc:260] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.180263 30076 ts_tablet_manager.cc:1434] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.152s	user 0.011s	sys 0.001s
I20260709 12:27:50.188176 30076 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:50.190526 30077 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:50.203115 30077 ts_tablet_manager.cc:1403] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.311s	user 0.016s	sys 0.005s
I20260709 12:27:50.207664 30074 ts_tablet_manager.cc:1434] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.174s	user 0.022s	sys 0.004s
I20260709 12:27:50.220297 30077 raft_consensus.cc:359] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.223397 30077 raft_consensus.cc:385] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.163265 30204 leader_election.cc:290] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.223887 30077 raft_consensus.cc:740] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.168924 30133 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.221454 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:50.233798 30074 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:50.234839 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:50.242651 30025 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:50.240729 30136 raft_consensus.cc:515] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.239040 29955 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:50.248684 30077 consensus_queue.cc:260] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.257112 30075 tablet_bootstrap.cc:654] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.260231 30077 ts_tablet_manager.cc:1434] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.056s	user 0.011s	sys 0.000s
I20260709 12:27:50.261693 30136 leader_election.cc:290] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:50.262629 30077 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:50.262871 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc3696cf98a402bb9d717b584cb229c" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:50.264127 29804 raft_consensus.cc:3060] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.267009 30084 raft_consensus.cc:493] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.268244 30084 raft_consensus.cc:515] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.258737 30112 consensus_queue.cc:1048] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:27:50.275442 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc3696cf98a402bb9d717b584cb229c" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:50.272908 30157 consensus_queue.cc:1048] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260709 12:27:50.278162 29992 leader_election.cc:343] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.273375 29772 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:50.285760 30076 tablet_bootstrap.cc:654] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.310717 30074 tablet_bootstrap.cc:654] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.303994 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:50.301121 30233 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:50.329685 30233 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.330193 30233 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.300343 30084 leader_election.cc:290] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:50.342646 30075 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:50.344214 30075 ts_tablet_manager.cc:1403] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.310s	user 0.011s	sys 0.013s
I20260709 12:27:50.360810 30157 raft_consensus.cc:493] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.363947 30157 raft_consensus.cc:515] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.372938 29880 raft_consensus.cc:3060] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.350565 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:50.392746 29772 leader_election.cc:304] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:50.403522 30136 consensus_queue.cc:1048] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.406908 30157 leader_election.cc:290] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:50.409976 30077 tablet_bootstrap.cc:654] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.412173 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:50.416513 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:50.419950 29846 leader_election.cc:343] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:50.421172 29847 leader_election.cc:343] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.422313 30075 raft_consensus.cc:359] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.423564 29847 leader_election.cc:304] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:50.426093 30075 raft_consensus.cc:385] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.430584 30075 raft_consensus.cc:740] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.434434 30075 consensus_queue.cc:260] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.435750 30157 raft_consensus.cc:2749] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:50.437327 30107 raft_consensus.cc:2749] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:50.463617 30085 consensus_queue.cc:1048] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.464887 29804 raft_consensus.cc:2468] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:50.466971 29994 leader_election.cc:304] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:50.469102 30230 raft_consensus.cc:2749] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:50.471098 30085 raft_consensus.cc:2804] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.471884 30085 raft_consensus.cc:697] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:50.479861 29955 raft_consensus.cc:3060] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.473167 30085 consensus_queue.cc:237] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.479810 29702 catalog_manager.cc:5697] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:50.508173 29846 consensus_peers.cc:597] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:50.510509 29702 catalog_manager.cc:5697] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:50.526206 29772 consensus_peers.cc:597] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:50.540063 30075 ts_tablet_manager.cc:1434] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.195s	user 0.007s	sys 0.005s
I20260709 12:27:50.563727 30165 consensus_queue.cc:1048] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.566788 30075 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:50.572005 30233 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.597337 30077 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:50.599189 30157 raft_consensus.cc:493] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.599920 30157 raft_consensus.cc:515] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.602391 30077 ts_tablet_manager.cc:1403] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.340s	user 0.035s	sys 0.044s
I20260709 12:27:50.606839 30165 raft_consensus.cc:493] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.605476 30157 leader_election.cc:290] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.608325 30200 consensus_queue.cc:1048] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.609706 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:50.610268 30200 consensus_queue.cc:1048] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.607702 30165 raft_consensus.cc:515] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.619932 30077 raft_consensus.cc:359] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.620551 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:50.620884 30077 raft_consensus.cc:385] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.621201 30077 raft_consensus.cc:740] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.621299 29954 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.626787 30107 consensus_queue.cc:1048] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:27:50.638372 30231 raft_consensus.cc:493] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.638772 30200 consensus_queue.cc:1048] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.639027 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:50.639334 30231 raft_consensus.cc:515] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.644997 30076 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:50.645797 30076 ts_tablet_manager.cc:1403] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.458s	user 0.020s	sys 0.012s
I20260709 12:27:50.631091 30077 consensus_queue.cc:260] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.652149 29954 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:50.654650 29772 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:50.657343 30165 leader_election.cc:290] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:50.658686 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:50.659734 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:50.660888 30230 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.661528 30230 raft_consensus.cc:697] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:50.662072 29918 leader_election.cc:304] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:50.662660 30230 consensus_queue.cc:237] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.666338 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611c12107cde4b0e81c03ffd67cb64e2" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:50.667258 30074 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:50.667951 30074 ts_tablet_manager.cc:1403] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.446s	user 0.030s	sys 0.039s
I20260709 12:27:50.671762 30074 raft_consensus.cc:359] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.672627 30074 raft_consensus.cc:385] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.672933 30074 raft_consensus.cc:740] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.674446 30231 leader_election.cc:290] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.675601 30233 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.675987 29847 leader_election.cc:304] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:50.679648 30107 raft_consensus.cc:2749] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:50.679953 30074 consensus_queue.cc:260] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.682411 30165 raft_consensus.cc:493] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.687870 30165 raft_consensus.cc:515] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:50.688287 29772 leader_election.cc:343] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.701884 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:50.702395 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:50.702847 30074 ts_tablet_manager.cc:1434] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.002s
I20260709 12:27:50.704169 30076 raft_consensus.cc:359] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.704929 30074 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:50.705103 30076 raft_consensus.cc:385] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.705543 30076 raft_consensus.cc:740] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.706442 30076 consensus_queue.cc:260] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.709698 30028 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.711457 30076 ts_tablet_manager.cc:1434] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.065s	user 0.007s	sys 0.001s
I20260709 12:27:50.719010 30076 tablet_bootstrap.cc:492] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:50.725699 30165 leader_election.cc:290] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:50.729316 30075 tablet_bootstrap.cc:654] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.735809 30085 raft_consensus.cc:493] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.738670 30077 ts_tablet_manager.cc:1434] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.136s	user 0.008s	sys 0.003s
W20260709 12:27:50.730926 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 59521us
I20260709 12:27:50.737493 30085 raft_consensus.cc:515] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.749706 30225 raft_consensus.cc:2749] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:50.753278 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:50.754233 29804 raft_consensus.cc:2468] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:50.754422 30074 tablet_bootstrap.cc:654] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.755779 29994 leader_election.cc:304] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:50.756937 30245 raft_consensus.cc:2804] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:50.757431 30245 raft_consensus.cc:493] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.757117 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:50.757862 30245 raft_consensus.cc:3060] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.758217 29881 raft_consensus.cc:2468] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:50.757272 30028 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:50.757246 30085 leader_election.cc:290] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:50.768263 30245 raft_consensus.cc:515] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:50.743996 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 68134.2us
I20260709 12:27:50.771647 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:50.773761 29920 leader_election.cc:343] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.776376 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:50.777690 29881 raft_consensus.cc:2468] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:50.782697 29918 leader_election.cc:304] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:27:50.783699 30165 raft_consensus.cc:2804] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:50.784150 30165 raft_consensus.cc:493] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.784526 30165 raft_consensus.cc:3060] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.808032 30076 tablet_bootstrap.cc:654] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:50.795131 29772 leader_election.cc:343] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.809692 29772 leader_election.cc:304] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:50.810874 30230 raft_consensus.cc:2749] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:50.812988 30230 consensus_queue.cc:1048] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.818924 30077 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:50.829077 30074 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:50.829878 30074 ts_tablet_manager.cc:1403] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.125s	user 0.017s	sys 0.029s
I20260709 12:27:50.834446 30075 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:50.835192 30075 ts_tablet_manager.cc:1403] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.293s	user 0.018s	sys 0.000s
I20260709 12:27:50.837481 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:50.838331 29881 raft_consensus.cc:3060] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.831549 30165 raft_consensus.cc:515] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.838829 30075 raft_consensus.cc:359] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.841188 30075 raft_consensus.cc:385] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.841599 30075 raft_consensus.cc:740] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.838418 29703 catalog_manager.cc:5697] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:50.843966 30075 consensus_queue.cc:260] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.844503 30245 leader_election.cc:290] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:50.844553 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:50.846956 30075 ts_tablet_manager.cc:1434] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:27:50.845590 30077 tablet_bootstrap.cc:654] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.848377 30075 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:50.848750 30165 leader_election.cc:290] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:50.852136 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:50.853647 29805 raft_consensus.cc:3060] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:50.862468 29920 leader_election.cc:343] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.880736 30075 tablet_bootstrap.cc:654] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.880530 29881 raft_consensus.cc:2468] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:50.884328 30165 raft_consensus.cc:493] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.885210 30165 raft_consensus.cc:515] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.887773 30165 leader_election.cc:290] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.888873 29805 raft_consensus.cc:2468] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:50.890599 29994 leader_election.cc:304] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:50.890851 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:50.891835 30028 raft_consensus.cc:2468] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:50.894469 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:50.889626 30074 raft_consensus.cc:359] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.896564 30074 raft_consensus.cc:385] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.896842 30074 raft_consensus.cc:740] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.899593 29921 leader_election.cc:304] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:50.901044 30165 raft_consensus.cc:2804] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:50.901597 30165 raft_consensus.cc:493] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.902041 30165 raft_consensus.cc:3060] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:50.901145 30245 raft_consensus.cc:2804] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.913434 30076 tablet_bootstrap.cc:492] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:50.914309 30076 ts_tablet_manager.cc:1403] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.202s	user 0.025s	sys 0.013s
I20260709 12:27:50.891920 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
W20260709 12:27:50.916155 29918 leader_election.cc:343] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.917690 30074 consensus_queue.cc:260] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.913597 30245 raft_consensus.cc:697] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:50.919746 30245 consensus_queue.cc:237] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:50.921857 29881 raft_consensus.cc:3060] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.927102 30165 raft_consensus.cc:515] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.926522 29702 catalog_manager.cc:5697] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:50.930907 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:50.931260 29702 catalog_manager.cc:5697] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:50.932750 30074 ts_tablet_manager.cc:1434] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.103s	user 0.008s	sys 0.000s
I20260709 12:27:50.934389 30074 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:50.937512 30165 leader_election.cc:290] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:50.938408 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34b3b0e1b5d14def8b90e20a7a38ee24" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:50.939422 30028 raft_consensus.cc:3060] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:50.925366 30076 raft_consensus.cc:359] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.941221 30076 raft_consensus.cc:385] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.941697 30076 raft_consensus.cc:740] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.955184 29881 raft_consensus.cc:2468] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:50.965472 29918 leader_election.cc:304] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:27:50.958758 30076 consensus_queue.cc:260] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:50.967778 29880 raft_consensus.cc:3060] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 2
I20260709 12:27:50.967860 30165 raft_consensus.cc:2804] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.968529 30165 raft_consensus.cc:697] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:50.969499 30165 consensus_queue.cc:237] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:50.973407 30076 ts_tablet_manager.cc:1434] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.059s	user 0.009s	sys 0.000s
I20260709 12:27:50.977442 30077 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:50.978242 30077 ts_tablet_manager.cc:1403] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.208s	user 0.017s	sys 0.011s
I20260709 12:27:50.978492 30076 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:50.981670 30077 raft_consensus.cc:359] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.986279 30077 raft_consensus.cc:385] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.986649 30077 raft_consensus.cc:740] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.987509 30077 consensus_queue.cc:260] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:50.988225 30074 tablet_bootstrap.cc:654] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.989516 30028 raft_consensus.cc:2468] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:27:50.993316 30075 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:50.999914 30075 ts_tablet_manager.cc:1403] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.152s	user 0.007s	sys 0.020s
I20260709 12:27:51.000720 29921 leader_election.cc:304] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:51.001739 30204 raft_consensus.cc:2804] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:51.002328 30204 raft_consensus.cc:697] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:50.993476 30077 ts_tablet_manager.cc:1434] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:27:51.004908 30157 raft_consensus.cc:493] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.005620 30077 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:51.005579 30157 raft_consensus.cc:515] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.007659 29704 catalog_manager.cc:5697] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:51.010102 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc3696cf98a402bb9d717b584cb229c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:51.011221 29805 raft_consensus.cc:2393] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:27:51.011303 29702 catalog_manager.cc:5697] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:51.013629 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc3696cf98a402bb9d717b584cb229c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.015785 30074 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:51.016451 30074 ts_tablet_manager.cc:1403] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.083s	user 0.014s	sys 0.000s
I20260709 12:27:51.018031 29847 leader_election.cc:304] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:51.019027 30200 raft_consensus.cc:3060] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.996556 29880 raft_consensus.cc:2468] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:27:51.019949 30074 raft_consensus.cc:359] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.020779 30074 raft_consensus.cc:385] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.021060 30074 raft_consensus.cc:740] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.022027 30074 consensus_queue.cc:260] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.023342 29805 raft_consensus.cc:3060] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.997558 30076 tablet_bootstrap.cc:654] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.003474 30204 consensus_queue.cc:237] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.028962 30077 tablet_bootstrap.cc:654] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.003612 30075 raft_consensus.cc:359] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.030630 30157 leader_election.cc:290] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:51.030913 30075 raft_consensus.cc:385] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.031335 30075 raft_consensus.cc:740] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.032361 30074 ts_tablet_manager.cc:1434] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.000s
I20260709 12:27:51.032200 30075 consensus_queue.cc:260] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.033924 30074 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:51.042797 30074 tablet_bootstrap.cc:654] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.043009 30085 consensus_queue.cc:1048] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:27:51.062640 30075 ts_tablet_manager.cc:1434] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.062s	user 0.008s	sys 0.001s
I20260709 12:27:51.063664 30074 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:51.064216 30075 tablet_bootstrap.cc:492] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:51.064337 30074 ts_tablet_manager.cc:1403] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.031s	user 0.020s	sys 0.000s
I20260709 12:27:51.068921 30074 raft_consensus.cc:359] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.069840 30074 raft_consensus.cc:385] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.070250 30074 raft_consensus.cc:740] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.071291 30074 consensus_queue.cc:260] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.073485 30074 ts_tablet_manager.cc:1434] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:51.093408 30074 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:51.113445 29701 catalog_manager.cc:5697] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 2, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:51.125700 30075 tablet_bootstrap.cc:654] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.122763 30200 raft_consensus.cc:2749] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:51.131784 30085 consensus_queue.cc:1048] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:51.138200 30076 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:51.138962 30076 ts_tablet_manager.cc:1403] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.161s	user 0.016s	sys 0.012s
I20260709 12:27:51.142324 30077 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:51.143021 30077 ts_tablet_manager.cc:1403] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.138s	user 0.019s	sys 0.009s
I20260709 12:27:51.151767 30077 raft_consensus.cc:359] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.153070 30077 raft_consensus.cc:385] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.153571 30077 raft_consensus.cc:740] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.157922 30077 consensus_queue.cc:260] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.160830 30074 tablet_bootstrap.cc:654] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.155985 30076 raft_consensus.cc:359] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.166904 30076 raft_consensus.cc:385] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.167306 30076 raft_consensus.cc:740] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.170910 30076 consensus_queue.cc:260] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.188310 30076 ts_tablet_manager.cc:1434] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.001s
I20260709 12:27:51.191395 30076 tablet_bootstrap.cc:492] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:51.202872 30076 tablet_bootstrap.cc:654] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.205857 30077 ts_tablet_manager.cc:1434] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.062s	user 0.011s	sys 0.000s
I20260709 12:27:51.207563 30077 tablet_bootstrap.cc:492] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:51.210321 30075 tablet_bootstrap.cc:492] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:51.211076 30075 ts_tablet_manager.cc:1403] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.147s	user 0.025s	sys 0.015s
W20260709 12:27:51.212165 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:51.219167 30233 consensus_queue.cc:1048] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:51.220081 30075 raft_consensus.cc:359] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.220943 30075 raft_consensus.cc:385] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.221238 30075 raft_consensus.cc:740] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.222072 30075 consensus_queue.cc:260] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.224803 30075 ts_tablet_manager.cc:1434] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:27:51.234192 30075 tablet_bootstrap.cc:492] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:51.241689 30074 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:51.242935 30074 ts_tablet_manager.cc:1403] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.150s	user 0.021s	sys 0.015s
I20260709 12:27:51.244146 30076 tablet_bootstrap.cc:492] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:51.244849 30076 ts_tablet_manager.cc:1403] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.054s	user 0.014s	sys 0.010s
I20260709 12:27:51.246558 30233 consensus_queue.cc:1048] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:51.246676 30074 raft_consensus.cc:359] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.247756 30074 raft_consensus.cc:385] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.248095 30074 raft_consensus.cc:740] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.248312 30076 raft_consensus.cc:359] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.249104 30076 raft_consensus.cc:385] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.249030 30074 consensus_queue.cc:260] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.251755 30074 ts_tablet_manager.cc:1434] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:27:51.253109 30074 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:51.254073 30075 tablet_bootstrap.cc:654] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.260859 30077 tablet_bootstrap.cc:654] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.260893 30074 tablet_bootstrap.cc:654] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.249572 30076 raft_consensus.cc:740] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.274051 30076 consensus_queue.cc:260] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.277371 30076 ts_tablet_manager.cc:1434] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:27:51.279297 30076 tablet_bootstrap.cc:492] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:51.280191 30075 tablet_bootstrap.cc:492] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:51.280798 30075 ts_tablet_manager.cc:1403] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.055s	user 0.011s	sys 0.007s
I20260709 12:27:51.284359 30075 raft_consensus.cc:359] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.285120 30075 raft_consensus.cc:385] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.285391 30075 raft_consensus.cc:740] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.286520 30076 tablet_bootstrap.cc:654] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.288419 30233 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.287820 30075 consensus_queue.cc:260] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.289351 30233 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.296301 30233 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:51.299960 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:51.300868 29879 raft_consensus.cc:2468] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:51.304486 29769 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:51.304958 30075 ts_tablet_manager.cc:1434] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.001s
I20260709 12:27:51.306818 30075 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:51.307024 30084 raft_consensus.cc:2804] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:51.307559 30084 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.307740 30085 raft_consensus.cc:493] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.307963 30084 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.308434 30085 raft_consensus.cc:515] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.310511 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.311153 30074 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:51.311755 30074 ts_tablet_manager.cc:1403] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.059s	user 0.009s	sys 0.004s
I20260709 12:27:51.312937 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:51.314230 29805 raft_consensus.cc:2468] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:51.314268 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.315071 29952 raft_consensus.cc:2468] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:51.315246 30085 leader_election.cc:290] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:51.315356 30074 raft_consensus.cc:359] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.316030 29994 leader_election.cc:304] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:51.316268 30074 raft_consensus.cc:385] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.316773 30074 raft_consensus.cc:740] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.317858 30085 raft_consensus.cc:2804] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:51.317619 30074 consensus_queue.cc:260] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.318432 30085 raft_consensus.cc:493] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.318918 30085 raft_consensus.cc:3060] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.320122 30074 ts_tablet_manager.cc:1434] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:51.320350 30245 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.320856 30245 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.321501 30074 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:51.323378 30245 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:27:51.316798 29772 leader_election.cc:343] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.326318 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:51.327364 29805 raft_consensus.cc:2468] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:51.325977 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.331372 30074 tablet_bootstrap.cc:654] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:51.332345 29995 leader_election.cc:343] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.334131 29994 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:51.335738 30245 raft_consensus.cc:2804] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:51.340103 30245 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.349529 30245 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.351522 30085 raft_consensus.cc:515] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.359019 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:51.359836 29806 raft_consensus.cc:3060] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.361521 30075 tablet_bootstrap.cc:654] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.365087 30085 leader_election.cc:290] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:51.367421 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd88542c59c449a0904ad8b002efc5a0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:51.368264 29951 raft_consensus.cc:3060] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.373703 30077 tablet_bootstrap.cc:492] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:51.374454 30077 ts_tablet_manager.cc:1403] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.167s	user 0.012s	sys 0.003s
I20260709 12:27:51.377956 30077 raft_consensus.cc:359] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.378789 30077 raft_consensus.cc:385] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.379087 30077 raft_consensus.cc:740] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.381099 30077 consensus_queue.cc:260] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.394698 29806 raft_consensus.cc:2468] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:51.398140 30085 consensus_queue.cc:1048] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:51.414170 30084 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.398788 29951 raft_consensus.cc:2468] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:51.415992 30204 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.399780 30077 ts_tablet_manager.cc:1434] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:27:51.416749 30245 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.413200 29994 leader_election.cc:304] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:51.424763 30076 tablet_bootstrap.cc:492] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:51.425457 30076 ts_tablet_manager.cc:1403] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.147s	user 0.016s	sys 0.010s
I20260709 12:27:51.428845 30085 raft_consensus.cc:2804] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:51.429206 30077 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:51.438170 30245 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:51.438771 30076 raft_consensus.cc:359] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.418741 30204 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.441603 30076 raft_consensus.cc:385] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.442096 29806 raft_consensus.cc:3060] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.442131 30076 raft_consensus.cc:740] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.443655 30076 consensus_queue.cc:260] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.444649 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:51.445443 29880 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.446826 30076 ts_tablet_manager.cc:1434] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.021s	user 0.011s	sys 0.000s
I20260709 12:27:51.453902 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:51.464291 29805 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.452185 30245 consensus_queue.cc:1048] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:27:51.470562 30084 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:51.474087 30076 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:51.458037 30085 raft_consensus.cc:697] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:51.479812 30204 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:51.480825 30085 consensus_queue.cc:237] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.481791 29803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:51.482141 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:51.483139 30231 raft_consensus.cc:493] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.483716 30231 raft_consensus.cc:515] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.478804 29880 raft_consensus.cc:2468] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:51.487524 30231 leader_election.cc:290] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:27:51.486333 29772 leader_election.cc:343] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.515957 30074 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:51.516628 30074 ts_tablet_manager.cc:1403] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.195s	user 0.016s	sys 0.029s
I20260709 12:27:51.518332 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:51.520339 30084 raft_consensus.cc:493] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.521117 30084 raft_consensus.cc:515] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.523895 30084 leader_election.cc:290] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:51.524489 30084 raft_consensus.cc:493] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.524719 30157 raft_consensus.cc:493] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.525033 30084 raft_consensus.cc:515] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.525663 30157 raft_consensus.cc:515] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.527868 30076 tablet_bootstrap.cc:654] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.529583 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.531122 30157 leader_election.cc:290] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:51.534821 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.535981 29805 raft_consensus.cc:2468] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:51.537621 29994 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:51.538833 30289 raft_consensus.cc:2804] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:51.529222 29769 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:51.533354 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.539356 29880 raft_consensus.cc:1275] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:27:51.541531 29847 leader_election.cc:343] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.539403 30289 raft_consensus.cc:697] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:51.543144 30204 consensus_queue.cc:1048] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:51.543256 30289 consensus_queue.cc:237] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:51.545717 30084 leader_election.cc:290] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:27:51.545922 29847 leader_election.cc:343] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.546470 29847 leader_election.cc:304] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:51.549161 30077 tablet_bootstrap.cc:654] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.549791 30200 raft_consensus.cc:2749] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:51.551024 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b914c950a335404b8369b7d5868ad408" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:51.551262 29769 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 63623.3us
W20260709 12:27:51.552901 29772 leader_election.cc:343] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.553761 30233 raft_consensus.cc:2804] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:51.554400 30233 raft_consensus.cc:697] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:51.555197 30204 consensus_queue.cc:1048] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:51.555488 30233 consensus_queue.cc:237] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.539266 30074 raft_consensus.cc:359] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.562282 30074 raft_consensus.cc:385] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.562831 30074 raft_consensus.cc:740] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.573155 30074 consensus_queue.cc:260] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.612640 30074 ts_tablet_manager.cc:1434] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.096s	user 0.012s	sys 0.000s
I20260709 12:27:51.639252 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:51.642402 30074 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:51.620136 30245 raft_consensus.cc:493] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.644925 30245 raft_consensus.cc:515] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.647149 30075 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:51.647794 30245 leader_election.cc:290] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:51.648901 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:51.649019 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.650167 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.647886 30075 ts_tablet_manager.cc:1403] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.341s	user 0.037s	sys 0.034s
W20260709 12:27:51.649206 29995 leader_election.cc:343] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.623833 30200 raft_consensus.cc:493] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:51.621167 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 61087.5us
I20260709 12:27:51.618937 30165 consensus_queue.cc:1048] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20260709 12:27:51.675153 30297 raft_consensus.cc:493] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.677018 30297 raft_consensus.cc:515] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.678157 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b914c950a335404b8369b7d5868ad408" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:51.673354 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 44174.8us
I20260709 12:27:51.667738 30200 raft_consensus.cc:515] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:51.697474 30074 tablet_bootstrap.cc:654] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.702303 30200 leader_election.cc:290] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:51.702847 30075 raft_consensus.cc:359] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.706768 30075 raft_consensus.cc:385] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.683825 30297 leader_election.cc:290] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:51.707252 30075 raft_consensus.cc:740] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
W20260709 12:27:51.719441 30288 log.cc:927] Time spent T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:27:51.719771 30075 consensus_queue.cc:260] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.686774 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:51.726212 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 51284.5us
I20260709 12:27:51.738624 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:51.753122 30075 ts_tablet_manager.cc:1434] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.102s	user 0.006s	sys 0.003s
I20260709 12:27:51.742182 29952 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.758406 29769 leader_election.cc:304] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:51.776973 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.778043 30028 raft_consensus.cc:1275] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:51.778313 29921 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
W20260709 12:27:51.779805 29992 leader_election.cc:343] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.780769 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.756623 30289 raft_consensus.cc:493] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.785641 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.786173 30075 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
W20260709 12:27:51.787322 30294 log.cc:927] Time spent T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:27:51.791712 30077 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:51.792419 30077 ts_tablet_manager.cc:1403] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.364s	user 0.016s	sys 0.015s
I20260709 12:27:51.794891 29704 catalog_manager.cc:5697] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:51.759146 29876 raft_consensus.cc:2468] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:51.809096 30296 consensus_queue.cc:1048] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:51.816475 30077 raft_consensus.cc:359] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.817704 30077 raft_consensus.cc:385] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.818141 30077 raft_consensus.cc:740] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.818961 30077 consensus_queue.cc:260] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.793326 30233 raft_consensus.cc:2749] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:51.793164 29955 raft_consensus.cc:2468] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:51.816706 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:51.813938 29769 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 63966.6us
I20260709 12:27:51.799541 30289 raft_consensus.cc:515] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:51.828428 29772 leader_election.cc:343] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.871349 30077 ts_tablet_manager.cc:1434] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.079s	user 0.010s	sys 0.000s
I20260709 12:27:51.845011 30296 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.875416 30076 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:51.876609 30076 ts_tablet_manager.cc:1403] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.403s	user 0.021s	sys 0.014s
I20260709 12:27:51.885721 29918 leader_election.cc:304] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:51.876626 30025 raft_consensus.cc:2468] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
W20260709 12:27:51.872372 29769 leader_election.cc:343] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:51.891021 30075 tablet_bootstrap.cc:654] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.894105 30297 raft_consensus.cc:2804] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:27:51.894490 29995 leader_election.cc:343] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:51.868345 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 45702us
I20260709 12:27:51.897770 30077 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:51.877676 30296 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:51.896767 30297 raft_consensus.cc:493] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.901691 29995 leader_election.cc:304] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:51.916594 30085 raft_consensus.cc:2749] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:51.920348 30296 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:51.920608 30077 tablet_bootstrap.cc:654] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.908351 30297 raft_consensus.cc:3060] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.927345 30289 leader_election.cc:290] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:51.930967 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:51.934316 30231 raft_consensus.cc:493] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.934653 30298 consensus_queue.cc:1048] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:27:51.935173 30231 raft_consensus.cc:515] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.944669 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:51.937641 30231 leader_election.cc:290] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:51.950996 30074 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:51.951975 30074 ts_tablet_manager.cc:1403] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.310s	user 0.015s	sys 0.020s
I20260709 12:27:51.900743 29769 leader_election.cc:304] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:51.977957 30245 raft_consensus.cc:493] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.978964 30245 raft_consensus.cc:515] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:51.973271 30231 raft_consensus.cc:2749] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.002153 29847 leader_election.cc:304] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:52.002211 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:52.005474 29876 raft_consensus.cc:2393] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
W20260709 12:27:51.977311 29772 leader_election.cc:343] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.005966 30303 raft_consensus.cc:2804] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:51.982257 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:52.006496 30303 raft_consensus.cc:493] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.006939 30303 raft_consensus.cc:3060] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.023625 30297 raft_consensus.cc:515] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.025774 29953 raft_consensus.cc:2393] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
W20260709 12:27:52.028184 29995 consensus_peers.cc:597] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:52.030164 30076 raft_consensus.cc:359] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.031163 30076 raft_consensus.cc:385] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.031487 30076 raft_consensus.cc:740] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.032369 30076 consensus_queue.cc:260] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:52.036826 29769 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 166223us
I20260709 12:27:52.040069 30076 ts_tablet_manager.cc:1434] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.163s	user 0.007s	sys 0.001s
W20260709 12:27:52.043850 29769 leader_election.cc:343] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.044729 29769 leader_election.cc:304] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.048014 30297 leader_election.cc:290] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:52.048482 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:52.038637 30074 raft_consensus.cc:359] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.055069 30074 raft_consensus.cc:385] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.055795 30074 raft_consensus.cc:740] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.054867 30025 raft_consensus.cc:3060] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.070421 30075 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:52.071170 30075 ts_tablet_manager.cc:1403] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.289s	user 0.023s	sys 0.010s
I20260709 12:27:52.074848 30075 raft_consensus.cc:359] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.075731 30075 raft_consensus.cc:385] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.060230 30231 raft_consensus.cc:2749] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.055018 30245 leader_election.cc:290] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:52.078722 30076 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:52.062451 30074 consensus_queue.cc:260] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.076043 30075 raft_consensus.cc:740] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.076542 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:52.091812 30075 consensus_queue.cc:260] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:27:52.097131 29769 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 53376us
I20260709 12:27:52.088495 29804 raft_consensus.cc:2468] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:52.089000 30245 consensus_queue.cc:1048] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:27:52.101042 29994 leader_election.cc:304] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:52.104171 30077 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:52.104796 30077 ts_tablet_manager.cc:1403] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.212s	user 0.046s	sys 0.037s
I20260709 12:27:52.083105 30316 consensus_queue.cc:1048] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260709 12:27:52.110765 30289 raft_consensus.cc:2804] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:52.111377 30289 raft_consensus.cc:493] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.111801 30289 raft_consensus.cc:3060] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.111565 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:52.112413 29876 raft_consensus.cc:2468] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:52.121663 30077 raft_consensus.cc:359] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.125020 30077 raft_consensus.cc:385] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.125397 30077 raft_consensus.cc:740] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.127242 30077 consensus_queue.cc:260] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.131716 30077 ts_tablet_manager.cc:1434] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.027s	user 0.010s	sys 0.000s
I20260709 12:27:52.133047 30077 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:52.129725 30303 raft_consensus.cc:515] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.139325 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:52.141623 30077 tablet_bootstrap.cc:654] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.141332 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:52.142455 29950 raft_consensus.cc:2393] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:52.143167 29881 raft_consensus.cc:2393] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:52.146914 30074 ts_tablet_manager.cc:1434] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.195s	user 0.008s	sys 0.004s
I20260709 12:27:52.149415 30075 ts_tablet_manager.cc:1434] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.078s	user 0.007s	sys 0.002s
I20260709 12:27:52.149329 29702 catalog_manager.cc:5697] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.161434 29950 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.166297 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742ffdad627c4e71865fe5b85509976d" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:52.168426 30289 raft_consensus.cc:515] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.174244 29992 leader_election.cc:304] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.175755 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:52.139297 30303 leader_election.cc:290] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:52.185516 29806 raft_consensus.cc:3060] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.195482 30075 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:52.199920 30074 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:52.200745 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:52.208033 30076 tablet_bootstrap.cc:654] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.212317 30317 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.214262 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:52.214577 30317 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.215109 29876 raft_consensus.cc:3060] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.179674 30025 raft_consensus.cc:2468] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:52.217835 30245 raft_consensus.cc:2749] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.219709 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:52.220227 29847 leader_election.cc:304] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:52.222255 30299 raft_consensus.cc:2749] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.223740 30165 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.234181 29798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:52.240542 29995 leader_election.cc:343] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.242051 29921 leader_election.cc:304] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.243949 30297 raft_consensus.cc:2804] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:52.244539 30297 raft_consensus.cc:697] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:52.244788 29876 raft_consensus.cc:2468] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:52.245688 30297 consensus_queue.cc:237] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.248019 29992 leader_election.cc:304] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:52.249342 30074 tablet_bootstrap.cc:654] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.252422 30245 raft_consensus.cc:2804] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:27:52.252480 29769 leader_election.cc:343] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.258052 29769 leader_election.cc:304] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.259222 30084 raft_consensus.cc:2749] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.254799 30245 raft_consensus.cc:697] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:52.260879 30231 consensus_queue.cc:1048] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:52.262233 30245 consensus_queue.cc:237] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.266389 30084 consensus_queue.cc:1048] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20260709 12:27:52.325794 30075 tablet_bootstrap.cc:654] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.329486 29703 catalog_manager.cc:5697] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.334730 30077 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:52.335597 30077 ts_tablet_manager.cc:1403] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.203s	user 0.012s	sys 0.005s
I20260709 12:27:52.327085 30289 leader_election.cc:290] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:52.322115 30317 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:52.331281 29920 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.345260 30297 raft_consensus.cc:2749] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.329108 30233 raft_consensus.cc:493] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:52.326820 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 56403.1us
I20260709 12:27:52.346732 29806 raft_consensus.cc:2468] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:52.349582 30233 raft_consensus.cc:515] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.360653 30233 leader_election.cc:290] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:52.363857 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:52.366663 29994 leader_election.cc:343] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.367450 29994 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:52.369473 30085 raft_consensus.cc:2749] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.372449 30076 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:52.373225 30076 ts_tablet_manager.cc:1403] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.295s	user 0.018s	sys 0.009s
I20260709 12:27:52.375326 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:52.381597 29769 leader_election.cc:343] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.383342 30077 raft_consensus.cc:359] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:52.384232 30323 log.cc:927] Time spent T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Append to log took a long time: real 0.054s	user 0.002s	sys 0.000s
I20260709 12:27:52.384420 30077 raft_consensus.cc:385] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.384737 30077 raft_consensus.cc:740] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.390578 30076 raft_consensus.cc:359] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.394079 30074 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:52.394817 30074 ts_tablet_manager.cc:1403] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.198s	user 0.028s	sys 0.031s
I20260709 12:27:52.394141 30076 raft_consensus.cc:385] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.385618 30077 consensus_queue.cc:260] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.396426 30076 raft_consensus.cc:740] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.411974 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:52.426977 30074 raft_consensus.cc:359] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.427989 30074 raft_consensus.cc:385] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.428313 30074 raft_consensus.cc:740] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.429293 29700 catalog_manager.cc:5697] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:52.458732 29772 leader_election.cc:343] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.467555 29772 leader_election.cc:304] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.429167 30074 consensus_queue.cc:260] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.475875 30076 consensus_queue.cc:260] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.476683 30204 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.477730 30074 ts_tablet_manager.cc:1434] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.083s	user 0.008s	sys 0.000s
I20260709 12:27:52.492635 30077 ts_tablet_manager.cc:1434] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.157s	user 0.008s	sys 0.000s
W20260709 12:27:52.478892 29920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 47669.4us
I20260709 12:27:52.478328 30231 raft_consensus.cc:2749] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.491173 30204 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.501809 29955 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:52.475374 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 131842us
I20260709 12:27:52.510504 30204 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:52.510699 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:52.512890 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:52.516444 30077 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:52.525065 30076 ts_tablet_manager.cc:1434] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.151s	user 0.011s	sys 0.001s
I20260709 12:27:52.528182 30076 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:52.535701 29920 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:52.544613 30075 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:52.545351 30075 ts_tablet_manager.cc:1403] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.366s	user 0.020s	sys 0.005s
W20260709 12:27:52.553537 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:52.555290 30074 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:52.561065 30204 raft_consensus.cc:2749] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:52.578214 30076 tablet_bootstrap.cc:654] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.558132 30075 raft_consensus.cc:359] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.578835 30075 raft_consensus.cc:385] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.579160 30075 raft_consensus.cc:740] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.580116 30075 consensus_queue.cc:260] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.583832 30075 ts_tablet_manager.cc:1434] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.038s	user 0.009s	sys 0.002s
I20260709 12:27:52.586520 30075 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:52.588254 30077 tablet_bootstrap.cc:654] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.600052 30074 tablet_bootstrap.cc:654] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.601737 29703 catalog_manager.cc:5697] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.610284 30075 tablet_bootstrap.cc:654] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.627429 30077 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:52.628079 30077 ts_tablet_manager.cc:1403] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.112s	user 0.014s	sys 0.009s
I20260709 12:27:52.629632 30076 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:52.630494 30076 ts_tablet_manager.cc:1403] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.104s	user 0.031s	sys 0.024s
I20260709 12:27:52.631317 30077 raft_consensus.cc:359] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.632148 30077 raft_consensus.cc:385] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.632419 30077 raft_consensus.cc:740] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.634193 30076 raft_consensus.cc:359] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.635437 30077 consensus_queue.cc:260] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.635121 30076 raft_consensus.cc:385] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.638401 30076 raft_consensus.cc:740] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.638721 30204 raft_consensus.cc:493] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.639328 30204 raft_consensus.cc:515] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.639983 30077 ts_tablet_manager.cc:1434] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:27:52.641518 30077 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:52.643203 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b914c950a335404b8369b7d5868ad408" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:52.644183 29804 raft_consensus.cc:2468] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:52.645998 30075 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:52.646716 30075 ts_tablet_manager.cc:1403] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.061s	user 0.016s	sys 0.000s
I20260709 12:27:52.648514 29920 leader_election.cc:304] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:52.649689 30204 leader_election.cc:290] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:52.650328 30204 raft_consensus.cc:2804] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:52.650686 30204 raft_consensus.cc:493] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.651034 30204 raft_consensus.cc:3060] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.652002 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b914c950a335404b8369b7d5868ad408" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:52.651036 30075 raft_consensus.cc:359] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.654942 30075 raft_consensus.cc:385] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.655434 30075 raft_consensus.cc:740] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.639317 30076 consensus_queue.cc:260] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.658845 29876 raft_consensus.cc:2468] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:52.660256 30077 tablet_bootstrap.cc:654] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.665146 30076 ts_tablet_manager.cc:1434] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.000s
I20260709 12:27:52.665386 30074 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:52.667807 30074 ts_tablet_manager.cc:1403] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.134s	user 0.019s	sys 0.016s
I20260709 12:27:52.659878 30075 consensus_queue.cc:260] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.669764 30165 consensus_queue.cc:1048] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:52.671830 30075 ts_tablet_manager.cc:1434] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:27:52.671527 30074 raft_consensus.cc:359] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.672394 30074 raft_consensus.cc:385] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.672745 30074 raft_consensus.cc:740] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.673161 30075 tablet_bootstrap.cc:492] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:52.673519 30074 consensus_queue.cc:260] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.676050 30074 ts_tablet_manager.cc:1434] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:52.679008 30076 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:52.680019 30297 consensus_queue.cc:1048] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:52.686774 30075 tablet_bootstrap.cc:654] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.699318 30329 raft_consensus.cc:493] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.700048 30329 raft_consensus.cc:515] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.702729 30085 consensus_queue.cc:1048] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:52.704259 30329 leader_election.cc:290] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:52.705077 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:52.706568 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:52.707296 29769 leader_election.cc:343] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.707399 30025 raft_consensus.cc:2468] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:52.710024 30074 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:52.710549 29772 leader_election.cc:304] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:52.710980 30075 tablet_bootstrap.cc:492] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:52.711820 30075 ts_tablet_manager.cc:1403] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.004s
I20260709 12:27:52.712162 30295 raft_consensus.cc:2804] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:52.712643 30295 raft_consensus.cc:493] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.712990 30295 raft_consensus.cc:3060] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.719210 30075 raft_consensus.cc:359] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.720321 30075 raft_consensus.cc:385] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.687036 30204 raft_consensus.cc:515] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.728394 30074 tablet_bootstrap.cc:654] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.720716 30075 raft_consensus.cc:740] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.730551 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b914c950a335404b8369b7d5868ad408" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:52.731436 29804 raft_consensus.cc:3060] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.731690 30075 consensus_queue.cc:260] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.733233 30204 leader_election.cc:290] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:52.733477 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b914c950a335404b8369b7d5868ad408" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:52.734536 30075 ts_tablet_manager.cc:1434] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.022s	user 0.010s	sys 0.000s
I20260709 12:27:52.734263 29881 raft_consensus.cc:3060] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.746302 30076 tablet_bootstrap.cc:654] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.767464 29881 raft_consensus.cc:2468] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:52.768478 29804 raft_consensus.cc:2468] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:52.770599 29920 leader_election.cc:304] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:52.775135 30075 tablet_bootstrap.cc:492] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:52.775748 30165 raft_consensus.cc:2804] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:52.776427 30165 raft_consensus.cc:697] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:52.777478 30165 consensus_queue.cc:237] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.808688 30295 raft_consensus.cc:515] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.818243 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:52.819027 30026 raft_consensus.cc:3060] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.817466 30074 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:52.830435 30074 ts_tablet_manager.cc:1403] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.126s	user 0.025s	sys 0.015s
I20260709 12:27:52.834661 30074 raft_consensus.cc:359] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.836759 30026 raft_consensus.cc:2468] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:52.839195 30295 leader_election.cc:290] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:52.837147 30077 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:52.841163 30077 ts_tablet_manager.cc:1403] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.200s	user 0.015s	sys 0.003s
I20260709 12:27:52.841307 29772 leader_election.cc:304] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:52.842666 30295 raft_consensus.cc:2804] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:52.844144 30074 raft_consensus.cc:385] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.844530 30074 raft_consensus.cc:740] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.845404 30074 consensus_queue.cc:260] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.847224 30077 raft_consensus.cc:359] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.848060 30074 ts_tablet_manager.cc:1434] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:27:52.848181 30077 raft_consensus.cc:385] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.848820 30077 raft_consensus.cc:740] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.849570 30074 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:52.843276 30295 raft_consensus.cc:697] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:52.849720 30077 consensus_queue.cc:260] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.850605 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:52.850824 30295 consensus_queue.cc:237] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:52.852583 29769 leader_election.cc:343] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.853268 30077 ts_tablet_manager.cc:1434] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.005s
I20260709 12:27:52.849843 30245 consensus_queue.cc:1048] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:27:52.857288 30077 tablet_bootstrap.cc:492] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:52.859568 30076 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:52.860294 30076 ts_tablet_manager.cc:1403] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.182s	user 0.004s	sys 0.010s
I20260709 12:27:52.863816 30076 raft_consensus.cc:359] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.864724 30076 raft_consensus.cc:385] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.865041 30076 raft_consensus.cc:740] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.865916 30075 tablet_bootstrap.cc:654] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.870388 30076 consensus_queue.cc:260] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:52.873674 30074 tablet_bootstrap.cc:654] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.875017 30076 ts_tablet_manager.cc:1434] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.005s
I20260709 12:27:52.877637 30076 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:52.882714 30299 raft_consensus.cc:493] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.883405 30299 raft_consensus.cc:515] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.885478 30076 tablet_bootstrap.cc:654] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.887086 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:52.888098 30026 raft_consensus.cc:2468] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:52.889914 29847 leader_election.cc:304] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:52.891202 30200 raft_consensus.cc:2804] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:52.891645 30200 raft_consensus.cc:493] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.892002 30200 raft_consensus.cc:3060] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.900321 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:52.901314 29804 raft_consensus.cc:2468] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:52.907639 30077 tablet_bootstrap.cc:654] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.908034 29704 catalog_manager.cc:5697] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:52.903409 30299 leader_election.cc:290] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:52.924969 30200 raft_consensus.cc:515] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.930150 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:52.931000 30026 raft_consensus.cc:3060] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.939254 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:52.940389 29804 raft_consensus.cc:3060] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.952075 30026 raft_consensus.cc:2468] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:52.949120 30200 leader_election.cc:290] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:52.958792 29847 leader_election.cc:304] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:52.963543 30200 raft_consensus.cc:2804] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:52.964425 30200 raft_consensus.cc:697] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:52.966135 30074 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:52.967546 30329 raft_consensus.cc:493] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.968411 30329 raft_consensus.cc:515] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:52.971685 30074 ts_tablet_manager.cc:1403] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.118s	user 0.018s	sys 0.000s
I20260709 12:27:52.965549 30200 consensus_queue.cc:237] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:52.977065 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:52.979650 29952 raft_consensus.cc:2468] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:52.981616 30075 tablet_bootstrap.cc:492] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:52.982435 30075 ts_tablet_manager.cc:1403] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.211s	user 0.018s	sys 0.003s
I20260709 12:27:52.985062 29772 leader_election.cc:304] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:52.988132 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:52.993506 30075 raft_consensus.cc:359] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.994729 30075 raft_consensus.cc:385] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.995162 30075 raft_consensus.cc:740] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
W20260709 12:27:52.996444 29772 leader_election.cc:343] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.996445 30075 consensus_queue.cc:260] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:52.978721 30329 leader_election.cc:290] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:52.997671 30076 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:52.997135 30084 raft_consensus.cc:2804] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:52.998238 30084 raft_consensus.cc:493] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.998333 30076 ts_tablet_manager.cc:1403] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.121s	user 0.027s	sys 0.014s
I20260709 12:27:52.998673 30084 raft_consensus.cc:3060] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.001360 30075 ts_tablet_manager.cc:1434] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 12:27:53.002049 30076 raft_consensus.cc:359] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.002810 30075 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:53.002939 30076 raft_consensus.cc:385] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.018016 30076 raft_consensus.cc:740] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.004702 30074 raft_consensus.cc:359] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.018692 30074 raft_consensus.cc:385] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.019047 30074 raft_consensus.cc:740] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.020716 30076 consensus_queue.cc:260] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.020008 30074 consensus_queue.cc:260] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.023895 30076 ts_tablet_manager.cc:1434] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.025s	user 0.011s	sys 0.006s
I20260709 12:27:53.031448 29804 raft_consensus.cc:2468] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:53.033999 30077 tablet_bootstrap.cc:492] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:53.034701 30077 ts_tablet_manager.cc:1403] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.178s	user 0.027s	sys 0.001s
I20260709 12:27:53.031296 30084 raft_consensus.cc:515] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.039101 30074 ts_tablet_manager.cc:1434] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.067s	user 0.010s	sys 0.002s
I20260709 12:27:53.039565 30076 tablet_bootstrap.cc:492] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:53.035746 29703 catalog_manager.cc:5697] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.040153 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:53.040501 30084 leader_election.cc:290] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.040642 30074 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:53.045224 29952 raft_consensus.cc:3060] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.046437 30165 raft_consensus.cc:493] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.047060 30165 raft_consensus.cc:515] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.050875 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:53.052101 29799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:53.053197 29799 raft_consensus.cc:2468] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:53.054736 29920 leader_election.cc:304] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:53.055943 30204 raft_consensus.cc:2804] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.056392 30204 raft_consensus.cc:493] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.056495 30299 raft_consensus.cc:493] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.056835 30204 raft_consensus.cc:3060] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.057226 30299 raft_consensus.cc:515] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.057881 30074 tablet_bootstrap.cc:654] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.061127 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:53.049595 30165 leader_election.cc:290] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:53.061600 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:53.062153 29804 raft_consensus.cc:2468] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:53.063711 29846 leader_election.cc:304] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:53.066530 30299 leader_election.cc:290] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:53.068707 30299 raft_consensus.cc:2804] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.069223 30299 raft_consensus.cc:493] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.070699 30077 raft_consensus.cc:359] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.072710 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.072796 30077 raft_consensus.cc:385] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.073861 30077 raft_consensus.cc:740] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
W20260709 12:27:53.077941 29772 leader_election.cc:343] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.082267 30077 consensus_queue.cc:260] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.078202 30204 raft_consensus.cc:515] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.069633 30299 raft_consensus.cc:3060] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.097257 30075 tablet_bootstrap.cc:654] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.103812 30076 tablet_bootstrap.cc:654] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:53.096314 29918 leader_election.cc:343] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:53.121174 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 57592us
I20260709 12:27:53.123499 30077 ts_tablet_manager.cc:1434] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.088s	user 0.008s	sys 0.002s
I20260709 12:27:53.125118 30077 tablet_bootstrap.cc:492] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:53.127516 29952 raft_consensus.cc:2468] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
W20260709 12:27:53.108986 29847 leader_election.cc:343] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.139477 30204 leader_election.cc:290] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:53.144389 29799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:53.145292 29799 raft_consensus.cc:3060] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.146523 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
W20260709 12:27:53.148214 29918 leader_election.cc:343] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.159425 30289 raft_consensus.cc:493] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.160130 30289 raft_consensus.cc:515] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.162539 29772 leader_election.cc:304] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:53.163846 30084 raft_consensus.cc:2804] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.164565 30084 raft_consensus.cc:697] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:53.149442 29701 catalog_manager.cc:5697] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.165500 30084 consensus_queue.cc:237] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.168562 30289 leader_election.cc:290] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:53.168777 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:53.171003 29992 leader_election.cc:343] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.179275 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.181182 30077 tablet_bootstrap.cc:654] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:53.182308 29995 leader_election.cc:343] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.183220 29995 leader_election.cc:304] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:53.186430 30289 raft_consensus.cc:2749] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.173013 30299 raft_consensus.cc:515] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.183164 29799 raft_consensus.cc:2468] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:53.202273 29920 leader_election.cc:304] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:53.204355 30204 raft_consensus.cc:2804] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.205013 30204 raft_consensus.cc:697] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:53.214982 30204 consensus_queue.cc:237] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.224994 30074 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:53.225656 30074 ts_tablet_manager.cc:1403] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.185s	user 0.041s	sys 0.026s
I20260709 12:27:53.244580 30076 tablet_bootstrap.cc:492] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:53.245309 30076 ts_tablet_manager.cc:1403] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.218s	user 0.013s	sys 0.010s
I20260709 12:27:53.246897 30074 raft_consensus.cc:359] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.247723 30074 raft_consensus.cc:385] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.247987 30074 raft_consensus.cc:740] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.248270 30076 raft_consensus.cc:359] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.249025 30076 raft_consensus.cc:385] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.248757 30074 consensus_queue.cc:260] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.249325 30076 raft_consensus.cc:740] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.251890 30076 consensus_queue.cc:260] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.252398 29799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:53.253221 30075 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:53.253185 29799 raft_consensus.cc:3060] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.253906 30075 ts_tablet_manager.cc:1403] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.251s	user 0.029s	sys 0.007s
I20260709 12:27:53.254350 30076 ts_tablet_manager.cc:1434] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.004s
I20260709 12:27:53.254101 29701 catalog_manager.cc:5697] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.255626 30076 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:53.266515 30074 ts_tablet_manager.cc:1434] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260709 12:27:53.267916 30074 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:53.268323 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:53.257232 30075 raft_consensus.cc:359] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:27:53.270040 29847 leader_election.cc:343] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.270378 30075 raft_consensus.cc:385] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.270740 30075 raft_consensus.cc:740] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.271080 29704 catalog_manager.cc:5697] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.275609 30165 consensus_queue.cc:1048] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.277948 30076 tablet_bootstrap.cc:654] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.279626 30075 consensus_queue.cc:260] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.285383 30075 ts_tablet_manager.cc:1434] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 12:27:53.286975 30075 tablet_bootstrap.cc:492] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:53.302870 30074 tablet_bootstrap.cc:654] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.304368 30329 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.305020 30329 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.307327 30329 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.309873 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:53.310921 30026 raft_consensus.cc:2468] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:53.314613 29772 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:53.315464 30075 tablet_bootstrap.cc:654] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.320040 30329 raft_consensus.cc:2804] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.320726 30329 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.321856 30329 raft_consensus.cc:3060] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.317649 30299 leader_election.cc:290] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:53.329438 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:53.332537 29772 leader_election.cc:343] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.337090 30074 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:53.332726 30076 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:53.337800 30074 ts_tablet_manager.cc:1403] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.070s	user 0.014s	sys 0.000s
I20260709 12:27:53.337982 30076 ts_tablet_manager.cc:1403] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.083s	user 0.020s	sys 0.010s
I20260709 12:27:53.352097 30356 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.348268 30076 raft_consensus.cc:359] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.352952 30076 raft_consensus.cc:385] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.352837 30356 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.353397 30076 raft_consensus.cc:740] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.356740 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:53.359558 29847 leader_election.cc:343] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.356807 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:53.361726 29799 raft_consensus.cc:2468] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:53.361990 30076 consensus_queue.cc:260] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.362819 30329 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.366886 30329 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.369969 30076 ts_tablet_manager.cc:1434] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.001s
I20260709 12:27:53.371495 30076 tablet_bootstrap.cc:492] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
W20260709 12:27:53.371656 29846 leader_election.cc:343] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.372368 29846 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:53.373445 30299 raft_consensus.cc:2749] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.375768 30074 raft_consensus.cc:359] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.376936 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:53.373697 29846 leader_election.cc:304] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:53.402817 30077 tablet_bootstrap.cc:492] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:53.382203 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:53.403785 30026 raft_consensus.cc:3060] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.404353 30077 ts_tablet_manager.cc:1403] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.280s	user 0.023s	sys 0.018s
I20260709 12:27:53.392620 29880 raft_consensus.cc:3060] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:53.408025 29772 leader_election.cc:343] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.410280 30299 raft_consensus.cc:2804] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.357491 30356 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:53.387933 30204 consensus_queue.cc:1048] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.411113 30299 raft_consensus.cc:697] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:53.415234 30299 consensus_queue.cc:237] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.426463 30357 raft_consensus.cc:493] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.426813 30026 raft_consensus.cc:2468] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:53.427241 30357 raft_consensus.cc:515] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.430807 30357 leader_election.cc:290] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.431643 30075 tablet_bootstrap.cc:492] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:53.432319 30075 ts_tablet_manager.cc:1403] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.146s	user 0.016s	sys 0.000s
I20260709 12:27:53.433039 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:53.434223 29772 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:53.435236 30295 raft_consensus.cc:2804] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.435825 30295 raft_consensus.cc:697] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:53.436866 30295 consensus_queue.cc:237] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.442437 30074 raft_consensus.cc:385] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.442925 30074 raft_consensus.cc:740] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.443423 30329 consensus_queue.cc:1048] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.443753 30074 consensus_queue.cc:260] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.445031 30077 raft_consensus.cc:359] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.449637 30076 tablet_bootstrap.cc:654] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.447238 30074 ts_tablet_manager.cc:1434] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.109s	user 0.008s	sys 0.000s
I20260709 12:27:53.449909 30077 raft_consensus.cc:385] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:53.453488 29847 leader_election.cc:343] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.457108 30077 raft_consensus.cc:740] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.455986 30074 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:53.465309 30317 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.466276 30317 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.495734 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.496094 30077 consensus_queue.cc:260] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.499208 29952 raft_consensus.cc:2468] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:53.505931 30317 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:53.508623 30245 raft_consensus.cc:493] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.509306 30245 raft_consensus.cc:515] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.513320 30075 raft_consensus.cc:359] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.522538 30075 raft_consensus.cc:385] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.524075 29847 leader_election.cc:304] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:53.528729 30075 raft_consensus.cc:740] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.528828 30356 raft_consensus.cc:2804] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.534080 30356 raft_consensus.cc:493] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.534480 30356 raft_consensus.cc:3060] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.516254 30357 consensus_queue.cc:1048] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:27:53.532932 30204 raft_consensus.cc:493] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.545670 30075 consensus_queue.cc:260] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.541862 30204 raft_consensus.cc:515] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.549301 30077 ts_tablet_manager.cc:1434] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.120s	user 0.011s	sys 0.009s
I20260709 12:27:53.550034 30074 tablet_bootstrap.cc:654] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.551767 29704 catalog_manager.cc:5697] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.579860 30077 tablet_bootstrap.cc:492] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:53.578675 30299 raft_consensus.cc:493] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.583691 30299 raft_consensus.cc:515] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.591305 29799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:53.584602 30084 consensus_queue.cc:1048] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:27:53.595599 30075 ts_tablet_manager.cc:1434] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.163s	user 0.015s	sys 0.007s
I20260709 12:27:53.591903 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:53.602397 29995 leader_election.cc:343] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.612242 30204 leader_election.cc:290] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.613075 30299 leader_election.cc:290] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
W20260709 12:27:53.614948 29994 leader_election.cc:343] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.615095 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:53.615609 29994 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:53.616047 29881 raft_consensus.cc:2468] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:53.615095 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:53.618252 30025 raft_consensus.cc:2468] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:53.636529 30245 leader_election.cc:290] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:53.634419 30329 raft_consensus.cc:493] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.635059 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:53.638120 29847 leader_election.cc:304] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:53.643618 30026 raft_consensus.cc:2468] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:53.643419 30329 raft_consensus.cc:515] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.622290 29918 leader_election.cc:304] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:53.623569 30357 consensus_queue.cc:1048] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:27:53.645984 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.622344 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.647013 29955 raft_consensus.cc:2468] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
W20260709 12:27:53.649564 29995 leader_election.cc:343] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.652506 30317 raft_consensus.cc:493] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.652812 30329 leader_election.cc:290] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:53.653107 30317 raft_consensus.cc:515] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.632926 30076 tablet_bootstrap.cc:492] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:53.654609 30076 ts_tablet_manager.cc:1403] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.283s	user 0.023s	sys 0.007s
I20260709 12:27:53.638852 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:53.657052 30200 raft_consensus.cc:2804] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.627929 29700 catalog_manager.cc:5697] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.660753 30200 raft_consensus.cc:493] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.661389 30200 raft_consensus.cc:3060] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.639966 30374 raft_consensus.cc:2749] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.658154 30077 tablet_bootstrap.cc:654] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.638631 30075 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
W20260709 12:27:53.669100 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 94986.6us
I20260709 12:27:53.650058 30204 raft_consensus.cc:2804] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:27:53.675577 29918 consensus_peers.cc:597] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:53.676054 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.672046 30204 raft_consensus.cc:493] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.677552 30204 raft_consensus.cc:3060] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.679049 30076 raft_consensus.cc:359] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.679857 30076 raft_consensus.cc:385] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.680148 30076 raft_consensus.cc:740] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
W20260709 12:27:53.681102 29992 leader_election.cc:343] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.681792 29992 leader_election.cc:304] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:53.682425 30317 leader_election.cc:290] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:53.681198 30076 consensus_queue.cc:260] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.621752 30356 raft_consensus.cc:515] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.701797 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:53.711534 30074 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:53.712607 29881 raft_consensus.cc:2393] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:53.712607 30074 ts_tablet_manager.cc:1403] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.257s	user 0.020s	sys 0.013s
W20260709 12:27:53.715579 29772 leader_election.cc:343] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.717700 30356 leader_election.cc:290] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.689975 30245 raft_consensus.cc:2749] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.734721 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:53.735673 29952 raft_consensus.cc:2468] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
W20260709 12:27:53.752691 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 65453.8us
I20260709 12:27:53.754390 30076 ts_tablet_manager.cc:1434] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.099s	user 0.009s	sys 0.000s
I20260709 12:27:53.765347 30076 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:53.771354 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:53.772249 30026 raft_consensus.cc:3060] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.750442 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:53.793675 30074 raft_consensus.cc:359] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.794622 30074 raft_consensus.cc:385] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.794901 30074 raft_consensus.cc:740] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.795696 30074 consensus_queue.cc:260] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:53.779006 30204 raft_consensus.cc:515] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.796630 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57229c57228146d6ab456a83f82ef167" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:53.796474 29995 leader_election.cc:304] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:53.797516 29952 raft_consensus.cc:3060] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.798933 30204 leader_election.cc:290] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.802181 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:53.802963 30025 raft_consensus.cc:3060] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:53.806579 29769 leader_election.cc:343] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.806780 30369 consensus_queue.cc:1048] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.807330 29769 leader_election.cc:304] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:53.771471 30200 raft_consensus.cc:515] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.815758 30074 ts_tablet_manager.cc:1434] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.102s	user 0.008s	sys 0.000s
I20260709 12:27:53.816354 30200 leader_election.cc:290] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:53.825413 30075 tablet_bootstrap.cc:654] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.825839 30074 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:53.832442 30076 tablet_bootstrap.cc:654] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.823834 30385 raft_consensus.cc:2749] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:53.851027 29772 consensus_peers.cc:597] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 -> Peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Couldn't send request to peer 1e2ceabfe50c47b49529b663fa133b86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:53.851549 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 77263.3us
W20260709 12:27:53.786677 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 87477.5us
I20260709 12:27:53.853802 30026 raft_consensus.cc:2468] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:53.860699 30374 raft_consensus.cc:2764] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:27:53.879437 30388 consensus_queue.cc:1048] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.879400 30077 tablet_bootstrap.cc:492] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:53.882573 30077 ts_tablet_manager.cc:1403] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.309s	user 0.032s	sys 0.023s
I20260709 12:27:53.913725 29952 raft_consensus.cc:2468] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
W20260709 12:27:53.944510 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 188399us
I20260709 12:27:53.944427 30077 raft_consensus.cc:359] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.947635 30077 raft_consensus.cc:385] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.929669 30025 raft_consensus.cc:2468] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:53.947974 30077 raft_consensus.cc:740] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.933671 30385 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.952481 30077 consensus_queue.cc:260] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:53.954945 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:53.964640 30385 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:53.971230 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:53.971948 30076 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:53.972597 30076 ts_tablet_manager.cc:1403] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.208s	user 0.026s	sys 0.009s
I20260709 12:27:53.972597 29881 raft_consensus.cc:2393] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:53.976344 30383 consensus_queue.cc:1048] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.978618 30385 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:53.986764 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:53.987856 29879 raft_consensus.cc:2468] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:53.990845 30076 raft_consensus.cc:359] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.991777 30076 raft_consensus.cc:385] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.992069 30076 raft_consensus.cc:740] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.992801 30076 consensus_queue.cc:260] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:53.994889 30165 raft_consensus.cc:493] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.995502 30165 raft_consensus.cc:515] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:53.996038 30076 ts_tablet_manager.cc:1434] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:27:53.992700 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:54.003154 30076 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:54.007583 30165 leader_election.cc:290] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.008850 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:54.009754 30026 raft_consensus.cc:2468] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:54.013346 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:54.014451 29952 raft_consensus.cc:2393] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:54.017884 29769 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
W20260709 12:27:54.019990 29772 consensus_peers.cc:597] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:54.020918 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cefd124c3a724d06897bd37636aeb5b6" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:54.035091 30076 tablet_bootstrap.cc:654] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.035645 30026 raft_consensus.cc:2393] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:54.024657 30200 raft_consensus.cc:493] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.038394 30200 raft_consensus.cc:515] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.041078 30200 leader_election.cc:290] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.029006 30385 consensus_queue.cc:1048] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20260709 12:27:54.030572 29772 leader_election.cc:343] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.028429 29921 leader_election.cc:304] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:54.046154 29921 leader_election.cc:304] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:54.031770 30084 raft_consensus.cc:2804] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:54.047448 30084 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.047822 30084 raft_consensus.cc:3060] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.048149 30074 tablet_bootstrap.cc:654] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.024901 29847 leader_election.cc:304] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:54.054595 30388 raft_consensus.cc:2804] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.054773 30369 raft_consensus.cc:2804] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.055274 30388 raft_consensus.cc:697] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:54.055459 30369 raft_consensus.cc:697] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:54.056526 30369 consensus_queue.cc:237] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.058653 30297 raft_consensus.cc:2804] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:54.059134 30297 raft_consensus.cc:493] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.059487 30297 raft_consensus.cc:3060] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.056348 30388 consensus_queue.cc:237] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.061525 30077 ts_tablet_manager.cc:1434] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.178s	user 0.008s	sys 0.001s
I20260709 12:27:54.065735 29847 leader_election.cc:304] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.075048 30084 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.076176 30356 raft_consensus.cc:2749] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.116426 30077 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:54.118100 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:54.119068 29880 raft_consensus.cc:3060] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.117988 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:54.122400 30084 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:27:54.126353 30061 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.16.102.62:33633, user_credentials={real_user=slave}} blocked reactor thread for 43551.2us
I20260709 12:27:54.127874 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:54.128826 30026 raft_consensus.cc:2468] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:54.130163 30383 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.130826 30383 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.134254 30075 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:54.134994 30075 ts_tablet_manager.cc:1403] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.497s	user 0.040s	sys 0.030s
I20260709 12:27:54.121853 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:54.147159 30383 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.151139 30075 raft_consensus.cc:359] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.152139 30075 raft_consensus.cc:385] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.151844 29847 leader_election.cc:304] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:54.152467 30075 raft_consensus.cc:740] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.160292 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:54.161195 30026 raft_consensus.cc:2468] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:54.161492 30297 raft_consensus.cc:515] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.164054 30297 leader_election.cc:290] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
W20260709 12:27:54.170364 29772 leader_election.cc:343] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.171617 29772 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
W20260709 12:27:54.134552 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 53602.5us
I20260709 12:27:54.175160 30076 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:54.175836 30076 ts_tablet_manager.cc:1403] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.173s	user 0.011s	sys 0.008s
I20260709 12:27:54.178561 30075 consensus_queue.cc:260] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.179776 30399 raft_consensus.cc:2804] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:54.180229 30399 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.180593 30399 raft_consensus.cc:3060] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.181425 30075 ts_tablet_manager.cc:1434] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.002s
I20260709 12:27:54.191205 30075 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:54.173209 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:54.155630 30299 raft_consensus.cc:2804] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:54.206060 30299 raft_consensus.cc:493] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.212761 30299 raft_consensus.cc:3060] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.207300 30028 raft_consensus.cc:3060] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.215124 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:54.216504 29847 consensus_peers.cc:597] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:54.217126 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:54.226586 29847 leader_election.cc:343] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.230181 30299 raft_consensus.cc:515] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.233130 30388 raft_consensus.cc:493] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.233986 30388 raft_consensus.cc:515] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.236850 30388 leader_election.cc:290] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:54.237623 30076 raft_consensus.cc:359] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.243614 30076 raft_consensus.cc:385] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.244165 30076 raft_consensus.cc:740] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.238158 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:54.255383 30076 consensus_queue.cc:260] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.244566 30299 leader_election.cc:290] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.262295 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
W20260709 12:27:54.255311 29772 leader_election.cc:343] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.263224 30029 raft_consensus.cc:3060] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:54.254421 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 44676.8us
W20260709 12:27:54.266202 29918 leader_election.cc:343] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.279752 30399 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.280391 29880 raft_consensus.cc:2468] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:54.282160 29769 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:54.283808 30385 raft_consensus.cc:2804] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.284461 30385 raft_consensus.cc:697] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:54.285485 30385 consensus_queue.cc:237] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.286387 30075 tablet_bootstrap.cc:654] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.292855 30077 tablet_bootstrap.cc:654] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.293736 29879 raft_consensus.cc:3060] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.300530 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:54.300814 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:54.319648 29703 catalog_manager.cc:5697] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.330179 30399 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.331627 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:54.333576 30026 raft_consensus.cc:3060] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.344826 30076 ts_tablet_manager.cc:1434] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.169s	user 0.004s	sys 0.006s
I20260709 12:27:54.343629 29701 catalog_manager.cc:5697] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.352108 30076 tablet_bootstrap.cc:492] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:54.339140 30028 raft_consensus.cc:2468] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:54.357606 29921 leader_election.cc:304] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:54.359072 30369 raft_consensus.cc:2804] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.359597 30369 raft_consensus.cc:697] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:54.360496 30369 consensus_queue.cc:237] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:54.361933 29847 leader_election.cc:343] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.362263 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:54.362917 29847 leader_election.cc:304] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.345238 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:54.386091 30074 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:54.387033 30074 ts_tablet_manager.cc:1403] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.566s	user 0.016s	sys 0.023s
W20260709 12:27:54.387777 29772 leader_election.cc:343] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:54.382169 29918 leader_election.cc:343] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.407657 30075 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:54.420096 30075 ts_tablet_manager.cc:1403] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.229s	user 0.016s	sys 0.029s
I20260709 12:27:54.420780 30029 raft_consensus.cc:2468] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:54.424252 30076 tablet_bootstrap.cc:654] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.424759 29847 leader_election.cc:304] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.426359 30356 raft_consensus.cc:2804] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.427027 30356 raft_consensus.cc:697] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:54.428056 30356 consensus_queue.cc:237] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.436640 30074 raft_consensus.cc:359] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.437808 30074 raft_consensus.cc:385] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.442075 30074 raft_consensus.cc:740] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.443130 30074 consensus_queue.cc:260] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.445849 30075 raft_consensus.cc:359] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.447893 30075 raft_consensus.cc:385] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.448168 30075 raft_consensus.cc:740] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.452621 30074 ts_tablet_manager.cc:1434] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.065s	user 0.005s	sys 0.004s
I20260709 12:27:54.454383 30075 consensus_queue.cc:260] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.452028 30026 raft_consensus.cc:2468] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:54.460098 30074 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:54.462802 29772 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.463181 30356 raft_consensus.cc:493] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.468042 30077 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:54.471889 30356 raft_consensus.cc:515] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.472440 30077 ts_tablet_manager.cc:1403] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.356s	user 0.011s	sys 0.013s
I20260709 12:27:54.466164 30075 ts_tablet_manager.cc:1434] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.046s	user 0.009s	sys 0.000s
I20260709 12:27:54.466101 30385 raft_consensus.cc:2804] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.479425 30385 raft_consensus.cc:697] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:54.482565 30077 raft_consensus.cc:359] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.485456 30077 raft_consensus.cc:385] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.485941 30077 raft_consensus.cc:740] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.488288 30356 leader_election.cc:290] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:54.489080 30075 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:54.490690 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:54.491760 29802 raft_consensus.cc:2468] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:54.486984 30077 consensus_queue.cc:260] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.493383 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:54.497051 29846 leader_election.cc:304] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:54.504338 30299 raft_consensus.cc:2749] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.480545 30385 consensus_queue.cc:237] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.514732 30165 raft_consensus.cc:493] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.515268 30299 raft_consensus.cc:2804] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:54.515430 30165 raft_consensus.cc:515] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.516166 30299 raft_consensus.cc:493] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.516736 30299 raft_consensus.cc:3060] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.519735 30165 leader_election.cc:290] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:54.519920 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:54.505766 29847 leader_election.cc:343] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.535156 30399 raft_consensus.cc:493] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.535883 30399 raft_consensus.cc:515] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.527604 30077 ts_tablet_manager.cc:1434] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.054s	user 0.008s	sys 0.000s
I20260709 12:27:54.540169 30077 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:54.541978 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:54.543797 29918 leader_election.cc:304] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:54.545365 30399 leader_election.cc:290] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:54.547748 30165 raft_consensus.cc:2749] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.547775 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:54.550520 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:54.552424 29772 leader_election.cc:343] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.555747 30074 tablet_bootstrap.cc:654] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.566289 30076 tablet_bootstrap.cc:492] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
W20260709 12:27:54.566493 29769 leader_election.cc:343] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.566962 30076 ts_tablet_manager.cc:1403] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.215s	user 0.038s	sys 0.037s
I20260709 12:27:54.567049 29769 leader_election.cc:304] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:54.571768 30399 raft_consensus.cc:2749] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.567988 29952 raft_consensus.cc:3060] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.576336 30076 raft_consensus.cc:359] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.577288 30076 raft_consensus.cc:385] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.577610 30076 raft_consensus.cc:740] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.578531 30076 consensus_queue.cc:260] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.581228 30076 ts_tablet_manager.cc:1434] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.003s
I20260709 12:27:54.580605 29704 catalog_manager.cc:5697] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.589432 30076 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:54.591563 30075 tablet_bootstrap.cc:654] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.601367 30299 raft_consensus.cc:515] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.604461 30299 leader_election.cc:290] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:54.606300 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:54.607141 29802 raft_consensus.cc:3060] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.607470 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
W20260709 12:27:54.609371 29847 leader_election.cc:343] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:54.611984 30074 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:54.630038 30392 raft_consensus.cc:493] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.630554 30074 ts_tablet_manager.cc:1403] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.173s	user 0.019s	sys 0.004s
I20260709 12:27:54.630693 30392 raft_consensus.cc:515] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:54.633348 30392 leader_election.cc:290] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.633971 30074 raft_consensus.cc:359] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.634738 30074 raft_consensus.cc:385] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.635015 30074 raft_consensus.cc:740] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.636842 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:54.636847 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:54.638096 30029 raft_consensus.cc:2393] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:54.629662 30374 raft_consensus.cc:493] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.639370 29802 raft_consensus.cc:2468] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:54.639787 30374 raft_consensus.cc:515] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.640543 29701 catalog_manager.cc:5697] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.641161 29846 leader_election.cc:304] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.645632 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:54.618911 30076 tablet_bootstrap.cc:654] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.640933 29847 leader_election.cc:304] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.646984 30392 raft_consensus.cc:3060] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.647823 29700 catalog_manager.cc:5697] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:54.618911 30077 tablet_bootstrap.cc:654] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.644078 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:54.644719 30374 leader_election.cc:290] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:54.645730 30299 raft_consensus.cc:2804] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:54.642491 30074 consensus_queue.cc:260] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.655334 29994 leader_election.cc:304] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:54.656808 30374 raft_consensus.cc:2749] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.657742 30074 ts_tablet_manager.cc:1434] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 12:27:54.659096 30074 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:54.668668 30077 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:54.669431 30077 ts_tablet_manager.cc:1403] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.130s	user 0.010s	sys 0.003s
I20260709 12:27:54.672238 30299 raft_consensus.cc:697] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:54.674417 30076 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:54.675115 30076 ts_tablet_manager.cc:1403] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.086s	user 0.013s	sys 0.001s
I20260709 12:27:54.675611 29703 catalog_manager.cc:5697] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.674568 30299 consensus_queue.cc:237] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.672575 30165 consensus_queue.cc:1048] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.686492 30076 raft_consensus.cc:359] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.687403 30076 raft_consensus.cc:385] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.687672 30076 raft_consensus.cc:740] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.689986 30381 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.690872 30075 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:54.690594 30381 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.691499 30075 ts_tablet_manager.cc:1403] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.203s	user 0.019s	sys 0.000s
I20260709 12:27:54.693048 30381 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:54.692054 30077 raft_consensus.cc:359] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.699641 30077 raft_consensus.cc:385] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.699456 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:54.700062 30077 raft_consensus.cc:740] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.701732 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:54.688462 30076 consensus_queue.cc:260] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.703799 29879 raft_consensus.cc:2393] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:54.706881 30074 tablet_bootstrap.cc:654] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.681627 30392 raft_consensus.cc:2749] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.704259 30076 ts_tablet_manager.cc:1434] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:27:54.719955 29994 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:54.722973 30317 raft_consensus.cc:3060] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.736040 30317 raft_consensus.cc:2749] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.719837 30077 consensus_queue.cc:260] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.740213 30029 raft_consensus.cc:3060] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.749789 30077 ts_tablet_manager.cc:1434] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.080s	user 0.009s	sys 0.001s
I20260709 12:27:54.751765 30076 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:54.758795 30077 tablet_bootstrap.cc:492] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:54.766775 30406 consensus_queue.cc:1048] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.768180 30076 tablet_bootstrap.cc:654] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.782534 30075 raft_consensus.cc:359] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.773059 30077 tablet_bootstrap.cc:654] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:54.783342 30200 consensus_queue.cc:1048] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.785495 30075 raft_consensus.cc:385] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.786845 30075 raft_consensus.cc:740] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.788471 30299 consensus_queue.cc:1048] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:54.806892 30075 consensus_queue.cc:260] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.829888 30295 consensus_queue.cc:1048] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.838773 30406 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.838706 30385 consensus_queue.cc:1048] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.839527 30406 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.843098 30406 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:54.851384 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:54.818686 29703 catalog_manager.cc:5697] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.869410 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:54.870345 30317 raft_consensus.cc:493] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.871222 30317 raft_consensus.cc:515] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:54.875285 29920 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:54.876737 30406 raft_consensus.cc:2749] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:54.881541 29953 raft_consensus.cc:3060] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.899902 30075 ts_tablet_manager.cc:1434] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.208s	user 0.010s	sys 0.001s
I20260709 12:27:54.905737 30074 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:54.906745 30074 ts_tablet_manager.cc:1403] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.248s	user 0.027s	sys 0.003s
I20260709 12:27:54.914764 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:54.923278 29802 raft_consensus.cc:2468] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:54.893190 30200 consensus_queue.cc:1048] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 12:27:54.937188 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:54.938381 29876 raft_consensus.cc:2468] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:54.921761 30317 leader_election.cc:290] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:54.951803 29994 leader_election.cc:304] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:54.951951 30404 consensus_queue.cc:1048] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.952942 30374 raft_consensus.cc:2804] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:54.953380 30374 raft_consensus.cc:493] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.953734 30374 raft_consensus.cc:3060] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:54.954046 30399 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.954769 30399 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:54.962857 30077 tablet_bootstrap.cc:492] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:54.963740 30077 ts_tablet_manager.cc:1403] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.213s	user 0.028s	sys 0.035s
I20260709 12:27:54.964152 30075 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:54.968576 30076 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:54.980407 30074 raft_consensus.cc:359] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:54.981433 30074 raft_consensus.cc:385] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:54.981755 30074 raft_consensus.cc:740] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:54.973124 30076 ts_tablet_manager.cc:1403] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.222s	user 0.015s	sys 0.003s
I20260709 12:27:54.957736 30399 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:54.995184 30074 consensus_queue.cc:260] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.007432 30074 ts_tablet_manager.cc:1434] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.000s
W20260709 12:27:55.016866 29847 consensus_peers.cc:597] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:55.022164 30077 raft_consensus.cc:359] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:55.023285 30077 raft_consensus.cc:385] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.026329 30074 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:55.026202 30404 consensus_queue.cc:1048] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:55.034763 30074 tablet_bootstrap.cc:654] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.012144 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:55.047719 29772 leader_election.cc:343] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.023540 30077 raft_consensus.cc:740] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.051139 30077 consensus_queue.cc:260] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:55.067154 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:55.061683 30077 ts_tablet_manager.cc:1434] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.098s	user 0.006s	sys 0.003s
I20260709 12:27:55.064379 30076 raft_consensus.cc:359] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.074250 30076 raft_consensus.cc:385] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.074594 30076 raft_consensus.cc:740] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.076339 30076 consensus_queue.cc:260] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:55.080608 29772 leader_election.cc:343] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.081223 29772 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:55.091012 30076 ts_tablet_manager.cc:1434] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.101s	user 0.009s	sys 0.000s
I20260709 12:27:55.105681 30077 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:55.087662 30295 raft_consensus.cc:2749] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:55.111289 30076 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:55.087647 30392 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.114631 30074 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:55.115265 30074 ts_tablet_manager.cc:1403] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.098s	user 0.024s	sys 0.029s
W20260709 12:27:55.109273 30439 log.cc:927] Time spent T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:27:55.127015 30438 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.127719 30438 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.120167 30392 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:55.156015 29992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 43707.1us
W20260709 12:27:55.162580 30433 log.cc:927] Time spent T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:27:55.163437 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 65840.8us
W20260709 12:27:55.205853 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:55.205552 30077 tablet_bootstrap.cc:654] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.209474 30075 tablet_bootstrap.cc:654] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.175307 30374 raft_consensus.cc:515] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:55.221366 29921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 69888.6us
I20260709 12:27:55.244073 30374 leader_election.cc:290] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:55.261461 30076 tablet_bootstrap.cc:654] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.326973 30074 raft_consensus.cc:359] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.328150 30074 raft_consensus.cc:385] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:55.283682 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 52974.2us
I20260709 12:27:55.346365 30074 raft_consensus.cc:740] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.352797 30074 consensus_queue.cc:260] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.383373 30076 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:55.384114 30076 ts_tablet_manager.cc:1403] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.284s	user 0.013s	sys 0.007s
W20260709 12:27:55.394168 30441 log.cc:927] Time spent T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:27:55.388041 30392 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:55.379562 30387 consensus_queue.cc:1048] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260709 12:27:55.422891 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:55.423866 29802 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:55.424536 29877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:55.427726 29877 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
W20260709 12:27:55.405864 29921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 156038us
I20260709 12:27:55.432886 30077 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:55.441712 30077 ts_tablet_manager.cc:1403] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.353s	user 0.019s	sys 0.010s
I20260709 12:27:55.442291 30076 raft_consensus.cc:359] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.443528 30076 raft_consensus.cc:385] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.443892 30076 raft_consensus.cc:740] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.444933 30076 consensus_queue.cc:260] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.430996 29846 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:55.452438 30455 raft_consensus.cc:2804] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:55.453055 30455 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.453477 30455 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.456552 30077 raft_consensus.cc:359] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.457469 30077 raft_consensus.cc:385] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.458401 29877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:55.459188 29877 raft_consensus.cc:3060] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.457767 30077 raft_consensus.cc:740] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.469945 30077 consensus_queue.cc:260] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.486533 30076 ts_tablet_manager.cc:1434] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.102s	user 0.009s	sys 0.001s
I20260709 12:27:55.494005 30076 tablet_bootstrap.cc:492] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:55.532718 30075 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:55.534965 30075 ts_tablet_manager.cc:1403] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.619s	user 0.030s	sys 0.013s
I20260709 12:27:55.526073 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:55.554162 30470 raft_consensus.cc:493] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.554921 30470 raft_consensus.cc:515] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:55.557552 30470 leader_election.cc:290] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:55.578502 29769 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:55.565208 30455 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.596308 30074 ts_tablet_manager.cc:1434] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.481s	user 0.020s	sys 0.021s
I20260709 12:27:55.577630 29802 raft_consensus.cc:3060] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.590546 30385 raft_consensus.cc:2804] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:55.609336 30074 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:55.615782 30076 tablet_bootstrap.cc:654] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.615624 30464 consensus_queue.cc:1048] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:55.621598 30461 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.622730 30075 raft_consensus.cc:359] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.626929 30075 raft_consensus.cc:385] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.627377 30075 raft_consensus.cc:740] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.608217 30385 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.631129 30077 ts_tablet_manager.cc:1434] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.189s	user 0.029s	sys 0.036s
I20260709 12:27:55.631255 30075 consensus_queue.cc:260] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.651073 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:55.655635 30074 tablet_bootstrap.cc:654] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.640654 30385 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.661931 29877 raft_consensus.cc:2468] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:55.630587 30461 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.665205 30461 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:27:55.656330 29920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 231270us
I20260709 12:27:55.669976 30075 ts_tablet_manager.cc:1434] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.135s	user 0.008s	sys 0.001s
I20260709 12:27:55.666916 30455 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:55.667953 30077 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:55.680740 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:55.683679 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:55.689985 29949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:55.690310 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:55.695268 29992 leader_election.cc:304] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:55.665525 30438 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:55.700142 30376 raft_consensus.cc:2804] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:55.700778 30376 raft_consensus.cc:697] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:55.702157 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:55.690382 30392 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.706835 30392 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.708313 30077 tablet_bootstrap.cc:654] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.709419 30392 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:55.710719 29880 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:55.721143 30385 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.723152 29806 raft_consensus.cc:2393] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:55.701745 30376 consensus_queue.cc:237] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.729023 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:55.730551 29950 raft_consensus.cc:3060] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.735426 29802 raft_consensus.cc:2468] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
W20260709 12:27:55.739668 29772 consensus_peers.cc:597] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:55.740852 30075 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:55.744020 30449 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.747705 30449 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.748224 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:55.753170 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:55.755661 29772 leader_election.cc:343] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:55.756174 29847 consensus_peers.cc:597] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:55.762535 29995 leader_election.cc:343] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:55.759429 29847 leader_election.cc:343] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:55.771803 29847 leader_election.cc:343] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.780771 30076 tablet_bootstrap.cc:492] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:55.781898 30076 ts_tablet_manager.cc:1403] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.288s	user 0.015s	sys 0.005s
I20260709 12:27:55.799053 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:55.799466 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:55.800060 29802 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
W20260709 12:27:55.801510 29994 leader_election.cc:343] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.802911 29994 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:55.804107 30385 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:55.804498 29918 leader_election.cc:304] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:55.804545 30317 raft_consensus.cc:2749] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:55.807749 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:55.753750 30449 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:55.813395 29769 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
W20260709 12:27:55.831161 29847 leader_election.cc:343] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.861644 30076 raft_consensus.cc:359] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:55.866751 30075 tablet_bootstrap.cc:654] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.865056 30076 raft_consensus.cc:385] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.879657 30077 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:55.880381 30077 ts_tablet_manager.cc:1403] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.213s	user 0.025s	sys 0.012s
I20260709 12:27:55.872187 30076 raft_consensus.cc:740] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.885840 30074 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:55.886677 30074 ts_tablet_manager.cc:1403] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.283s	user 0.018s	sys 0.001s
I20260709 12:27:55.886721 30076 consensus_queue.cc:260] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:55.890237 30445 raft_consensus.cc:2804] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:55.890808 30445 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.891220 30445 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.869838 29846 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:55.894668 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
W20260709 12:27:55.896620 29772 leader_election.cc:343] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.898257 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:55.899267 29876 raft_consensus.cc:2393] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:55.906919 30077 raft_consensus.cc:359] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.907912 30077 raft_consensus.cc:385] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.908285 30077 raft_consensus.cc:740] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.914935 30076 ts_tablet_manager.cc:1434] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.132s	user 0.007s	sys 0.002s
I20260709 12:27:55.916374 30076 tablet_bootstrap.cc:492] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
W20260709 12:27:55.917204 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 68768.8us
I20260709 12:27:55.909070 30077 consensus_queue.cc:260] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.922438 29769 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:55.921590 30200 raft_consensus.cc:2749] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:55.924283 30450 raft_consensus.cc:493] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.924863 30450 raft_consensus.cc:515] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:55.926874 29846 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:55.928040 30200 raft_consensus.cc:2804] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:55.928643 30200 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.929019 30200 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:55.931188 30450 leader_election.cc:290] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:55.933071 30077 ts_tablet_manager.cc:1434] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.052s	user 0.006s	sys 0.001s
I20260709 12:27:55.934386 30074 raft_consensus.cc:359] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.936692 30074 raft_consensus.cc:385] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:55.937052 30074 raft_consensus.cc:740] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:55.946153 30074 consensus_queue.cc:260] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.950616 30077 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:55.941017 30461 raft_consensus.cc:2749] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:55.961184 30456 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.960788 30074 ts_tablet_manager.cc:1434] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.074s	user 0.008s	sys 0.002s
I20260709 12:27:55.961786 30456 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:55.957990 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:55.962021 30471 raft_consensus.cc:2749] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:55.965224 30445 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:55.968540 30076 tablet_bootstrap.cc:654] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:55.988055 30074 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:55.991673 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:55.993340 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
W20260709 12:27:55.995415 29772 leader_election.cc:343] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:55.978025 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:55.985016 30445 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:56.012828 30077 tablet_bootstrap.cc:654] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.011981 29920 leader_election.cc:304] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:56.018080 30471 raft_consensus.cc:2749] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:56.020531 30456 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:56.021432 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.023787 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:56.029217 30074 tablet_bootstrap.cc:654] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.030301 29952 raft_consensus.cc:3060] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:56.029937 30200 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:56.045783 30200 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:56.050547 29876 raft_consensus.cc:2393] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:56.054190 29769 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:56.055572 30075 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:56.055965 30074 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:56.056298 30075 ts_tablet_manager.cc:1403] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.316s	user 0.019s	sys 0.012s
I20260709 12:27:56.056627 30074 ts_tablet_manager.cc:1403] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.069s	user 0.025s	sys 0.003s
I20260709 12:27:56.064448 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:56.049860 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
W20260709 12:27:56.068110 29847 leader_election.cc:343] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.069420 30075 raft_consensus.cc:359] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.070420 30075 raft_consensus.cc:385] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.055606 30461 raft_consensus.cc:2749] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:56.070816 30075 raft_consensus.cc:740] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.049875 29920 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:56.074533 29806 raft_consensus.cc:2393] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:56.084903 30075 consensus_queue.cc:260] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.088042 30075 ts_tablet_manager.cc:1434] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 12:27:56.089917 30075 tablet_bootstrap.cc:492] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:56.094560 29846 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:56.096208 30457 raft_consensus.cc:2749] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:56.099217 30075 tablet_bootstrap.cc:654] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.076694 30076 tablet_bootstrap.cc:492] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:56.078668 30074 raft_consensus.cc:359] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:56.101996 30076 ts_tablet_manager.cc:1403] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.186s	user 0.013s	sys 0.012s
I20260709 12:27:56.102191 30074 raft_consensus.cc:385] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.102591 30074 raft_consensus.cc:740] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.103411 30074 consensus_queue.cc:260] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:56.106204 30074 ts_tablet_manager.cc:1434] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.002s
I20260709 12:27:56.106279 30076 raft_consensus.cc:359] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.107206 30076 raft_consensus.cc:385] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.093150 29704 catalog_manager.cc:5697] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:56.107484 30076 raft_consensus.cc:740] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.108767 30074 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:56.108532 30076 consensus_queue.cc:260] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.111490 30076 ts_tablet_manager.cc:1434] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:27:56.112815 30076 tablet_bootstrap.cc:492] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:56.122735 30074 tablet_bootstrap.cc:654] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.141573 30077 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:56.142307 30077 ts_tablet_manager.cc:1403] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.201s	user 0.027s	sys 0.022s
I20260709 12:27:56.148063 30374 consensus_queue.cc:1048] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:56.149127 30077 raft_consensus.cc:359] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.149983 30077 raft_consensus.cc:385] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.150272 30077 raft_consensus.cc:740] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.151119 30077 consensus_queue.cc:260] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.169380 30077 ts_tablet_manager.cc:1434] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.006s
I20260709 12:27:56.184705 30075 tablet_bootstrap.cc:492] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:56.185739 30075 ts_tablet_manager.cc:1403] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.096s	user 0.028s	sys 0.006s
I20260709 12:27:56.194932 29949 raft_consensus.cc:3060] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:56.249943 30076 tablet_bootstrap.cc:654] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.258350 30077 tablet_bootstrap.cc:492] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:56.255177 30457 raft_consensus.cc:493] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.295568 30074 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:56.296311 30074 ts_tablet_manager.cc:1403] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.188s	user 0.013s	sys 0.009s
I20260709 12:27:56.273262 30445 raft_consensus.cc:493] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.301512 30445 raft_consensus.cc:515] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:56.288105 30457 raft_consensus.cc:515] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.500304 30376 consensus_queue.cc:1048] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:56.501605 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 172172us
W20260709 12:27:56.503141 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:56.355360 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:56.508544 30449 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.516077 30449 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.518800 30449 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:56.338639 30475 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.520568 30075 raft_consensus.cc:359] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.519042 30074 raft_consensus.cc:359] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.521886 30075 raft_consensus.cc:385] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.522449 30075 raft_consensus.cc:740] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.523584 30075 consensus_queue.cc:260] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.522022 30074 raft_consensus.cc:385] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.526129 30475 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.529400 30475 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:56.525337 30074 raft_consensus.cc:740] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.542784 30074 consensus_queue.cc:260] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.558295 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:56.562754 30457 leader_election.cc:290] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:56.564819 30077 tablet_bootstrap.cc:654] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.538760 30471 raft_consensus.cc:2749] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:56.579732 30074 ts_tablet_manager.cc:1434] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.283s	user 0.012s	sys 0.002s
I20260709 12:27:56.577067 30455 raft_consensus.cc:493] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.591655 30075 ts_tablet_manager.cc:1434] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.405s	user 0.012s	sys 0.003s
I20260709 12:27:56.584338 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:56.594686 29847 leader_election.cc:343] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:56.598181 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 101117us
W20260709 12:27:56.581054 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 51694.5us
I20260709 12:27:56.601919 30074 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:56.598664 30445 leader_election.cc:290] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:56.607851 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:56.608763 29804 raft_consensus.cc:2468] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:56.589139 30456 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.611693 30456 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.616459 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.617424 30029 raft_consensus.cc:2468] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:56.599017 30455 raft_consensus.cc:515] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.619541 29847 leader_election.cc:304] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:56.620388 30455 leader_election.cc:290] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:56.621155 30455 raft_consensus.cc:2804] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:56.621904 30456 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:56.623677 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:56.617568 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:56.624451 30076 tablet_bootstrap.cc:492] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:56.627915 30076 ts_tablet_manager.cc:1403] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.515s	user 0.030s	sys 0.031s
I20260709 12:27:56.621567 30455 raft_consensus.cc:493] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:56.631994 29769 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 75718.8us
W20260709 12:27:56.632900 29769 leader_election.cc:343] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.625555 29994 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:56.636080 30449 raft_consensus.cc:2804] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:56.636514 30449 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:56.636907 30449 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:56.639971 30076 raft_consensus.cc:359] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.640869 30076 raft_consensus.cc:385] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.641180 30076 raft_consensus.cc:740] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.641026 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.634539 29951 raft_consensus.cc:2468] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:56.643376 30076 consensus_queue.cc:260] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.643966 30074 tablet_bootstrap.cc:654] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.648587 30075 tablet_bootstrap.cc:492] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:56.633049 30455 raft_consensus.cc:3060] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:56.671126 29847 leader_election.cc:343] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.660013 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:56.680176 29847 leader_election.cc:343] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.680848 29847 leader_election.cc:304] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:56.682860 30387 raft_consensus.cc:2749] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:56.650471 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.685123 30029 raft_consensus.cc:2468] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
W20260709 12:27:56.682928 29995 leader_election.cc:343] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.688244 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.689015 30027 raft_consensus.cc:2468] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:56.712252 30076 ts_tablet_manager.cc:1434] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.084s	user 0.008s	sys 0.002s
W20260709 12:27:56.711367 29772 leader_election.cc:343] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.720331 29772 leader_election.cc:304] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:56.722555 29772 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:56.740341 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:56.730057 30499 raft_consensus.cc:2749] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:56.742405 30505 raft_consensus.cc:493] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.743294 30505 raft_consensus.cc:515] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.729768 30445 raft_consensus.cc:2804] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:56.745245 30445 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:56.745623 30445 raft_consensus.cc:3060] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:56.748636 30505 leader_election.cc:290] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:56.732874 29921 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:56.751914 30165 raft_consensus.cc:2804] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:56.752507 30165 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:56.754729 30165 raft_consensus.cc:3060] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:56.755584 30076 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:56.758302 30077 tablet_bootstrap.cc:492] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:56.759014 30077 ts_tablet_manager.cc:1403] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.524s	user 0.053s	sys 0.029s
I20260709 12:27:56.759093 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3b379d58464fa6b60b8a33472252d4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:56.761356 29847 leader_election.cc:343] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.764744 30075 tablet_bootstrap.cc:654] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:56.787870 30077 raft_consensus.cc:359] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.788833 30077 raft_consensus.cc:385] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.789115 30077 raft_consensus.cc:740] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.806279 30077 consensus_queue.cc:260] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.809530 30076 tablet_bootstrap.cc:654] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:56.805833 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 75709.5us
W20260709 12:27:56.805758 29992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 130598us
I20260709 12:27:56.826296 30077 ts_tablet_manager.cc:1434] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.067s	user 0.008s	sys 0.002s
I20260709 12:27:56.827880 30074 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:56.828548 30074 ts_tablet_manager.cc:1403] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.240s	user 0.010s	sys 0.007s
I20260709 12:27:56.858011 30449 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.857273 30074 raft_consensus.cc:359] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:56.862869 30074 raft_consensus.cc:385] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:56.863232 30074 raft_consensus.cc:740] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:56.865633 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3b379d58464fa6b60b8a33472252d4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.869916 30077 tablet_bootstrap.cc:492] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:56.875242 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:56.875871 29949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:56.876138 29806 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:56.878005 29995 leader_election.cc:343] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.863881 30449 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:56.866966 30074 consensus_queue.cc:260] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:56.974459 29847 leader_election.cc:343] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:56.979625 29847 leader_election.cc:304] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:56.983232 30503 raft_consensus.cc:493] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.983932 30503 raft_consensus.cc:515] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:56.987887 30503 leader_election.cc:290] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:56.990445 30074 ts_tablet_manager.cc:1434] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.162s	user 0.018s	sys 0.044s
I20260709 12:27:56.989283 30445 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:56.995970 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:56.997336 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:56.999094 30027 raft_consensus.cc:2468] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:56.993669 30392 raft_consensus.cc:2749] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.002321 29921 leader_election.cc:304] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:57.007215 30471 raft_consensus.cc:2804] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.007789 30471 raft_consensus.cc:493] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.008152 30471 raft_consensus.cc:3060] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.009500 29802 raft_consensus.cc:2393] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:57.005216 30455 raft_consensus.cc:515] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.010744 29879 raft_consensus.cc:2393] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:57.015782 29806 raft_consensus.cc:2468] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:57.020236 29994 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:57.023277 30165 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:57.016149 30445 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:57.025343 30317 raft_consensus.cc:2804] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.026075 30317 raft_consensus.cc:697] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:57.027014 30317 consensus_queue.cc:237] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:57.033138 30077 tablet_bootstrap.cc:654] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.034565 30074 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:57.037717 30502 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.043071 30165 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:57.046880 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:57.051215 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:57.052057 30027 raft_consensus.cc:3060] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.052683 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:57.059912 30475 raft_consensus.cc:493] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.060799 29806 raft_consensus.cc:2393] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:57.060662 30475 raft_consensus.cc:515] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.063215 30475 leader_election.cc:290] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:57.065191 29920 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:57.052922 30075 tablet_bootstrap.cc:492] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:57.069882 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:57.070921 29951 raft_consensus.cc:2393] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:57.068078 30075 ts_tablet_manager.cc:1403] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.439s	user 0.033s	sys 0.021s
W20260709 12:27:57.066677 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 53105.2us
I20260709 12:27:57.075594 30503 raft_consensus.cc:2749] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.070042 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:57.092388 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:57.092512 29769 leader_election.cc:343] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.047349 30502 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:27:57.096840 29772 leader_election.cc:343] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.097457 29772 leader_election.cc:304] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:57.098348 30502 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:57.098421 30445 raft_consensus.cc:2749] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.102910 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:57.103483 30075 raft_consensus.cc:359] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.106137 30075 raft_consensus.cc:385] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.106451 30075 raft_consensus.cc:740] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
W20260709 12:27:57.108371 29772 leader_election.cc:343] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.121614 29949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:57.129707 30075 consensus_queue.cc:260] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:57.130565 29772 leader_election.cc:343] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.132701 29772 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:57.135341 30462 raft_consensus.cc:2749] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.136952 30076 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:57.137667 30076 ts_tablet_manager.cc:1403] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.404s	user 0.021s	sys 0.005s
I20260709 12:27:57.137972 30075 ts_tablet_manager.cc:1434] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.002s
I20260709 12:27:57.141917 30074 tablet_bootstrap.cc:654] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.144335 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:57.135281 30455 leader_election.cc:290] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:57.146898 30077 tablet_bootstrap.cc:492] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:57.147637 30077 ts_tablet_manager.cc:1403] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.278s	user 0.013s	sys 0.013s
I20260709 12:27:57.151054 29704 catalog_manager.cc:5697] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.153051 30076 raft_consensus.cc:359] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.158152 30076 raft_consensus.cc:385] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.158574 30076 raft_consensus.cc:740] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.159210 30076 consensus_queue.cc:260] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.154548 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:57.163530 30024 raft_consensus.cc:3060] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.165560 30027 raft_consensus.cc:2468] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:57.151103 30077 raft_consensus.cc:359] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.167728 30077 raft_consensus.cc:385] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.168031 30077 raft_consensus.cc:740] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.168859 30077 consensus_queue.cc:260] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.181916 30075 tablet_bootstrap.cc:492] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:57.182760 29772 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:57.183909 30462 raft_consensus.cc:2804] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.184460 30462 raft_consensus.cc:697] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:57.185359 30462 consensus_queue.cc:237] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:57.188563 30074 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:57.189234 30074 ts_tablet_manager.cc:1403] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.155s	user 0.013s	sys 0.004s
I20260709 12:27:57.195309 30076 ts_tablet_manager.cc:1434] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.057s	user 0.004s	sys 0.003s
I20260709 12:27:57.206625 30077 ts_tablet_manager.cc:1434] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.059s	user 0.010s	sys 0.000s
I20260709 12:27:57.208156 30455 raft_consensus.cc:493] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.209045 30076 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:57.209929 30024 raft_consensus.cc:2468] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:57.210021 30455 raft_consensus.cc:515] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.217386 29847 leader_election.cc:304] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:57.218758 30506 raft_consensus.cc:2804] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.219293 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:57.219623 30455 leader_election.cc:290] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.219938 30506 raft_consensus.cc:697] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:57.223836 30076 tablet_bootstrap.cc:654] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.225162 29949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:57.224681 30506 consensus_queue.cc:237] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:57.227088 29847 leader_election.cc:343] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.227766 30077 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:57.219341 30075 tablet_bootstrap.cc:654] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.220096 29802 raft_consensus.cc:2468] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:57.239934 29846 leader_election.cc:304] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:57.231564 30471 raft_consensus.cc:515] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.243726 30455 raft_consensus.cc:2804] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.248051 30455 raft_consensus.cc:493] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.248494 30455 raft_consensus.cc:3060] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.253513 30074 raft_consensus.cc:359] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.257157 30074 raft_consensus.cc:385] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.257517 30074 raft_consensus.cc:740] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.251814 29701 catalog_manager.cc:5697] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:57.258333 30074 consensus_queue.cc:260] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.252583 29954 raft_consensus.cc:2393] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:57.280434 30074 ts_tablet_manager.cc:1434] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.091s	user 0.009s	sys 0.000s
I20260709 12:27:57.281776 30074 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:57.284312 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:57.285980 30027 raft_consensus.cc:2393] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:57.283070 30471 leader_election.cc:290] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:57.301524 30077 tablet_bootstrap.cc:654] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.307017 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35000da0d1054109927271f881d5b697" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:57.306880 30074 tablet_bootstrap.cc:654] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.322710 29918 leader_election.cc:304] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:57.327888 30165 raft_consensus.cc:2749] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.339877 30075 tablet_bootstrap.cc:492] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:57.340584 30075 ts_tablet_manager.cc:1403] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.159s	user 0.021s	sys 0.004s
I20260709 12:27:57.343680 30455 raft_consensus.cc:515] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.347646 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:57.348907 29802 raft_consensus.cc:3060] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.385017 30076 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:57.386003 30076 ts_tablet_manager.cc:1403] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.177s	user 0.019s	sys 0.008s
I20260709 12:27:57.392673 29700 catalog_manager.cc:5697] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.409471 30471 raft_consensus.cc:493] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.410276 30471 raft_consensus.cc:515] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.413808 30471 leader_election.cc:290] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:57.416100 29802 raft_consensus.cc:2468] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:57.416939 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:57.418339 29846 leader_election.cc:304] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:57.420208 30505 raft_consensus.cc:2804] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.421604 30505 raft_consensus.cc:697] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:57.423178 30505 consensus_queue.cc:237] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:57.427037 29921 leader_election.cc:343] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.434067 30075 raft_consensus.cc:359] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.435402 30075 raft_consensus.cc:385] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.437495 30075 raft_consensus.cc:740] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.438552 30075 consensus_queue.cc:260] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.441361 30075 ts_tablet_manager.cc:1434] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.000s
I20260709 12:27:57.444972 30076 raft_consensus.cc:359] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.446167 30076 raft_consensus.cc:385] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.446555 30076 raft_consensus.cc:740] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.447640 30076 consensus_queue.cc:260] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.450852 30076 ts_tablet_manager.cc:1434] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.064s	user 0.009s	sys 0.000s
I20260709 12:27:57.452389 30076 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:57.453653 30455 leader_election.cc:290] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.454275 30077 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:57.454929 30077 ts_tablet_manager.cc:1403] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.227s	user 0.024s	sys 0.013s
I20260709 12:27:57.454092 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:57.458794 29879 raft_consensus.cc:2468] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 0.
I20260709 12:27:57.459203 29949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:57.461174 30449 consensus_queue.cc:1048] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:57.462641 30077 raft_consensus.cc:359] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:57.463814 30077 raft_consensus.cc:385] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.464210 30077 raft_consensus.cc:740] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
W20260709 12:27:57.465407 29847 leader_election.cc:343] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.468137 29918 leader_election.cc:304] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:57.469579 30456 raft_consensus.cc:2804] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.470075 30456 raft_consensus.cc:493] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.470448 30456 raft_consensus.cc:3060] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.471129 30077 consensus_queue.cc:260] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:57.483098 30075 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:57.489053 30077 ts_tablet_manager.cc:1434] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.001s
I20260709 12:27:57.490603 30077 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:57.504401 30462 raft_consensus.cc:493] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.505390 30462 raft_consensus.cc:515] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.523217 30075 tablet_bootstrap.cc:654] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.528951 30077 tablet_bootstrap.cc:654] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.541462 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aaee6416f659496fb39a11ee1ae00ca3" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:57.545924 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aaee6416f659496fb39a11ee1ae00ca3" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:57.548493 29772 leader_election.cc:343] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.555683 30074 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:57.556447 30074 ts_tablet_manager.cc:1403] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.275s	user 0.037s	sys 0.035s
I20260709 12:27:57.558851 30509 raft_consensus.cc:493] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.559559 30509 raft_consensus.cc:515] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.564452 30076 tablet_bootstrap.cc:654] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.545599 30462 leader_election.cc:290] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:57.574424 30509 leader_election.cc:290] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.574704 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:57.575490 29881 raft_consensus.cc:2468] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:57.580945 30456 raft_consensus.cc:515] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.587060 30076 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:57.587291 29992 leader_election.cc:304] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:57.587817 30076 ts_tablet_manager.cc:1403] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.136s	user 0.009s	sys 0.006s
I20260709 12:27:57.588121 30317 raft_consensus.cc:493] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.588747 30317 raft_consensus.cc:515] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.591292 30317 leader_election.cc:290] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.592859 30317 raft_consensus.cc:2804] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.595175 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:57.634526 29881 raft_consensus.cc:3060] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.631752 29701 catalog_manager.cc:5697] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.591508 30074 raft_consensus.cc:359] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.637063 30074 raft_consensus.cc:385] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:57.637805 29769 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 84951.5us
I20260709 12:27:57.647503 30317 raft_consensus.cc:493] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:57.640353 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 63270.5us
I20260709 12:27:57.648053 30317 raft_consensus.cc:3060] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.648530 30456 leader_election.cc:290] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:57.652889 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3b379d58464fa6b60b8a33472252d4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:57.653858 29879 raft_consensus.cc:2468] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:57.655388 29992 leader_election.cc:304] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:57.655952 29951 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.656327 30509 raft_consensus.cc:2804] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.656812 30509 raft_consensus.cc:493] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.657223 30509 raft_consensus.cc:3060] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.637492 30074 raft_consensus.cc:740] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.659250 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:57.660367 29954 raft_consensus.cc:2393] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:57.660202 30074 consensus_queue.cc:260] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.662573 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3b379d58464fa6b60b8a33472252d4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:57.669021 29769 leader_election.cc:343] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.669767 29769 leader_election.cc:304] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:57.678992 29881 raft_consensus.cc:2468] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:27:57.683343 30074 ts_tablet_manager.cc:1434] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.127s	user 0.012s	sys 0.000s
I20260709 12:27:57.710628 30077 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:57.711699 30077 ts_tablet_manager.cc:1403] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.221s	user 0.013s	sys 0.010s
I20260709 12:27:57.679797 30500 raft_consensus.cc:2749] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.721630 30076 raft_consensus.cc:359] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.725697 30076 raft_consensus.cc:385] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:57.714640 29995 leader_election.cc:343] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.726214 30076 raft_consensus.cc:740] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
W20260709 12:27:57.735025 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:57.735184 30076 consensus_queue.cc:260] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.723739 30471 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.750197 30074 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:57.758561 30075 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:57.759361 30075 ts_tablet_manager.cc:1403] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.277s	user 0.026s	sys 0.021s
I20260709 12:27:57.759694 30374 consensus_queue.cc:1048] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:57.714468 29918 leader_election.cc:304] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
W20260709 12:27:57.762528 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 78248.8us
I20260709 12:27:57.763293 30075 raft_consensus.cc:359] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.763949 30165 raft_consensus.cc:2804] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.764318 30075 raft_consensus.cc:385] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.764737 30075 raft_consensus.cc:740] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.765748 30075 consensus_queue.cc:260] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.774187 30076 ts_tablet_manager.cc:1434] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.186s	user 0.012s	sys 0.001s
I20260709 12:27:57.785455 30075 ts_tablet_manager.cc:1434] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.001s
W20260709 12:27:57.789273 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 71548.6us
I20260709 12:27:57.806968 30165 raft_consensus.cc:697] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:27:57.808056 30076 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:57.752609 30471 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:57.808494 30165 consensus_queue.cc:237] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:57.814916 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 60300.9us
I20260709 12:27:57.791844 30509 raft_consensus.cc:515] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.869396 30540 raft_consensus.cc:493] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.894776 30317 raft_consensus.cc:515] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:57.876296 30475 raft_consensus.cc:493] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.908296 30076 tablet_bootstrap.cc:654] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.915841 30317 leader_election.cc:290] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.897290 30540 raft_consensus.cc:515] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.918633 30509 leader_election.cc:290] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.919566 30540 leader_election.cc:290] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:57.898667 30500 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.904331 30475 raft_consensus.cc:515] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.920449 30500 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:57.922873 30500 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:57.923198 30475 leader_election.cc:290] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.925302 30077 raft_consensus.cc:359] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.926697 30077 raft_consensus.cc:385] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.927201 30077 raft_consensus.cc:740] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.939683 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:57.941323 30074 tablet_bootstrap.cc:654] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.941314 29881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:57.942371 29881 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:57.942654 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:57.928551 30506 raft_consensus.cc:493] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.943593 30022 raft_consensus.cc:2393] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:27:57.943950 29806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:57.943804 30506 raft_consensus.cc:515] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.944744 29806 raft_consensus.cc:2468] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:57.946689 30075 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:57.947923 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:57.948598 29846 leader_election.cc:304] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:57.950961 30505 raft_consensus.cc:2804] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.951546 30505 raft_consensus.cc:493] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.951941 30505 raft_consensus.cc:3060] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.952054 30506 leader_election.cc:290] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:57.955305 30077 consensus_queue.cc:260] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:57.956478 29769 leader_election.cc:343] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.963836 29769 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:57.965261 30475 raft_consensus.cc:2804] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:57.965735 30475 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.966161 30475 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:57.967270 30074 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:57.967931 30074 ts_tablet_manager.cc:1403] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.241s	user 0.013s	sys 0.008s
I20260709 12:27:57.971369 30074 raft_consensus.cc:359] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.972147 30074 raft_consensus.cc:385] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.972425 30074 raft_consensus.cc:740] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.973361 30074 consensus_queue.cc:260] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:57.975579 30074 ts_tablet_manager.cc:1434] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:57.976783 30074 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:57.977164 30075 tablet_bootstrap.cc:654] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.979928 29954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3b379d58464fa6b60b8a33472252d4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:57.986270 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c3b379d58464fa6b60b8a33472252d4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:57.987216 29876 raft_consensus.cc:3060] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.988240 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811ccdd2daa94877be1d910bd3745085" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:57.989207 29879 raft_consensus.cc:2393] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:27:57.992967 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:57.993892 29801 raft_consensus.cc:2468] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:57.995759 30471 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:58.002183 29992 leader_election.cc:304] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:58.003274 30077 ts_tablet_manager.cc:1434] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.291s	user 0.009s	sys 0.003s
I20260709 12:27:58.003961 30537 raft_consensus.cc:2749] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:58.015606 30505 raft_consensus.cc:515] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.017084 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:58.019518 30505 leader_election.cc:290] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:58.035935 30021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:58.040477 30471 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.041219 30471 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:58.042186 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:58.041226 30075 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:58.044716 30075 ts_tablet_manager.cc:1403] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.098s	user 0.015s	sys 0.000s
I20260709 12:27:58.012444 29846 leader_election.cc:304] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:58.046819 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:58.047569 30392 raft_consensus.cc:2804] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:58.047848 29801 raft_consensus.cc:2393] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1ae5481d76bf4a90a0ee824a14223421 in this term.
I20260709 12:27:58.048081 30392 raft_consensus.cc:493] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.048529 30392 raft_consensus.cc:3060] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:58.052860 29995 leader_election.cc:343] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.056972 29799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:58.058002 29799 raft_consensus.cc:3060] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.049155 30075 raft_consensus.cc:359] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.062790 30075 raft_consensus.cc:385] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.064495 30471 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:58.064312 29875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:58.065523 29875 raft_consensus.cc:2393] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:27:58.067999 30475 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.069875 30029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:58.071009 30074 tablet_bootstrap.cc:654] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:58.072284 29875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:58.063197 30075 raft_consensus.cc:740] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.073123 30545 consensus_queue.cc:1048] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.087224 29921 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:58.073146 29875 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.073773 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:58.073695 29955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:58.093446 30542 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.094097 30542 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:58.096845 30542 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:58.098578 30075 consensus_queue.cc:260] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.098737 30544 consensus_queue.cc:1048] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.101940 30075 ts_tablet_manager.cc:1434] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.057s	user 0.009s	sys 0.000s
I20260709 12:27:58.102224 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:58.103217 29879 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 1.
I20260709 12:27:58.110357 30077 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
W20260709 12:27:58.104543 29847 leader_election.cc:343] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:58.117935 29847 leader_election.cc:343] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.114006 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:58.103430 30075 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:58.104346 30475 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:58.134407 29799 raft_consensus.cc:2468] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:58.134346 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:58.136425 29951 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 0.
I20260709 12:27:58.138875 29769 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:58.139314 29846 leader_election.cc:304] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:58.139890 29875 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:27:58.140249 30475 raft_consensus.cc:2804] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:58.140736 30475 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.141104 30475 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.141747 29769 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:58.142880 30535 raft_consensus.cc:2804] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.143438 30535 raft_consensus.cc:697] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:58.151012 30024 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.144439 30535 consensus_queue.cc:237] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.167824 29918 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:58.169991 30541 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.162256 30506 raft_consensus.cc:2804] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:58.118983 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:58.174636 30506 raft_consensus.cc:697] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:58.176152 30506 consensus_queue.cc:237] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.177928 30457 consensus_queue.cc:1048] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.181792 30076 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:58.182569 30076 ts_tablet_manager.cc:1403] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.386s	user 0.026s	sys 0.025s
W20260709 12:27:58.192705 29772 leader_election.cc:343] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.198928 29772 leader_election.cc:304] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:58.198069 30200 consensus_queue.cc:1048] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.194465 29876 raft_consensus.cc:2468] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:58.200348 30392 raft_consensus.cc:515] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:58.203366 30076 raft_consensus.cc:359] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.207283 30076 raft_consensus.cc:385] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.207772 30076 raft_consensus.cc:740] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.214747 29992 leader_election.cc:304] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:58.208712 30076 consensus_queue.cc:260] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:58.217023 29847 leader_election.cc:343] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:58.222566 29847 consensus_peers.cc:597] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:58.228184 30449 raft_consensus.cc:2804] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:58.229167 30543 raft_consensus.cc:2749] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:58.231688 30076 ts_tablet_manager.cc:1434] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.001s
I20260709 12:27:58.229723 30449 raft_consensus.cc:697] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:58.236217 30449 consensus_queue.cc:237] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:58.240353 30077 tablet_bootstrap.cc:654] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:58.245282 30075 tablet_bootstrap.cc:654] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:58.214789 30445 raft_consensus.cc:2749] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:58.245699 30475 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:58.254493 30024 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:27:58.280310 30445 consensus_queue.cc:1048] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.104s
I20260709 12:27:58.427902 30541 raft_consensus.cc:2749] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:58.431236 30074 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:58.432050 30074 ts_tablet_manager.cc:1403] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.456s	user 0.014s	sys 0.006s
W20260709 12:27:58.432235 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 140976us
I20260709 12:27:58.437503 30074 raft_consensus.cc:359] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.438390 30074 raft_consensus.cc:385] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.438733 30074 raft_consensus.cc:740] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.439507 30074 consensus_queue.cc:260] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.443189 30475 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:58.443763 30392 leader_election.cc:290] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:27:58.445439 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 127140us
I20260709 12:27:58.443225 30074 ts_tablet_manager.cc:1434] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:27:58.513547 30074 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:58.515556 30075 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:58.516232 30075 ts_tablet_manager.cc:1403] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.413s	user 0.013s	sys 0.016s
I20260709 12:27:58.458278 30076 tablet_bootstrap.cc:492] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:58.459477 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:58.462430 29875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:27:58.518713 29875 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.533113 29801 raft_consensus.cc:3060] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.537506 30076 tablet_bootstrap.cc:654] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:58.473153 29701 catalog_manager.cc:5697] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.496172 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:58.499070 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:58.548036 29951 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.471927 29704 catalog_manager.cc:5697] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 1 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.545028 30074 tablet_bootstrap.cc:654] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:58.552626 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 79731.4us
I20260709 12:27:58.560664 29951 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:27:58.563881 30537 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.564623 30537 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.567188 30537 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:58.464529 30077 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:58.581629 30077 ts_tablet_manager.cc:1403] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.475s	user 0.010s	sys 0.008s
I20260709 12:27:58.602640 29772 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:58.556849 29875 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:27:58.615469 30075 raft_consensus.cc:359] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.616760 30075 raft_consensus.cc:385] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.617239 30075 raft_consensus.cc:740] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.618032 30075 consensus_queue.cc:260] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.611145 30534 raft_consensus.cc:2804] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.635578 30075 ts_tablet_manager.cc:1434] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.119s	user 0.010s	sys 0.000s
I20260709 12:27:58.608283 30471 consensus_queue.cc:1048] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 12:27:58.628685 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:58.650367 30534 raft_consensus.cc:697] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:27:58.646524 29801 raft_consensus.cc:2468] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
W20260709 12:27:58.694610 30574 log.cc:927] Time spent T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:27:58.706612 30074 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:58.707528 30074 ts_tablet_manager.cc:1403] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.194s	user 0.024s	sys 0.010s
W20260709 12:27:58.714365 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 67030.8us
I20260709 12:27:58.726899 30076 tablet_bootstrap.cc:492] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:58.727524 30076 ts_tablet_manager.cc:1403] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.270s	user 0.005s	sys 0.018s
I20260709 12:27:58.728125 30077 raft_consensus.cc:359] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.729022 30077 raft_consensus.cc:385] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.748026 30536 consensus_queue.cc:1048] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.735244 30534 consensus_queue.cc:237] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:58.751012 29878 raft_consensus.cc:1275] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:58.741761 30075 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:58.755242 29846 leader_election.cc:304] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:58.756626 30200 raft_consensus.cc:2804] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:58.757355 30200 raft_consensus.cc:697] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:58.750607 30076 raft_consensus.cc:359] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:58.758126 30076 raft_consensus.cc:385] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.758598 30076 raft_consensus.cc:740] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.758497 30200 consensus_queue.cc:237] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:58.748245 30077 raft_consensus.cc:740] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.761595 30077 consensus_queue.cc:260] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:27:58.773291 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 141688us
W20260709 12:27:58.774278 29847 leader_election.cc:343] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.776124 30076 consensus_queue.cc:260] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:58.780269 30074 raft_consensus.cc:359] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.781242 30074 raft_consensus.cc:385] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:58.781561 30074 raft_consensus.cc:740] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.786418 30077 ts_tablet_manager.cc:1434] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.204s	user 0.010s	sys 0.000s
I20260709 12:27:58.790467 29951 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:58.790791 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:58.792965 30577 consensus_queue.cc:1048] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 12:27:58.794973 30076 ts_tablet_manager.cc:1434] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.067s	user 0.010s	sys 0.000s
I20260709 12:27:58.784881 30074 consensus_queue.cc:260] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.806625 30505 raft_consensus.cc:493] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.821919 30543 consensus_queue.cc:1048] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.815770 30505 raft_consensus.cc:515] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.828655 30505 leader_election.cc:290] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:58.829414 30028 raft_consensus.cc:3060] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.828439 29703 catalog_manager.cc:5697] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:58.815922 29802 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:58.833215 29994 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:27:58.834954 30449 raft_consensus.cc:2804] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:58.835469 30449 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.835824 30449 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.840474 30075 tablet_bootstrap.cc:654] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:58.865013 30074 ts_tablet_manager.cc:1434] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.157s	user 0.008s	sys 0.002s
I20260709 12:27:58.871317 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aaee6416f659496fb39a11ee1ae00ca3" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:58.872414 29802 raft_consensus.cc:2468] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
W20260709 12:27:58.874190 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 44167.7us
I20260709 12:27:58.878544 30077 tablet_bootstrap.cc:492] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:58.879787 30076 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:58.874735 29846 leader_election.cc:304] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:58.863224 30540 consensus_queue.cc:1048] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:27:58.883109 30578 raft_consensus.cc:493] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.883455 30544 raft_consensus.cc:2804] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:58.883970 30544 raft_consensus.cc:493] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.883783 30578 raft_consensus.cc:515] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:58.884377 30544 raft_consensus.cc:3060] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.886461 30578 leader_election.cc:290] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:58.890702 30576 raft_consensus.cc:493] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.891409 30576 raft_consensus.cc:515] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:58.892664 30449 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:58.894335 30074 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:27:58.905169 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5988ddbc0f84d47b9b94c1375502692" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:58.916766 29876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:58.920269 29769 leader_election.cc:343] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:58.926740 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:58.883226 30025 raft_consensus.cc:1275] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:27:58.929587 29847 leader_election.cc:343] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.936815 30074 tablet_bootstrap.cc:654] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:58.940613 30548 consensus_queue.cc:1048] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.943918 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:58.953454 30576 leader_election.cc:290] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
W20260709 12:27:58.956162 29995 consensus_peers.cc:597] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:58.960511 29772 leader_election.cc:343] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.961661 29772 leader_election.cc:304] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:58.988426 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5988ddbc0f84d47b9b94c1375502692" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:58.970022 30449 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:27:58.994915 29847 leader_election.cc:343] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:58.995720 29847 leader_election.cc:304] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:58.966660 30475 raft_consensus.cc:2749] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:59.003661 30505 raft_consensus.cc:2749] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:59.034353 30076 tablet_bootstrap.cc:654] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:59.025444 30492 log.cc:927] Time spent T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:27:59.043684 29799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:59.044579 29799 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.044762 30544 raft_consensus.cc:515] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.023785 30374 raft_consensus.cc:493] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.047959 30374 raft_consensus.cc:515] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.025116 30581 consensus_queue.cc:1048] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
W20260709 12:27:59.072707 30595 log.cc:927] Time spent T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:27:59.074132 30583 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.076520 30583 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.084640 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:59.081702 30583 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
W20260709 12:27:59.090562 29772 leader_election.cc:343] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.108632 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aaee6416f659496fb39a11ee1ae00ca3" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:27:59.110716 29847 leader_election.cc:343] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.112082 29953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:59.112895 29953 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.091668 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:59.150779 30075 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:59.151975 30075 ts_tablet_manager.cc:1403] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.411s	user 0.030s	sys 0.015s
W20260709 12:27:59.153167 29772 leader_election.cc:343] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.153896 29772 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
W20260709 12:27:59.162782 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 96274.4us
I20260709 12:27:59.174626 30075 raft_consensus.cc:359] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.175118 29799 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:59.163722 30542 raft_consensus.cc:2749] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:59.180279 29700 catalog_manager.cc:5697] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:59.184257 30075 raft_consensus.cc:385] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.187954 30075 raft_consensus.cc:740] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
W20260709 12:27:59.193396 30593 log.cc:927] Time spent T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:27:59.195708 30544 leader_election.cc:290] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:59.197495 29953 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:59.199083 29995 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:27:59.202577 30374 leader_election.cc:290] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:59.203555 30374 raft_consensus.cc:2804] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:59.204164 30374 raft_consensus.cc:697] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:59.207466 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aaee6416f659496fb39a11ee1ae00ca3" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
W20260709 12:27:59.213887 29847 leader_election.cc:343] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.215230 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:59.218273 30074 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: No bootstrap required, opened a new log
I20260709 12:27:59.219018 30074 ts_tablet_manager.cc:1403] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.340s	user 0.023s	sys 0.016s
I20260709 12:27:59.240983 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9fd2b0fbb1d49da957faacba2d1b38f" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:59.245000 29994 leader_election.cc:304] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:59.192467 30075 consensus_queue.cc:260] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.253933 30074 raft_consensus.cc:359] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.255808 30074 raft_consensus.cc:385] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.256415 30074 raft_consensus.cc:740] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.257794 30074 consensus_queue.cc:260] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.265969 30075 ts_tablet_manager.cc:1434] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.114s	user 0.010s	sys 0.000s
I20260709 12:27:59.205184 30374 consensus_queue.cc:237] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.249763 30537 raft_consensus.cc:2749] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:59.279551 30077 tablet_bootstrap.cc:654] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:59.290640 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aaee6416f659496fb39a11ee1ae00ca3" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:59.291841 29805 raft_consensus.cc:3060] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.297016 29876 raft_consensus.cc:1275] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:59.298784 30576 consensus_queue.cc:1048] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:59.305672 30075 tablet_bootstrap.cc:492] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:59.322547 29805 raft_consensus.cc:2468] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:59.325455 29846 leader_election.cc:304] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:59.326946 29950 raft_consensus.cc:1275] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:59.326946 30506 raft_consensus.cc:2804] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:59.328929 30506 raft_consensus.cc:697] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:59.330919 30074 ts_tablet_manager.cc:1434] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.112s	user 0.009s	sys 0.000s
I20260709 12:27:59.325469 30545 consensus_queue.cc:1048] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:27:59.335544 30545 consensus_queue.cc:1048] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:59.331583 30506 consensus_queue.cc:237] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:27:59.339625 29847 consensus_peers.cc:597] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:59.340579 30076 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:59.341496 30076 ts_tablet_manager.cc:1403] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.477s	user 0.029s	sys 0.011s
I20260709 12:27:59.345229 30076 raft_consensus.cc:359] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.347571 30076 raft_consensus.cc:385] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.348814 30076 raft_consensus.cc:740] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.333083 29701 catalog_manager.cc:5697] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:59.355659 30076 consensus_queue.cc:260] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.360800 30076 ts_tablet_manager.cc:1434] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:27:59.364998 30076 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6: Bootstrap starting.
I20260709 12:27:59.424238 30075 tablet_bootstrap.cc:654] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:59.424221 29703 catalog_manager.cc:5697] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:59.455233 30076 tablet_bootstrap.cc:654] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:59.462199 30077 tablet_bootstrap.cc:492] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:59.462879 29704 catalog_manager.cc:5697] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:59.462924 30077 ts_tablet_manager.cc:1403] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.597s	user 0.010s	sys 0.012s
I20260709 12:27:59.460634 30506 raft_consensus.cc:493] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.466429 30506 raft_consensus.cc:515] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.469081 30506 leader_election.cc:290] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:59.470017 30021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:59.471094 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
W20260709 12:27:59.471797 29847 leader_election.cc:343] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.471891 29805 raft_consensus.cc:2468] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:59.473230 30077 raft_consensus.cc:359] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:59.474462 30077 raft_consensus.cc:385] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.477247 29846 leader_election.cc:304] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:27:59.478144 30599 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.479003 30599 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.479648 30576 raft_consensus.cc:2804] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:59.480357 30576 raft_consensus.cc:493] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:59.480733 30576 raft_consensus.cc:3060] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.482746 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:27:59.487401 29995 leader_election.cc:343] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.492463 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:59.494510 29805 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:59.496088 29994 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:59.497175 30537 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:59.497613 30537 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:59.498169 30537 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.501248 30075 tablet_bootstrap.cc:492] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:59.502002 30075 ts_tablet_manager.cc:1403] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.198s	user 0.013s	sys 0.006s
I20260709 12:27:59.503440 30077 raft_consensus.cc:740] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.493499 30076 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6: No bootstrap required, opened a new log
I20260709 12:27:59.493191 30599 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:59.505240 30076 ts_tablet_manager.cc:1403] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent bootstrapping tablet: real 0.141s	user 0.018s	sys 0.000s
I20260709 12:27:59.506938 30077 consensus_queue.cc:260] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:59.518850 30077 ts_tablet_manager.cc:1434] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260709 12:27:59.520223 30077 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:59.525717 29953 raft_consensus.cc:3060] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.528352 30076 raft_consensus.cc:359] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.529071 30076 raft_consensus.cc:385] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.529323 30076 raft_consensus.cc:740] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.530107 30076 consensus_queue.cc:260] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.536705 30077 tablet_bootstrap.cc:654] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:59.542904 30076 ts_tablet_manager.cc:1434] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 12:27:59.553613 30506 raft_consensus.cc:493] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.554481 30506 raft_consensus.cc:515] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.555472 30075 raft_consensus.cc:359] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:59.556239 30075 raft_consensus.cc:385] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.556545 30075 raft_consensus.cc:740] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.559147 30506 leader_election.cc:290] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:59.560045 30537 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.561372 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:59.564288 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:59.601667 29805 raft_consensus.cc:2468] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 0.
I20260709 12:27:59.562671 30075 consensus_queue.cc:260] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:59.585420 29700 catalog_manager.cc:5697] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:59.604499 29846 leader_election.cc:304] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:59.609899 30075 ts_tablet_manager.cc:1434] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.105s	user 0.005s	sys 0.002s
I20260709 12:27:59.611256 30075 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:59.611580 30537 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:27:59.611912 30077 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:59.612566 30077 ts_tablet_manager.cc:1403] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.093s	user 0.021s	sys 0.026s
I20260709 12:27:59.616039 30077 raft_consensus.cc:359] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.616873 30077 raft_consensus.cc:385] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.617185 30077 raft_consensus.cc:740] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.617635 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:59.617956 30077 consensus_queue.cc:260] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:27:59.629897 29995 leader_election.cc:343] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.632762 30077 ts_tablet_manager.cc:1434] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.001s
I20260709 12:27:59.640424 30075 tablet_bootstrap.cc:654] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:59.649242 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:59.650141 29801 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.626943 30586 raft_consensus.cc:2804] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:59.651733 30586 raft_consensus.cc:493] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:59.652251 30586 raft_consensus.cc:3060] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:59.664942 29847 leader_election.cc:343] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.668962 30077 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:59.677659 30541 raft_consensus.cc:493] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.684058 30589 raft_consensus.cc:493] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.687330 30589 raft_consensus.cc:515] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:27:59.689870 30589 leader_election.cc:290] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:59.686808 30541 raft_consensus.cc:515] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.697403 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:27:59.701964 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5988ddbc0f84d47b9b94c1375502692" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:27:59.702838 29879 raft_consensus.cc:2468] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:27:59.693104 30077 tablet_bootstrap.cc:654] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:59.707170 29992 leader_election.cc:304] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:59.708930 30541 leader_election.cc:290] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:27:59.710661 30591 raft_consensus.cc:2804] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:59.711421 30591 raft_consensus.cc:493] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:59.711898 30591 raft_consensus.cc:3060] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.699257 29801 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:27:59.716666 29994 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:59.718421 30589 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:59.719141 30589 raft_consensus.cc:697] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:27:59.720233 30589 consensus_queue.cc:237] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:27:59.722378 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 82534.8us
I20260709 12:27:59.725952 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5988ddbc0f84d47b9b94c1375502692" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:59.736397 30545 raft_consensus.cc:493] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.737398 30545 raft_consensus.cc:515] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:27:59.740341 30545 leader_election.cc:290] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:27:59.762231 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:27:59.767412 30576 raft_consensus.cc:515] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.769711 30576 leader_election.cc:290] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:27:59.773285 29995 leader_election.cc:343] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:59.778483 29772 leader_election.cc:343] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.795866 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:59.803225 30021 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:27:59.812875 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:27:59.824219 29772 leader_election.cc:343] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.825003 29772 leader_election.cc:304] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:27:59.792151 30601 consensus_queue.cc:1048] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260709 12:27:59.823190 29801 raft_consensus.cc:3060] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.828867 30542 raft_consensus.cc:2749] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:59.847265 30077 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:27:59.847987 30077 ts_tablet_manager.cc:1403] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.179s	user 0.012s	sys 0.031s
W20260709 12:27:59.849113 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 52783.4us
I20260709 12:27:59.832736 30586 raft_consensus.cc:515] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.856990 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:27:59.855209 29918 leader_election.cc:304] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:27:59.859051 29953 raft_consensus.cc:3060] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.863051 30586 leader_election.cc:290] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:27:59.863914 30075 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:27:59.874398 30075 ts_tablet_manager.cc:1403] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.263s	user 0.020s	sys 0.011s
W20260709 12:27:59.872934 29847 leader_election.cc:343] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:59.888974 29805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:27:59.889701 29805 raft_consensus.cc:3060] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:59.890224 30077 raft_consensus.cc:359] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.891414 30077 raft_consensus.cc:385] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.891808 30077 raft_consensus.cc:740] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.897362 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:27:59.898953 30075 raft_consensus.cc:359] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.900161 30075 raft_consensus.cc:385] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:59.900514 30075 raft_consensus.cc:740] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.901139 30540 consensus_queue.cc:1048] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:59.892939 30077 consensus_queue.cc:260] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:27:59.901541 30075 consensus_queue.cc:260] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:27:59.916270 30077 ts_tablet_manager.cc:1434] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.003s
I20260709 12:27:59.926172 30077 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:27:59.927671 30075 ts_tablet_manager.cc:1434] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.002s
I20260709 12:27:59.928645 29805 raft_consensus.cc:2468] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:27:59.930365 29846 leader_election.cc:304] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:27:59.935526 30540 raft_consensus.cc:2804] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:59.936151 30540 raft_consensus.cc:697] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:27:59.936903 30591 raft_consensus.cc:515] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:27:59.958341 29847 leader_election.cc:343] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:59.964702 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 105872us
I20260709 12:27:59.974964 30075 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:27:59.937184 30540 consensus_queue.cc:237] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:00.014077 29921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 67158.8us
I20260709 12:28:00.042055 30026 raft_consensus.cc:3060] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:00.053479 30075 tablet_bootstrap.cc:654] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:00.073783 30077 tablet_bootstrap.cc:654] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:00.081388 29703 catalog_manager.cc:5697] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:00.071616 30580 raft_consensus.cc:493] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.099310 30471 raft_consensus.cc:2749] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:00.067857 29801 raft_consensus.cc:2468] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:28:00.122502 30580 raft_consensus.cc:515] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:00.115185 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5988ddbc0f84d47b9b94c1375502692" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:00.065559 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5988ddbc0f84d47b9b94c1375502692" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:00.124385 29879 raft_consensus.cc:3060] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:00.125114 30580 leader_election.cc:290] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:00.062582 30591 leader_election.cc:290] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:28:00.134539 29752 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:28:00.145685 29995 leader_election.cc:343] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:00.192469 30075 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:28:00.193524 30075 ts_tablet_manager.cc:1403] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.230s	user 0.010s	sys 0.009s
I20260709 12:28:00.156580 30387 consensus_queue.cc:1048] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260709 12:28:00.227380 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:28:00.273119 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 47186us
I20260709 12:28:00.289050 29802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ad7f32688184e219c1d203375c23682" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:28:00.276901 29879 raft_consensus.cc:2468] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:00.304322 30077 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:28:00.305039 30077 ts_tablet_manager.cc:1403] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.386s	user 0.013s	sys 0.018s
I20260709 12:28:00.324712 29846 leader_election.cc:304] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:00.326030 30582 raft_consensus.cc:2804] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:00.339418 30075 raft_consensus.cc:359] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.340246 30075 raft_consensus.cc:385] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:00.340528 30075 raft_consensus.cc:740] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.341167 30075 consensus_queue.cc:260] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.337990 30582 raft_consensus.cc:697] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:28:00.343457 30075 ts_tablet_manager.cc:1434] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.148s	user 0.008s	sys 0.000s
I20260709 12:28:00.347095 30582 consensus_queue.cc:237] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:00.352123 30075 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86: Bootstrap starting.
I20260709 12:28:00.354626 30077 raft_consensus.cc:359] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.355774 30077 raft_consensus.cc:385] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:00.356144 30077 raft_consensus.cc:740] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.357100 30077 consensus_queue.cc:260] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.363453 30075 tablet_bootstrap.cc:654] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:00.364001 30077 ts_tablet_manager.cc:1434] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.059s	user 0.011s	sys 0.000s
I20260709 12:28:00.365739 30077 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:28:00.372529 30075 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86: No bootstrap required, opened a new log
I20260709 12:28:00.373459 30075 ts_tablet_manager.cc:1403] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.006s
I20260709 12:28:00.448585 29953 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:00.436792 29920 leader_election.cc:304] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:00.491818 30075 raft_consensus.cc:359] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:00.493466 30075 raft_consensus.cc:385] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:00.493623 30471 raft_consensus.cc:2749] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:28:00.506191 29920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 70043.3us
I20260709 12:28:00.510870 29992 leader_election.cc:304] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:00.520823 30077 tablet_bootstrap.cc:654] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:00.528322 30536 consensus_queue.cc:1048] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:00.538688 30075 raft_consensus.cc:740] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.579061 29946 raft_consensus.cc:3060] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:00.557251 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 367760us
I20260709 12:28:00.559931 30075 consensus_queue.cc:260] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:28:00.614831 29992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 104510us
I20260709 12:28:00.615792 30536 raft_consensus.cc:2804] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:00.631860 30387 consensus_queue.cc:1048] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:00.638700 30075 ts_tablet_manager.cc:1434] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86: Time spent starting tablet: real 0.265s	user 0.008s	sys 0.008s
I20260709 12:28:00.640946 30536 raft_consensus.cc:697] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:00.648092 30536 consensus_queue.cc:237] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:28:00.664319 29772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 49050.7us
I20260709 12:28:00.683005 30536 raft_consensus.cc:493] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.683913 30536 raft_consensus.cc:515] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.688367 30536 leader_election.cc:290] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:00.698364 30536 raft_consensus.cc:493] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.701438 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:28:00.704267 29995 leader_election.cc:343] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:00.709576 29847 consensus_peers.cc:597] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:00.701867 30536 raft_consensus.cc:515] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:00.734112 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:28:00.735294 29804 raft_consensus.cc:2468] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 0.
I20260709 12:28:00.745604 30599 consensus_queue.cc:1048] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:28:00.747689 29700 catalog_manager.cc:5697] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:00.755616 30536 leader_election.cc:290] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:00.771770 30541 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.772619 30541 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.787817 29804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:28:00.789090 29804 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:00.799728 30077 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:28:00.777771 30541 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:00.807933 30077 ts_tablet_manager.cc:1403] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.443s	user 0.021s	sys 0.023s
I20260709 12:28:00.810675 29920 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:00.813403 30456 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:00.813910 30456 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.814342 30456 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:00.809178 30601 consensus_queue.cc:1048] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:28:00.820562 29994 leader_election.cc:304] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:00.821009 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:00.822039 30589 raft_consensus.cc:2804] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:00.822518 30589 raft_consensus.cc:493] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.822908 30589 raft_consensus.cc:3060] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:00.831133 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:00.862426 29798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:28:00.875706 29798 raft_consensus.cc:2393] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 1: Already voted for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in this term.
I20260709 12:28:00.878943 30456 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.883671 29994 leader_election.cc:304] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:00.887550 29952 raft_consensus.cc:1240] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Rejecting Update request from peer 1e2ceabfe50c47b49529b663fa133b86 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:28:00.905211 30374 raft_consensus.cc:3060] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:00.910468 30506 consensus_queue.cc:1048] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:00.893724 29700 catalog_manager.cc:5697] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:00.914534 30077 raft_consensus.cc:359] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:00.915647 30077 raft_consensus.cc:385] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:00.916028 30077 raft_consensus.cc:740] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.924810 30077 consensus_queue.cc:260] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:00.927042 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:28:00.938838 30456 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:00.940091 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:00.943060 29921 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:00.944264 30456 raft_consensus.cc:2749] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:28:00.951112 30599 consensus_queue.cc:1059] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:00.952427 30599 raft_consensus.cc:3055] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:00.952852 30599 raft_consensus.cc:740] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:00.954720 30599 consensus_queue.cc:260] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:00.956441 30599 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:00.962863 30077 ts_tablet_manager.cc:1434] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.116s	user 0.008s	sys 0.000s
I20260709 12:28:00.973037 30077 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:28:00.972960 29704 catalog_manager.cc:5697] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:01.024185 30589 raft_consensus.cc:515] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:01.026348 30374 raft_consensus.cc:2749] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:01.037379 29800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:28:01.038296 29800 raft_consensus.cc:3060] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:01.040896 30077 tablet_bootstrap.cc:654] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:01.041589 30629 consensus_queue.cc:1048] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:01.050191 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:01.047453 30589 leader_election.cc:290] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:01.071069 29995 leader_election.cc:343] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:01.097136 29995 consensus_peers.cc:597] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:01.098212 30374 consensus_queue.cc:1048] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:01.127154 29952 raft_consensus.cc:3060] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:01.168448 29800 raft_consensus.cc:2468] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:01.243687 29994 leader_election.cc:304] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:01.244055 30077 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:28:01.244685 30077 ts_tablet_manager.cc:1403] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.280s	user 0.034s	sys 0.025s
I20260709 12:28:01.249929 30599 raft_consensus.cc:2804] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:01.250810 30599 raft_consensus.cc:697] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:01.252303 30599 consensus_queue.cc:237] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:01.286728 30077 raft_consensus.cc:359] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:01.289498 30077 raft_consensus.cc:385] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:01.293166 30077 raft_consensus.cc:740] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:28:01.294811 30077 consensus_queue.cc:260] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:01.316322 30541 raft_consensus.cc:493] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.316992 30541 raft_consensus.cc:515] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:01.319933 30541 leader_election.cc:290] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:01.325071 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:01.331123 29801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f1befeab89b48729c8ba035e0ef71a8" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:28:01.317015 30077 ts_tablet_manager.cc:1434] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.072s	user 0.010s	sys 0.002s
I20260709 12:28:01.337146 29801 raft_consensus.cc:2393] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 1: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:28:01.346530 29920 leader_election.cc:304] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:01.334637 29701 catalog_manager.cc:5697] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 1 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:01.358644 30077 tablet_bootstrap.cc:492] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac: Bootstrap starting.
I20260709 12:28:01.362303 30541 raft_consensus.cc:3060] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:01.375507 30077 tablet_bootstrap.cc:654] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:01.384826 30541 raft_consensus.cc:2749] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:01.408970 30077 tablet_bootstrap.cc:492] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac: No bootstrap required, opened a new log
I20260709 12:28:01.411005 30077 ts_tablet_manager.cc:1403] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent bootstrapping tablet: real 0.053s	user 0.015s	sys 0.002s
I20260709 12:28:01.414870 30077 raft_consensus.cc:359] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:01.415720 30077 raft_consensus.cc:385] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:01.416178 30077 raft_consensus.cc:740] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Initialized, Role: FOLLOWER
I20260709 12:28:01.418133 30077 consensus_queue.cc:260] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:01.421747 30077 ts_tablet_manager.cc:1434] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.001s
I20260709 12:28:01.473227 29952 raft_consensus.cc:3060] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:01.689656 30599 consensus_queue.cc:1048] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:01.706143 30374 consensus_queue.cc:1048] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:02.293974 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
W20260709 12:28:02.338568 29846 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20260709 12:28:02.346983 29846 consensus_peers.cc:597] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.357029 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:02.358467 16792 tablet_replica.cc:333] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.359753 16792 raft_consensus.cc:2243] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.360442 16792 pending_rounds.cc:70] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Trying to abort 1 pending ops.
I20260709 12:28:02.360811 16792 pending_rounds.cc:77] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305625926632751104 op_type: NO_OP noop_request { }
I20260709 12:28:02.361315 16792 raft_consensus.cc:2889] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:02.361759 16792 raft_consensus.cc:2272] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.367892 16792 tablet_replica.cc:333] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.369426 16792 raft_consensus.cc:2243] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.370373 16792 raft_consensus.cc:2272] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:02.376106 29994 consensus_peers.cc:597] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.378240 16792 tablet_replica.cc:333] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
W20260709 12:28:02.379894 29994 consensus_peers.cc:597] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.379524 16792 raft_consensus.cc:2243] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.382057 16792 raft_consensus.cc:2272] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.386052 16792 tablet_replica.cc:333] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.386906 16792 raft_consensus.cc:2243] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.387619 16792 raft_consensus.cc:2272] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.391654 16792 tablet_replica.cc:333] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.393033 16792 raft_consensus.cc:2243] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.393857 16792 raft_consensus.cc:2272] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.397027 16792 tablet_replica.cc:333] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.397667 16792 raft_consensus.cc:2243] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.398768 16792 raft_consensus.cc:2272] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.401213 16792 tablet_replica.cc:333] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.402005 16792 raft_consensus.cc:2243] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.403039 16792 raft_consensus.cc:2272] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.405685 16792 tablet_replica.cc:333] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.406754 16792 raft_consensus.cc:2243] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.407531 16792 raft_consensus.cc:2272] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.411486 16792 tablet_replica.cc:333] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.412664 16792 raft_consensus.cc:2243] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.413944 16792 raft_consensus.cc:2272] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.416812 16792 tablet_replica.cc:333] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.417551 16792 raft_consensus.cc:2243] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.418746 16792 raft_consensus.cc:2272] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.421589 16792 tablet_replica.cc:333] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.422501 16792 raft_consensus.cc:2243] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.423074 16792 raft_consensus.cc:2272] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.426999 16792 tablet_replica.cc:333] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.428180 16792 raft_consensus.cc:2243] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.428968 16792 raft_consensus.cc:2272] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:02.429280 29994 consensus_peers.cc:597] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.432762 16792 tablet_replica.cc:333] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.434149 16792 raft_consensus.cc:2243] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.434888 16792 raft_consensus.cc:2272] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.438813 16792 tablet_replica.cc:333] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.439914 16792 raft_consensus.cc:2243] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.440991 16792 raft_consensus.cc:2272] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.444015 16792 tablet_replica.cc:333] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.444937 16792 raft_consensus.cc:2243] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.445649 16792 raft_consensus.cc:2272] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.449255 16792 tablet_replica.cc:333] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.450318 16792 raft_consensus.cc:2243] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.450830 16792 raft_consensus.cc:2272] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.453446 16792 tablet_replica.cc:333] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.454268 16792 raft_consensus.cc:2243] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.455411 16792 raft_consensus.cc:2272] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.458699 16792 tablet_replica.cc:333] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.459686 16792 raft_consensus.cc:2243] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.460265 16792 raft_consensus.cc:2272] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.462697 16792 tablet_replica.cc:333] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.463307 16792 raft_consensus.cc:2243] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.463847 16792 raft_consensus.cc:2272] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.467110 16792 tablet_replica.cc:333] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.468159 16792 raft_consensus.cc:2243] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.468811 16792 raft_consensus.cc:2272] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.471530 16792 tablet_replica.cc:333] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.472280 16792 raft_consensus.cc:2243] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.473220 16792 raft_consensus.cc:2272] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.475857 16792 tablet_replica.cc:333] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.476501 16792 raft_consensus.cc:2243] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.477444 16792 raft_consensus.cc:2272] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.479895 16792 tablet_replica.cc:333] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.480654 16792 raft_consensus.cc:2243] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.481657 16792 raft_consensus.cc:2272] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.484366 16792 tablet_replica.cc:333] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.485317 16792 raft_consensus.cc:2243] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.485956 16792 raft_consensus.cc:2272] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.489675 16792 tablet_replica.cc:333] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
W20260709 12:28:02.490607 29994 consensus_peers.cc:597] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.490782 16792 raft_consensus.cc:2243] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.492096 16792 raft_consensus.cc:2272] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.495904 16792 tablet_replica.cc:333] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.496924 16792 raft_consensus.cc:2243] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.498340 16792 raft_consensus.cc:2272] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.501268 16792 tablet_replica.cc:333] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.502050 16792 raft_consensus.cc:2243] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.503012 16792 raft_consensus.cc:2272] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.505927 16792 tablet_replica.cc:333] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.506788 16792 raft_consensus.cc:2243] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.507766 16792 raft_consensus.cc:2272] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.510712 16792 tablet_replica.cc:333] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.511427 16792 raft_consensus.cc:2243] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.512728 16792 raft_consensus.cc:2272] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.516281 16792 tablet_replica.cc:333] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.517331 16792 raft_consensus.cc:2243] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.518074 16792 raft_consensus.cc:2272] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.521548 16792 tablet_replica.cc:333] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
W20260709 12:28:02.521833 29846 consensus_peers.cc:597] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.522500 16792 raft_consensus.cc:2243] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:02.522998 29846 consensus_peers.cc:597] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.523384 16792 raft_consensus.cc:2272] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.527204 16792 tablet_replica.cc:333] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.528107 16792 raft_consensus.cc:2243] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.528829 16792 raft_consensus.cc:2272] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.532596 16792 tablet_replica.cc:333] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.533598 16792 raft_consensus.cc:2243] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.534354 16792 raft_consensus.cc:2272] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.537803 16792 tablet_replica.cc:333] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.538803 16792 raft_consensus.cc:2243] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.539532 16792 raft_consensus.cc:2272] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.543370 16792 tablet_replica.cc:333] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.544380 16792 raft_consensus.cc:2243] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.545073 16792 raft_consensus.cc:2272] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.548420 16792 tablet_replica.cc:333] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.549193 16792 raft_consensus.cc:2243] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:28:02.549427 29920 consensus_peers.cc:597] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.550447 16792 raft_consensus.cc:2272] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.553845 16792 tablet_replica.cc:333] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
W20260709 12:28:02.554519 29846 consensus_peers.cc:597] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.554752 16792 raft_consensus.cc:2243] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.555642 16792 raft_consensus.cc:2272] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.559288 16792 tablet_replica.cc:333] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.560206 16792 raft_consensus.cc:2243] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.560879 16792 raft_consensus.cc:2272] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.564311 16792 tablet_replica.cc:333] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.565248 16792 raft_consensus.cc:2243] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.565974 16792 raft_consensus.cc:2272] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.569754 16792 tablet_replica.cc:333] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.570690 16792 raft_consensus.cc:2243] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.572044 16792 raft_consensus.cc:2272] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.575704 16792 tablet_replica.cc:333] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.576539 16792 raft_consensus.cc:2243] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.577906 16792 raft_consensus.cc:2272] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.581247 16792 tablet_replica.cc:333] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.582211 16792 raft_consensus.cc:2243] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.583738 16792 raft_consensus.cc:2272] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:02.586719 29920 consensus_peers.cc:597] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.587127 16792 tablet_replica.cc:333] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.588044 16792 raft_consensus.cc:2243] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.588703 16792 raft_consensus.cc:2272] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.592034 16792 tablet_replica.cc:333] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.592813 16792 raft_consensus.cc:2243] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.593379 16792 raft_consensus.cc:2272] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.596652 16792 tablet_replica.cc:333] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: stopping tablet replica
I20260709 12:28:02.597371 16792 raft_consensus.cc:2243] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.598021 16792 raft_consensus.cc:2272] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:02.604645 29846 consensus_peers.cc:597] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.611860 29846 consensus_peers.cc:597] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.617079 29994 consensus_peers.cc:597] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.617991 29994 consensus_peers.cc:597] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.628851 29994 consensus_peers.cc:597] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.657967 29846 consensus_peers.cc:597] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.661542 29846 consensus_peers.cc:597] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.681725 29846 connection.cc:573] client connection to 127.16.102.1:42003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260709 12:28:02.690176 29846 consensus_peers.cc:597] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.704839 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
W20260709 12:28:02.718487 29846 consensus_peers.cc:597] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.749615 30599 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.750480 30599 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:02.751911 29994 consensus_peers.cc:597] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.754479 30599 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:02.755236 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:02.756121 29952 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:02.757459 29995 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:02.758528 30599 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:02.759022 30599 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:02.760160 30599 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:02.761241 29994 leader_election.cc:336] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:02.778199 29846 consensus_peers.cc:597] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.778990 30599 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:02.786818 29994 consensus_peers.cc:597] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.787742 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:02.788611 29952 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:02.796345 29994 leader_election.cc:336] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:02.797319 30599 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:02.797958 29952 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 3.
I20260709 12:28:02.800042 29995 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:02.801527 30599 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:02.802147 30599 raft_consensus.cc:697] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:02.803135 30599 consensus_queue.cc:237] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:02.811201 29846 consensus_peers.cc:597] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.814143 29920 consensus_peers.cc:597] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.822270 29701 catalog_manager.cc:5697] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:02.831947 29846 consensus_peers.cc:597] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.852106 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:02.852401 29994 consensus_peers.cc:597] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.856493 29846 consensus_peers.cc:597] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:02.868420 30687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:02.871194 30688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:02.881598 30690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:02.884536 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:02.891706 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:02.892096 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:02.892359 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600082892334 us; error 0 us; skew 500 ppm
I20260709 12:28:02.893121 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:02.928544 16792 webserver.cc:533] Webserver started at http://127.16.102.1:35873/ using document root <none> and password file <none>
I20260709 12:28:02.929533 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:02.929854 16792 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:28:02.936508 16792 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:02.981465 30696 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:02.983603 16792 fs_manager.cc:730] Time spent opening block manager: real 0.035s	user 0.017s	sys 0.016s
I20260709 12:28:02.984112 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "1ae5481d76bf4a90a0ee824a14223421"
format_stamp: "Formatted at 2026-07-09 12:27:45 on dist-test-slave-7xt3"
I20260709 12:28:02.984668 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:03.011412 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:03.013461 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:03.015775 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:03.021368 30703 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:28:03.205543 29946 raft_consensus.cc:1275] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:03.207937 30685 consensus_queue.cc:1048] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:03.268456 29994 consensus_peers.cc:597] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.321048 30694 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.321731 30694 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.325225 30694 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.325613 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:28:03.329407 29920 leader_election.cc:336] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.329954 29920 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.330747 30694 raft_consensus.cc:2749] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.437975 30694 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.438660 30694 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.441329 30694 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.442098 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:28:03.448241 29920 leader_election.cc:336] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.448818 29920 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.449651 30694 raft_consensus.cc:2749] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.452865 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:28:03.453372 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.433s	user 0.009s	sys 0.000s
I20260709 12:28:03.453748 16792 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:28:03.470306 30703 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:03.482534 30374 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.483237 30374 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.485524 30374 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.487674 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.489091 30703 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:03.490443 30703 tablet_bootstrap.cc:492] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:03.491200 30703 ts_tablet_manager.cc:1403] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.003s
I20260709 12:28:03.494240 30703 raft_consensus.cc:359] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:03.494601 29994 leader_election.cc:336] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.495296 29994 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:03.495980 30703 raft_consensus.cc:740] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:03.496110 30685 raft_consensus.cc:2749] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.497056 30703 consensus_queue.cc:260] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.499776 30703 ts_tablet_manager.cc:1434] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:28:03.500777 30703 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:03.502249 30676 raft_consensus.cc:493] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.502866 30676 raft_consensus.cc:515] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.509903 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
W20260709 12:28:03.511112 29846 leader_election.cc:336] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.512096 30676 leader_election.cc:290] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.512449 29847 leader_election.cc:304] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.513422 30676 raft_consensus.cc:2749] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.535306 30703 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:03.536517 30703 tablet_bootstrap.cc:492] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:03.537351 30703 ts_tablet_manager.cc:1403] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.037s	user 0.021s	sys 0.008s
I20260709 12:28:03.540980 30703 raft_consensus.cc:359] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.541844 30703 raft_consensus.cc:740] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:03.542718 30703 consensus_queue.cc:260] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.545260 30703 ts_tablet_manager.cc:1434] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:03.546213 30703 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:03.560756 30703 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:03.561643 30703 tablet_bootstrap.cc:492] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:03.562322 30703 ts_tablet_manager.cc:1403] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.016s	user 0.015s	sys 0.000s
I20260709 12:28:03.564544 30703 raft_consensus.cc:359] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.565196 30703 raft_consensus.cc:740] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:03.566038 30694 raft_consensus.cc:493] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.565956 30703 consensus_queue.cc:260] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.566682 30694 raft_consensus.cc:515] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.568323 30703 ts_tablet_manager.cc:1434] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:28:03.569031 30694 leader_election.cc:290] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.569409 30703 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:03.573733 30676 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.574473 30676 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.575248 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.576992 30676 leader_election.cc:290] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:28:03.580188 29920 leader_election.cc:336] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.580844 29920 leader_election.cc:304] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.581797 30694 raft_consensus.cc:2749] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.582813 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:28:03.590327 29846 leader_election.cc:336] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.590677 30685 raft_consensus.cc:493] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.600987 30685 raft_consensus.cc:515] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.593683 30374 raft_consensus.cc:493] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.602717 30374 raft_consensus.cc:515] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.605530 30374 leader_election.cc:290] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:03.610028 29994 leader_election.cc:336] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:03.611653 29994 leader_election.cc:336] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.616729 30694 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.635329 30582 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.620054 29846 leader_election.cc:304] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:03.636144 30582 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:03.637255 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 46985.5us
I20260709 12:28:03.622018 30685 leader_election.cc:290] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.627430 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.638932 30582 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:03.639382 30676 raft_consensus.cc:2749] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.633538 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.640769 29952 raft_consensus.cc:2468] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:03.642899 30694 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.646502 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.646526 29995 leader_election.cc:304] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:03.656911 30374 raft_consensus.cc:2749] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.658540 29995 leader_election.cc:304] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:03.669414 30685 raft_consensus.cc:2804] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:28:03.671073 29846 leader_election.cc:336] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.671729 30685 raft_consensus.cc:493] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.672417 30685 raft_consensus.cc:3060] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.674803 30582 raft_consensus.cc:493] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.675601 30582 raft_consensus.cc:515] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.678261 30582 leader_election.cc:290] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:03.671764 29846 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:03.662027 30694 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.689533 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.690243 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.692857 30582 raft_consensus.cc:2749] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:03.702471 29920 leader_election.cc:336] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.703809 30681 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.705058 30681 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.708848 30681 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.710028 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:03.715755 29920 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.722504 30541 raft_consensus.cc:2749] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.723547 30703 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:28:03.718868 29994 leader_election.cc:336] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.724786 30703 tablet_bootstrap.cc:492] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:03.725600 30703 ts_tablet_manager.cc:1403] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.156s	user 0.049s	sys 0.084s
I20260709 12:28:03.725797 30456 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.726491 30456 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:03.726797 29846 leader_election.cc:336] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.727346 29846 leader_election.cc:304] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:03.728734 30456 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.728904 30703 raft_consensus.cc:359] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.729621 30703 raft_consensus.cc:740] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:03.730393 30703 consensus_queue.cc:260] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.732538 30703 ts_tablet_manager.cc:1434] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:28:03.733153 30582 raft_consensus.cc:2749] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.733384 30703 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:03.744978 30681 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.745882 30681 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.751535 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.752854 29946 raft_consensus.cc:2468] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:03.755122 30374 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.755728 30374 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.758239 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.749428 30681 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.764031 30582 raft_consensus.cc:493] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.759080 29995 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:03.773283 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.763202 30456 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.764688 30582 raft_consensus.cc:515] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.775820 30456 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.759019 30027 raft_consensus.cc:2468] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:03.757722 30374 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.780965 30374 raft_consensus.cc:2804] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.781741 30374 raft_consensus.cc:493] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.782199 30374 raft_consensus.cc:3060] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.786890 30676 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.787621 30676 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.794839 30582 leader_election.cc:290] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.797056 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.802163 30582 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.802886 30582 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.805166 30582 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.780965 29946 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:03.810850 30676 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:03.819276 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.820909 30694 raft_consensus.cc:493] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.822021 30694 raft_consensus.cc:515] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.825197 30694 leader_election.cc:290] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.827889 30541 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
W20260709 12:28:03.829077 29846 leader_election.cc:336] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.828912 30541 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:03.829766 29846 leader_election.cc:336] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.830236 29846 leader_election.cc:304] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.764978 29994 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:03.831936 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.831967 30541 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.835446 30599 raft_consensus.cc:2749] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.836509 29921 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:03.836676 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.837728 30694 raft_consensus.cc:2804] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.837641 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.843456 30694 raft_consensus.cc:493] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.836890 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:03.844024 30694 raft_consensus.cc:3060] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.843686 30685 raft_consensus.cc:515] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.844426 30026 raft_consensus.cc:2468] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:03.837520 30027 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
W20260709 12:28:03.845507 29920 leader_election.cc:336] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.780344 30456 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:03.846560 29920 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.846570 30456 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.847779 30456 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:03.847792 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 128964us
I20260709 12:28:03.842705 30541 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.850160 30703 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:03.849977 30541 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.850896 30456 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.851348 30703 tablet_bootstrap.cc:492] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:03.852257 30703 ts_tablet_manager.cc:1403] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.119s	user 0.040s	sys 0.047s
I20260709 12:28:03.848385 29847 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:03.853201 30541 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.841619 30374 raft_consensus.cc:515] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.856041 30703 raft_consensus.cc:359] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.856907 30703 raft_consensus.cc:740] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:03.860625 30676 raft_consensus.cc:2749] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.863023 30676 raft_consensus.cc:2749] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:03.864817 30374 leader_election.cc:290] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:03.865353 29846 leader_election.cc:336] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.867071 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:03.867465 30714 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.867314 30685 leader_election.cc:290] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.868263 29846 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:03.868103 30714 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.857761 30703 consensus_queue.cc:260] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.870134 30599 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.868113 29874 raft_consensus.cc:2468] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:03.870872 30599 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.872751 30703 ts_tablet_manager.cc:1434] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 12:28:03.872902 30714 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.873178 30599 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.873839 30685 raft_consensus.cc:493] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.874298 30703 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:03.874516 30685 raft_consensus.cc:515] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.876827 30685 leader_election.cc:290] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.883533 29918 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:03.885962 30456 raft_consensus.cc:2804] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.886421 30456 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.886571 29995 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:03.889917 30685 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.891500 16792 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 12:28:03.897935 30717 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.927156 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.473s	user 0.372s	sys 0.004s
W20260709 12:28:03.927676 29994 leader_election.cc:336] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:03.920708 29846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 55374.4us
I20260709 12:28:03.901273 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:03.936691 29880 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
W20260709 12:28:03.936900 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 50432.2us
I20260709 12:28:03.890148 30676 raft_consensus.cc:2749] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:03.887149 30681 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.927397 30717 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:03.937839 30716 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.938584 30681 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.911325 30714 raft_consensus.cc:493] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.940577 30717 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:03.940840 29921 leader_election.cc:304] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:03.939078 30716 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.943377 30716 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.944814 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97428fa3f86d463fa4d309cffb192127" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:03.945410 30541 raft_consensus.cc:2804] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.945907 30541 raft_consensus.cc:493] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.946266 30541 raft_consensus.cc:3060] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.947347 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:03.948132 29874 raft_consensus.cc:2468] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:03.948776 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d30c5d36c6e4ad2a38bb757a7773f78" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:03.949510 29950 raft_consensus.cc:3060] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.949985 29946 raft_consensus.cc:3060] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.951203 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:03.952003 29952 raft_consensus.cc:2468] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:03.886780 30456 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.952255 30694 raft_consensus.cc:515] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.956303 29921 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:03.941208 30714 raft_consensus.cc:515] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:03.951386 29920 leader_election.cc:336] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:03.959201 29920 leader_election.cc:336] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.959663 29920 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
W20260709 12:28:03.957645 29994 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:03.959910 30714 leader_election.cc:290] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.940976 30681 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.931187 30685 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.962261 30713 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.962788 30713 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.963255 30713 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.968796 30714 raft_consensus.cc:2749] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:03.972363 29992 leader_election.cc:304] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:03.973752 30681 raft_consensus.cc:2804] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:03.974336 30681 raft_consensus.cc:493] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.974813 30681 raft_consensus.cc:3060] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:03.980118 30694 leader_election.cc:290] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:03.981880 30685 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.983726 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:03.984637 29874 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:03.985092 30703 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:03.989734 30716 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:03.994194 30717 raft_consensus.cc:493] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:03.986416 30703 tablet_bootstrap.cc:492] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:03.994946 30717 raft_consensus.cc:515] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:03.995957 30703 ts_tablet_manager.cc:1403] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.122s	user 0.025s	sys 0.037s
I20260709 12:28:03.997575 30717 leader_election.cc:290] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:03.999413 30703 raft_consensus.cc:359] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.001130 29992 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
W20260709 12:28:04.002658 29994 leader_election.cc:336] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.002765 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:28:04.003932 29994 leader_election.cc:336] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.004043 29880 raft_consensus.cc:2468] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:04.004522 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:04.005323 30027 raft_consensus.cc:2468] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:04.006188 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e047383bb70448908508e00810ebff84" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:04.006884 30022 raft_consensus.cc:3060] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.007246 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:04.007956 29952 raft_consensus.cc:2468] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:04.000217 30703 raft_consensus.cc:740] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:04.011179 30599 raft_consensus.cc:2804] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:04.011756 30599 raft_consensus.cc:493] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
W20260709 12:28:04.004508 29994 leader_election.cc:336] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.012233 30599 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:04.012670 29994 leader_election.cc:336] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.013024 30713 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.015632 30713 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.016886 30703 consensus_queue.cc:260] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.018236 30022 raft_consensus.cc:2468] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:04.018602 30681 raft_consensus.cc:515] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.022399 30599 raft_consensus.cc:515] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:28:04.019218 29920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 60052.8us
I20260709 12:28:04.021006 30703 ts_tablet_manager.cc:1434] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.001s
I20260709 12:28:04.024302 29950 raft_consensus.cc:2468] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:04.027722 29992 leader_election.cc:304] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:04.028780 30685 raft_consensus.cc:2804] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.025007 30681 leader_election.cc:290] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.029211 30685 raft_consensus.cc:493] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.033980 30717 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.034180 29995 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.034682 30717 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.036937 30681 raft_consensus.cc:2804] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.026464 30703 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:04.030799 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "444cae907f1d4bdab292e8a3f59f7f11" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.039779 29880 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:04.041173 30676 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.041913 30676 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.043208 29918 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:04.044610 30676 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.045035 30716 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.046216 30721 raft_consensus.cc:2804] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:04.046813 30721 raft_consensus.cc:493] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
W20260709 12:28:04.047533 29994 leader_election.cc:336] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.048049 30721 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:04.048209 29994 leader_election.cc:336] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.048965 29994 leader_election.cc:336] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.025113 30599 leader_election.cc:290] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.031852 29921 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:04.032720 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:04.037052 30717 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.033974 30685 raft_consensus.cc:3060] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.052738 29950 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:04.056888 29877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:04.057727 29877 raft_consensus.cc:2468] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:04.037997 30541 raft_consensus.cc:515] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.029520 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "268205acac824554b94b834e990e1ed4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.063923 29874 raft_consensus.cc:3060] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.064658 29946 raft_consensus.cc:2468] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
W20260709 12:28:04.064702 29920 leader_election.cc:336] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.064790 30681 raft_consensus.cc:493] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.068867 30681 raft_consensus.cc:3060] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:04.079403 29920 leader_election.cc:336] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.064828 30694 raft_consensus.cc:2804] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.080420 30694 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
W20260709 12:28:04.080632 29994 leader_election.cc:336] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.080902 30694 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.067987 30456 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.071818 29918 leader_election.cc:304] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:04.072376 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:04.082646 30024 raft_consensus.cc:2468] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 1.
I20260709 12:28:04.071491 30541 leader_election.cc:290] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:04.083560 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 48176.4us
W20260709 12:28:04.086386 29920 leader_election.cc:336] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.074148 29846 leader_election.cc:336] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.083680 30713 raft_consensus.cc:2804] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.070276 29880 raft_consensus.cc:2468] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 3.
I20260709 12:28:04.090373 30713 raft_consensus.cc:493] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.090888 30713 raft_consensus.cc:3060] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.090440 30685 raft_consensus.cc:515] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.091488 29992 leader_election.cc:304] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:04.092944 30685 leader_election.cc:290] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.093647 30685 raft_consensus.cc:2804] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:04.094973 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abceb9531803436f8fd8d02595d73cf4" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.094995 30685 raft_consensus.cc:697] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.095781 29880 raft_consensus.cc:3060] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.096443 30685 consensus_queue.cc:237] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.099345 29995 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.100530 30717 raft_consensus.cc:2804] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.101581 30717 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.078274 29874 raft_consensus.cc:2468] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:04.102267 30717 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:04.081708 29994 leader_election.cc:336] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.104980 29994 leader_election.cc:336] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.105785 30713 raft_consensus.cc:515] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.110735 29992 leader_election.cc:304] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.112685 30721 raft_consensus.cc:515] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:04.113966 29994 leader_election.cc:336] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.151716 29994 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.114770 30694 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.153581 30721 leader_election.cc:290] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.154476 30713 leader_election.cc:290] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.154951 30694 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.116182 30599 raft_consensus.cc:2804] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.117625 29995 leader_election.cc:304] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.157477 30599 raft_consensus.cc:697] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.156838 30681 raft_consensus.cc:515] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.118660 30456 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.127125 30717 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.158948 30716 raft_consensus.cc:2804] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.158597 30599 consensus_queue.cc:237] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.159531 30716 raft_consensus.cc:697] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.160642 30717 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:04.135692 30703 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:04.160673 30716 consensus_queue.cc:237] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.162284 30703 tablet_bootstrap.cc:492] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:04.163205 30703 ts_tablet_manager.cc:1403] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.137s	user 0.025s	sys 0.004s
W20260709 12:28:04.163642 29920 leader_election.cc:336] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.164608 29920 leader_election.cc:336] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.165174 29920 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.165287 29994 leader_election.cc:336] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.137706 29880 raft_consensus.cc:2468] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
W20260709 12:28:04.143925 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 54123.8us
I20260709 12:28:04.167054 29847 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.167498 29992 leader_election.cc:304] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.147579 30720 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
W20260709 12:28:04.158308 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 48064.6us
I20260709 12:28:04.169446 30586 raft_consensus.cc:2804] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.170116 30586 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.170387 29995 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.170112 30703 raft_consensus.cc:359] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.169474 30720 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.172248 30374 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.172749 30374 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.173314 30374 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.174403 30720 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.174612 29995 leader_election.cc:304] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.176158 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.176936 29874 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.170681 30586 raft_consensus.cc:3060] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.178421 30722 raft_consensus.cc:2804] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.179210 30722 raft_consensus.cc:697] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.180227 30722 consensus_queue.cc:237] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.176164 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c4c5d911ee4bcbbfb014114223cc07" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.182686 29880 raft_consensus.cc:3060] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.168991 30717 raft_consensus.cc:2804] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.187284 30717 raft_consensus.cc:697] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.187767 30703 raft_consensus.cc:740] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:04.188449 30717 consensus_queue.cc:237] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.189924 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8ada01658a746029b8fd58389f608a4" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.193308 29879 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:04.191267 29877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:04.194026 29921 leader_election.cc:304] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.194519 29877 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 1.
I20260709 12:28:04.195010 30586 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.195395 30720 raft_consensus.cc:2804] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.196051 30720 raft_consensus.cc:697] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:04.196982 30720 consensus_queue.cc:237] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.197359 30703 consensus_queue.cc:260] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.199753 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:04.200599 29952 raft_consensus.cc:2468] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 1.
I20260709 12:28:04.202204 30703 ts_tablet_manager.cc:1434] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.037s	user 0.009s	sys 0.000s
I20260709 12:28:04.203342 30703 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:04.206847 29994 leader_election.cc:336] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.213835 30374 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.217481 29880 raft_consensus.cc:2468] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:04.217538 29879 raft_consensus.cc:2468] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:04.224330 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:04.225461 29950 raft_consensus.cc:2393] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 2: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:28:04.227613 30374 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.210759 30681 leader_election.cc:290] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:04.211153 29920 leader_election.cc:336] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.212253 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:04.232074 29952 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 2: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:28:04.220891 29701 catalog_manager.cc:5697] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 2 to 3, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 3 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.233064 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:04.234375 30022 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 2: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
W20260709 12:28:04.235841 29920 leader_election.cc:336] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.236433 29920 leader_election.cc:336] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.240933 29918 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:04.242251 30586 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.242864 30456 raft_consensus.cc:2804] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.243388 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad89089259844673bbd8eeef685bb362" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:04.244477 30022 raft_consensus.cc:3060] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.243462 30456 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.250699 30456 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.251711 29918 leader_election.cc:304] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.253909 30694 raft_consensus.cc:2804] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.209384 29874 raft_consensus.cc:2468] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:04.254519 30694 raft_consensus.cc:697] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:04.256004 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:04.257140 30024 raft_consensus.cc:2393] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 2: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:28:04.255686 30694 consensus_queue.cc:237] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:04.263089 29846 leader_election.cc:336] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.266505 29918 leader_election.cc:304] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:04.267968 30721 raft_consensus.cc:2804] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:04.268603 30721 raft_consensus.cc:697] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:04.269675 30721 consensus_queue.cc:237] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.276057 29921 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:04.279625 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6872c9bba3c34283ac8c1a5a72884f7d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:04.280694 29952 raft_consensus.cc:3060] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.284080 30541 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:28:04.284611 29994 leader_election.cc:336] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.285298 29994 leader_election.cc:336] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.286263 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:04.287299 30024 raft_consensus.cc:3060] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.292980 29700 catalog_manager.cc:5697] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:04.296114 29920 leader_election.cc:336] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.297200 29920 leader_election.cc:336] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.297200 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:04.297775 29920 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:04.298419 29950 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.310695 30720 raft_consensus.cc:2749] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:28:04.298684 29995 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.305222 29918 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.312633 30541 raft_consensus.cc:2804] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.313211 30541 raft_consensus.cc:697] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:04.314203 30541 consensus_queue.cc:237] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.316035 30722 raft_consensus.cc:2804] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:04.316491 29995 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:04.316491 30722 raft_consensus.cc:493] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:04.318060 30722 raft_consensus.cc:3060] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.317523 30681 raft_consensus.cc:2749] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:28:04.321923 29920 leader_election.cc:336] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.324126 29952 raft_consensus.cc:2468] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:04.325304 29995 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:04.332722 30374 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:28:04.339078 29995 leader_election.cc:304] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.341116 30681 raft_consensus.cc:2804] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.343894 30681 raft_consensus.cc:697] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.346194 30681 consensus_queue.cc:237] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.379514 30456 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.390730 30456 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.381754 30024 raft_consensus.cc:2468] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a2c42e3a7d4dedbea2bd7d705862e6 in term 2.
I20260709 12:28:04.394695 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:04.395460 29874 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.395387 29847 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.396533 30733 raft_consensus.cc:2804] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.397114 30733 raft_consensus.cc:697] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 LEADER]: Becoming Leader. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
W20260709 12:28:04.426884 29920 leader_election.cc:336] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:04.381492 30022 raft_consensus.cc:2468] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:04.398254 30733 consensus_queue.cc:237] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.452123 29950 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:04.459261 29874 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:04.469424 29918 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.469971 29921 leader_election.cc:304] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.471257 29995 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.471683 30703 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:04.476363 30456 raft_consensus.cc:2804] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.477075 30456 raft_consensus.cc:697] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:04.478032 30456 consensus_queue.cc:237] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.479135 30721 raft_consensus.cc:2804] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.479825 30721 raft_consensus.cc:697] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:04.479979 30717 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:04.480598 30717 raft_consensus.cc:697] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.480947 30721 consensus_queue.cc:237] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.481611 30717 consensus_queue.cc:237] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.498665 30703 tablet_bootstrap.cc:492] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:04.499949 30703 ts_tablet_manager.cc:1403] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.297s	user 0.064s	sys 0.058s
W20260709 12:28:04.501119 30748 log.cc:927] Time spent T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:28:04.505074 30703 raft_consensus.cc:359] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.506191 30703 raft_consensus.cc:740] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:04.484632 30722 raft_consensus.cc:515] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.507332 30703 consensus_queue.cc:260] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.509058 30722 leader_election.cc:290] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:04.510129 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a71487088564b8489c63edd9d182c5a" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:04.510932 29950 raft_consensus.cc:3060] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:04.530310 30703 ts_tablet_manager.cc:1434] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 12:28:04.534914 30703 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:04.516343 29703 catalog_manager.cc:5697] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:04.546249 29950 raft_consensus.cc:2468] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:04.572196 29874 raft_consensus.cc:1275] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:04.576784 30717 consensus_queue.cc:1048] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:04.583003 29880 raft_consensus.cc:1275] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:04.586403 30717 consensus_queue.cc:1048] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:04.612627 30703 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:04.614681 30703 tablet_bootstrap.cc:492] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:04.615840 30703 ts_tablet_manager.cc:1403] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.081s	user 0.033s	sys 0.016s
I20260709 12:28:04.620404 30703 raft_consensus.cc:359] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.621670 30703 raft_consensus.cc:740] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:04.622833 30703 consensus_queue.cc:260] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:04.626005 30703 ts_tablet_manager.cc:1434] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:28:04.627434 30703 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:04.551692 29994 leader_election.cc:336] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:04.655941 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 104283us
I20260709 12:28:04.657135 29995 leader_election.cc:304] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:04.658334 30599 raft_consensus.cc:2804] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:28:04.658397 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 97401.6us
I20260709 12:28:04.659268 30599 raft_consensus.cc:697] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:04.660989 29950 raft_consensus.cc:1275] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:04.680063 29700 catalog_manager.cc:5697] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.683007 29700 catalog_manager.cc:5697] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.719066 29700 catalog_manager.cc:5697] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.694999 30717 consensus_queue.cc:1048] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:04.708480 29950 raft_consensus.cc:1275] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:04.728806 30717 consensus_queue.cc:1048] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:28:04.714010 29920 consensus_peers.cc:597] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:04.730456 29994 consensus_peers.cc:597] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.733031 30703 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:28:04.738258 29920 consensus_peers.cc:597] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.660401 30599 consensus_queue.cc:237] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:04.741926 30703 tablet_bootstrap.cc:492] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
W20260709 12:28:04.742479 29920 consensus_peers.cc:597] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.742759 30708 raft_consensus.cc:493] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.743535 30703 ts_tablet_manager.cc:1403] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.116s	user 0.023s	sys 0.013s
I20260709 12:28:04.743654 30708 raft_consensus.cc:515] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.760510 30703 raft_consensus.cc:359] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.763310 29700 catalog_manager.cc:5697] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:04.766824 29994 consensus_peers.cc:597] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:04.772102 29994 consensus_peers.cc:597] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.779637 29880 raft_consensus.cc:1275] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:04.773023 29703 catalog_manager.cc:5697] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 2 to 3, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 3 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:04.767599 30708 leader_election.cc:290] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:04.762044 30703 raft_consensus.cc:740] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:04.800580 29703 catalog_manager.cc:5697] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:04.805294 30703 consensus_queue.cc:260] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:04.812883 30703 ts_tablet_manager.cc:1434] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.069s	user 0.009s	sys 0.001s
I20260709 12:28:04.813704 29880 raft_consensus.cc:1275] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:04.820384 30022 raft_consensus.cc:1275] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:04.824652 30721 consensus_queue.cc:1048] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:04.831755 30703 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:04.766026 29700 catalog_manager.cc:5697] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.839568 29700 catalog_manager.cc:5697] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.806026 29703 catalog_manager.cc:5697] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:04.846380 30541 consensus_queue.cc:1048] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:28:04.849577 29703 catalog_manager.cc:5697] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:04.858385 30456 consensus_queue.cc:1048] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:04.858474 29703 catalog_manager.cc:5697] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:04.870235 29994 consensus_peers.cc:597] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.873224 29880 raft_consensus.cc:1275] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:28:04.872987 29994 consensus_peers.cc:597] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.878687 30722 consensus_queue.cc:1048] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20260709 12:28:04.914057 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 63462.4us
I20260709 12:28:04.916131 29880 raft_consensus.cc:1275] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:04.923934 30456 consensus_queue.cc:1048] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:04.918303 30708 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.925997 30708 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:04.930071 29920 consensus_peers.cc:597] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:04.937812 30708 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:28:04.951653 29994 consensus_peers.cc:597] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:04.959712 29994 consensus_peers.cc:597] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:05.089378 29950 raft_consensus.cc:1275] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:28:05.089484 29921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 57230.5us
W20260709 12:28:05.096223 29846 consensus_peers.cc:597] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.104326 29994 consensus_peers.cc:597] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.116225 29950 raft_consensus.cc:1275] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:05.121739 30722 consensus_queue.cc:1048] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:05.104022 30708 raft_consensus.cc:493] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.123816 30708 raft_consensus.cc:515] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:05.114181 29994 consensus_peers.cc:597] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.127398 30599 consensus_queue.cc:1048] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:28:05.137928 29846 consensus_peers.cc:597] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.176077 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 213116us
W20260709 12:28:05.199580 29994 consensus_peers.cc:597] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:05.200548 29994 consensus_peers.cc:597] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.207309 29994 consensus_peers.cc:597] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.219796 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 77942.3us
W20260709 12:28:05.305015 29846 consensus_peers.cc:597] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.305971 29846 consensus_peers.cc:597] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.308506 29846 consensus_peers.cc:597] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.294418 30708 leader_election.cc:290] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:28:05.330993 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 140264us
I20260709 12:28:05.331890 29874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:05.333088 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:05.302826 30798 raft_consensus.cc:493] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.331010 30708 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:05.383121 29994 consensus_peers.cc:597] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.343175 30708 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:05.410275 30805 raft_consensus.cc:493] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.411110 30805 raft_consensus.cc:515] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:05.427207 30805 leader_election.cc:290] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:05.455808 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:28:05.459632 29846 consensus_peers.cc:597] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:05.384196 30798 raft_consensus.cc:515] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:05.460517 29846 consensus_peers.cc:597] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.462558 29846 consensus_peers.cc:597] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.455896 29949 raft_consensus.cc:1275] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:05.447448 30022 raft_consensus.cc:1275] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer b5a2c42e3a7d4dedbea2bd7d705862e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:28:05.508329 29846 consensus_peers.cc:597] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.508934 29846 consensus_peers.cc:597] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.521651 30703 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:05.537873 30703 tablet_bootstrap.cc:492] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:05.543236 30708 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
W20260709 12:28:05.550412 30813 log.cc:927] Time spent T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:28:05.556128 30805 raft_consensus.cc:493] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.539763 30703 ts_tablet_manager.cc:1403] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.709s	user 0.025s	sys 0.008s
I20260709 12:28:05.577716 30586 consensus_queue.cc:1048] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:05.563548 30805 raft_consensus.cc:515] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:05.604017 29994 consensus_peers.cc:597] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:05.605064 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:05.606778 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49dfe78f7164655b82bfc664fbd11dd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:05.607008 29951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80eac7cb175d4f0d84a00bd792dfd476" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:05.608490 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:05.609750 29950 raft_consensus.cc:2393] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 2: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:28:05.610958 30700 leader_election.cc:304] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:05.613046 30708 raft_consensus.cc:2749] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:05.613898 30805 leader_election.cc:290] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:05.615304 29881 raft_consensus.cc:1275] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:05.616544 29949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:05.617493 29949 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 2: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:28:05.617353 30703 raft_consensus.cc:359] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:05.619563 30703 raft_consensus.cc:740] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:05.620653 30703 consensus_queue.cc:260] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:05.634897 29920 consensus_peers.cc:597] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.635604 29920 consensus_peers.cc:597] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:05.636062 29920 consensus_peers.cc:597] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:05.640549 30703 ts_tablet_manager.cc:1434] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.080s	user 0.003s	sys 0.005s
I20260709 12:28:05.641760 30703 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:05.618038 30700 leader_election.cc:304] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
W20260709 12:28:05.644176 29846 consensus_peers.cc:597] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.650655 29846 consensus_peers.cc:597] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.654115 30541 consensus_queue.cc:1048] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:28:05.657667 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 63593.2us
I20260709 12:28:05.692039 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c89d9015e54780811a19203939e9a5" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:05.652266 30805 raft_consensus.cc:2749] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:05.705636 30819 log.cc:927] Time spent T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:28:05.711125 29846 consensus_peers.cc:597] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.711926 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 53408.3us
I20260709 12:28:05.732237 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:05.740096 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:05.743826 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:05.750890 30027 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 2: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:28:05.752966 30028 raft_consensus.cc:2393] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 2: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:28:05.753954 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a7202a2bfb344cdbb49a8cb58ac6b52" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:05.754786 30717 consensus_queue.cc:1048] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:28:05.760258 29846 consensus_peers.cc:597] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.763585 29920 consensus_peers.cc:597] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.722057 30798 leader_election.cc:290] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:05.773958 30028 raft_consensus.cc:1275] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:28:05.790587 29920 consensus_peers.cc:597] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.809077 29872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6fa078c85764bb8b88bc6afff3c5bda" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:05.805963 30700 leader_election.cc:304] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:05.814016 30798 raft_consensus.cc:2749] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:05.822384 29994 consensus_peers.cc:597] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:05.824337 29994 consensus_peers.cc:597] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.813745 30700 leader_election.cc:304] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:05.882740 30798 raft_consensus.cc:2749] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:05.882969 30700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 69346.7us
I20260709 12:28:05.884199 30700 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:05.831019 30790 consensus_queue.cc:1048] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:28:05.901684 30805 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.902454 30805 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:05.906760 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
W20260709 12:28:05.897559 29994 consensus_peers.cc:597] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.910112 30805 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:05.892868 30798 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:05.945748 29846 consensus_peers.cc:597] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:05.928586 30700 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
W20260709 12:28:06.100808 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 94791.8us
I20260709 12:28:06.107522 30599 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.108259 30599 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:05.998365 30805 raft_consensus.cc:493] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:06.122089 30833 log.cc:927] Time spent T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:28:06.100203 30831 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:06.129062 30830 log.cc:927] Time spent T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:28:06.137948 30599 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:06.118990 30805 raft_consensus.cc:515] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.141111 30805 leader_election.cc:290] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:06.099606 30708 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:06.146167 30831 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.168502 29946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:06.207931 30708 raft_consensus.cc:2749] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:06.274840 30790 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.269282 30703 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:28:06.285792 29920 leader_election.cc:336] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:06.288820 29700 catalog_manager.cc:5697] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:06.260102 30027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:06.318892 30027 raft_consensus.cc:2468] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
W20260709 12:28:06.223451 30824 log.cc:927] Time spent T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:28:06.188865 30798 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:06.325731 30703 tablet_bootstrap.cc:492] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
W20260709 12:28:06.274466 30700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 362110us
I20260709 12:28:06.330330 30703 ts_tablet_manager.cc:1403] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.689s	user 0.050s	sys 0.039s
I20260709 12:28:06.238854 30831 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:06.350750 30703 raft_consensus.cc:359] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.353435 30703 raft_consensus.cc:740] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:06.356812 30703 consensus_queue.cc:260] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.320197 30790 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:06.377013 30790 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:06.370636 29921 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:06.382139 29946 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:06.341960 30700 leader_election.cc:304] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
W20260709 12:28:06.388387 30700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.102.4:40301, user_credentials={real_user=slave}} blocked reactor thread for 52218us
I20260709 12:28:06.390057 30798 raft_consensus.cc:2749] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:06.397181 30703 ts_tablet_manager.cc:1434] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.063s	user 0.009s	sys 0.000s
I20260709 12:28:06.399467 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:06.404795 30703 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:06.431109 30024 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:06.432219 30024 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:06.409423 30831 raft_consensus.cc:2804] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:06.433120 30831 raft_consensus.cc:493] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.433522 30831 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:06.453063 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3dafcab3681f4882a86829813cbc7772" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:06.439121 29921 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
W20260709 12:28:06.459911 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 51552.5us
I20260709 12:28:06.471778 29995 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:06.473223 30803 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:06.473707 30803 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.474121 30803 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:06.460956 30694 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:28:06.482007 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 118409us
I20260709 12:28:06.478929 30694 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.508076 30700 leader_election.cc:304] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:06.509907 30798 raft_consensus.cc:2749] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:06.486423 30694 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:06.521188 30028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c799e9531994c7cba86a172713405e8" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:06.527585 30831 raft_consensus.cc:515] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.531231 30831 leader_election.cc:290] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:06.533216 30798 raft_consensus.cc:493] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.533932 30798 raft_consensus.cc:515] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:28:06.545760 29994 consensus_peers.cc:597] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:06.546695 29994 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:06.561260 30700 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:06.562889 30708 raft_consensus.cc:2749] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:06.564658 30798 leader_election.cc:290] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:06.566244 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:06.550964 30803 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:06.568930 30803 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:06.576404 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4a3b54cd9af44d69698e15cc4bccafa" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:06.577221 30025 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:06.593662 29920 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:06.607438 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
W20260709 12:28:06.612419 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 64665.7us
W20260709 12:28:06.616304 29992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 57881.8us
W20260709 12:28:06.618099 29994 consensus_peers.cc:597] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:06.632423 30694 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:06.642212 29950 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e2ceabfe50c47b49529b663fa133b86 in current term 3: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:28:06.642490 30694 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:06.643514 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:06.644454 30022 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7df42fd4269f4d16aeebecaa4ea958ac in current term 3: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
W20260709 12:28:06.641430 29920 leader_election.cc:336] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:06.671085 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f31657dfccf4d45ab006ef5aadcfb4c" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:06.698197 30703 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:06.702076 30025 raft_consensus.cc:2468] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:06.705904 30703 tablet_bootstrap.cc:492] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:06.713938 30703 ts_tablet_manager.cc:1403] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.310s	user 0.026s	sys 0.002s
W20260709 12:28:06.718595 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 44218.4us
I20260709 12:28:06.722323 29921 leader_election.cc:304] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:06.723476 30852 raft_consensus.cc:2804] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:06.724035 30852 raft_consensus.cc:697] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:06.724896 30852 consensus_queue.cc:237] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.733723 30700 leader_election.cc:304] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:06.735371 30798 raft_consensus.cc:2749] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:06.747611 29920 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:06.748358 29920 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:06.748824 30703 raft_consensus.cc:359] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:06.749300 30694 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:28:06.749693 30703 raft_consensus.cc:740] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:06.750732 30703 consensus_queue.cc:260] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:06.756270 29994 leader_election.cc:336] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:06.756759 29994 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:06.761157 30703 ts_tablet_manager.cc:1434] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.001s
I20260709 12:28:06.762346 30703 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:06.762122 30857 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:28:06.784552 29701 catalog_manager.cc:5697] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 3, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 3 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:06.790380 30798 raft_consensus.cc:493] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.791113 30798 raft_consensus.cc:515] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:06.794875 30798 leader_election.cc:290] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:06.797735 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:06.800061 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c08a657511124ab1ad80a227eeec2736" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:06.803812 30700 leader_election.cc:304] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:06.813369 30798 raft_consensus.cc:2749] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:06.851485 30703 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:06.854431 30703 tablet_bootstrap.cc:492] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:06.855732 30703 ts_tablet_manager.cc:1403] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.094s	user 0.015s	sys 0.007s
I20260709 12:28:06.860214 30703 raft_consensus.cc:359] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:06.861506 30703 raft_consensus.cc:740] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:06.862836 30703 consensus_queue.cc:260] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:06.880039 30703 ts_tablet_manager.cc:1434] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:28:06.887826 30703 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:06.958086 30703 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:06.959409 30703 tablet_bootstrap.cc:492] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:06.960333 30703 ts_tablet_manager.cc:1403] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.073s	user 0.018s	sys 0.008s
I20260709 12:28:06.963809 30703 raft_consensus.cc:359] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:06.964692 30703 raft_consensus.cc:740] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:06.965842 30703 consensus_queue.cc:260] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:06.968576 30703 ts_tablet_manager.cc:1434] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:06.974581 30703 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.057590 30703 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.058856 30703 tablet_bootstrap.cc:492] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.059731 30703 ts_tablet_manager.cc:1403] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.085s	user 0.030s	sys 0.013s
I20260709 12:28:07.063227 30703 raft_consensus.cc:359] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.065197 30703 raft_consensus.cc:740] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.066226 30703 consensus_queue.cc:260] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.069370 30703 ts_tablet_manager.cc:1434] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:28:07.071871 30703 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.094187 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:42003
I20260709 12:28:07.094324 30883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:42003 every 8 connection(s)
I20260709 12:28:07.116370 30703 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.141839 30703 tablet_bootstrap.cc:492] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.143671 30703 ts_tablet_manager.cc:1403] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.072s	user 0.014s	sys 0.014s
I20260709 12:28:07.169911 30703 raft_consensus.cc:359] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.171314 30703 raft_consensus.cc:740] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.172340 30703 consensus_queue.cc:260] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.175218 30703 ts_tablet_manager.cc:1434] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.001s
I20260709 12:28:07.176580 30703 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:07.171343 29994 consensus_peers.cc:597] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:07.198374 30022 raft_consensus.cc:1275] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:28:07.211745 29994 consensus_peers.cc:597] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:07.226217 30852 consensus_queue.cc:1048] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:07.229269 30836 raft_consensus.cc:3060] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:07.282577 30887 heartbeater.cc:344] Connected to a master server at 127.16.102.62:33633
I20260709 12:28:07.283254 30887 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:07.285580 30887 heartbeater.cc:507] Master 127.16.102.62:33633 requested a full tablet report, sending...
I20260709 12:28:07.291301 30703 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.292563 30838 raft_consensus.cc:3060] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:07.292611 30703 tablet_bootstrap.cc:492] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.293895 30703 ts_tablet_manager.cc:1403] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.117s	user 0.052s	sys 0.035s
W20260709 12:28:07.294034 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 44614.4us
I20260709 12:28:07.309939 30838 raft_consensus.cc:1275] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:28:07.312450 29994 consensus_peers.cc:597] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:07.318001 30790 consensus_queue.cc:1048] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:28:07.320626 30703 raft_consensus.cc:359] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.321453 30703 raft_consensus.cc:740] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.322294 30703 consensus_queue.cc:260] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.325932 30703 ts_tablet_manager.cc:1434] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 12:28:07.326813 30703 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:07.395205 29994 consensus_peers.cc:597] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:07.409700 30836 raft_consensus.cc:3060] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:07.419797 29701 ts_manager.cc:194] Re-registered known tserver with Master: 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:07.431630 30703 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.433395 30703 tablet_bootstrap.cc:492] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.434249 30703 ts_tablet_manager.cc:1403] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.108s	user 0.013s	sys 0.009s
W20260709 12:28:07.438171 29994 consensus_peers.cc:597] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:07.444644 30703 raft_consensus.cc:359] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.445848 30703 raft_consensus.cc:740] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.446686 30703 consensus_queue.cc:260] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.451625 30703 ts_tablet_manager.cc:1434] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.004s
I20260709 12:28:07.455751 30703 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.471889 29701 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53082
I20260709 12:28:07.478940 30887 heartbeater.cc:499] Master 127.16.102.62:33633 was elected leader, sending a full tablet report...
I20260709 12:28:07.490285 30798 raft_consensus.cc:493] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.490923 30798 raft_consensus.cc:515] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.494273 29879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" is_pre_election: true
I20260709 12:28:07.494638 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01f4a9dd66ad4df0a8f5a598a7b432fd" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:07.496327 30697 leader_election.cc:304] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:07.496381 30798 leader_election.cc:290] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:07.497430 30893 raft_consensus.cc:2749] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:07.492705 30703 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.499236 30703 tablet_bootstrap.cc:492] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.500041 30703 ts_tablet_manager.cc:1403] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.045s	user 0.013s	sys 0.006s
I20260709 12:28:07.504040 30703 raft_consensus.cc:359] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.506024 30703 raft_consensus.cc:740] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.506918 30703 consensus_queue.cc:260] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.508744 30703 ts_tablet_manager.cc:1434] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:28:07.509711 30703 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:07.522305 29846 consensus_peers.cc:597] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:07.535473 30703 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.536777 30703 tablet_bootstrap.cc:492] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.537647 30703 ts_tablet_manager.cc:1403] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.028s	user 0.016s	sys 0.002s
I20260709 12:28:07.539984 30703 raft_consensus.cc:359] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.540702 30703 raft_consensus.cc:740] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.541507 30703 consensus_queue.cc:260] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.544049 30703 ts_tablet_manager.cc:1434] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.003s
I20260709 12:28:07.545061 30703 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.564049 30703 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.565215 30703 tablet_bootstrap.cc:492] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.566054 30703 ts_tablet_manager.cc:1403] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.007s
I20260709 12:28:07.569070 30703 raft_consensus.cc:359] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.569882 30703 raft_consensus.cc:740] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.570636 30703 consensus_queue.cc:260] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.573277 30703 ts_tablet_manager.cc:1434] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:28:07.577986 30703 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:07.579248 29920 consensus_peers.cc:597] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:07.598917 30703 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.600221 30703 tablet_bootstrap.cc:492] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.600975 30703 ts_tablet_manager.cc:1403] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.003s
I20260709 12:28:07.603332 30703 raft_consensus.cc:359] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.603873 30703 raft_consensus.cc:740] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.604431 30703 consensus_queue.cc:260] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.606184 30703 ts_tablet_manager.cc:1434] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:28:07.607584 30703 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.626646 30703 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.627841 30703 tablet_bootstrap.cc:492] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.628697 30703 ts_tablet_manager.cc:1403] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.000s
I20260709 12:28:07.633093 30703 raft_consensus.cc:359] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.633960 30703 raft_consensus.cc:740] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.634892 30703 consensus_queue.cc:260] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.638464 30703 ts_tablet_manager.cc:1434] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:28:07.639519 30703 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.660084 30703 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.661401 30703 tablet_bootstrap.cc:492] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.662343 30703 ts_tablet_manager.cc:1403] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.000s
I20260709 12:28:07.665539 30703 raft_consensus.cc:359] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.669714 30703 raft_consensus.cc:740] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.670998 30703 consensus_queue.cc:260] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.672545 30843 raft_consensus.cc:3060] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:07.674451 30703 ts_tablet_manager.cc:1434] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:28:07.675820 30703 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:07.681856 29846 consensus_peers.cc:597] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:07.707067 30703 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.708385 30703 tablet_bootstrap.cc:492] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.709278 30703 ts_tablet_manager.cc:1403] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.034s	user 0.021s	sys 0.002s
I20260709 12:28:07.712363 30703 raft_consensus.cc:359] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.713198 30703 raft_consensus.cc:740] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.714146 30703 consensus_queue.cc:260] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.715340 30843 raft_consensus.cc:3060] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:07.716890 30703 ts_tablet_manager.cc:1434] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.001s
I20260709 12:28:07.718003 30703 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.751497 30703 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.752754 30703 tablet_bootstrap.cc:492] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.753610 30703 ts_tablet_manager.cc:1403] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.036s	user 0.018s	sys 0.006s
I20260709 12:28:07.757485 30703 raft_consensus.cc:359] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.758736 30703 raft_consensus.cc:740] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.759554 30703 consensus_queue.cc:260] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.761912 30703 ts_tablet_manager.cc:1434] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:28:07.762849 30703 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.807192 30823 raft_consensus.cc:3060] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:07.814232 30703 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.815615 30703 tablet_bootstrap.cc:492] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.815929 30843 raft_consensus.cc:3060] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:07.816464 30703 ts_tablet_manager.cc:1403] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.054s	user 0.016s	sys 0.024s
I20260709 12:28:07.819617 30703 raft_consensus.cc:359] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.820853 30703 raft_consensus.cc:740] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.821853 30703 consensus_queue.cc:260] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.824025 30836 raft_consensus.cc:3060] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:07.824527 30703 ts_tablet_manager.cc:1434] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:07.825572 30703 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.849547 30703 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.852290 30703 tablet_bootstrap.cc:492] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.853132 30703 ts_tablet_manager.cc:1403] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.009s
I20260709 12:28:07.856663 30703 raft_consensus.cc:359] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.857513 30703 raft_consensus.cc:740] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.858582 30703 consensus_queue.cc:260] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.861243 30703 ts_tablet_manager.cc:1434] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.003s
I20260709 12:28:07.862319 30703 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.887555 30703 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.888761 30703 tablet_bootstrap.cc:492] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.889549 30703 ts_tablet_manager.cc:1403] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.027s	user 0.016s	sys 0.005s
I20260709 12:28:07.893400 30703 raft_consensus.cc:359] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.894246 30703 raft_consensus.cc:740] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.895120 30703 consensus_queue.cc:260] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:07.897554 30703 ts_tablet_manager.cc:1434] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20260709 12:28:07.898736 30703 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:07.900309 29994 consensus_peers.cc:597] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:07.928185 29920 consensus_peers.cc:597] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:07.946398 30703 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.947830 30703 tablet_bootstrap.cc:492] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.948783 30703 ts_tablet_manager.cc:1403] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.050s	user 0.024s	sys 0.013s
I20260709 12:28:07.951468 30703 raft_consensus.cc:359] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.952212 30703 raft_consensus.cc:740] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.952961 30703 consensus_queue.cc:260] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:07.954872 30703 ts_tablet_manager.cc:1434] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:28:07.955842 30703 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:07.984809 30703 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:07.986205 30703 tablet_bootstrap.cc:492] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:07.987171 30703 ts_tablet_manager.cc:1403] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.031s	user 0.018s	sys 0.008s
I20260709 12:28:07.990487 30703 raft_consensus.cc:359] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.991354 30703 raft_consensus.cc:740] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:07.992117 30703 consensus_queue.cc:260] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:07.994637 30703 ts_tablet_manager.cc:1434] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:28:07.995676 30703 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.040702 30703 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.041703 30703 tablet_bootstrap.cc:492] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.042482 30703 ts_tablet_manager.cc:1403] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.047s	user 0.024s	sys 0.007s
I20260709 12:28:08.044881 30703 raft_consensus.cc:359] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.045476 30703 raft_consensus.cc:740] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.046310 30703 consensus_queue.cc:260] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.048599 30703 ts_tablet_manager.cc:1434] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:28:08.052254 30703 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.082844 30703 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.084527 30703 tablet_bootstrap.cc:492] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.085314 30703 ts_tablet_manager.cc:1403] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.012s
I20260709 12:28:08.087680 30843 raft_consensus.cc:3060] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:08.088666 30703 raft_consensus.cc:359] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.089485 30703 raft_consensus.cc:740] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.090232 30703 consensus_queue.cc:260] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.094276 30703 ts_tablet_manager.cc:1434] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:28:08.095276 30703 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.104103 30836 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:08.128099 29994 consensus_peers.cc:597] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:08.139046 30823 raft_consensus.cc:3060] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:08.176231 30826 raft_consensus.cc:3060] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:08.194625 30703 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.196053 30703 tablet_bootstrap.cc:492] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.196918 30703 ts_tablet_manager.cc:1403] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.102s	user 0.049s	sys 0.036s
W20260709 12:28:08.172890 29994 consensus_peers.cc:597] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:08.200111 30703 raft_consensus.cc:359] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.201155 30703 raft_consensus.cc:740] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.202349 30703 consensus_queue.cc:260] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.204890 30703 ts_tablet_manager.cc:1434] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:28:08.208832 30703 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.241686 30703 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.243113 30703 tablet_bootstrap.cc:492] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.244529 30703 ts_tablet_manager.cc:1403] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.038s	user 0.026s	sys 0.000s
I20260709 12:28:08.248473 30703 raft_consensus.cc:359] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.249543 30703 raft_consensus.cc:740] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.250399 30703 consensus_queue.cc:260] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.253708 30703 ts_tablet_manager.cc:1434] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260709 12:28:08.254807 30703 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.284580 30703 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.285842 30703 tablet_bootstrap.cc:492] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.286664 30703 ts_tablet_manager.cc:1403] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.032s	user 0.022s	sys 0.001s
I20260709 12:28:08.289853 30703 raft_consensus.cc:359] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.290668 30703 raft_consensus.cc:740] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.291469 30703 consensus_queue.cc:260] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.294186 30703 ts_tablet_manager.cc:1434] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:08.295185 30703 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.316653 30703 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:28:08.317652 30703 tablet_bootstrap.cc:492] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.318378 30703 ts_tablet_manager.cc:1403] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.023s	user 0.020s	sys 0.000s
I20260709 12:28:08.320950 30703 raft_consensus.cc:359] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.321902 30703 raft_consensus.cc:740] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.322738 30703 consensus_queue.cc:260] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.325110 30703 ts_tablet_manager.cc:1434] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:28:08.326220 30703 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.347031 30703 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.348390 30703 tablet_bootstrap.cc:492] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.349231 30703 ts_tablet_manager.cc:1403] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.023s	user 0.018s	sys 0.001s
I20260709 12:28:08.352429 30703 raft_consensus.cc:359] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.353165 30703 raft_consensus.cc:740] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.354363 30703 consensus_queue.cc:260] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.357191 30703 ts_tablet_manager.cc:1434] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:28:08.360723 30703 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.367251 30823 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:08.412021 30836 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:28:08.422236 30703 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.423296 30703 tablet_bootstrap.cc:492] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.424091 30703 ts_tablet_manager.cc:1403] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.064s	user 0.029s	sys 0.027s
I20260709 12:28:08.427193 30703 raft_consensus.cc:359] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.427955 30703 raft_consensus.cc:740] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.428742 30703 consensus_queue.cc:260] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.431066 30703 ts_tablet_manager.cc:1434] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:08.432031 30703 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.444156 30798 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.444917 30798 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.447515 30836 raft_consensus.cc:3060] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:08.453137 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:08.454352 29950 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 3: Already voted for candidate 7df42fd4269f4d16aeebecaa4ea958ac in this term.
I20260709 12:28:08.464125 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:08.465201 30026 raft_consensus.cc:2393] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ae5481d76bf4a90a0ee824a14223421 in current term 3: Already voted for candidate 1e2ceabfe50c47b49529b663fa133b86 in this term.
I20260709 12:28:08.467310 30700 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421; no voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:08.460480 30798 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:08.472950 30798 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:08.483403 30823 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:08.484596 30703 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.486002 30703 tablet_bootstrap.cc:492] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.486904 30703 ts_tablet_manager.cc:1403] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.055s	user 0.015s	sys 0.016s
I20260709 12:28:08.490211 30703 raft_consensus.cc:359] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.491070 30703 raft_consensus.cc:740] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.491935 30703 consensus_queue.cc:260] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.494418 30703 ts_tablet_manager.cc:1434] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:08.496634 30703 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
W20260709 12:28:08.502812 29846 consensus_peers.cc:597] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:08.514451 30798 raft_consensus.cc:2749] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:08.521229 30703 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.522555 30703 tablet_bootstrap.cc:492] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.523404 30703 ts_tablet_manager.cc:1403] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.027s	user 0.019s	sys 0.003s
I20260709 12:28:08.526561 30703 raft_consensus.cc:359] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.527385 30703 raft_consensus.cc:740] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.528172 30703 consensus_queue.cc:260] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.530615 30703 ts_tablet_manager.cc:1434] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20260709 12:28:08.531454 30703 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.584565 30703 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.585955 30703 tablet_bootstrap.cc:492] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.591997 30703 ts_tablet_manager.cc:1403] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.061s	user 0.025s	sys 0.012s
I20260709 12:28:08.595192 30703 raft_consensus.cc:359] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.596760 30703 raft_consensus.cc:740] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.597616 30703 consensus_queue.cc:260] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:08.607415 30703 ts_tablet_manager.cc:1434] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:28:08.609201 30703 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.656706 30703 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.658385 30703 tablet_bootstrap.cc:492] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.659178 30703 ts_tablet_manager.cc:1403] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.050s	user 0.014s	sys 0.013s
I20260709 12:28:08.662428 30703 raft_consensus.cc:359] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.663260 30703 raft_consensus.cc:740] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.664088 30703 consensus_queue.cc:260] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:08.666465 30703 ts_tablet_manager.cc:1434] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:28:08.667492 30703 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap starting.
I20260709 12:28:08.686522 30703 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:08.687752 30703 tablet_bootstrap.cc:492] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Bootstrap complete.
I20260709 12:28:08.688611 30703 ts_tablet_manager.cc:1403] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.001s
I20260709 12:28:08.691584 30703 raft_consensus.cc:359] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.692225 30703 raft_consensus.cc:740] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Initialized, Role: FOLLOWER
I20260709 12:28:08.693076 30703 consensus_queue.cc:260] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:08.695294 30703 ts_tablet_manager.cc:1434] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20260709 12:28:08.887912 30823 raft_consensus.cc:3060] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.362485 30790 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:09.363080 30790 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:09.365262 30790 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:09.366096 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:09.366559 30823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421" is_pre_election: true
I20260709 12:28:09.366971 30026 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:09.368235 30823 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:09.369005 29921 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:09.370338 30790 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:28:09.370744 30790 raft_consensus.cc:493] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.371112 30790 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:09.380052 30790 raft_consensus.cc:515] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:09.382444 30790 leader_election.cc:290] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:09.383167 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:09.383628 30823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba9fb933bf47466999dfd7892e6a6638" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
I20260709 12:28:09.383863 30026 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:09.384289 30823 raft_consensus.cc:3060] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:09.392371 30026 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 4.
I20260709 12:28:09.392371 30823 raft_consensus.cc:2468] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 4.
I20260709 12:28:09.393956 29920 leader_election.cc:304] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:09.394726 30790 raft_consensus.cc:2804] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:09.395237 30790 raft_consensus.cc:697] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:09.396142 30790 consensus_queue.cc:237] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:09.408038 29703 catalog_manager.cc:5697] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 4, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 4 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:09.791610 30823 raft_consensus.cc:1275] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:28:09.793710 30852 consensus_queue.cc:1048] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:09.817091 30026 raft_consensus.cc:1275] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:28:09.830299 30790 consensus_queue.cc:1048] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:28:10.691618 29994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 44799.4us
W20260709 12:28:10.719838 29847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 46933.5us
I20260709 12:28:17.172466 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:17.239274 30026 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:17.240357 30026 raft_consensus.cc:606] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:17.243676 30026 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:17.244215 30026 raft_consensus.cc:606] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:17.259334 29878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536"
dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:49664
I20260709 12:28:17.260121 29878 raft_consensus.cc:606] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:17.263262 30026 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:17.263943 30026 raft_consensus.cc:606] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:17.270499 30026 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:17.271142 30026 raft_consensus.cc:606] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:17.285363 29952 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334"
dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
mode: GRACEFUL
new_leader_uuid: "1e2ceabfe50c47b49529b663fa133b86"
 from {username='slave'} at 127.0.0.1:43076
I20260709 12:28:17.286185 29952 raft_consensus.cc:606] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Received request to transfer leadership to TS 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:17.289940 30026 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:17.290709 30026 raft_consensus.cc:606] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:17.292184 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260709 12:28:17.295018 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:17.299923 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:17.365953 30977 raft_consensus.cc:993] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.366581 30977 raft_consensus.cc:1081] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.368633 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:17.369234 30836 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.369596 30836 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:17.377756 30836 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.380637 30836 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:17.382078 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:17.383267 29952 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:17.384500 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:17.385361 30022 raft_consensus.cc:3055] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:17.385706 30022 raft_consensus.cc:740] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:17.386802 30022 consensus_queue.cc:260] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.388942 30022 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:17.395881 29952 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 3.
I20260709 12:28:17.397796 30700 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:17.404241 30022 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 3.
I20260709 12:28:17.409906 30989 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:17.411432 30989 raft_consensus.cc:697] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:17.412925 30989 consensus_queue.cc:237] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.428732 29701 catalog_manager.cc:5697] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 2 to 3, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 3 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:17.448975 30971 raft_consensus.cc:993] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.449558 30971 raft_consensus.cc:1081] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.451838 30823 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48806
I20260709 12:28:17.452533 30823 raft_consensus.cc:493] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.452980 30823 raft_consensus.cc:3060] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.471269 30986 raft_consensus.cc:993] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.472052 30977 raft_consensus.cc:1081] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.472186 30823 raft_consensus.cc:515] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:17.474135 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:17.474656 30823 leader_election.cc:290] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:17.474828 30836 raft_consensus.cc:493] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.475308 30836 raft_consensus.cc:3060] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.475888 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:17.476562 29878 raft_consensus.cc:3055] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:17.476778 29878 raft_consensus.cc:740] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:28:17.477629 29878 consensus_queue.cc:260] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:17.478901 29878 raft_consensus.cc:3060] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.479688 29952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "259a887eac9c4ed0b816a4ac0bc2c536" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:17.480542 29952 raft_consensus.cc:3060] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.487274 30836 raft_consensus.cc:515] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:17.490675 29952 raft_consensus.cc:2468] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.491396 30022 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:17.491547 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889b2ffc0b554cbea97dc55e0aa95094" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:17.492139 29878 raft_consensus.cc:2468] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.492280 30836 leader_election.cc:290] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:17.492172 30022 raft_consensus.cc:3055] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:17.493314 30022 raft_consensus.cc:740] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:17.492352 29880 raft_consensus.cc:3060] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.493759 30700 leader_election.cc:304] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:17.494560 30022 consensus_queue.cc:260] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:17.495169 30989 raft_consensus.cc:2804] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:17.496013 30989 raft_consensus.cc:697] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:17.496129 30022 raft_consensus.cc:3060] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.497170 30989 consensus_queue.cc:237] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:17.508759 29880 raft_consensus.cc:2468] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.511933 30697 leader_election.cc:304] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:17.513330 30989 raft_consensus.cc:2804] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:17.513942 30989 raft_consensus.cc:697] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:17.514937 30989 consensus_queue.cc:237] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:17.514456 29701 catalog_manager.cc:5697] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:17.518899 30022 raft_consensus.cc:2468] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.520686 30977 raft_consensus.cc:993] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.522445 30716 raft_consensus.cc:1081] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.526538 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:17.527184 30836 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.527534 30836 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.528641 30716 raft_consensus.cc:993] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 7df42fd4269f4d16aeebecaa4ea958ac to start an election
I20260709 12:28:17.529335 30716 raft_consensus.cc:1081] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 7df42fd4269f4d16aeebecaa4ea958ac to start an election
I20260709 12:28:17.531522 29952 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5"
dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
 from {username='slave'} at 127.0.0.1:57180
I20260709 12:28:17.532184 29952 raft_consensus.cc:493] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.532562 29952 raft_consensus.cc:3060] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.540359 30836 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:17.621033 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 76429.9us
I20260709 12:28:17.621129 29701 catalog_manager.cc:5697] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:17.623417 29952 raft_consensus.cc:515] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:28:17.625567 29995 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:39029, user_credentials={real_user=slave}} blocked reactor thread for 81854.5us
I20260709 12:28:17.640751 30973 raft_consensus.cc:993] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: : Instructing follower 1e2ceabfe50c47b49529b663fa133b86 to start an election
I20260709 12:28:17.641217 30836 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:17.643476 30979 raft_consensus.cc:1081] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Signalling peer 1e2ceabfe50c47b49529b663fa133b86 to start an election
I20260709 12:28:17.655306 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:17.656076 29878 raft_consensus.cc:3060] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.662214 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:17.662456 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:17.663172 29950 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.663297 30026 raft_consensus.cc:3055] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:17.663797 30026 raft_consensus.cc:740] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:17.666191 30026 consensus_queue.cc:260] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.667778 30026 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:17.673519 30998 log.cc:927] Time spent T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421: Append to log took a long time: real 0.056s	user 0.004s	sys 0.000s
I20260709 12:28:17.663103 29952 leader_election.cc:290] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:17.694279 30022 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
 from {username='slave'} at 127.0.0.1:36550
I20260709 12:28:17.695152 30022 raft_consensus.cc:493] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.695534 30022 raft_consensus.cc:3060] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.688868 29878 raft_consensus.cc:2468] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:17.703631 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15ead15e87d345f48f74ca65fb97d5c5" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:17.704546 30025 raft_consensus.cc:3055] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:17.704901 30025 raft_consensus.cc:740] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:17.705869 30025 consensus_queue.cc:260] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:17.707319 30025 raft_consensus.cc:3060] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.714572 29950 raft_consensus.cc:2468] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.716097 30700 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:17.717237 30989 raft_consensus.cc:2804] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:17.718304 30989 raft_consensus.cc:697] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:17.719592 30989 consensus_queue.cc:237] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.725857 29918 leader_election.cc:304] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:17.729657 31002 raft_consensus.cc:2804] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:17.730378 31002 raft_consensus.cc:697] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:17.731454 31002 consensus_queue.cc:237] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:17.773375 29701 catalog_manager.cc:5697] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:17.783541 30022 raft_consensus.cc:515] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:17.791419 30026 raft_consensus.cc:2468] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.786465 29703 catalog_manager.cc:5697] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 2 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:17.815778 31007 log.cc:927] Time spent T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:28:17.823616 30025 raft_consensus.cc:2468] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:17.824314 30977 raft_consensus.cc:993] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.824935 30977 raft_consensus.cc:1081] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:17.827354 30823 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:17.827975 30823 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:17.828168 29952 raft_consensus.cc:1275] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:17.828320 30823 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.831267 30022 leader_election.cc:290] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:17.832199 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:17.832605 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71ae82107a97483ba4fa78a788adb334" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:17.832957 29878 raft_consensus.cc:3060] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.833298 29947 raft_consensus.cc:3055] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:17.833698 29947 raft_consensus.cc:740] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
W20260709 12:28:17.834559 29920 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.1:42003, user_credentials={real_user=slave}} blocked reactor thread for 109656us
I20260709 12:28:17.834642 29947 consensus_queue.cc:260] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:17.836283 29947 raft_consensus.cc:3060] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.840529 30823 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.844099 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:17.844962 29950 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.848351 30989 consensus_queue.cc:1048] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:17.848812 29947 raft_consensus.cc:2468] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:17.850284 29995 leader_election.cc:304] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:17.851444 30985 raft_consensus.cc:2804] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:17.851997 30985 raft_consensus.cc:697] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:17.852320 29878 raft_consensus.cc:2468] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:17.853060 30985 consensus_queue.cc:237] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:17.859843 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:17.860737 30025 raft_consensus.cc:3055] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:17.861105 30025 raft_consensus.cc:740] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:17.863708 30025 consensus_queue.cc:260] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.866992 30025 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:17.883394 30025 raft_consensus.cc:2468] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:17.884810 30700 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:28:17.886212 30989 raft_consensus.cc:2804] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:17.909232 30989 raft_consensus.cc:697] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:17.912918 29878 raft_consensus.cc:1275] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:17.914835 30823 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:17.922843 30025 raft_consensus.cc:1275] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:17.925536 30989 consensus_queue.cc:237] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:17.925781 31015 consensus_queue.cc:1048] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:28:17.927520 29701 catalog_manager.cc:5697] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:17.931589 31014 consensus_queue.cc:1048] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:17.958762 29880 raft_consensus.cc:1275] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:17.965342 29701 catalog_manager.cc:5697] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:17.968353 30989 consensus_queue.cc:1048] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:17.978822 29947 raft_consensus.cc:1275] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:17.980988 30989 consensus_queue.cc:1048] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:18.003007 29950 raft_consensus.cc:2468] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:18.067186 30022 raft_consensus.cc:1275] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.070444 31015 consensus_queue.cc:1048] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:18.181007 30022 raft_consensus.cc:1275] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.182615 31014 consensus_queue.cc:1048] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:18.195611 29947 raft_consensus.cc:1275] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.198741 31014 consensus_queue.cc:1048] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:18.241760 30022 raft_consensus.cc:1275] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.243319 31006 consensus_queue.cc:1048] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:18.261471 29880 raft_consensus.cc:1275] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 7df42fd4269f4d16aeebecaa4ea958ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.261458 29947 raft_consensus.cc:1275] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.263306 31003 consensus_queue.cc:1048] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:18.264323 30984 consensus_queue.cc:1048] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:18.287395 29878 raft_consensus.cc:1275] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.289167 30977 consensus_queue.cc:1048] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:18.337550 30025 raft_consensus.cc:1275] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.339432 31014 consensus_queue.cc:1048] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:18.345460 29950 raft_consensus.cc:1275] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:18.347965 31014 consensus_queue.cc:1048] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:18.363081 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:18.444056 29878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "325254299d324d0b950855283ec2fcf0"
dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
mode: GRACEFUL
new_leader_uuid: "1e2ceabfe50c47b49529b663fa133b86"
 from {username='slave'} at 127.0.0.1:49664
I20260709 12:28:18.444770 29878 raft_consensus.cc:606] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Received request to transfer leadership to TS 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:18.448809 30022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:18.449506 30022 raft_consensus.cc:606] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:18.452250 30022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "997e1531077444f3b82718c8b75b357e"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:18.452883 30022 raft_consensus.cc:606] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:18.455540 30022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:18.456089 30022 raft_consensus.cc:606] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:18.458659 29950 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300"
dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:43076
I20260709 12:28:18.459241 29950 raft_consensus.cc:606] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:18.462067 29950 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505"
dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:43076
I20260709 12:28:18.462642 29950 raft_consensus.cc:606] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:18.464563 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:28:18.467337 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:18.468308 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:18.522403 30986 raft_consensus.cc:993] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.523048 30986 raft_consensus.cc:1081] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.524786 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "997e1531077444f3b82718c8b75b357e"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:18.525472 30836 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:18.525898 30836 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:18.533069 30836 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:18.535900 30836 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 4 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:18.536077 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:18.536927 29950 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:18.537042 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:18.537782 30025 raft_consensus.cc:3055] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:18.538174 30025 raft_consensus.cc:740] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:18.538996 30025 consensus_queue.cc:260] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:18.540105 30025 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:18.541868 31003 raft_consensus.cc:993] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.542492 30973 raft_consensus.cc:1081] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.544566 30826 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48788
I20260709 12:28:18.545166 30826 raft_consensus.cc:493] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:18.545493 30826 raft_consensus.cc:3060] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.546643 29950 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 4.
I20260709 12:28:18.548061 30700 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:18.549034 30990 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:18.549510 30990 raft_consensus.cc:697] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 4 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:18.550395 30990 consensus_queue.cc:237] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:18.555043 30826 raft_consensus.cc:515] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:18.558178 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:18.558288 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b820ed22868a4fe98bd2c74221e3e300" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:18.558933 29878 raft_consensus.cc:3060] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.558977 29950 raft_consensus.cc:3055] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:18.559579 29950 raft_consensus.cc:740] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:18.560549 29950 consensus_queue.cc:260] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:18.561839 29950 raft_consensus.cc:3060] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.563242 29703 catalog_manager.cc:5697] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 3 to 4, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 4 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.564512 30826 leader_election.cc:290] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:18.567018 29878 raft_consensus.cc:2468] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:18.568064 30025 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 4.
I20260709 12:28:18.569581 30697 leader_election.cc:304] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:18.570497 30990 raft_consensus.cc:2804] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:18.571034 30990 raft_consensus.cc:697] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:18.571458 29950 raft_consensus.cc:2468] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:18.572063 30990 consensus_queue.cc:237] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:18.606000 29703 catalog_manager.cc:5697] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.616400 31002 raft_consensus.cc:993] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.617026 31003 raft_consensus.cc:1081] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.618983 30826 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48788
I20260709 12:28:18.619541 30826 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:18.619869 30826 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:18.627257 30826 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:18.629107 30826 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:18.630227 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:18.630913 29878 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:18.631202 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff934fd9fe834c2ab8737af23cdc9505" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:18.631783 29950 raft_consensus.cc:3055] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:18.632068 29950 raft_consensus.cc:740] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:18.632853 29950 consensus_queue.cc:260] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:18.634028 29950 raft_consensus.cc:3060] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:18.637354 29878 raft_consensus.cc:2468] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 3.
I20260709 12:28:18.638793 30697 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:18.639606 30990 raft_consensus.cc:2804] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:18.640117 30990 raft_consensus.cc:697] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:18.640964 29950 raft_consensus.cc:2468] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 3.
I20260709 12:28:18.641184 30990 consensus_queue.cc:237] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:18.653103 29703 catalog_manager.cc:5697] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 2 to 3, leader changed from 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 3 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.700289 31004 raft_consensus.cc:993] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: : Instructing follower 1e2ceabfe50c47b49529b663fa133b86 to start an election
I20260709 12:28:18.701197 30999 raft_consensus.cc:1081] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Signalling peer 1e2ceabfe50c47b49529b663fa133b86 to start an election
I20260709 12:28:18.703866 30025 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "325254299d324d0b950855283ec2fcf0"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
 from {username='slave'} at 127.0.0.1:36558
I20260709 12:28:18.704599 30025 raft_consensus.cc:493] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:18.704972 30025 raft_consensus.cc:3060] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.720021 30025 raft_consensus.cc:515] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:18.722500 30025 leader_election.cc:290] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:18.723242 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:18.724067 29878 raft_consensus.cc:3055] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:18.724383 29878 raft_consensus.cc:740] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:28:18.724702 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "325254299d324d0b950855283ec2fcf0" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:18.725418 29950 raft_consensus.cc:3060] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.725315 29878 consensus_queue.cc:260] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:18.727312 29878 raft_consensus.cc:3060] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.735467 29950 raft_consensus.cc:2468] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:18.737425 29995 leader_election.cc:304] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:18.738408 30986 raft_consensus.cc:2804] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:18.738953 30986 raft_consensus.cc:697] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:18.739970 30986 consensus_queue.cc:237] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:18.748736 30716 raft_consensus.cc:993] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.749315 30716 raft_consensus.cc:1081] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:18.751433 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:18.752136 30836 raft_consensus.cc:493] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:18.752568 30836 raft_consensus.cc:3060] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.752595 29878 raft_consensus.cc:2468] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
I20260709 12:28:18.763633 30836 raft_consensus.cc:515] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:18.763089 29703 catalog_manager.cc:5697] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 1 to 2, leader changed from b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 2 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.766727 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:18.766923 30836 leader_election.cc:290] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:18.767434 30025 raft_consensus.cc:3055] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:18.767783 30025 raft_consensus.cc:740] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:18.768700 30025 consensus_queue.cc:260] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:18.770154 30025 raft_consensus.cc:3060] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.770251 29878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a33c77154df4be8aee967d4d7df39b0" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:18.772379 29878 raft_consensus.cc:3060] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:18.779649 30025 raft_consensus.cc:2468] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:18.781257 30700 leader_election.cc:304] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:28:18.782017 29878 raft_consensus.cc:2468] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:18.782389 30990 raft_consensus.cc:2804] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:18.783095 30990 raft_consensus.cc:697] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:18.784173 30990 consensus_queue.cc:237] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:18.797104 29703 catalog_manager.cc:5697] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.967247 30025 raft_consensus.cc:1275] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:28:18.969341 31013 consensus_queue.cc:1048] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:18.993402 29947 raft_consensus.cc:1275] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:28:18.995951 30990 consensus_queue.cc:1048] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:19.048004 30985 raft_consensus.cc:993] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.048591 30986 raft_consensus.cc:1081] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.050345 30826 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:19.050884 30826 raft_consensus.cc:493] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:19.051208 30826 raft_consensus.cc:3060] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.062369 30826 raft_consensus.cc:515] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:19.065080 30826 leader_election.cc:290] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:19.065469 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:19.065675 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4bd785dff644d0a8e536827c0304894" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:19.069537 30025 raft_consensus.cc:3055] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:19.069624 29880 raft_consensus.cc:3060] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.070115 30025 raft_consensus.cc:740] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:19.071096 30025 consensus_queue.cc:260] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:19.077672 30025 raft_consensus.cc:3060] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.084133 29950 raft_consensus.cc:1275] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.086354 30990 consensus_queue.cc:1048] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:19.091303 30025 raft_consensus.cc:2468] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.094097 30700 leader_election.cc:304] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:28:19.081928 29880 raft_consensus.cc:2468] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.097229 30990 raft_consensus.cc:2804] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:19.098060 30990 raft_consensus.cc:697] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:19.099480 30990 consensus_queue.cc:237] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:19.114384 29880 raft_consensus.cc:1275] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.116240 31015 consensus_queue.cc:1048] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:19.129374 29703 catalog_manager.cc:5697] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:19.137996 29878 raft_consensus.cc:1275] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:19.147747 30990 consensus_queue.cc:1048] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:19.165083 29950 raft_consensus.cc:1275] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.167178 30986 consensus_queue.cc:1048] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:19.177323 29947 raft_consensus.cc:1275] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:19.179247 31013 consensus_queue.cc:1048] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:19.185088 30025 raft_consensus.cc:1275] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.187307 31015 consensus_queue.cc:1048] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:19.227694 29880 raft_consensus.cc:1275] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.229218 31015 consensus_queue.cc:1048] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:19.230239 29878 raft_consensus.cc:1275] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1e2ceabfe50c47b49529b663fa133b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.239892 30984 consensus_queue.cc:1048] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:19.499738 30022 raft_consensus.cc:1275] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.501379 31013 consensus_queue.cc:1048] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:19.513371 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:19.526340 29880 raft_consensus.cc:1275] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:19.528285 31046 consensus_queue.cc:1048] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:19.603024 29947 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1f944367bb734866841edc565647828b"
dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:43076
I20260709 12:28:19.603690 29947 raft_consensus.cc:606] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:19.606971 29880 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c"
dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:49664
I20260709 12:28:19.607594 29880 raft_consensus.cc:606] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:19.610736 30022 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d"
dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:55100
I20260709 12:28:19.611362 30022 raft_consensus.cc:606] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:19.614293 29880 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98"
dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
mode: GRACEFUL
new_leader_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:49664
I20260709 12:28:19.614975 29880 raft_consensus.cc:606] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Received request to transfer leadership to TS 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:19.616418 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:28:19.618120 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:19.618875 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:19.757345 30973 raft_consensus.cc:993] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.757930 31003 raft_consensus.cc:1081] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.759670 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1f944367bb734866841edc565647828b"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48788
I20260709 12:28:19.760263 30836 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:19.760667 30836 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:19.767962 30836 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:19.770134 30836 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:19.771044 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:19.771257 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f944367bb734866841edc565647828b" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:19.772009 29880 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:19.772030 29950 raft_consensus.cc:3055] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:19.772580 29950 raft_consensus.cc:740] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:19.773661 29950 consensus_queue.cc:260] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:19.775100 29950 raft_consensus.cc:3060] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:19.782361 29880 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 3.
I20260709 12:28:19.784328 30697 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6; no voters: 
I20260709 12:28:19.785303 30990 raft_consensus.cc:2804] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:19.785620 29950 raft_consensus.cc:2468] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 3.
I20260709 12:28:19.785892 30990 raft_consensus.cc:697] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 3 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:19.787282 30990 consensus_queue.cc:237] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:19.799188 29704 catalog_manager.cc:5697] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 2 to 3, leader changed from 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 3 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:19.829479 30999 raft_consensus.cc:993] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.830139 31055 raft_consensus.cc:1081] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.831943 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48806
I20260709 12:28:19.832628 30836 raft_consensus.cc:493] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:19.833034 30836 raft_consensus.cc:3060] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.840333 30836 raft_consensus.cc:515] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:19.843519 30025 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:19.843948 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adaa2559c49e4cba8e82a850b5004d98" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:19.844398 30025 raft_consensus.cc:3060] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.844627 29880 raft_consensus.cc:3055] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:19.842811 30836 leader_election.cc:290] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:19.844960 29880 raft_consensus.cc:740] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:28:19.845911 29880 consensus_queue.cc:260] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:19.846932 29880 raft_consensus.cc:3060] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.855891 30025 raft_consensus.cc:2468] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.856194 29880 raft_consensus.cc:2468] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.857712 30700 leader_election.cc:304] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86; no voters: 
I20260709 12:28:19.858765 31015 raft_consensus.cc:2804] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:19.859452 31015 raft_consensus.cc:697] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:19.860476 31015 consensus_queue.cc:237] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:19.889324 29704 catalog_manager.cc:5697] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:19.893187 31055 raft_consensus.cc:993] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.893854 30999 raft_consensus.cc:1081] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.895735 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48806
I20260709 12:28:19.896354 30836 raft_consensus.cc:493] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:19.896713 30836 raft_consensus.cc:3060] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.906481 30836 raft_consensus.cc:515] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:19.908210 30836 leader_election.cc:290] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:19.909300 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:19.909204 29880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66a7253beff14dc2a82c02db69def01c" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6"
I20260709 12:28:19.910274 29947 raft_consensus.cc:3060] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.910383 29880 raft_consensus.cc:3055] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:19.910822 29880 raft_consensus.cc:740] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b5a2c42e3a7d4dedbea2bd7d705862e6, State: Running, Role: LEADER
I20260709 12:28:19.911706 29880 consensus_queue.cc:260] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:19.912904 29880 raft_consensus.cc:3060] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.920485 29947 raft_consensus.cc:2468] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.921670 29880 raft_consensus.cc:2468] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.921947 30700 leader_election.cc:304] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:19.922940 31015 raft_consensus.cc:2804] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:19.923597 31015 raft_consensus.cc:697] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:19.924657 31015 consensus_queue.cc:237] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:19.936476 29704 catalog_manager.cc:5697] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:19.953794 30984 raft_consensus.cc:993] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: : Instructing follower 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.954440 30986 raft_consensus.cc:1081] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Signalling peer 1ae5481d76bf4a90a0ee824a14223421 to start an election
I20260709 12:28:19.959868 30836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d"
dest_uuid: "1ae5481d76bf4a90a0ee824a14223421"
 from {username='slave'} at 127.0.0.1:48800
I20260709 12:28:19.960510 30836 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:19.960897 30836 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.981956 30836 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:19.984234 30836 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:19.985496 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:19.985885 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1ae5481d76bf4a90a0ee824a14223421" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:19.986372 30026 raft_consensus.cc:3055] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:19.986701 29950 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.986826 30026 raft_consensus.cc:740] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
I20260709 12:28:19.987847 30026 consensus_queue.cc:260] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:19.989444 30026 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.996191 29950 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.996313 30026 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae5481d76bf4a90a0ee824a14223421 in term 2.
I20260709 12:28:19.997530 30700 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:19.998376 31015 raft_consensus.cc:2804] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:19.998838 31015 raft_consensus.cc:697] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae5481d76bf4a90a0ee824a14223421, State: Running, Role: LEADER
I20260709 12:28:19.999842 31015 consensus_queue.cc:237] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:20.013406 29704 catalog_manager.cc:5697] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 reported cstate change: term changed from 1 to 2, leader changed from 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4) to 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "1ae5481d76bf4a90a0ee824a14223421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:20.214819 29880 raft_consensus.cc:1275] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:20.216317 31015 consensus_queue.cc:1048] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:20.233321 29950 raft_consensus.cc:1275] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:20.243305 31013 consensus_queue.cc:1048] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:20.276069 30025 raft_consensus.cc:1275] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.277676 31013 consensus_queue.cc:1048] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:20.307037 29880 raft_consensus.cc:1275] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.309051 31013 consensus_queue.cc:1048] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:20.384572 29947 raft_consensus.cc:1275] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.386055 30990 consensus_queue.cc:1048] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:20.401844 30025 raft_consensus.cc:1275] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.407977 30990 consensus_queue.cc:1048] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:20.423100 29950 raft_consensus.cc:1275] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.430207 31015 consensus_queue.cc:1048] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:20.457705 29880 raft_consensus.cc:1275] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Refusing update from remote peer 1ae5481d76bf4a90a0ee824a14223421: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.459370 31013 consensus_queue.cc:1048] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:20.700721 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:28:20.815608 29918 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:35595, user_credentials={real_user=slave}} blocked reactor thread for 57334.1us
I20260709 12:28:20.972999 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:20.974714 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:20.975554 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:21.031536 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:21.076975 16792 tablet_server.cc:179] TabletServer@127.16.102.1:42003 shutting down...
W20260709 12:28:21.100564 29920 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 365 similar messages]
W20260709 12:28:21.108021 29920 consensus_peers.cc:597] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.116763 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:21.118391 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.120174 16792 raft_consensus.cc:2243] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.122431 16792 raft_consensus.cc:2272] T 9ad7f32688184e219c1d203375c23682 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.127946 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.129362 16792 raft_consensus.cc:2243] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.130290 16792 raft_consensus.cc:2272] T abceb9531803436f8fd8d02595d73cf4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.130946 29994 consensus_peers.cc:597] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.135126 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.136343 16792 raft_consensus.cc:2243] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.137879 16792 raft_consensus.cc:2272] T 997e1531077444f3b82718c8b75b357e P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.142493 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.143486 16792 raft_consensus.cc:2243] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.144904 16792 raft_consensus.cc:2272] T 889b2ffc0b554cbea97dc55e0aa95094 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.148294 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.149582 16792 raft_consensus.cc:2243] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.150450 16792 raft_consensus.cc:2272] T 444cae907f1d4bdab292e8a3f59f7f11 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.152010 29994 consensus_peers.cc:597] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.154803 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.156224 16792 raft_consensus.cc:2243] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.157845 16792 raft_consensus.cc:2272] T b820ed22868a4fe98bd2c74221e3e300 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.162317 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.163434 16792 raft_consensus.cc:2243] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.164772 16792 raft_consensus.cc:2272] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.171257 29994 consensus_peers.cc:597] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.174682 29994 consensus_peers.cc:597] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.181434 29846 consensus_peers.cc:597] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.181928 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.183043 16792 raft_consensus.cc:2243] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.183724 16792 raft_consensus.cc:2272] T 97428fa3f86d463fa4d309cffb192127 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.187727 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.188704 16792 raft_consensus.cc:2243] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 LEADER]: Raft consensus shutting down.
W20260709 12:28:21.189229 29994 consensus_peers.cc:597] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.190065 16792 raft_consensus.cc:2272] T 1b5584cc51f14a9a8dcee35a9998b380 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.196727 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.197940 16792 raft_consensus.cc:2243] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.198686 16792 raft_consensus.cc:2272] T d8ada01658a746029b8fd58389f608a4 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.201643 29994 consensus_peers.cc:597] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.203588 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.204778 16792 raft_consensus.cc:2243] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.206225 16792 raft_consensus.cc:2272] T ff934fd9fe834c2ab8737af23cdc9505 P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.210655 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.211869 16792 raft_consensus.cc:2243] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.212850 16792 raft_consensus.cc:2272] T f8c4c5d911ee4bcbbfb014114223cc07 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.219367 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.220479 16792 raft_consensus.cc:2243] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.223887 16792 raft_consensus.cc:2272] T a4bd785dff644d0a8e536827c0304894 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.226919 29920 consensus_peers.cc:597] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.228688 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.229803 16792 raft_consensus.cc:2243] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.230399 16792 raft_consensus.cc:2272] T 80a7001c5f324ba1955a1eef4ea15e93 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.234254 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.235152 16792 raft_consensus.cc:2243] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.236392 16792 raft_consensus.cc:2272] T 66a7253beff14dc2a82c02db69def01c P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.241837 16792 tablet_replica.cc:333] stopping tablet replica
W20260709 12:28:21.242620 29920 consensus_peers.cc:597] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.243063 16792 raft_consensus.cc:2243] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.243912 16792 raft_consensus.cc:2272] T b914c950a335404b8369b7d5868ad408 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.248183 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.249223 16792 raft_consensus.cc:2243] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.249924 16792 raft_consensus.cc:2272] T 6872c9bba3c34283ac8c1a5a72884f7d P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.254179 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.255105 16792 raft_consensus.cc:2243] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.256523 16792 raft_consensus.cc:2272] T 259a887eac9c4ed0b816a4ac0bc2c536 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.260598 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.261719 16792 raft_consensus.cc:2243] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.263347 16792 raft_consensus.cc:2272] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.268879 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.270208 16792 raft_consensus.cc:2243] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.271682 16792 raft_consensus.cc:2272] T adaa2559c49e4cba8e82a850b5004d98 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.277987 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.279121 16792 raft_consensus.cc:2243] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.280392 16792 raft_consensus.cc:2272] T 645003ea84b04f4a9e6d95a0bca63384 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.283955 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.284973 16792 raft_consensus.cc:2243] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.285653 16792 raft_consensus.cc:2272] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.289471 29846 consensus_peers.cc:597] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.289783 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.291039 16792 raft_consensus.cc:2243] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.291741 16792 raft_consensus.cc:2272] T aaee6416f659496fb39a11ee1ae00ca3 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.296105 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.297209 16792 raft_consensus.cc:2243] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.297971 16792 raft_consensus.cc:2272] T ad89089259844673bbd8eeef685bb362 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.301440 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.302600 16792 raft_consensus.cc:2243] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.303426 16792 raft_consensus.cc:2272] T 4c799e9531994c7cba86a172713405e8 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.308152 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.309194 16792 raft_consensus.cc:2243] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.310070 16792 raft_consensus.cc:2272] T 268205acac824554b94b834e990e1ed4 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.310932 29846 consensus_peers.cc:597] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.314019 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.314998 16792 raft_consensus.cc:2243] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.315543 16792 raft_consensus.cc:2272] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.319139 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.319938 16792 raft_consensus.cc:2243] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.320662 16792 raft_consensus.cc:2272] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.325184 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.326303 16792 raft_consensus.cc:2243] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.326992 16792 raft_consensus.cc:2272] T 80eac7cb175d4f0d84a00bd792dfd476 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.328152 29920 consensus_peers.cc:597] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.330297 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.331261 16792 raft_consensus.cc:2243] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.331885 16792 raft_consensus.cc:2272] T e047383bb70448908508e00810ebff84 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.334867 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.335788 16792 raft_consensus.cc:2243] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.336316 16792 raft_consensus.cc:2272] T b9fd2b0fbb1d49da957faacba2d1b38f P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.339507 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.340276 16792 raft_consensus.cc:2243] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.340955 16792 raft_consensus.cc:2272] T 6a71487088564b8489c63edd9d182c5a P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.344081 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.345062 16792 raft_consensus.cc:2243] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.345608 16792 raft_consensus.cc:2272] T 3dafcab3681f4882a86829813cbc7772 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.351140 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.351972 16792 raft_consensus.cc:2243] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.352480 16792 raft_consensus.cc:2272] T e6fa078c85764bb8b88bc6afff3c5bda P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.355888 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.356834 16792 raft_consensus.cc:2243] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.357439 16792 raft_consensus.cc:2272] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.358839 29846 consensus_peers.cc:597] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.361414 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.362576 16792 raft_consensus.cc:2243] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:21.362603 29846 consensus_peers.cc:597] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.363530 16792 raft_consensus.cc:2272] T d4a3b54cd9af44d69698e15cc4bccafa P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.366036 29846 consensus_peers.cc:597] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.367374 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.368196 16792 raft_consensus.cc:2243] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.368781 16792 raft_consensus.cc:2272] T dd88542c59c449a0904ad8b002efc5a0 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.372762 16792 tablet_replica.cc:333] stopping tablet replica
W20260709 12:28:21.373311 29994 consensus_peers.cc:597] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.373494 16792 raft_consensus.cc:2243] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.374191 16792 raft_consensus.cc:2272] T c08a657511124ab1ad80a227eeec2736 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.377043 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.377868 16792 raft_consensus.cc:2243] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.379009 16792 raft_consensus.cc:2272] T 4a33c77154df4be8aee967d4d7df39b0 P 1ae5481d76bf4a90a0ee824a14223421 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.382277 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.383095 16792 raft_consensus.cc:2243] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.383674 16792 raft_consensus.cc:2272] T b2c89d9015e54780811a19203939e9a5 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.387306 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.388037 16792 raft_consensus.cc:2243] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.388593 16792 raft_consensus.cc:2272] T ffc3696cf98a402bb9d717b584cb229c P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.392262 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.393316 16792 raft_consensus.cc:2243] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.394143 16792 raft_consensus.cc:2272] T 4f1befeab89b48729c8ba035e0ef71a8 P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.397102 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.397802 16792 raft_consensus.cc:2243] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.398926 16792 raft_consensus.cc:2272] T 1f944367bb734866841edc565647828b P 1ae5481d76bf4a90a0ee824a14223421 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.402487 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.403407 16792 raft_consensus.cc:2243] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.404247 16792 raft_consensus.cc:2272] T ba9fb933bf47466999dfd7892e6a6638 P 1ae5481d76bf4a90a0ee824a14223421 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.404995 29920 consensus_peers.cc:597] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.407274 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:21.407981 16792 raft_consensus.cc:2243] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.408509 16792 raft_consensus.cc:2272] T d49dfe78f7164655b82bfc664fbd11dd P 1ae5481d76bf4a90a0ee824a14223421 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.430522 29846 consensus_peers.cc:597] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.447688 29846 consensus_peers.cc:597] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.454423 29846 consensus_peers.cc:597] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.491688 29994 consensus_peers.cc:597] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.501662 29920 consensus_peers.cc:597] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.505029 30698 connection.cc:466] server connection from 127.0.0.1:48800 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48800 (request call id 516) could send its response
I20260709 12:28:21.530328 16792 tablet_server.cc:196] TabletServer@127.16.102.1:42003 shutdown complete.
W20260709 12:28:21.555045 29846 consensus_peers.cc:597] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.557189 29994 consensus_peers.cc:597] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.561766 29994 consensus_peers.cc:597] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.640232 29994 consensus_peers.cc:597] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.674476 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
W20260709 12:28:21.687278 29992 consensus_peers.cc:597] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.692880 29992 consensus_peers.cc:597] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.790066 29992 consensus_peers.cc:597] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.790922 29992 consensus_peers.cc:597] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.807796 29918 consensus_peers.cc:597] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.829609 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:21.830960 16792 tablet_replica.cc:333] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.832266 16792 raft_consensus.cc:2243] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.833123 16792 raft_consensus.cc:2272] T d5988ddbc0f84d47b9b94c1375502692 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.835748 16792 tablet_replica.cc:333] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.836460 16792 raft_consensus.cc:2243] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.837841 16792 raft_consensus.cc:2272] T e6fa078c85764bb8b88bc6afff3c5bda P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.841547 16792 tablet_replica.cc:333] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.842433 16792 raft_consensus.cc:2243] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.843151 16792 raft_consensus.cc:2272] T 811ccdd2daa94877be1d910bd3745085 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.846180 16792 tablet_replica.cc:333] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.847100 16792 raft_consensus.cc:2243] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.848289 16792 raft_consensus.cc:2272] T 35000da0d1054109927271f881d5b697 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.850925 16792 tablet_replica.cc:333] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.851606 16792 raft_consensus.cc:2243] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.852382 16792 raft_consensus.cc:2272] T 889b2ffc0b554cbea97dc55e0aa95094 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.855736 16792 tablet_replica.cc:333] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.856772 16792 raft_consensus.cc:2243] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:21.856631 29992 consensus_peers.cc:597] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:21.856642 29918 consensus_peers.cc:597] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.857676 16792 raft_consensus.cc:2272] T 71ae82107a97483ba4fa78a788adb334 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.860320 16792 tablet_replica.cc:333] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.861052 16792 raft_consensus.cc:2243] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.862139 16792 raft_consensus.cc:2272] T b9fd2b0fbb1d49da957faacba2d1b38f P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.864616 16792 tablet_replica.cc:333] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.865332 16792 raft_consensus.cc:2243] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.865895 16792 raft_consensus.cc:2272] T 3dafcab3681f4882a86829813cbc7772 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.868809 16792 tablet_replica.cc:333] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.869621 16792 raft_consensus.cc:2243] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.870900 16792 raft_consensus.cc:2272] T 42b6f3a05b40452e9c171932ca603b19 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.873579 16792 tablet_replica.cc:333] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.874365 16792 raft_consensus.cc:2243] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.875492 16792 raft_consensus.cc:2272] T 9ad7f32688184e219c1d203375c23682 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.878187 16792 tablet_replica.cc:333] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.878964 16792 raft_consensus.cc:2243] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.879536 16792 raft_consensus.cc:2272] T cefd124c3a724d06897bd37636aeb5b6 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.882467 16792 tablet_replica.cc:333] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.883343 16792 raft_consensus.cc:2243] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.884268 16792 raft_consensus.cc:2272] T aaee6416f659496fb39a11ee1ae00ca3 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.887691 16792 tablet_replica.cc:333] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.888849 16792 raft_consensus.cc:2243] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.889904 16792 raft_consensus.cc:2272] T 4c799e9531994c7cba86a172713405e8 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.892807 16792 tablet_replica.cc:333] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.893582 16792 raft_consensus.cc:2243] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.894203 16792 raft_consensus.cc:2272] T b914c950a335404b8369b7d5868ad408 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.896998 16792 tablet_replica.cc:333] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.898089 16792 raft_consensus.cc:2243] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.898979 16792 raft_consensus.cc:2272] T 4a33c77154df4be8aee967d4d7df39b0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.902223 16792 tablet_replica.cc:333] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.903012 16792 raft_consensus.cc:2243] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.904021 16792 raft_consensus.cc:2272] T b2c89d9015e54780811a19203939e9a5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.907177 16792 tablet_replica.cc:333] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.907963 16792 raft_consensus.cc:2243] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.908905 16792 raft_consensus.cc:2272] T c08a657511124ab1ad80a227eeec2736 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.911912 16792 tablet_replica.cc:333] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.912621 16792 raft_consensus.cc:2243] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.913208 16792 raft_consensus.cc:2272] T 15ead15e87d345f48f74ca65fb97d5c5 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.916275 16792 tablet_replica.cc:333] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.917297 16792 raft_consensus.cc:2243] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.918684 16792 raft_consensus.cc:2272] T 57229c57228146d6ab456a83f82ef167 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.921685 29918 consensus_peers.cc:597] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.922606 16792 tablet_replica.cc:333] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.923965 16792 raft_consensus.cc:2243] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.924779 16792 raft_consensus.cc:2272] T 742ffdad627c4e71865fe5b85509976d P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.928956 16792 tablet_replica.cc:333] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.930142 16792 raft_consensus.cc:2243] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.930846 16792 raft_consensus.cc:2272] T ffc3696cf98a402bb9d717b584cb229c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.934355 29918 consensus_peers.cc:597] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.935122 16792 tablet_replica.cc:333] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.936142 16792 raft_consensus.cc:2243] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.936704 16792 raft_consensus.cc:2272] T 611c12107cde4b0e81c03ffd67cb64e2 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.939301 16792 tablet_replica.cc:333] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.940083 16792 raft_consensus.cc:2243] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.940712 16792 raft_consensus.cc:2272] T 66a7253beff14dc2a82c02db69def01c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.943387 16792 tablet_replica.cc:333] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.944106 16792 raft_consensus.cc:2243] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.944756 16792 raft_consensus.cc:2272] T f8c4c5d911ee4bcbbfb014114223cc07 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.948067 16792 tablet_replica.cc:333] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.948945 16792 raft_consensus.cc:2243] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:21.949569 29918 consensus_peers.cc:597] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.949682 16792 raft_consensus.cc:2272] T a4bd785dff644d0a8e536827c0304894 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.952569 16792 tablet_replica.cc:333] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.953343 16792 raft_consensus.cc:2243] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.954278 16792 raft_consensus.cc:2272] T 80a7001c5f324ba1955a1eef4ea15e93 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.956887 16792 tablet_replica.cc:333] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.957578 16792 raft_consensus.cc:2243] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.958132 16792 raft_consensus.cc:2272] T 34b3b0e1b5d14def8b90e20a7a38ee24 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.960667 16792 tablet_replica.cc:333] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.961366 16792 raft_consensus.cc:2243] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.961891 16792 raft_consensus.cc:2272] T 7a554edb2449436ea89ed389c4aed651 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.964766 16792 tablet_replica.cc:333] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.965718 16792 raft_consensus.cc:2243] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.966526 16792 raft_consensus.cc:2272] T abceb9531803436f8fd8d02595d73cf4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.969614 16792 tablet_replica.cc:333] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.970392 16792 raft_consensus.cc:2243] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.971009 16792 raft_consensus.cc:2272] T 259a887eac9c4ed0b816a4ac0bc2c536 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.974074 16792 tablet_replica.cc:333] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.974961 16792 raft_consensus.cc:2243] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.975625 16792 raft_consensus.cc:2272] T adaa2559c49e4cba8e82a850b5004d98 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.979208 16792 tablet_replica.cc:333] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.980228 16792 raft_consensus.cc:2243] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.981568 16792 raft_consensus.cc:2272] T 325254299d324d0b950855283ec2fcf0 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.984791 16792 tablet_replica.cc:333] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.985733 16792 raft_consensus.cc:2243] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:21.986840 16792 raft_consensus.cc:2272] T 01f4a9dd66ad4df0a8f5a598a7b432fd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:21.990491 16792 tablet_replica.cc:333] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.991395 16792 raft_consensus.cc:2243] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.992074 16792 raft_consensus.cc:2272] T d8ada01658a746029b8fd58389f608a4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:21.995719 29918 consensus_peers.cc:597] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:21.995611 16792 tablet_replica.cc:333] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:21.996884 16792 raft_consensus.cc:2243] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:21.997512 16792 raft_consensus.cc:2272] T ff934fd9fe834c2ab8737af23cdc9505 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.000495 16792 tablet_replica.cc:333] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.001451 16792 raft_consensus.cc:2243] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.002264 16792 raft_consensus.cc:2272] T 444cae907f1d4bdab292e8a3f59f7f11 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.005929 16792 tablet_replica.cc:333] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.006624 16792 raft_consensus.cc:2243] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.007298 16792 raft_consensus.cc:2272] T b820ed22868a4fe98bd2c74221e3e300 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.010532 16792 tablet_replica.cc:333] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.011397 16792 raft_consensus.cc:2243] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.012533 16792 raft_consensus.cc:2272] T d49dfe78f7164655b82bfc664fbd11dd P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:22.012368 29918 consensus_peers.cc:597] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:22.015894 16792 tablet_replica.cc:333] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.016655 16792 raft_consensus.cc:2243] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.017491 16792 raft_consensus.cc:2272] T 1f944367bb734866841edc565647828b P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:22.018103 29846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.020745 16792 tablet_replica.cc:333] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.021494 16792 raft_consensus.cc:2243] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.022001 16792 raft_consensus.cc:2272] T 4c3b379d58464fa6b60b8a33472252d4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.024507 16792 tablet_replica.cc:333] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.025146 16792 raft_consensus.cc:2243] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.025964 16792 raft_consensus.cc:2272] T 80eac7cb175d4f0d84a00bd792dfd476 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.028812 16792 tablet_replica.cc:333] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.029539 16792 raft_consensus.cc:2243] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.030480 16792 raft_consensus.cc:2272] T 2f31657dfccf4d45ab006ef5aadcfb4c P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:22.031554 29918 consensus_peers.cc:597] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:22.033445 16792 tablet_replica.cc:333] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.034157 16792 raft_consensus.cc:2243] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.034824 16792 raft_consensus.cc:2272] T 268205acac824554b94b834e990e1ed4 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.037676 16792 tablet_replica.cc:333] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6: stopping tablet replica
I20260709 12:28:22.038283 16792 raft_consensus.cc:2243] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:22.038684 29918 consensus_peers.cc:597] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:22.038867 16792 raft_consensus.cc:2272] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P b5a2c42e3a7d4dedbea2bd7d705862e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:22.046034 29918 consensus_peers.cc:597] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.072955 29992 consensus_peers.cc:597] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.082198 29846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:22.139755 29846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:22.142865 29992 consensus_peers.cc:597] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.160707 29992 consensus_peers.cc:597] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:22.207173 31094 raft_consensus.cc:493] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.207948 31094 raft_consensus.cc:515] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
W20260709 12:28:22.210669 29844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.210723 31094 leader_election.cc:290] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:22.214248 29918 leader_election.cc:336] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:22.235316 29992 consensus_peers.cc:597] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.242275 29918 consensus_peers.cc:597] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Couldn't send request to peer b5a2c42e3a7d4dedbea2bd7d705862e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:22.263432 30986 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.264055 30986 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.265895 30986 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.267114 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:28:22.270618 29994 leader_election.cc:336] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.271057 29994 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:22.271761 30986 raft_consensus.cc:2749] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:28:22.312239 29845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.341379 30986 raft_consensus.cc:493] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.342034 30986 raft_consensus.cc:515] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:22.344424 30986 leader_election.cc:290] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:22.356081 30986 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
W20260709 12:28:22.357000 29844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:22.366261 29844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.367928 31094 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.368633 31094 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.372144 31094 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:22.358520 29992 leader_election.cc:336] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:28:22.380313 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:22.356699 30986 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.385581 30986 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
W20260709 12:28:22.386833 29994 leader_election.cc:336] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.387734 29994 leader_election.cc:304] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.389662 30986 raft_consensus.cc:2749] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:22.390267 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:22.390590 30986 raft_consensus.cc:493] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.393217 30986 raft_consensus.cc:515] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:28:22.394480 29845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.397933 30986 leader_election.cc:290] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:22.401965 29992 leader_election.cc:336] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:22.402513 29994 leader_election.cc:336] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.403123 29994 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:22.403970 30986 raft_consensus.cc:2749] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:22.410187 29994 leader_election.cc:336] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.410705 29994 leader_election.cc:304] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.411653 31080 raft_consensus.cc:2749] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:22.420084 30986 raft_consensus.cc:493] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.420771 30986 raft_consensus.cc:515] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:22.423280 30986 leader_election.cc:290] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:22.426753 29992 leader_election.cc:336] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:22.427789 29994 leader_election.cc:336] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.428225 29994 leader_election.cc:304] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.429031 30986 raft_consensus.cc:2749] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:22.462450 31094 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.463124 31094 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.465126 31094 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.466045 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:22.499465 31094 raft_consensus.cc:493] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.499982 31094 raft_consensus.cc:515] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:22.502149 31094 leader_election.cc:290] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:22.505360 29918 leader_election.cc:336] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:28:22.505415 31094 raft_consensus.cc:493] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.506368 31094 raft_consensus.cc:515] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:22.508062 30986 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.508733 30986 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.509805 31094 leader_election.cc:290] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:22.514665 29918 leader_election.cc:336] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:28:22.514791 31098 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.515564 31098 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.516341 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:22.517091 29947 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 4.
I20260709 12:28:22.514972 30986 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.519797 31098 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.519922 29995 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.521215 31094 raft_consensus.cc:493] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.521417 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
I20260709 12:28:22.522032 31098 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:28:22.522154 29947 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 2.
W20260709 12:28:22.522315 29994 leader_election.cc:336] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.522663 31098 raft_consensus.cc:493] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.523252 31098 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:22.521924 31094 raft_consensus.cc:515] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:22.525053 29995 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.525552 31094 leader_election.cc:290] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:22.526098 30986 raft_consensus.cc:2804] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:22.526563 30986 raft_consensus.cc:493] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.527024 30986 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:22.528900 29844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:22.529829 29918 leader_election.cc:336] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:22.530745 29846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.538507 31101 raft_consensus.cc:493] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.539244 31101 raft_consensus.cc:515] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
W20260709 12:28:22.543035 29994 leader_election.cc:336] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.542663 30986 raft_consensus.cc:515] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.542554 31098 raft_consensus.cc:515] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.549412 30986 leader_election.cc:290] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.549459 31098 leader_election.cc:290] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 5 election: Requested vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.553177 29947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ca5a52b63dd4c46bb69db5b73b48b3d" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:22.554270 29947 raft_consensus.cc:3060] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:22.558449 31101 leader_election.cc:290] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:22.563599 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "997e1531077444f3b82718c8b75b357e" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac"
I20260709 12:28:22.564517 29950 raft_consensus.cc:3060] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:28:22.571688 29846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.571914 29947 raft_consensus.cc:2468] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 3.
W20260709 12:28:22.577845 29992 leader_election.cc:336] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:22.578778 29920 consensus_peers.cc:597] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.579648 29920 consensus_peers.cc:597] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.579973 29994 leader_election.cc:336] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:22.580631 29920 leader_election.cc:336] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.581190 29920 leader_election.cc:304] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
W20260709 12:28:22.580680 29994 leader_election.cc:336] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.582198 29995 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:22.582533 31094 raft_consensus.cc:2749] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:28:22.582954 29994 leader_election.cc:336] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.583278 31101 raft_consensus.cc:2804] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:22.583568 29994 leader_election.cc:304] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.583854 31101 raft_consensus.cc:697] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
W20260709 12:28:22.584081 29920 leader_election.cc:336] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.584704 29920 leader_election.cc:304] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:22.584733 31102 raft_consensus.cc:2749] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:22.585005 31101 consensus_queue.cc:237] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.586207 31094 raft_consensus.cc:2749] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:22.587706 29920 leader_election.cc:336] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.588279 29920 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, 1e2ceabfe50c47b49529b663fa133b86
I20260709 12:28:22.588698 29950 raft_consensus.cc:2468] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2ceabfe50c47b49529b663fa133b86 in term 5.
I20260709 12:28:22.590137 31093 raft_consensus.cc:2749] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:28:22.591212 29995 leader_election.cc:304] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:22.594883 31098 raft_consensus.cc:2804] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:28:22.595665 31098 raft_consensus.cc:697] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 5 LEADER]: Becoming Leader. State: Replica: 1e2ceabfe50c47b49529b663fa133b86, State: Running, Role: LEADER
W20260709 12:28:22.597222 29920 leader_election.cc:336] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.596953 31098 consensus_queue.cc:237] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.597784 29920 leader_election.cc:304] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.598660 31093 raft_consensus.cc:2749] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:22.598918 29920 leader_election.cc:336] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.599462 29920 leader_election.cc:304] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.600212 31093 raft_consensus.cc:2749] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:22.601266 29920 leader_election.cc:336] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.601683 29920 leader_election.cc:304] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.602416 31093 raft_consensus.cc:2749] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:22.604084 29704 catalog_manager.cc:5697] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 2 to 3, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 3 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:22.630492 31093 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.631259 31093 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.633886 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
I20260709 12:28:22.634271 31093 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.635164 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:22.635939 30026 raft_consensus.cc:2468] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:22.639074 29921 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.640113 31094 raft_consensus.cc:2804] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:22.640539 31094 raft_consensus.cc:493] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.642539 31094 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:22.645470 31093 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.646176 31093 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:22.647459 29920 leader_election.cc:336] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.648509 31093 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.655151 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:22.656222 30026 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:22.666206 29921 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.667678 31093 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:28:22.661513 31094 raft_consensus.cc:515] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.668154 31093 raft_consensus.cc:493] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.669868 31093 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:22.670933 31094 leader_election.cc:290] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.672813 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4b038ac5e344bf96cf8c672adb2dbb" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:22.672945 31102 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.673797 30026 raft_consensus.cc:3060] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:22.674182 31102 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.680194 31102 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.688756 29950 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "1e2ceabfe50c47b49529b663fa133b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" is_pre_election: true
W20260709 12:28:22.690505 29994 leader_election.cc:336] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.695040 29995 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, 7df42fd4269f4d16aeebecaa4ea958ac
I20260709 12:28:22.696267 31101 raft_consensus.cc:2749] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:22.698451 29920 leader_election.cc:336] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.713152 30026 raft_consensus.cc:2468] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:22.714937 29921 leader_election.cc:304] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.709630 31093 raft_consensus.cc:515] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.715965 31099 raft_consensus.cc:2804] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:22.716545 31099 raft_consensus.cc:697] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:22.718075 31093 leader_election.cc:290] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.717675 31099 consensus_queue.cc:237] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.719808 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b5584cc51f14a9a8dcee35a9998b380" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:22.720530 30026 raft_consensus.cc:3060] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:28:22.725338 29920 leader_election.cc:336] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.730799 30026 raft_consensus.cc:2468] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 4.
I20260709 12:28:22.732372 29921 leader_election.cc:304] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.733943 29704 catalog_manager.cc:5697] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 reported cstate change: term changed from 4 to 5, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4). New cstate: current_term: 5 leader_uuid: "1e2ceabfe50c47b49529b663fa133b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:22.738076 31083 raft_consensus.cc:2804] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:22.739039 31083 raft_consensus.cc:697] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:22.740530 31083 consensus_queue.cc:237] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:22.755537 29920 leader_election.cc:336] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.773343 29703 catalog_manager.cc:5697] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 2 to 3, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 3 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:22.775404 31083 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.776132 31083 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.779768 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:22.780465 31083 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.780596 30026 raft_consensus.cc:2468] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 2.
I20260709 12:28:22.783532 29921 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 
I20260709 12:28:22.790484 31094 raft_consensus.cc:2804] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:28:22.791033 31094 raft_consensus.cc:493] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.791501 31094 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:22.795711 29920 leader_election.cc:336] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.802870 31094 raft_consensus.cc:515] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
I20260709 12:28:22.806440 31094 leader_election.cc:290] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Requested vote from peers 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301), 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003)
I20260709 12:28:22.806747 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "645003ea84b04f4a9e6d95a0bca63384" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86"
I20260709 12:28:22.807585 30026 raft_consensus.cc:3060] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:22.813316 29703 catalog_manager.cc:5697] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 3 to 4, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 4 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:22.815395 29920 leader_election.cc:336] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.824098 30026 raft_consensus.cc:2468] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df42fd4269f4d16aeebecaa4ea958ac in term 3.
I20260709 12:28:22.826071 29921 leader_election.cc:304] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86, 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421
I20260709 12:28:22.826450 31094 raft_consensus.cc:493] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:22.827273 31094 raft_consensus.cc:515] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:22.827653 31083 raft_consensus.cc:493] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1ae5481d76bf4a90a0ee824a14223421)
I20260709 12:28:22.828353 31083 raft_consensus.cc:515] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } }
I20260709 12:28:22.830086 31094 leader_election.cc:290] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 1e2ceabfe50c47b49529b663fa133b86 (127.16.102.4:40301)
I20260709 12:28:22.831702 31083 leader_election.cc:290] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:22.836885 31094 raft_consensus.cc:2804] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:22.837580 31094 raft_consensus.cc:697] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Becoming Leader. State: Replica: 7df42fd4269f4d16aeebecaa4ea958ac, State: Running, Role: LEADER
I20260709 12:28:22.838774 31094 consensus_queue.cc:237] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } }
W20260709 12:28:22.840865 29918 leader_election.cc:336] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
W20260709 12:28:22.841521 29918 leader_election.cc:336] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:22.843024 30026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42b6f3a05b40452e9c171932ca603b19" candidate_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e2ceabfe50c47b49529b663fa133b86" is_pre_election: true
I20260709 12:28:22.844872 29921 leader_election.cc:304] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1e2ceabfe50c47b49529b663fa133b86, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.846459 31083 raft_consensus.cc:2749] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:22.848493 29920 leader_election.cc:336] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:22.848956 29920 leader_election.cc:304] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7df42fd4269f4d16aeebecaa4ea958ac; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:22.849701 31094 raft_consensus.cc:2749] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:28:22.852876 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
I20260709 12:28:22.859872 29703 catalog_manager.cc:5697] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac reported cstate change: term changed from 2 to 3, leader changed from 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1) to 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3). New cstate: current_term: 3 leader_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:22.928103 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:22.929154 16792 tablet_replica.cc:333] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.930212 16792 raft_consensus.cc:2243] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.930910 16792 raft_consensus.cc:2272] T 997e1531077444f3b82718c8b75b357e P 7df42fd4269f4d16aeebecaa4ea958ac [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.933810 16792 tablet_replica.cc:333] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.934654 16792 raft_consensus.cc:2243] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.935222 16792 raft_consensus.cc:2272] T e6fa078c85764bb8b88bc6afff3c5bda P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.938488 16792 tablet_replica.cc:333] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.939528 16792 raft_consensus.cc:2243] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.940208 16792 raft_consensus.cc:2272] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.943645 16792 tablet_replica.cc:333] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.944442 16792 raft_consensus.cc:2243] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.945461 16792 raft_consensus.cc:2272] T 811ccdd2daa94877be1d910bd3745085 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.948643 16792 tablet_replica.cc:333] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.949406 16792 raft_consensus.cc:2243] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.950017 16792 raft_consensus.cc:2272] T 35000da0d1054109927271f881d5b697 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.952729 16792 tablet_replica.cc:333] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.953532 16792 raft_consensus.cc:2243] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.954212 16792 raft_consensus.cc:2272] T d5988ddbc0f84d47b9b94c1375502692 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.957585 16792 tablet_replica.cc:333] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.958806 16792 raft_consensus.cc:2243] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.959511 16792 raft_consensus.cc:2272] T 97428fa3f86d463fa4d309cffb192127 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:22.961094 29995 consensus_peers.cc:597] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:22.962625 16792 tablet_replica.cc:333] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.963554 16792 raft_consensus.cc:2243] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.964176 16792 raft_consensus.cc:2272] T 71ae82107a97483ba4fa78a788adb334 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.967011 16792 tablet_replica.cc:333] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.968266 16792 raft_consensus.cc:2243] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.968901 16792 raft_consensus.cc:2272] T 9ad7f32688184e219c1d203375c23682 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.972404 16792 tablet_replica.cc:333] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.973464 16792 raft_consensus.cc:2243] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.974813 16792 raft_consensus.cc:2272] T cefd124c3a724d06897bd37636aeb5b6 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.978256 16792 tablet_replica.cc:333] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.979028 16792 raft_consensus.cc:2243] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.979588 16792 raft_consensus.cc:2272] T 6a71487088564b8489c63edd9d182c5a P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.982122 16792 tablet_replica.cc:333] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.982853 16792 raft_consensus.cc:2243] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.983366 16792 raft_consensus.cc:2272] T 42b6f3a05b40452e9c171932ca603b19 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.985742 16792 tablet_replica.cc:333] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.986624 16792 raft_consensus.cc:2243] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.987596 16792 raft_consensus.cc:2272] T e047383bb70448908508e00810ebff84 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.991348 16792 tablet_replica.cc:333] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.992344 16792 raft_consensus.cc:2243] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:22.993541 16792 raft_consensus.cc:2272] T 3dafcab3681f4882a86829813cbc7772 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:22.996341 16792 tablet_replica.cc:333] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:22.997097 16792 raft_consensus.cc:2243] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:22.997598 16792 raft_consensus.cc:2272] T 6872c9bba3c34283ac8c1a5a72884f7d P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.000221 16792 tablet_replica.cc:333] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.001000 16792 raft_consensus.cc:2243] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.002076 16792 raft_consensus.cc:2272] T b914c950a335404b8369b7d5868ad408 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.004875 16792 tablet_replica.cc:333] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.006119 16792 raft_consensus.cc:2243] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.007468 16792 raft_consensus.cc:2272] T 742ffdad627c4e71865fe5b85509976d P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:23.009783 29994 consensus_peers.cc:597] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.010905 16792 tablet_replica.cc:333] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.011727 16792 raft_consensus.cc:2243] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.012552 16792 pending_rounds.cc:70] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: Trying to abort 1 pending ops.
I20260709 12:28:23.012828 16792 pending_rounds.cc:77] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305626020752969728 op_type: NO_OP noop_request { }
I20260709 12:28:23.013206 16792 raft_consensus.cc:2889] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:23.013456 16792 raft_consensus.cc:2272] T 3a4b038ac5e344bf96cf8c672adb2dbb P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.016281 16792 tablet_replica.cc:333] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.016949 16792 raft_consensus.cc:2243] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.017434 16792 raft_consensus.cc:2272] T 259a887eac9c4ed0b816a4ac0bc2c536 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.019855 16792 tablet_replica.cc:333] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.020493 16792 raft_consensus.cc:2243] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.021234 16792 pending_rounds.cc:70] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: Trying to abort 1 pending ops.
I20260709 12:28:23.021421 16792 pending_rounds.cc:77] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305626021238587392 op_type: NO_OP noop_request { }
I20260709 12:28:23.021672 16792 raft_consensus.cc:2889] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:23.022014 16792 raft_consensus.cc:2272] T 645003ea84b04f4a9e6d95a0bca63384 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.024907 16792 tablet_replica.cc:333] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.025573 16792 raft_consensus.cc:2243] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.026151 16792 raft_consensus.cc:2272] T 325254299d324d0b950855283ec2fcf0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.028501 16792 tablet_replica.cc:333] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.029134 16792 raft_consensus.cc:2243] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.029541 16792 raft_consensus.cc:2272] T 4f1befeab89b48729c8ba035e0ef71a8 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.031955 16792 tablet_replica.cc:333] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.032595 16792 raft_consensus.cc:2243] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.032634 30986 raft_consensus.cc:493] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.033187 16792 raft_consensus.cc:2272] T d49dfe78f7164655b82bfc664fbd11dd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.033259 30986 raft_consensus.cc:515] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:23.035408 30986 leader_election.cc:290] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:23.036275 16792 tablet_replica.cc:333] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.037321 16792 raft_consensus.cc:2243] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.037979 16792 raft_consensus.cc:2272] T 1f944367bb734866841edc565647828b P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.038430 31101 raft_consensus.cc:493] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.039024 31101 raft_consensus.cc:515] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:28:23.039647 29992 leader_election.cc:336] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.041383 16792 tablet_replica.cc:333] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.042512 16792 raft_consensus.cc:2243] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.041275 31101 leader_election.cc:290] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
I20260709 12:28:23.043835 16792 raft_consensus.cc:2272] T ba9fb933bf47466999dfd7892e6a6638 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.047145 16792 tablet_replica.cc:333] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
W20260709 12:28:23.047041 29994 leader_election.cc:336] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:23.047477 29992 leader_election.cc:336] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.047758 29994 leader_election.cc:304] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.048310 16792 raft_consensus.cc:2243] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.048652 31101 raft_consensus.cc:2749] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.049546 16792 raft_consensus.cc:2272] T d8ada01658a746029b8fd58389f608a4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:23.049576 29995 consensus_peers.cc:597] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:23.050995 29995 leader_election.cc:336] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:28:23.051441 29995 leader_election.cc:304] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.052134 30986 raft_consensus.cc:2749] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.053090 16792 tablet_replica.cc:333] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.053901 16792 raft_consensus.cc:2243] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.054471 16792 raft_consensus.cc:2272] T ff934fd9fe834c2ab8737af23cdc9505 P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:23.055677 29995 consensus_peers.cc:597] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.057168 16792 tablet_replica.cc:333] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.058038 16792 raft_consensus.cc:2243] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.058773 16792 pending_rounds.cc:70] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: Trying to abort 1 pending ops.
I20260709 12:28:23.058980 16792 pending_rounds.cc:77] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac: Aborting op as it isn't in flight: id { term: 4 index: 4 } timestamp: 7305626020865327104 op_type: NO_OP noop_request { }
I20260709 12:28:23.059306 16792 raft_consensus.cc:2889] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:23.059595 16792 raft_consensus.cc:2272] T 1b5584cc51f14a9a8dcee35a9998b380 P 7df42fd4269f4d16aeebecaa4ea958ac [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:23.059998 29995 consensus_peers.cc:597] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.062378 16792 tablet_replica.cc:333] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.063156 16792 raft_consensus.cc:2243] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.064055 16792 raft_consensus.cc:2272] T ad89089259844673bbd8eeef685bb362 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.065227 30986 raft_consensus.cc:493] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.065852 30986 raft_consensus.cc:515] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:23.067180 16792 tablet_replica.cc:333] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.068125 30986 leader_election.cc:290] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:23.068224 16792 raft_consensus.cc:2243] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.069123 16792 raft_consensus.cc:2272] T 4c3b379d58464fa6b60b8a33472252d4 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.072566 16792 tablet_replica.cc:333] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
W20260709 12:28:23.073323 29994 leader_election.cc:336] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:23.073495 16792 raft_consensus.cc:2243] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:23.073937 29992 leader_election.cc:336] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.074221 16792 raft_consensus.cc:2272] T 80eac7cb175d4f0d84a00bd792dfd476 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.074458 29992 leader_election.cc:304] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.075395 31101 raft_consensus.cc:2749] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:28:23.076031 29995 consensus_peers.cc:597] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.077633 16792 tablet_replica.cc:333] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.078431 16792 raft_consensus.cc:2243] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.078971 16792 raft_consensus.cc:2272] T 01f4a9dd66ad4df0a8f5a598a7b432fd P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.081444 16792 tablet_replica.cc:333] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.082257 16792 raft_consensus.cc:2243] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.082914 16792 raft_consensus.cc:2272] T 66a7253beff14dc2a82c02db69def01c P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.085810 16792 tablet_replica.cc:333] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.086686 16792 raft_consensus.cc:2243] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.087761 16792 raft_consensus.cc:2272] T f8c4c5d911ee4bcbbfb014114223cc07 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.091143 16792 tablet_replica.cc:333] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.092039 16792 raft_consensus.cc:2243] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.093034 16792 raft_consensus.cc:2272] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.095607 16792 tablet_replica.cc:333] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.096302 16792 raft_consensus.cc:2243] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.096827 16792 raft_consensus.cc:2272] T c08a657511124ab1ad80a227eeec2736 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.099334 16792 tablet_replica.cc:333] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.099990 16792 raft_consensus.cc:2243] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.100517 16792 raft_consensus.cc:2272] T 57229c57228146d6ab456a83f82ef167 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.102982 16792 tablet_replica.cc:333] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.103621 16792 raft_consensus.cc:2243] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.104555 16792 raft_consensus.cc:2272] T 15ead15e87d345f48f74ca65fb97d5c5 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.106937 16792 tablet_replica.cc:333] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.107530 16792 raft_consensus.cc:2243] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.107993 16792 raft_consensus.cc:2272] T dd88542c59c449a0904ad8b002efc5a0 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.110529 16792 tablet_replica.cc:333] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.111462 16792 raft_consensus.cc:2243] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.112250 16792 raft_consensus.cc:2272] T b820ed22868a4fe98bd2c74221e3e300 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.115643 16792 tablet_replica.cc:333] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.116595 16792 raft_consensus.cc:2243] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.117650 16792 raft_consensus.cc:2272] T 7a554edb2449436ea89ed389c4aed651 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.120344 16792 tablet_replica.cc:333] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.121131 16792 raft_consensus.cc:2243] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.122443 16792 raft_consensus.cc:2272] T 611c12107cde4b0e81c03ffd67cb64e2 P 7df42fd4269f4d16aeebecaa4ea958ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.125481 16792 tablet_replica.cc:333] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.126210 16792 raft_consensus.cc:2243] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.126852 16792 raft_consensus.cc:2272] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 7df42fd4269f4d16aeebecaa4ea958ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.129441 16792 tablet_replica.cc:333] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac: stopping tablet replica
I20260709 12:28:23.130084 16792 raft_consensus.cc:2243] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.131102 16792 raft_consensus.cc:2272] T d4a3b54cd9af44d69698e15cc4bccafa P 7df42fd4269f4d16aeebecaa4ea958ac [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:23.137673 29995 consensus_peers.cc:597] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:23.164742 29994 consensus_peers.cc:597] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Couldn't send request to peer 1ae5481d76bf4a90a0ee824a14223421. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.177714 30986 raft_consensus.cc:493] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.178508 30986 raft_consensus.cc:515] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:23.181043 30986 leader_election.cc:290] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:23.186048 29992 leader_election.cc:336] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
W20260709 12:28:23.186770 29994 leader_election.cc:336] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
I20260709 12:28:23.187186 29994 leader_election.cc:304] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.187891 30986 raft_consensus.cc:2749] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.195430 30986 raft_consensus.cc:493] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.196161 30986 raft_consensus.cc:515] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
I20260709 12:28:23.198685 30986 leader_election.cc:290] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:28:23.202414 29995 leader_election.cc:336] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:23.203809 29992 leader_election.cc:336] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.205861 29992 leader_election.cc:304] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.206776 30986 raft_consensus.cc:2749] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.208606 31101 raft_consensus.cc:493] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.209260 31101 raft_consensus.cc:515] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } } peers { permanent_uuid: "7df42fd4269f4d16aeebecaa4ea958ac" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 39029 } } peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } }
W20260709 12:28:23.209951 29921 connection.cc:466] server connection from 127.0.0.1:57180 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57180 (request call id 696) could send its response
I20260709 12:28:23.212122 31101 leader_election.cc:290] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595), 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029)
W20260709 12:28:23.214360 29995 consensus_peers.cc:597] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:23.217434 29992 leader_election.cc:336] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
W20260709 12:28:23.220155 29995 leader_election.cc:336] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111)
I20260709 12:28:23.220633 29995 leader_election.cc:304] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 7df42fd4269f4d16aeebecaa4ea958ac, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.221472 31101 raft_consensus.cc:2749] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.232363 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:28:23.235248 30986 raft_consensus.cc:493] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.236001 30986 raft_consensus.cc:515] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:23.238324 30986 leader_election.cc:290] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:23.241602 29995 consensus_peers.cc:597] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:23.243907 29994 leader_election.cc:336] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:23.244911 29992 leader_election.cc:336] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.245376 29992 leader_election.cc:304] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.246201 31101 raft_consensus.cc:2749] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.267903 31101 raft_consensus.cc:493] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.268615 31101 raft_consensus.cc:515] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:23.270934 31101 leader_election.cc:290] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
W20260709 12:28:23.276095 29994 leader_election.cc:336] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:23.276836 29992 leader_election.cc:336] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.277290 29992 leader_election.cc:304] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.278097 30986 raft_consensus.cc:2749] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:23.283942 29995 consensus_peers.cc:597] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:23.297533 29995 consensus_peers.cc:597] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:23.326761 29995 consensus_peers.cc:597] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.342206 16792 tablet_server.cc:179] TabletServer@127.16.102.4:0 shutting down...
W20260709 12:28:23.393594 29995 consensus_peers.cc:597] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 -> Peer 7df42fd4269f4d16aeebecaa4ea958ac (127.16.102.3:39029): Couldn't send request to peer 7df42fd4269f4d16aeebecaa4ea958ac. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.3:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:23.395087 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:23.396267 16792 tablet_replica.cc:333] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.397485 16792 raft_consensus.cc:2243] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.398821 16792 pending_rounds.cc:70] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: Trying to abort 1 pending ops.
I20260709 12:28:23.399118 16792 pending_rounds.cc:77] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86: Aborting op as it isn't in flight: id { term: 5 index: 4 } timestamp: 7305626020254720000 op_type: NO_OP noop_request { }
I20260709 12:28:23.399430 16792 raft_consensus.cc:2889] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 5 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:23.399762 16792 raft_consensus.cc:2272] T 997e1531077444f3b82718c8b75b357e P 1e2ceabfe50c47b49529b663fa133b86 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.403328 16792 tablet_replica.cc:333] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.404076 16792 raft_consensus.cc:2243] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.405050 16792 pending_rounds.cc:70] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: Trying to abort 1 pending ops.
I20260709 12:28:23.405269 16792 pending_rounds.cc:77] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305626020199591936 op_type: NO_OP noop_request { }
I20260709 12:28:23.405583 16792 raft_consensus.cc:2889] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:23.405928 16792 raft_consensus.cc:2272] T 9ca5a52b63dd4c46bb69db5b73b48b3d P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.408651 16792 tablet_replica.cc:333] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.409495 16792 raft_consensus.cc:2243] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.410586 16792 raft_consensus.cc:2272] T 444cae907f1d4bdab292e8a3f59f7f11 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.412988 16792 tablet_replica.cc:333] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.413647 16792 raft_consensus.cc:2243] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.414130 16792 raft_consensus.cc:2272] T 811ccdd2daa94877be1d910bd3745085 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.416438 16792 tablet_replica.cc:333] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.417228 16792 raft_consensus.cc:2243] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.418269 16792 raft_consensus.cc:2272] T 4c3b379d58464fa6b60b8a33472252d4 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.420763 16792 tablet_replica.cc:333] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.421509 16792 raft_consensus.cc:2243] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.422021 16792 raft_consensus.cc:2272] T 35000da0d1054109927271f881d5b697 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.424538 16792 tablet_replica.cc:333] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.425248 16792 raft_consensus.cc:2243] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.425797 16792 raft_consensus.cc:2272] T 889b2ffc0b554cbea97dc55e0aa95094 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.428763 16792 tablet_replica.cc:333] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.429456 16792 raft_consensus.cc:2243] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.430480 16792 raft_consensus.cc:2272] T 0d30c5d36c6e4ad2a38bb757a7773f78 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.433213 16792 tablet_replica.cc:333] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.433454 31101 raft_consensus.cc:493] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5a2c42e3a7d4dedbea2bd7d705862e6)
I20260709 12:28:23.434128 16792 raft_consensus.cc:2243] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.434198 31101 raft_consensus.cc:515] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2ceabfe50c47b49529b663fa133b86" member_type: VOTER last_known_addr { host: "127.16.102.4" port: 40301 } } peers { permanent_uuid: "1ae5481d76bf4a90a0ee824a14223421" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 42003 } } peers { permanent_uuid: "b5a2c42e3a7d4dedbea2bd7d705862e6" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35595 } }
I20260709 12:28:23.435334 16792 raft_consensus.cc:2272] T d5988ddbc0f84d47b9b94c1375502692 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.437237 31101 leader_election.cc:290] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003), b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595)
I20260709 12:28:23.439417 16792 tablet_replica.cc:333] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.440537 16792 raft_consensus.cc:2243] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.441883 16792 raft_consensus.cc:2272] T 97428fa3f86d463fa4d309cffb192127 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:23.443266 29994 leader_election.cc:336] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae5481d76bf4a90a0ee824a14223421 (127.16.102.1:42003): Network error: Client connection negotiation failed: client connection to 127.16.102.1:42003: connect: Connection refused (error 111)
W20260709 12:28:23.443971 29992 leader_election.cc:336] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5a2c42e3a7d4dedbea2bd7d705862e6 (127.16.102.2:35595): Network error: Client connection negotiation failed: client connection to 127.16.102.2:35595: connect: Connection refused (error 111)
I20260709 12:28:23.444444 29992 leader_election.cc:304] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e2ceabfe50c47b49529b663fa133b86; no voters: 1ae5481d76bf4a90a0ee824a14223421, b5a2c42e3a7d4dedbea2bd7d705862e6
I20260709 12:28:23.445384 31101 raft_consensus.cc:2749] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:23.445565 16792 tablet_replica.cc:333] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.446668 16792 raft_consensus.cc:2243] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.447706 16792 raft_consensus.cc:2272] T 71ae82107a97483ba4fa78a788adb334 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.450623 16792 tablet_replica.cc:333] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.451447 16792 raft_consensus.cc:2243] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.452018 16792 raft_consensus.cc:2272] T 2f31657dfccf4d45ab006ef5aadcfb4c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.454720 16792 tablet_replica.cc:333] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.455422 16792 raft_consensus.cc:2243] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.456468 16792 raft_consensus.cc:2272] T 8a7202a2bfb344cdbb49a8cb58ac6b52 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.459594 16792 tablet_replica.cc:333] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.460409 16792 raft_consensus.cc:2243] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.461426 16792 raft_consensus.cc:2272] T 268205acac824554b94b834e990e1ed4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.464083 16792 tablet_replica.cc:333] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.464797 16792 raft_consensus.cc:2243] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.465369 16792 raft_consensus.cc:2272] T 742ffdad627c4e71865fe5b85509976d P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.468143 16792 tablet_replica.cc:333] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.468932 16792 raft_consensus.cc:2243] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.469514 16792 raft_consensus.cc:2272] T cefd124c3a724d06897bd37636aeb5b6 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.472229 16792 tablet_replica.cc:333] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.473065 16792 raft_consensus.cc:2243] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.473716 16792 raft_consensus.cc:2272] T 3a4b038ac5e344bf96cf8c672adb2dbb P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.476449 16792 tablet_replica.cc:333] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.477248 16792 raft_consensus.cc:2243] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.477881 16792 raft_consensus.cc:2272] T adaa2559c49e4cba8e82a850b5004d98 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.480450 16792 tablet_replica.cc:333] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.481237 16792 raft_consensus.cc:2243] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.482429 16792 raft_consensus.cc:2272] T 325254299d324d0b950855283ec2fcf0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.485134 16792 tablet_replica.cc:333] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.486047 16792 raft_consensus.cc:2243] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.486645 16792 raft_consensus.cc:2272] T 645003ea84b04f4a9e6d95a0bca63384 P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.489212 16792 tablet_replica.cc:333] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.489971 16792 raft_consensus.cc:2243] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.490473 16792 raft_consensus.cc:2272] T a4bd785dff644d0a8e536827c0304894 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.493132 16792 tablet_replica.cc:333] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.493782 16792 raft_consensus.cc:2243] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.494340 16792 raft_consensus.cc:2272] T 80a7001c5f324ba1955a1eef4ea15e93 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.496922 16792 tablet_replica.cc:333] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.497638 16792 raft_consensus.cc:2243] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.498144 16792 raft_consensus.cc:2272] T 57229c57228146d6ab456a83f82ef167 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.501000 16792 tablet_replica.cc:333] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.501917 16792 raft_consensus.cc:2243] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.502604 16792 raft_consensus.cc:2272] T 15ead15e87d345f48f74ca65fb97d5c5 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.505558 16792 tablet_replica.cc:333] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.506305 16792 raft_consensus.cc:2243] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.507320 16792 raft_consensus.cc:2272] T dd88542c59c449a0904ad8b002efc5a0 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.509927 16792 tablet_replica.cc:333] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.510763 16792 raft_consensus.cc:2243] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.511786 16792 raft_consensus.cc:2272] T abceb9531803436f8fd8d02595d73cf4 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.514410 16792 tablet_replica.cc:333] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.515111 16792 raft_consensus.cc:2243] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.515610 16792 raft_consensus.cc:2272] T 7a554edb2449436ea89ed389c4aed651 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.518271 16792 tablet_replica.cc:333] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.518988 16792 raft_consensus.cc:2243] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.519524 16792 raft_consensus.cc:2272] T b2c89d9015e54780811a19203939e9a5 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.522166 16792 tablet_replica.cc:333] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.522912 16792 raft_consensus.cc:2243] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.523432 16792 raft_consensus.cc:2272] T 4a33c77154df4be8aee967d4d7df39b0 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.526093 16792 tablet_replica.cc:333] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.526906 16792 raft_consensus.cc:2243] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.527422 16792 raft_consensus.cc:2272] T aaee6416f659496fb39a11ee1ae00ca3 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.529858 16792 tablet_replica.cc:333] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.530625 16792 raft_consensus.cc:2243] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.531149 16792 raft_consensus.cc:2272] T 4c799e9531994c7cba86a172713405e8 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.533663 16792 tablet_replica.cc:333] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.534417 16792 raft_consensus.cc:2243] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.534904 16792 raft_consensus.cc:2272] T ad89089259844673bbd8eeef685bb362 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.537297 16792 tablet_replica.cc:333] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.538051 16792 raft_consensus.cc:2243] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.538594 16792 raft_consensus.cc:2272] T 1b5584cc51f14a9a8dcee35a9998b380 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.540901 16792 tablet_replica.cc:333] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.541570 16792 raft_consensus.cc:2243] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.542565 16792 raft_consensus.cc:2272] T ffc3696cf98a402bb9d717b584cb229c P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.544921 16792 tablet_replica.cc:333] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.545535 16792 raft_consensus.cc:2243] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.546490 16792 raft_consensus.cc:2272] T 4f1befeab89b48729c8ba035e0ef71a8 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.548796 16792 tablet_replica.cc:333] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.549401 16792 raft_consensus.cc:2243] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.549929 16792 raft_consensus.cc:2272] T ba9fb933bf47466999dfd7892e6a6638 P 1e2ceabfe50c47b49529b663fa133b86 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.552214 16792 tablet_replica.cc:333] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.552762 16792 raft_consensus.cc:2243] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.553236 16792 raft_consensus.cc:2272] T d4a3b54cd9af44d69698e15cc4bccafa P 1e2ceabfe50c47b49529b663fa133b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.555402 16792 tablet_replica.cc:333] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.555998 16792 raft_consensus.cc:2243] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.556456 16792 raft_consensus.cc:2272] T 611c12107cde4b0e81c03ffd67cb64e2 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.558946 16792 tablet_replica.cc:333] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.559538 16792 raft_consensus.cc:2243] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.560009 16792 raft_consensus.cc:2272] T 34b3b0e1b5d14def8b90e20a7a38ee24 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.562211 16792 tablet_replica.cc:333] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.562786 16792 raft_consensus.cc:2243] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.563655 16792 raft_consensus.cc:2272] T 6872c9bba3c34283ac8c1a5a72884f7d P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.565884 16792 tablet_replica.cc:333] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.566448 16792 raft_consensus.cc:2243] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.566906 16792 raft_consensus.cc:2272] T b9fd2b0fbb1d49da957faacba2d1b38f P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.569067 16792 tablet_replica.cc:333] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.569666 16792 raft_consensus.cc:2243] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.570649 16792 raft_consensus.cc:2272] T 6a71487088564b8489c63edd9d182c5a P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.572975 16792 tablet_replica.cc:333] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.573563 16792 raft_consensus.cc:2243] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.574074 16792 raft_consensus.cc:2272] T 42b6f3a05b40452e9c171932ca603b19 P 1e2ceabfe50c47b49529b663fa133b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.576329 16792 tablet_replica.cc:333] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86: stopping tablet replica
I20260709 12:28:23.576850 16792 raft_consensus.cc:2243] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:23.577323 16792 raft_consensus.cc:2272] T e047383bb70448908508e00810ebff84 P 1e2ceabfe50c47b49529b663fa133b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.655210 16792 tablet_server.cc:196] TabletServer@127.16.102.4:0 shutdown complete.
I20260709 12:28:23.737185 16792 master.cc:562] Master@127.16.102.62:33633 shutting down...
I20260709 12:28:23.770478 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:23.771242 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:23.771631 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9599a6d4076543719333b3d5fc20475a: stopping tablet replica
I20260709 12:28:23.799284 16792 master.cc:584] Master@127.16.102.62:33633 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (39162 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:28:23.906388 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:45863
I20260709 12:28:23.907810 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:23.915395 31116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:23.916065 31119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:23.916122 31117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:23.917083 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:23.918664 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:23.918875 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:23.919085 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600103919066 us; error 0 us; skew 500 ppm
I20260709 12:28:23.919725 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:23.922838 16792 webserver.cc:533] Webserver started at http://127.16.102.62:44485/ using document root <none> and password file <none>
I20260709 12:28:23.923454 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:23.923664 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:23.923997 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:23.925390 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "dce1c60063764b1ca97568bbe2f03dab"
format_stamp: "Formatted at 2026-07-09 12:28:23 on dist-test-slave-7xt3"
I20260709 12:28:23.932361 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:23.936770 31124 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:23.937963 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:23.938289 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "dce1c60063764b1ca97568bbe2f03dab"
format_stamp: "Formatted at 2026-07-09 12:28:23 on dist-test-slave-7xt3"
I20260709 12:28:23.938591 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:23.954766 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:23.956228 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:24.004657 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:45863
I20260709 12:28:24.004787 31175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:45863 every 8 connection(s)
I20260709 12:28:24.009604 31176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.023761 31176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab: Bootstrap starting.
I20260709 12:28:24.029745 31176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.035224 31176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab: No bootstrap required, opened a new log
I20260709 12:28:24.037987 31176 raft_consensus.cc:359] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce1c60063764b1ca97568bbe2f03dab" member_type: VOTER }
I20260709 12:28:24.038475 31176 raft_consensus.cc:385] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.038787 31176 raft_consensus.cc:740] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce1c60063764b1ca97568bbe2f03dab, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.039505 31176 consensus_queue.cc:260] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce1c60063764b1ca97568bbe2f03dab" member_type: VOTER }
I20260709 12:28:24.040035 31176 raft_consensus.cc:399] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:24.040300 31176 raft_consensus.cc:493] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:24.040663 31176 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:24.046681 31176 raft_consensus.cc:515] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce1c60063764b1ca97568bbe2f03dab" member_type: VOTER }
I20260709 12:28:24.047344 31176 leader_election.cc:304] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dce1c60063764b1ca97568bbe2f03dab; no voters: 
I20260709 12:28:24.048729 31176 leader_election.cc:290] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:24.049165 31179 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:24.050858 31179 raft_consensus.cc:697] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 1 LEADER]: Becoming Leader. State: Replica: dce1c60063764b1ca97568bbe2f03dab, State: Running, Role: LEADER
I20260709 12:28:24.051678 31179 consensus_queue.cc:237] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce1c60063764b1ca97568bbe2f03dab" member_type: VOTER }
I20260709 12:28:24.052186 31176 sys_catalog.cc:565] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:24.057297 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader dce1c60063764b1ca97568bbe2f03dab. Latest consensus state: current_term: 1 leader_uuid: "dce1c60063764b1ca97568bbe2f03dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce1c60063764b1ca97568bbe2f03dab" member_type: VOTER } }
I20260709 12:28:24.058207 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:24.060146 31180 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dce1c60063764b1ca97568bbe2f03dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce1c60063764b1ca97568bbe2f03dab" member_type: VOTER } }
I20260709 12:28:24.060830 31180 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:24.064073 31187 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:24.070281 31187 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:24.072597 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:24.080866 31187 catalog_manager.cc:1383] Generated new cluster ID: 3fc5c2feeb9f4f368d1db7b533aec91f
I20260709 12:28:24.081311 31187 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:24.105647 31187 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:24.107371 31187 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:24.124831 31187 catalog_manager.cc:6070] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab: Generated new TSK 0
I20260709 12:28:24.125597 31187 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:24.140240 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:24.148969 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:24.149772 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:24.151502 31200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:24.152583 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:24.154029 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:24.154327 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:24.154479 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600104154465 us; error 0 us; skew 500 ppm
I20260709 12:28:24.155066 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:24.158129 16792 webserver.cc:533] Webserver started at http://127.16.102.1:42945/ using document root <none> and password file <none>
I20260709 12:28:24.158751 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:24.158933 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:24.159188 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:24.160497 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "67a00a279c8f4e86840326b53037f2e6"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:24.166211 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20260709 12:28:24.170451 31205 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.171378 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:28:24.171756 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "67a00a279c8f4e86840326b53037f2e6"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:24.172058 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:24.193625 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:24.195261 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:24.197005 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:24.199867 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:24.200078 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.200431 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:24.200618 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.250826 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:38611
I20260709 12:28:24.251185 31267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:38611 every 8 connection(s)
I20260709 12:28:24.257603 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:24.266162 31272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:24.268260 31273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:24.271706 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:24.272991 31275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:24.274370 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:24.274611 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:24.274757 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600104274745 us; error 0 us; skew 500 ppm
I20260709 12:28:24.275286 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:24.279386 16792 webserver.cc:533] Webserver started at http://127.16.102.2:42323/ using document root <none> and password file <none>
I20260709 12:28:24.280120 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:24.280164 31268 heartbeater.cc:344] Connected to a master server at 127.16.102.62:45863
I20260709 12:28:24.280357 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:24.280778 31268 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:24.280838 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:24.281688 31268 heartbeater.cc:507] Master 127.16.102.62:45863 requested a full tablet report, sending...
I20260709 12:28:24.282512 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "6b512fc09fe2444a95f04d456239a9dd"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:24.284497 31141 ts_manager.cc:194] Registered new tserver with Master: 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611)
I20260709 12:28:24.287032 31141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46714
I20260709 12:28:24.288883 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:28:24.293991 31280 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.295312 16792 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:28:24.295701 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "6b512fc09fe2444a95f04d456239a9dd"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:24.296049 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:24.313350 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:24.314983 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:24.316745 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:24.319820 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:24.320072 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.320451 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:24.320631 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.371191 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:35733
I20260709 12:28:24.371341 31342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:35733 every 8 connection(s)
I20260709 12:28:24.377241 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:24.387086 31346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:24.388443 31347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:24.393961 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:24.394155 31349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:24.395098 31343 heartbeater.cc:344] Connected to a master server at 127.16.102.62:45863
I20260709 12:28:24.395659 31343 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:24.395757 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:24.396114 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:24.396354 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600104396327 us; error 0 us; skew 500 ppm
I20260709 12:28:24.396766 31343 heartbeater.cc:507] Master 127.16.102.62:45863 requested a full tablet report, sending...
I20260709 12:28:24.397276 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:24.399696 31141 ts_manager.cc:194] Registered new tserver with Master: 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:24.400696 16792 webserver.cc:533] Webserver started at http://127.16.102.3:37015/ using document root <none> and password file <none>
I20260709 12:28:24.401360 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:24.401577 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:24.401610 31141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46718
I20260709 12:28:24.401948 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:24.403709 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "451375e442ad416ebc7f5e58b8610f7f"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:24.408893 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20260709 12:28:24.412751 31354 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.413652 16792 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:28:24.414053 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "451375e442ad416ebc7f5e58b8610f7f"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:24.414415 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:24.436066 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:24.437580 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:24.439116 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:24.442003 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:24.442281 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.442548 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:24.442759 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:24.492244 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:32901
I20260709 12:28:24.492357 31416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:32901 every 8 connection(s)
I20260709 12:28:24.508272 31417 heartbeater.cc:344] Connected to a master server at 127.16.102.62:45863
I20260709 12:28:24.508749 31417 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:24.509598 31417 heartbeater.cc:507] Master 127.16.102.62:45863 requested a full tablet report, sending...
I20260709 12:28:24.511811 31141 ts_manager.cc:194] Registered new tserver with Master: 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:24.512377 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01616263s
I20260709 12:28:24.513846 31141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46732
I20260709 12:28:24.545346 31141 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46740:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:24.548748 31141 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:24.634510 31233 tablet_service.cc:1511] Processing CreateTablet for tablet b8c8e791b40f4efd9fc22519ed30b9ed (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:28:24.636233 31233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c8e791b40f4efd9fc22519ed30b9ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.638700 31232 tablet_service.cc:1511] Processing CreateTablet for tablet d6c0212c06fb4945ad761212c1868fb1 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:28:24.640676 31232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c0212c06fb4945ad761212c1868fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.643191 31307 tablet_service.cc:1511] Processing CreateTablet for tablet d6c0212c06fb4945ad761212c1868fb1 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:28:24.644687 31308 tablet_service.cc:1511] Processing CreateTablet for tablet b8c8e791b40f4efd9fc22519ed30b9ed (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:28:24.651391 31307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c0212c06fb4945ad761212c1868fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.652216 31231 tablet_service.cc:1511] Processing CreateTablet for tablet 36836434931140938a0c90e537ebd1e9 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:28:24.654119 31231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36836434931140938a0c90e537ebd1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.654613 31306 tablet_service.cc:1511] Processing CreateTablet for tablet 36836434931140938a0c90e537ebd1e9 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:28:24.652247 31308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c8e791b40f4efd9fc22519ed30b9ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.663352 31230 tablet_service.cc:1511] Processing CreateTablet for tablet d94e0ae944a1430a9b4def74a74cc1a3 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:28:24.665271 31230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94e0ae944a1430a9b4def74a74cc1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.666332 31306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36836434931140938a0c90e537ebd1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.667868 31305 tablet_service.cc:1511] Processing CreateTablet for tablet d94e0ae944a1430a9b4def74a74cc1a3 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:28:24.669658 31305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94e0ae944a1430a9b4def74a74cc1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.677636 31229 tablet_service.cc:1511] Processing CreateTablet for tablet aac47c2fc9c64d0e8627065d6c366bd0 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:28:24.679591 31229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac47c2fc9c64d0e8627065d6c366bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.681740 31304 tablet_service.cc:1511] Processing CreateTablet for tablet aac47c2fc9c64d0e8627065d6c366bd0 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:28:24.683682 31304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac47c2fc9c64d0e8627065d6c366bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.691371 31432 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:24.704838 31432 tablet_bootstrap.cc:654] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.706225 31232 tablet_service.cc:1511] Processing CreateTablet for tablet 6679bf1e8fbc4631903b685e7e5dde0a (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:28:24.707950 31232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6679bf1e8fbc4631903b685e7e5dde0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.708124 31433 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:24.711491 31303 tablet_service.cc:1511] Processing CreateTablet for tablet 6679bf1e8fbc4631903b685e7e5dde0a (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:28:24.713223 31303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6679bf1e8fbc4631903b685e7e5dde0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.721099 31433 tablet_bootstrap.cc:654] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.738440 31380 tablet_service.cc:1511] Processing CreateTablet for tablet 36836434931140938a0c90e537ebd1e9 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:28:24.739234 31228 tablet_service.cc:1511] Processing CreateTablet for tablet 32deb0d4043d46dfae3cc0aa450a34cb (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:28:24.740926 31228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32deb0d4043d46dfae3cc0aa450a34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.737059 31381 tablet_service.cc:1511] Processing CreateTablet for tablet d6c0212c06fb4945ad761212c1868fb1 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:28:24.742349 31379 tablet_service.cc:1511] Processing CreateTablet for tablet d94e0ae944a1430a9b4def74a74cc1a3 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:28:24.743716 31381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c0212c06fb4945ad761212c1868fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.744473 31378 tablet_service.cc:1511] Processing CreateTablet for tablet aac47c2fc9c64d0e8627065d6c366bd0 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:28:24.737023 31382 tablet_service.cc:1511] Processing CreateTablet for tablet b8c8e791b40f4efd9fc22519ed30b9ed (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:28:24.746841 31377 tablet_service.cc:1511] Processing CreateTablet for tablet 6679bf1e8fbc4631903b685e7e5dde0a (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:28:24.749537 31306 tablet_service.cc:1511] Processing CreateTablet for tablet 32deb0d4043d46dfae3cc0aa450a34cb (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:28:24.751151 31306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32deb0d4043d46dfae3cc0aa450a34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.751053 31376 tablet_service.cc:1511] Processing CreateTablet for tablet 32deb0d4043d46dfae3cc0aa450a34cb (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:28:24.754699 31378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac47c2fc9c64d0e8627065d6c366bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.756301 31232 tablet_service.cc:1511] Processing CreateTablet for tablet 653fa7d1e4f549e792dc2c6e2972a3d4 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:28:24.760371 31232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653fa7d1e4f549e792dc2c6e2972a3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.761929 31302 tablet_service.cc:1511] Processing CreateTablet for tablet 653fa7d1e4f549e792dc2c6e2972a3d4 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:28:24.758428 31380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36836434931140938a0c90e537ebd1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.764990 31375 tablet_service.cc:1511] Processing CreateTablet for tablet 653fa7d1e4f549e792dc2c6e2972a3d4 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:28:24.766295 31376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32deb0d4043d46dfae3cc0aa450a34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.767109 31375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653fa7d1e4f549e792dc2c6e2972a3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.769357 31302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653fa7d1e4f549e792dc2c6e2972a3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.770503 31377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6679bf1e8fbc4631903b685e7e5dde0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.771373 31227 tablet_service.cc:1511] Processing CreateTablet for tablet dc1054676023472eab95b1e45e804e71 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:28:24.771968 31379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94e0ae944a1430a9b4def74a74cc1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.772982 31227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1054676023472eab95b1e45e804e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.773218 31382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c8e791b40f4efd9fc22519ed30b9ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.777057 31301 tablet_service.cc:1511] Processing CreateTablet for tablet dc1054676023472eab95b1e45e804e71 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:28:24.778818 31301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1054676023472eab95b1e45e804e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.787098 31374 tablet_service.cc:1511] Processing CreateTablet for tablet dc1054676023472eab95b1e45e804e71 (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:28:24.788930 31374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1054676023472eab95b1e45e804e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.790748 31303 tablet_service.cc:1511] Processing CreateTablet for tablet 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:28:24.793208 31303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.796420 31226 tablet_service.cc:1511] Processing CreateTablet for tablet 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:28:24.798177 31226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.804752 31373 tablet_service.cc:1511] Processing CreateTablet for tablet 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f (DEFAULT_TABLE table=test-workload [id=47de08966f7b4dd2b86d111c6dec9800]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:28:24.806613 31373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:24.824489 31432 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:24.825259 31432 ts_tablet_manager.cc:1403] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.134s	user 0.014s	sys 0.007s
I20260709 12:28:24.843184 31436 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:24.848694 31432 raft_consensus.cc:359] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.852159 31436 tablet_bootstrap.cc:654] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.849566 31432 raft_consensus.cc:385] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.858608 31432 raft_consensus.cc:740] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.861012 31433 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:24.861718 31433 ts_tablet_manager.cc:1403] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.154s	user 0.013s	sys 0.003s
I20260709 12:28:24.882918 31436 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:24.883708 31436 ts_tablet_manager.cc:1403] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.041s	user 0.007s	sys 0.008s
I20260709 12:28:24.887447 31436 raft_consensus.cc:359] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.888321 31436 raft_consensus.cc:385] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.888648 31436 raft_consensus.cc:740] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.889564 31436 consensus_queue.cc:260] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.865437 31433 raft_consensus.cc:359] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.895399 31433 raft_consensus.cc:385] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.895820 31433 raft_consensus.cc:740] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.895839 31432 consensus_queue.cc:260] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.906237 31433 consensus_queue.cc:260] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.916077 31268 heartbeater.cc:499] Master 127.16.102.62:45863 was elected leader, sending a full tablet report...
I20260709 12:28:24.932699 31433 ts_tablet_manager.cc:1434] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.071s	user 0.023s	sys 0.004s
I20260709 12:28:24.933802 31343 heartbeater.cc:499] Master 127.16.102.62:45863 was elected leader, sending a full tablet report...
I20260709 12:28:24.932700 31432 ts_tablet_manager.cc:1434] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.107s	user 0.011s	sys 0.007s
I20260709 12:28:24.935667 31432 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:24.940116 31433 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:24.944114 31417 heartbeater.cc:499] Master 127.16.102.62:45863 was elected leader, sending a full tablet report...
I20260709 12:28:24.944796 31436 ts_tablet_manager.cc:1434] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.061s	user 0.008s	sys 0.015s
I20260709 12:28:24.946249 31436 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:24.946628 31432 tablet_bootstrap.cc:654] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.960578 31433 tablet_bootstrap.cc:654] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.967849 31436 tablet_bootstrap.cc:654] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.969099 31433 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:24.969885 31433 ts_tablet_manager.cc:1403] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.030s	user 0.010s	sys 0.008s
I20260709 12:28:24.971774 31432 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:24.972527 31432 ts_tablet_manager.cc:1403] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.037s	user 0.020s	sys 0.008s
I20260709 12:28:24.974399 31433 raft_consensus.cc:359] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.975545 31433 raft_consensus.cc:385] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.976343 31432 raft_consensus.cc:359] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.977257 31432 raft_consensus.cc:385] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.977633 31432 raft_consensus.cc:740] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.975957 31433 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.979244 31433 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.979244 31432 consensus_queue.cc:260] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.982657 31436 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:24.982725 31433 ts_tablet_manager.cc:1434] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.001s
I20260709 12:28:24.983484 31436 ts_tablet_manager.cc:1403] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.038s	user 0.011s	sys 0.010s
I20260709 12:28:24.983536 31432 ts_tablet_manager.cc:1434] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:28:24.984236 31433 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:24.984965 31432 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:24.989291 31438 raft_consensus.cc:493] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.990135 31438 raft_consensus.cc:515] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.992180 31436 raft_consensus.cc:359] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:24.993265 31433 tablet_bootstrap.cc:654] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:24.993353 31436 raft_consensus.cc:385] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:24.994056 31436 raft_consensus.cc:740] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.995227 31436 consensus_queue.cc:260] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
W20260709 12:28:24.997407 31418 tablet.cc:2406] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:28:25.003355 31438 leader_election.cc:290] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.011906 31432 tablet_bootstrap.cc:654] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.017918 31436 ts_tablet_manager.cc:1434] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.034s	user 0.010s	sys 0.000s
I20260709 12:28:25.019994 31436 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.024863 31440 raft_consensus.cc:493] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.025967 31440 raft_consensus.cc:515] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.032111 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.033738 31440 leader_election.cc:290] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.028359 31433 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.036329 31433 ts_tablet_manager.cc:1403] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.052s	user 0.025s	sys 0.016s
W20260709 12:28:25.036705 31206 leader_election.cc:343] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.040747 31433 raft_consensus.cc:359] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.042269 31433 raft_consensus.cc:385] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.042758 31433 raft_consensus.cc:740] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.044138 31433 consensus_queue.cc:260] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.050696 31433 ts_tablet_manager.cc:1434] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.004s
I20260709 12:28:25.052276 31433 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.064173 31432 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.064277 31433 tablet_bootstrap.cc:654] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
W20260709 12:28:25.066934 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:25.068068 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
W20260709 12:28:25.070079 31207 leader_election.cc:343] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.070652 31207 leader_election.cc:304] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6; no voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.072144 31438 raft_consensus.cc:2749] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.073448 31432 ts_tablet_manager.cc:1403] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.089s	user 0.030s	sys 0.029s
I20260709 12:28:25.074757 31436 tablet_bootstrap.cc:654] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.077636 31432 raft_consensus.cc:359] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.078583 31432 raft_consensus.cc:385] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.078964 31432 raft_consensus.cc:740] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.079922 31432 consensus_queue.cc:260] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.082300 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c9b3981ba8b4bf6af4bcd2c9e9e5e4f" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.083284 31243 raft_consensus.cc:2468] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 0.
I20260709 12:28:25.085058 31358 leader_election.cc:304] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:25.085585 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c9b3981ba8b4bf6af4bcd2c9e9e5e4f" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.086104 31440 raft_consensus.cc:2804] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.086632 31440 raft_consensus.cc:493] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.087018 31440 raft_consensus.cc:3060] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:25.088577 31355 leader_election.cc:343] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.090828 31436 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.091558 31432 ts_tablet_manager.cc:1434] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:28:25.091949 31436 ts_tablet_manager.cc:1403] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.072s	user 0.016s	sys 0.000s
I20260709 12:28:25.093617 31432 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.096170 31436 raft_consensus.cc:359] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.097131 31436 raft_consensus.cc:385] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.097680 31436 raft_consensus.cc:740] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.098910 31436 consensus_queue.cc:260] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.102823 31432 tablet_bootstrap.cc:654] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.103549 31433 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.103395 31440 raft_consensus.cc:515] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.104214 31438 raft_consensus.cc:493] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.104439 31433 ts_tablet_manager.cc:1403] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.052s	user 0.030s	sys 0.005s
I20260709 12:28:25.104801 31438 raft_consensus.cc:515] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.106525 31440 leader_election.cc:290] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.107827 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c9b3981ba8b4bf6af4bcd2c9e9e5e4f" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.108536 31243 raft_consensus.cc:3060] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.108878 31438 leader_election.cc:290] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.108669 31433 raft_consensus.cc:359] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.109551 31433 raft_consensus.cc:385] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.109917 31433 raft_consensus.cc:740] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.110891 31317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.110808 31433 consensus_queue.cc:260] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.112306 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c9b3981ba8b4bf6af4bcd2c9e9e5e4f" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:25.114617 31318 raft_consensus.cc:3060] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:25.112856 31206 leader_election.cc:343] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.113623 31433 ts_tablet_manager.cc:1434] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:28:25.120182 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.120828 31433 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.121649 31432 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
W20260709 12:28:25.122205 31207 leader_election.cc:343] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.122366 31432 ts_tablet_manager.cc:1403] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.011s
I20260709 12:28:25.122838 31207 leader_election.cc:304] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6; no voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.123978 31438 raft_consensus.cc:2749] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.126463 31432 raft_consensus.cc:359] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.127280 31432 raft_consensus.cc:385] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.127594 31432 raft_consensus.cc:740] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.128435 31432 consensus_queue.cc:260] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.130913 31432 ts_tablet_manager.cc:1434] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:28:25.131811 31433 tablet_bootstrap.cc:654] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.132212 31432 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.132722 31243 raft_consensus.cc:2468] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 1.
I20260709 12:28:25.135692 31358 leader_election.cc:304] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:25.136607 31436 ts_tablet_manager.cc:1434] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.044s	user 0.010s	sys 0.004s
I20260709 12:28:25.136823 31452 raft_consensus.cc:2804] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.137701 31452 raft_consensus.cc:697] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:25.139015 31452 consensus_queue.cc:237] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.139923 31436 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.148521 31436 tablet_bootstrap.cc:654] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.149735 31318 raft_consensus.cc:2468] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 1.
I20260709 12:28:25.152452 31432 tablet_bootstrap.cc:654] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.157634 31433 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.158353 31433 ts_tablet_manager.cc:1403] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.015s
I20260709 12:28:25.164039 31140 catalog_manager.cc:5697] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 1 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:25.169582 31436 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.164749 31433 raft_consensus.cc:359] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.170456 31433 raft_consensus.cc:385] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.170676 31436 ts_tablet_manager.cc:1403] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.013s
I20260709 12:28:25.170998 31433 raft_consensus.cc:740] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.172168 31433 consensus_queue.cc:260] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.174109 31432 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.174969 31432 ts_tablet_manager.cc:1403] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.043s	user 0.022s	sys 0.006s
I20260709 12:28:25.174804 31436 raft_consensus.cc:359] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.175770 31436 raft_consensus.cc:385] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.176148 31436 raft_consensus.cc:740] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.176363 31433 ts_tablet_manager.cc:1434] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.001s
I20260709 12:28:25.177297 31436 consensus_queue.cc:260] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.178473 31433 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.179863 31432 raft_consensus.cc:359] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.181052 31432 raft_consensus.cc:385] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.181173 31436 ts_tablet_manager.cc:1434] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.006s
I20260709 12:28:25.181612 31432 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.182852 31436 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.182998 31432 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.186772 31432 ts_tablet_manager.cc:1434] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:28:25.188022 31432 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.188225 31452 raft_consensus.cc:493] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.188869 31452 raft_consensus.cc:515] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.198167 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.200313 31436 tablet_bootstrap.cc:654] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.200332 31433 tablet_bootstrap.cc:654] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.205148 31452 leader_election.cc:290] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.208175 31432 tablet_bootstrap.cc:654] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
W20260709 12:28:25.204341 31358 leader_election.cc:343] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.208774 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
W20260709 12:28:25.211709 31355 leader_election.cc:343] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.212358 31355 leader_election.cc:304] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f; no voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.213717 31452 raft_consensus.cc:2749] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.214964 31436 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.215807 31436 ts_tablet_manager.cc:1403] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.000s
I20260709 12:28:25.220300 31433 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.220069 31436 raft_consensus.cc:359] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.221329 31433 ts_tablet_manager.cc:1403] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.043s	user 0.010s	sys 0.013s
I20260709 12:28:25.221272 31436 raft_consensus.cc:385] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.222128 31436 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.222258 31452 raft_consensus.cc:493] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.222981 31452 raft_consensus.cc:515] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.223543 31436 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.225659 31433 raft_consensus.cc:359] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.226752 31433 raft_consensus.cc:385] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.227162 31433 raft_consensus.cc:740] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.227911 31436 ts_tablet_manager.cc:1434] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20260709 12:28:25.228610 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.228295 31433 consensus_queue.cc:260] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.229866 31436 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.230295 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.231868 31433 ts_tablet_manager.cc:1434] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
W20260709 12:28:25.234874 31358 leader_election.cc:343] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:25.236409 31355 leader_election.cc:343] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.236965 31355 leader_election.cc:304] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f; no voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.238003 31440 raft_consensus.cc:2749] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.233505 31433 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.243503 31436 tablet_bootstrap.cc:654] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.244140 31438 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.244763 31438 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.247430 31438 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.252036 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.253041 31318 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a00a279c8f4e86840326b53037f2e6 in term 0.
I20260709 12:28:25.254835 31206 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.255106 31433 tablet_bootstrap.cc:654] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.256013 31438 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.256500 31438 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.256911 31438 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.258384 31436 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.259091 31436 ts_tablet_manager.cc:1403] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.030s	user 0.025s	sys 0.000s
I20260709 12:28:25.263573 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.264473 31392 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a00a279c8f4e86840326b53037f2e6 in term 0.
I20260709 12:28:25.266173 31433 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.225628 31452 leader_election.cc:290] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.266913 31433 ts_tablet_manager.cc:1403] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.004s
I20260709 12:28:25.270630 31433 raft_consensus.cc:359] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.271714 31433 raft_consensus.cc:385] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.272095 31433 raft_consensus.cc:740] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.273098 31433 consensus_queue.cc:260] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.263536 31436 raft_consensus.cc:359] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.275789 31436 raft_consensus.cc:385] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.276118 31436 raft_consensus.cc:740] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.276166 31433 ts_tablet_manager.cc:1434] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:28:25.276953 31436 consensus_queue.cc:260] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.277793 31433 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.279790 31436 ts_tablet_manager.cc:1434] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 12:28:25.281206 31436 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.285125 31432 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.286055 31432 ts_tablet_manager.cc:1403] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.098s	user 0.023s	sys 0.039s
I20260709 12:28:25.281214 31439 raft_consensus.cc:493] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.287243 31433 tablet_bootstrap.cc:654] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.287338 31439 raft_consensus.cc:515] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.288930 31436 tablet_bootstrap.cc:654] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.289981 31432 raft_consensus.cc:359] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.290995 31432 raft_consensus.cc:385] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.291365 31432 raft_consensus.cc:740] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.292294 31432 consensus_queue.cc:260] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.301146 31439 leader_election.cc:290] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.310595 31438 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.311389 31433 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.312191 31433 ts_tablet_manager.cc:1403] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.035s	user 0.013s	sys 0.003s
I20260709 12:28:25.314227 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:25.314960 31318 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.318746 31433 raft_consensus.cc:359] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.325759 31318 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a00a279c8f4e86840326b53037f2e6 in term 1.
I20260709 12:28:25.326587 31433 raft_consensus.cc:385] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.328060 31433 raft_consensus.cc:740] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.328552 31206 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.328860 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.329933 31392 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.333966 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36836434931140938a0c90e537ebd1e9" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.335248 31243 raft_consensus.cc:2468] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
I20260709 12:28:25.337932 31284 leader_election.cc:304] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.339365 31439 raft_consensus.cc:2804] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.339875 31439 raft_consensus.cc:493] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.340323 31439 raft_consensus.cc:3060] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.344041 31436 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.345046 31436 ts_tablet_manager.cc:1403] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.064s	user 0.024s	sys 0.023s
I20260709 12:28:25.326814 31438 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.349431 31433 consensus_queue.cc:260] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.352078 31469 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.352360 31439 raft_consensus.cc:515] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.353060 31432 ts_tablet_manager.cc:1434] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.067s	user 0.026s	sys 0.033s
I20260709 12:28:25.355242 31392 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a00a279c8f4e86840326b53037f2e6 in term 1.
I20260709 12:28:25.356297 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36836434931140938a0c90e537ebd1e9" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.357030 31243 raft_consensus.cc:3060] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.357599 31432 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.362394 31436 raft_consensus.cc:359] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.363437 31436 raft_consensus.cc:385] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.363785 31436 raft_consensus.cc:740] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.364713 31436 consensus_queue.cc:260] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.352861 31469 raft_consensus.cc:697] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 LEADER]: Becoming Leader. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Running, Role: LEADER
I20260709 12:28:25.366643 31469 consensus_queue.cc:237] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.367127 31436 ts_tablet_manager.cc:1434] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.006s
I20260709 12:28:25.368495 31436 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.369797 31472 raft_consensus.cc:493] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.370615 31433 ts_tablet_manager.cc:1434] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.058s	user 0.008s	sys 0.000s
I20260709 12:28:25.370411 31472 raft_consensus.cc:515] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.371860 31433 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.373036 31472 leader_election.cc:290] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.375735 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
W20260709 12:28:25.377974 31284 leader_election.cc:343] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.380533 31432 tablet_bootstrap.cc:654] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.384931 31452 raft_consensus.cc:493] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.385751 31452 raft_consensus.cc:515] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.385027 31436 tablet_bootstrap.cc:654] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.388963 31433 tablet_bootstrap.cc:654] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.388774 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.389732 31391 raft_consensus.cc:2468] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
I20260709 12:28:25.390357 31452 leader_election.cc:290] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.391536 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
W20260709 12:28:25.394757 31358 leader_election.cc:343] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.391445 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.396705 31318 raft_consensus.cc:2468] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 0.
I20260709 12:28:25.394765 31282 leader_election.cc:304] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.398607 31436 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.399098 31472 raft_consensus.cc:2804] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.398969 31355 leader_election.cc:304] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.399849 31472 raft_consensus.cc:493] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.400300 31472 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.401621 31452 raft_consensus.cc:2804] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.402431 31452 raft_consensus.cc:493] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.400038 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36836434931140938a0c90e537ebd1e9" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.403049 31452 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:25.404400 31282 leader_election.cc:343] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.408891 31472 raft_consensus.cc:515] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.410260 31436 ts_tablet_manager.cc:1403] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.042s	user 0.011s	sys 0.006s
I20260709 12:28:25.412098 31439 leader_election.cc:290] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.412832 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36836434931140938a0c90e537ebd1e9" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.413606 31436 raft_consensus.cc:359] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.414422 31436 raft_consensus.cc:385] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.415791 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.416481 31243 raft_consensus.cc:2468] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.416843 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.417552 31391 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:25.417464 31284 leader_election.cc:343] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.418807 31284 leader_election.cc:304] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.419782 31439 raft_consensus.cc:2804] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.414726 31436 raft_consensus.cc:740] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.417614 31139 catalog_manager.cc:5697] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "67a00a279c8f4e86840326b53037f2e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.421039 31433 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.421697 31433 ts_tablet_manager.cc:1403] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.050s	user 0.018s	sys 0.012s
I20260709 12:28:25.421334 31436 consensus_queue.cc:260] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.422334 31392 raft_consensus.cc:3060] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.425257 31433 raft_consensus.cc:359] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.426172 31433 raft_consensus.cc:385] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.426465 31433 raft_consensus.cc:740] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.427321 31433 consensus_queue.cc:260] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.426513 31472 leader_election.cc:290] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.430430 31472 raft_consensus.cc:493] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.430971 31452 raft_consensus.cc:515] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.431058 31472 raft_consensus.cc:515] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.434222 31452 leader_election.cc:290] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.435070 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.435020 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
W20260709 12:28:25.444028 31358 leader_election.cc:343] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.445135 31243 raft_consensus.cc:2468] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
I20260709 12:28:25.445680 31439 raft_consensus.cc:697] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:25.447307 31439 consensus_queue.cc:237] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.448128 31284 leader_election.cc:304] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.444761 31391 raft_consensus.cc:2468] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.451102 31479 raft_consensus.cc:2804] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.451573 31479 raft_consensus.cc:493] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.451807 31433 ts_tablet_manager.cc:1434] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.030s	user 0.013s	sys 0.007s
I20260709 12:28:25.451372 31282 leader_election.cc:304] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.452996 31433 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: Bootstrap starting.
I20260709 12:28:25.452591 31452 raft_consensus.cc:493] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.453900 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:25.454025 31452 raft_consensus.cc:515] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.454671 31318 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.451586 31432 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.457047 31452 leader_election.cc:290] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.452050 31479 raft_consensus.cc:3060] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.461867 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.463531 31433 tablet_bootstrap.cc:654] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.460384 31472 leader_election.cc:290] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.457374 31392 raft_consensus.cc:2468] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.458474 31432 ts_tablet_manager.cc:1403] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.101s	user 0.006s	sys 0.010s
I20260709 12:28:25.466903 31317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.468683 31436 ts_tablet_manager.cc:1434] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.058s	user 0.007s	sys 0.003s
W20260709 12:28:25.468959 31355 leader_election.cc:343] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.470984 31318 raft_consensus.cc:2468] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 1.
I20260709 12:28:25.472337 31355 leader_election.cc:304] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.474612 31436 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.478082 31452 raft_consensus.cc:2804] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.478849 31452 raft_consensus.cc:697] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:25.479866 31452 consensus_queue.cc:237] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.479585 31479 raft_consensus.cc:515] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.459427 31477 raft_consensus.cc:2804] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.459627 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.481688 31243 raft_consensus.cc:2468] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 0.
W20260709 12:28:25.463711 31282 leader_election.cc:343] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.483367 31358 leader_election.cc:304] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.483624 31477 raft_consensus.cc:697] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:25.484823 31440 raft_consensus.cc:2804] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.485296 31440 raft_consensus.cc:493] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.484735 31477 consensus_queue.cc:237] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.485656 31440 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.485715 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.486104 31432 raft_consensus.cc:359] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.486800 31243 raft_consensus.cc:3060] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.486949 31432 raft_consensus.cc:385] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.497893 31479 leader_election.cc:290] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.499620 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.498625 31432 raft_consensus.cc:740] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
W20260709 12:28:25.501358 31282 leader_election.cc:343] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.505573 31139 catalog_manager.cc:5697] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.506709 31436 tablet_bootstrap.cc:654] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.507727 31432 consensus_queue.cc:260] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.515944 31432 ts_tablet_manager.cc:1434] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.002s
I20260709 12:28:25.513665 31243 raft_consensus.cc:2468] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.517486 31284 leader_election.cc:304] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:25.518683 31432 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.520263 31440 raft_consensus.cc:515] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.523878 31477 raft_consensus.cc:2804] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.524627 31477 raft_consensus.cc:697] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:25.526837 31432 tablet_bootstrap.cc:654] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.528463 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.525651 31477 consensus_queue.cc:237] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.529247 31243 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.531876 31436 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.532596 31436 ts_tablet_manager.cc:1403] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.058s	user 0.013s	sys 0.000s
I20260709 12:28:25.535935 31436 raft_consensus.cc:359] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.536965 31436 raft_consensus.cc:385] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.537273 31436 raft_consensus.cc:740] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.540071 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:25.542876 31137 catalog_manager.cc:5697] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 1 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:25.545295 31469 raft_consensus.cc:493] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.546134 31469 raft_consensus.cc:515] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
W20260709 12:28:25.543051 31355 leader_election.cc:343] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.545460 31436 consensus_queue.cc:260] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.550982 31317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.555266 31469 leader_election.cc:290] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.555536 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.555992 31439 raft_consensus.cc:493] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.556565 31392 raft_consensus.cc:2393] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67a00a279c8f4e86840326b53037f2e6 in current term 1: Already voted for candidate 6b512fc09fe2444a95f04d456239a9dd in this term.
I20260709 12:28:25.556599 31439 raft_consensus.cc:515] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.559576 31432 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.559979 31439 leader_election.cc:290] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.560375 31432 ts_tablet_manager.cc:1403] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.042s	user 0.013s	sys 0.000s
I20260709 12:28:25.560989 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.561153 31440 leader_election.cc:290] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.565156 31436 ts_tablet_manager.cc:1434] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.003s
I20260709 12:28:25.567741 31436 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f: Bootstrap starting.
I20260709 12:28:25.570057 31207 leader_election.cc:304] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6; no voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.571321 31243 raft_consensus.cc:2468] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 1.
I20260709 12:28:25.571755 31469 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.574406 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.575292 31392 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
W20260709 12:28:25.575549 31284 leader_election.cc:343] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.577037 31432 raft_consensus.cc:359] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.577728 31432 raft_consensus.cc:385] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.578065 31432 raft_consensus.cc:740] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.576148 31358 leader_election.cc:304] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.578361 31282 leader_election.cc:304] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.579072 31432 consensus_queue.cc:260] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.580070 31440 raft_consensus.cc:2804] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.585397 31439 raft_consensus.cc:2804] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.585932 31432 ts_tablet_manager.cc:1434] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.025s	user 0.011s	sys 0.001s
I20260709 12:28:25.586404 31439 raft_consensus.cc:493] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.586894 31439 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.587590 31432 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.590437 31436 tablet_bootstrap.cc:654] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.590570 31452 consensus_queue.cc:1048] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:25.599049 31432 tablet_bootstrap.cc:654] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.605332 31433 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: No bootstrap required, opened a new log
I20260709 12:28:25.606087 31433 ts_tablet_manager.cc:1403] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: Time spent bootstrapping tablet: real 0.153s	user 0.040s	sys 0.045s
I20260709 12:28:25.608498 31469 raft_consensus.cc:2749] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.600131 31440 raft_consensus.cc:697] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:25.609802 31439 raft_consensus.cc:515] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.615000 31440 consensus_queue.cc:237] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.617000 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.625802 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.627059 31392 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.620215 31243 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.632350 31477 raft_consensus.cc:493] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.633312 31477 raft_consensus.cc:515] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.634030 31439 leader_election.cc:290] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.638622 31477 leader_election.cc:290] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.640127 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.640048 31241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.642423 31241 raft_consensus.cc:2468] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
I20260709 12:28:25.642486 31138 catalog_manager.cc:5697] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.644568 31284 leader_election.cc:304] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.641440 31391 raft_consensus.cc:2468] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
I20260709 12:28:25.645349 31138 catalog_manager.cc:5697] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.648247 31477 raft_consensus.cc:2804] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.648967 31477 raft_consensus.cc:493] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.649444 31477 raft_consensus.cc:3060] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.649714 31433 raft_consensus.cc:359] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.650727 31433 raft_consensus.cc:385] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.651079 31433 raft_consensus.cc:740] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.652055 31433 consensus_queue.cc:260] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.657573 31439 raft_consensus.cc:493] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.658937 31439 raft_consensus.cc:515] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.659613 31433 ts_tablet_manager.cc:1434] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: Time spent starting tablet: real 0.053s	user 0.007s	sys 0.001s
I20260709 12:28:25.662470 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.662871 31439 leader_election.cc:290] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
W20260709 12:28:25.664522 31284 leader_election.cc:343] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.664767 31392 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.664554 31477 raft_consensus.cc:515] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.667318 31477 leader_election.cc:290] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.669152 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.669983 31391 raft_consensus.cc:2468] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 0.
I20260709 12:28:25.668494 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:25.678895 31242 raft_consensus.cc:3060] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.680565 31282 leader_election.cc:304] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.687822 31243 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.688673 31477 raft_consensus.cc:2804] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.688824 31282 leader_election.cc:304] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.690999 31432 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.691170 31477 raft_consensus.cc:697] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:25.691918 31432 ts_tablet_manager.cc:1403] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.105s	user 0.018s	sys 0.028s
I20260709 12:28:25.691120 31439 raft_consensus.cc:2804] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:25.692821 31487 consensus_queue.cc:1048] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:25.693413 31439 raft_consensus.cc:493] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:25.693919 31439 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.694020 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.694867 31391 raft_consensus.cc:3060] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.692587 31477 consensus_queue.cc:237] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.706061 31439 raft_consensus.cc:515] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.705546 31139 catalog_manager.cc:5697] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 1 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:25.709635 31436 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f: No bootstrap required, opened a new log
I20260709 12:28:25.710350 31436 ts_tablet_manager.cc:1403] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f: Time spent bootstrapping tablet: real 0.143s	user 0.010s	sys 0.022s
I20260709 12:28:25.714671 31432 raft_consensus.cc:359] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.715595 31432 raft_consensus.cc:385] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.715874 31432 raft_consensus.cc:740] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.716770 31432 consensus_queue.cc:260] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.719383 31432 ts_tablet_manager.cc:1434] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.000s
I20260709 12:28:25.713634 31243 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
W20260709 12:28:25.724047 31284 leader_election.cc:343] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:25.726111 31242 raft_consensus.cc:2468] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.728657 31284 leader_election.cc:304] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:25.729601 31391 raft_consensus.cc:2468] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.730396 31432 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:25.736846 31477 raft_consensus.cc:2804] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.737614 31477 raft_consensus.cc:697] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:25.738673 31477 consensus_queue.cc:237] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.741350 31436 raft_consensus.cc:359] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.742238 31436 raft_consensus.cc:385] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.742573 31436 raft_consensus.cc:740] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.743585 31436 consensus_queue.cc:260] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.751169 31469 raft_consensus.cc:493] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.752122 31469 raft_consensus.cc:515] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.753965 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:25.752522 31439 leader_election.cc:290] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.753782 31138 catalog_manager.cc:5697] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.754976 31391 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.755883 31487 raft_consensus.cc:493] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.756572 31487 raft_consensus.cc:515] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.757187 31432 tablet_bootstrap.cc:654] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:25.754681 31436 ts_tablet_manager.cc:1434] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260709 12:28:25.758942 31487 leader_election.cc:290] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:25.759277 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.760972 31317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.761353 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.762329 31242 raft_consensus.cc:2393] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 451375e442ad416ebc7f5e58b8610f7f in current term 1: Already voted for candidate 6b512fc09fe2444a95f04d456239a9dd in this term.
I20260709 12:28:25.764214 31469 leader_election.cc:290] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.765076 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:25.766031 31318 raft_consensus.cc:2393] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67a00a279c8f4e86840326b53037f2e6 in current term 1: Already voted for candidate 451375e442ad416ebc7f5e58b8610f7f in this term.
I20260709 12:28:25.766148 31355 leader_election.cc:304] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f; no voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.767294 31206 leader_election.cc:304] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6; no voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:25.768811 31469 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.777036 31469 raft_consensus.cc:2749] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.778734 31487 raft_consensus.cc:3060] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.786392 31432 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: No bootstrap required, opened a new log
I20260709 12:28:25.787138 31432 ts_tablet_manager.cc:1403] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.057s	user 0.022s	sys 0.004s
I20260709 12:28:25.788368 31487 raft_consensus.cc:2749] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.788240 31391 raft_consensus.cc:2468] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:25.790019 31282 leader_election.cc:304] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.790488 31432 raft_consensus.cc:359] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.791419 31432 raft_consensus.cc:385] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:25.791697 31432 raft_consensus.cc:740] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:25.792469 31432 consensus_queue.cc:260] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.792482 31477 raft_consensus.cc:2804] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:25.793747 31477 raft_consensus.cc:697] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:25.794313 31438 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:25.794938 31432 ts_tablet_manager.cc:1434] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:25.794719 31477 consensus_queue.cc:237] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.808391 31469 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:25.809931 31138 catalog_manager.cc:5697] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.811933 31138 catalog_manager.cc:5697] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:25.824389 31477 raft_consensus.cc:493] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.825151 31477 raft_consensus.cc:515] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:25.827847 31477 leader_election.cc:290] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:25.829165 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:25.829313 31242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a00a279c8f4e86840326b53037f2e6" is_pre_election: true
I20260709 12:28:25.830473 31242 raft_consensus.cc:2393] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b512fc09fe2444a95f04d456239a9dd in current term 1: Already voted for candidate 451375e442ad416ebc7f5e58b8610f7f in this term.
I20260709 12:28:25.831696 31284 leader_election.cc:304] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b512fc09fe2444a95f04d456239a9dd; no voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:25.832495 31477 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:25.839972 31477 raft_consensus.cc:2749] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:25.848995 16792 ts_manager.cc:295] Set tserver state for 67a00a279c8f4e86840326b53037f2e6 to MAINTENANCE_MODE
I20260709 12:28:25.850160 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:28:25.872638 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:25.873755 16792 tablet_replica.cc:333] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.874943 16792 raft_consensus.cc:2243] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.875412 16792 raft_consensus.cc:2272] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.878015 16792 tablet_replica.cc:333] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.878873 16792 raft_consensus.cc:2243] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.879310 16792 raft_consensus.cc:2272] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.881807 16792 tablet_replica.cc:333] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.882496 16792 raft_consensus.cc:2243] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.882917 16792 raft_consensus.cc:2272] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.885010 16792 tablet_replica.cc:333] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.885591 16792 raft_consensus.cc:2243] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:25.886857 16792 raft_consensus.cc:2272] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.889571 16792 tablet_replica.cc:333] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.890309 16792 raft_consensus.cc:2243] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.890781 16792 raft_consensus.cc:2272] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.893218 16792 tablet_replica.cc:333] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.893962 16792 raft_consensus.cc:2243] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.894418 16792 raft_consensus.cc:2272] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.896503 16792 tablet_replica.cc:333] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.897109 16792 raft_consensus.cc:2243] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.897643 16792 raft_consensus.cc:2272] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.900382 16792 tablet_replica.cc:333] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.901059 16792 raft_consensus.cc:2243] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.901517 16792 raft_consensus.cc:2272] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.904274 16792 tablet_replica.cc:333] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
I20260709 12:28:25.905079 16792 raft_consensus.cc:2243] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.905731 16792 raft_consensus.cc:2272] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.908490 16792 tablet_replica.cc:333] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: stopping tablet replica
W20260709 12:28:25.908316 31284 consensus_peers.cc:597] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:25.909278 16792 raft_consensus.cc:2243] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:25.909749 16792 raft_consensus.cc:2272] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:25.944805 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
I20260709 12:28:25.976135 31477 consensus_queue.cc:1048] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:25.978996 31358 consensus_peers.cc:597] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:25.991091 31479 consensus_queue.cc:1048] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:25.998152 31487 consensus_queue.cc:1048] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:26.012122 31284 consensus_peers.cc:597] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:26.017767 31284 consensus_peers.cc:597] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:26.058266 31507 consensus_queue.cc:1048] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:26.068217 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:26.092031 31284 consensus_peers.cc:597] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:26.095611 31358 consensus_peers.cc:597] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:26.119577 31507 consensus_queue.cc:1048] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:26.148098 31358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111) [suppressed 226 similar messages]
W20260709 12:28:26.151185 31358 consensus_peers.cc:597] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:26.164017 31507 consensus_queue.cc:1048] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:26.179224 31284 consensus_peers.cc:597] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:26.256135 31507 consensus_queue.cc:1048] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:26.275341 31284 consensus_peers.cc:597] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:26.275490 31487 consensus_queue.cc:1048] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:27.069495 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:27.308523 31477 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67a00a279c8f4e86840326b53037f2e6)
I20260709 12:28:27.309250 31477 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:27.311265 31477 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:27.312500 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:27.313517 31391 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 1.
I20260709 12:28:27.314074 31494 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67a00a279c8f4e86840326b53037f2e6)
I20260709 12:28:27.314970 31282 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:27.315171 31494 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:27.316064 31477 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:28:27.316490 31284 leader_election.cc:336] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111)
I20260709 12:28:27.316520 31477 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67a00a279c8f4e86840326b53037f2e6)
I20260709 12:28:27.317278 31477 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:27.318845 31494 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:27.319078 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
W20260709 12:28:27.322176 31358 leader_election.cc:336] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111)
I20260709 12:28:27.326014 31477 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:27.327293 31318 raft_consensus.cc:2393] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 451375e442ad416ebc7f5e58b8610f7f in current term 2: Already voted for candidate 6b512fc09fe2444a95f04d456239a9dd in this term.
I20260709 12:28:27.328210 31477 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:27.329481 31355 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f; no voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:27.329689 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:27.330485 31391 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:27.333464 31284 leader_election.cc:336] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111)
I20260709 12:28:27.338521 31391 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 2.
I20260709 12:28:27.339131 31494 raft_consensus.cc:2749] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:27.340374 31282 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 67a00a279c8f4e86840326b53037f2e6
I20260709 12:28:27.341677 31477 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:27.342754 31477 raft_consensus.cc:697] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:27.343856 31477 consensus_queue.cc:237] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:27.353561 31139 catalog_manager.cc:5697] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 1 to 2, leader changed from 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 2 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:27.799765 31392 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:27.801651 31477 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:28:27.817850 31284 consensus_peers.cc:597] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:28.095185 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:28.317512 31358 consensus_peers.cc:597] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.411653 31284 consensus_peers.cc:597] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.433976 31284 consensus_peers.cc:597] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.447357 31284 consensus_peers.cc:597] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.620060 31284 consensus_peers.cc:597] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.622709 31358 consensus_peers.cc:597] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.757110 31284 consensus_peers.cc:597] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.773356 31358 consensus_peers.cc:597] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:28.845366 31284 consensus_peers.cc:597] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:29.076316 31413 debug-util.cc:398] Leaking SignalData structure 0x7b0800108780 after lost signal to thread 16795
W20260709 12:28:29.077665 31413 debug-util.cc:398] Leaking SignalData structure 0x7b0800162de0 after lost signal to thread 31175
W20260709 12:28:29.079171 31413 debug-util.cc:398] Leaking SignalData structure 0x7b0800201640 after lost signal to thread 31342
W20260709 12:28:29.080439 31413 debug-util.cc:398] Leaking SignalData structure 0x7b0800009500 after lost signal to thread 31416
W20260709 12:28:29.096524 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:29.936584 31284 consensus_peers.cc:597] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:30.097900 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:30.630095 31358 consensus_peers.cc:597] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:28:30.677487 31284 consensus_peers.cc:597] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:28:30.791553 31284 consensus_peers.cc:597] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:28:30.859503 31284 consensus_peers.cc:597] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:31.027567 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:31.039983 31532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:31.041956 31533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:31.048161 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:31.049083 31536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:31.050738 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:31.051075 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:31.051311 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600111051288 us; error 0 us; skew 500 ppm
I20260709 12:28:31.052037 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:31.056463 16792 webserver.cc:533] Webserver started at http://127.16.102.1:42945/ using document root <none> and password file <none>
I20260709 12:28:31.057220 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:31.057477 16792 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:28:31.063112 16792 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.003s
I20260709 12:28:31.069084 31540 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:31.070319 16792 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.001s
I20260709 12:28:31.070724 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "67a00a279c8f4e86840326b53037f2e6"
format_stamp: "Formatted at 2026-07-09 12:28:24 on dist-test-slave-7xt3"
I20260709 12:28:31.071290 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260709 12:28:31.099682 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:31.110023 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:31.111935 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:31.114082 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:31.123643 31547 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
I20260709 12:28:31.206032 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:28:31.206470 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.084s	user 0.003s	sys 0.000s
I20260709 12:28:31.206875 16792 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
W20260709 12:28:31.214587 31358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20260709 12:28:31.218415 31358 consensus_peers.cc:597] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:31.218135 31547 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
W20260709 12:28:31.230566 31284 consensus_peers.cc:597] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:31.239651 31547 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.241111 31547 tablet_bootstrap.cc:492] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.242187 31547 ts_tablet_manager.cc:1403] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.011s
I20260709 12:28:31.246173 31547 raft_consensus.cc:359] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.247588 31547 raft_consensus.cc:385] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.247995 31547 raft_consensus.cc:740] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.249169 31547 consensus_queue.cc:260] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.257586 31547 ts_tablet_manager.cc:1434] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.008s
I20260709 12:28:31.258661 31547 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.282824 31547 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.283905 31547 tablet_bootstrap.cc:492] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.284740 31547 ts_tablet_manager.cc:1403] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.009s
I20260709 12:28:31.288405 31547 raft_consensus.cc:359] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.289286 31547 raft_consensus.cc:740] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.290452 31547 consensus_queue.cc:260] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
W20260709 12:28:31.291975 31358 consensus_peers.cc:597] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:31.292886 31547 ts_tablet_manager.cc:1434] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.002s
I20260709 12:28:31.294534 31547 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
W20260709 12:28:31.300591 31284 consensus_peers.cc:597] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:31.305490 16792 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:28:31.306216 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.099s	user 0.080s	sys 0.013s
I20260709 12:28:31.320021 31547 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.321295 31547 tablet_bootstrap.cc:492] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.322310 31547 ts_tablet_manager.cc:1403] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.011s
I20260709 12:28:31.326856 31547 raft_consensus.cc:359] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.327943 31547 raft_consensus.cc:740] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.329362 31547 consensus_queue.cc:260] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.333169 31547 ts_tablet_manager.cc:1434] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 12:28:31.334637 31547 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
W20260709 12:28:31.356206 31284 consensus_peers.cc:597] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:31.363690 31549 raft_consensus.cc:493] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.364581 31549 raft_consensus.cc:515] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.375577 31549 leader_election.cc:290] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:31.390543 31547 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.391602 31547 tablet_bootstrap.cc:492] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.392345 31547 ts_tablet_manager.cc:1403] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.058s	user 0.017s	sys 0.005s
I20260709 12:28:31.395955 31547 raft_consensus.cc:359] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.396741 31547 raft_consensus.cc:740] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.397797 31547 consensus_queue.cc:260] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.400707 31547 ts_tablet_manager.cc:1434] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:28:31.401765 31547 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.410387 31391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451375e442ad416ebc7f5e58b8610f7f" is_pre_election: true
I20260709 12:28:31.412726 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "67a00a279c8f4e86840326b53037f2e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b512fc09fe2444a95f04d456239a9dd" is_pre_election: true
I20260709 12:28:31.415987 31541 leader_election.cc:304] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6; no voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:31.417805 31549 raft_consensus.cc:2749] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:31.429349 31547 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.431212 31547 tablet_bootstrap.cc:492] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.432428 31547 ts_tablet_manager.cc:1403] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.031s	user 0.017s	sys 0.008s
I20260709 12:28:31.436726 31547 raft_consensus.cc:359] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.438103 31547 raft_consensus.cc:740] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.439328 31547 consensus_queue.cc:260] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.442891 31547 ts_tablet_manager.cc:1434] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:28:31.444453 31547 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.468290 31547 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.469769 31547 tablet_bootstrap.cc:492] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.470919 31547 ts_tablet_manager.cc:1403] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.027s	user 0.025s	sys 0.000s
I20260709 12:28:31.475302 31547 raft_consensus.cc:359] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.476334 31547 raft_consensus.cc:740] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.477660 31547 consensus_queue.cc:260] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.481299 31547 ts_tablet_manager.cc:1434] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:28:31.481334 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:38611
I20260709 12:28:31.481791 31614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:38611 every 8 connection(s)
I20260709 12:28:31.482815 31547 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.509265 31547 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.511996 31547 tablet_bootstrap.cc:492] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.513000 31547 ts_tablet_manager.cc:1403] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.031s	user 0.017s	sys 0.008s
I20260709 12:28:31.514056 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:31.516958 31547 raft_consensus.cc:359] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.518213 31547 raft_consensus.cc:740] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.519564 31547 consensus_queue.cc:260] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.527438 31547 ts_tablet_manager.cc:1434] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:28:31.528765 31547 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.531118 31615 heartbeater.cc:344] Connected to a master server at 127.16.102.62:45863
I20260709 12:28:31.531844 31615 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:31.533239 31615 heartbeater.cc:507] Master 127.16.102.62:45863 requested a full tablet report, sending...
I20260709 12:28:31.548168 31138 ts_manager.cc:194] Re-registered known tserver with Master: 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611)
I20260709 12:28:31.556180 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:31.557377 31318 raft_consensus.cc:606] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:31.561760 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:31.562762 31318 raft_consensus.cc:606] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:31.563017 31138 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46746
I20260709 12:28:31.566859 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:31.567771 31318 raft_consensus.cc:606] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:31.568310 31615 heartbeater.cc:499] Master 127.16.102.62:45863 was elected leader, sending a full tablet report...
I20260709 12:28:31.569950 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:31.570904 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:31.571861 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:31.589323 31547 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.590643 31547 tablet_bootstrap.cc:492] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.591595 31547 ts_tablet_manager.cc:1403] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.063s	user 0.029s	sys 0.022s
I20260709 12:28:31.595335 31547 raft_consensus.cc:359] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.596204 31547 raft_consensus.cc:740] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.597173 31547 consensus_queue.cc:260] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.599936 31547 ts_tablet_manager.cc:1434] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:31.601171 31547 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.622160 31547 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.623498 31547 tablet_bootstrap.cc:492] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.624467 31547 ts_tablet_manager.cc:1403] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.024s	user 0.020s	sys 0.000s
I20260709 12:28:31.627998 31547 raft_consensus.cc:359] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.628957 31547 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.630045 31547 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.633010 31547 ts_tablet_manager.cc:1434] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:31.634212 31547 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap starting.
I20260709 12:28:31.654333 31547 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:31.655526 31547 tablet_bootstrap.cc:492] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Bootstrap complete.
I20260709 12:28:31.656492 31547 ts_tablet_manager.cc:1403] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.000s
I20260709 12:28:31.659962 31547 raft_consensus.cc:359] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.660887 31547 raft_consensus.cc:740] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a00a279c8f4e86840326b53037f2e6, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.661962 31547 consensus_queue.cc:260] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.664913 31547 ts_tablet_manager.cc:1434] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:31.670732 31528 raft_consensus.cc:993] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:31.671705 31531 raft_consensus.cc:1081] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:31.674422 31391 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:31.675249 31391 raft_consensus.cc:493] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:31.675726 31391 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.689410 31391 raft_consensus.cc:515] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.690214 31589 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.695756 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:31.696700 31318 raft_consensus.cc:3055] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:31.697116 31318 raft_consensus.cc:740] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:31.698356 31318 consensus_queue.cc:260] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.699939 31318 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.710418 31391 leader_election.cc:290] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:31.711490 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:31.712528 31588 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.721130 31318 raft_consensus.cc:2468] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:31.723664 31355 leader_election.cc:304] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:31.725178 31517 raft_consensus.cc:2804] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:31.728597 31588 raft_consensus.cc:2468] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:31.733955 31517 raft_consensus.cc:697] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:31.741312 31517 consensus_queue.cc:237] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.766292 31139 catalog_manager.cc:5697] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 1 to 2, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 2 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:31.890250 31588 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.947434 31531 raft_consensus.cc:993] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:31.948017 31528 raft_consensus.cc:1081] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:31.950161 31391 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:31.950802 31391 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:31.951175 31391 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:31.961948 31391 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.965260 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:31.966400 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:31.966836 31588 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:31.967198 31318 raft_consensus.cc:3055] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:31.967545 31318 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:31.968668 31318 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.970162 31318 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:31.977260 31588 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 3.
I20260709 12:28:31.978058 31391 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 3 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:31.978989 31358 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:31.980435 31627 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:31.981000 31627 raft_consensus.cc:697] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:31.982249 31627 consensus_queue.cc:237] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:31.984329 31528 raft_consensus.cc:993] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:31.984961 31528 raft_consensus.cc:1081] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:31.986819 31318 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 3.
I20260709 12:28:31.989140 31392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:31.990188 31392 raft_consensus.cc:493] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:31.990760 31392 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.995113 31138 catalog_manager.cc:5697] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 2 to 3, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 3 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:32.002732 31392 raft_consensus.cc:515] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:32.006505 31392 leader_election.cc:290] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:32.007048 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:32.007913 31318 raft_consensus.cc:3055] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:32.008255 31318 raft_consensus.cc:740] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:32.009531 31318 consensus_queue.cc:260] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:32.010672 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:32.011268 31318 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:32.012431 31588 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:32.022308 31588 raft_consensus.cc:2468] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:32.024044 31318 raft_consensus.cc:2468] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:32.024052 31358 leader_election.cc:304] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:32.025707 31627 raft_consensus.cc:2804] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:32.026353 31627 raft_consensus.cc:697] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:32.027499 31627 consensus_queue.cc:237] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:32.037592 31138 catalog_manager.cc:5697] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 1 to 2, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 2 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:32.101203 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:32.297950 31588 raft_consensus.cc:1275] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:32.299893 31627 consensus_queue.cc:1048] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:32.328614 31318 raft_consensus.cc:1275] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:32.330634 31625 consensus_queue.cc:1048] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:32.491036 31588 raft_consensus.cc:1275] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:32.492763 31625 consensus_queue.cc:1048] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:32.505096 31318 raft_consensus.cc:1275] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:32.508755 31627 consensus_queue.cc:1048] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:32.578056 31588 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:32.578148 31317 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:32.580046 31517 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:32.580837 31627 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:32.585081 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:32.619858 31392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
mode: GRACEFUL
new_leader_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:54300
I20260709 12:28:32.620647 31392 raft_consensus.cc:606] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Received request to transfer leadership to TS 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:32.623747 31392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
mode: GRACEFUL
new_leader_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:54300
I20260709 12:28:32.624426 31392 raft_consensus.cc:606] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Received request to transfer leadership to TS 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:32.627394 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dc1054676023472eab95b1e45e804e71"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:32.628005 31318 raft_consensus.cc:606] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:32.629427 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:32.630111 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:32.630861 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:33.038524 31643 raft_consensus.cc:993] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:33.039057 31531 raft_consensus.cc:1081] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:33.041006 31392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dc1054676023472eab95b1e45e804e71"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:33.041625 31392 raft_consensus.cc:493] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.042057 31392 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.050439 31392 raft_consensus.cc:515] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.051752 31627 raft_consensus.cc:993] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: : Instructing follower 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:33.052500 31650 raft_consensus.cc:1081] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Signalling peer 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:33.052966 31392 leader_election.cc:290] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:33.053387 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:33.053684 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:33.054348 31588 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.054473 31318 raft_consensus.cc:3055] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:33.056864 31318 raft_consensus.cc:740] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:33.057590 31317 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:39454
I20260709 12:28:33.058387 31317 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.058104 31318 consensus_queue.cc:260] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.058876 31317 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:33.059973 31318 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.066669 31317 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.067353 31588 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:33.069099 31358 leader_election.cc:304] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:33.070190 31650 raft_consensus.cc:2804] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:33.070324 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:33.070616 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:33.070942 31650 raft_consensus.cc:697] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:33.071552 31588 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:33.071617 31392 raft_consensus.cc:3055] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:33.072132 31392 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:33.072275 31650 consensus_queue.cc:237] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.070454 31318 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:33.073381 31392 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.074755 31392 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:33.082939 31588 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 4.
I20260709 12:28:33.084046 31317 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 4 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:33.084721 31392 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 4.
I20260709 12:28:33.085036 31284 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:33.086187 31531 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:33.086812 31531 raft_consensus.cc:697] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:33.087098 31139 catalog_manager.cc:5697] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 1 to 2, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 2 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:33.087738 31531 consensus_queue.cc:237] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.096302 31138 catalog_manager.cc:5697] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 3 to 4, leader changed from 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 4 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:33.106590 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:33.126432 31627 raft_consensus.cc:993] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: : Instructing follower 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:33.127100 31650 raft_consensus.cc:1081] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Signalling peer 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:33.128958 31317 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:39454
I20260709 12:28:33.129642 31317 raft_consensus.cc:493] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.130113 31317 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.136965 31317 raft_consensus.cc:515] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.139097 31317 leader_election.cc:290] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:33.140394 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:33.140394 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:33.141571 31392 raft_consensus.cc:3055] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:33.141572 31588 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.142083 31392 raft_consensus.cc:740] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:33.143438 31392 consensus_queue.cc:260] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.145148 31392 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.152187 31588 raft_consensus.cc:2468] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 2.
I20260709 12:28:33.153802 31284 leader_election.cc:304] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:33.154817 31531 raft_consensus.cc:2804] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:33.154867 31392 raft_consensus.cc:2468] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 2.
I20260709 12:28:33.155377 31531 raft_consensus.cc:697] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:33.156507 31531 consensus_queue.cc:237] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.165621 31138 catalog_manager.cc:5697] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 1 to 2, leader changed from 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 2 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:33.563524 31317 raft_consensus.cc:1275] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:33.565121 31650 consensus_queue.cc:1048] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:33.571728 31589 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 4 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:33.574282 31531 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:28:33.589929 31392 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 4 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:33.591894 31643 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:28:33.602301 31588 raft_consensus.cc:1275] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:33.605983 31627 consensus_queue.cc:1048] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:33.620694 31589 raft_consensus.cc:1275] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:33.627576 31643 consensus_queue.cc:1048] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:33.638388 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:33.650491 31392 raft_consensus.cc:1275] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:33.654755 31643 consensus_queue.cc:1048] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:33.662999 31317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:33.664058 31317 raft_consensus.cc:606] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:33.669291 31317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:33.670171 31317 raft_consensus.cc:606] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:33.680531 31392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dc1054676023472eab95b1e45e804e71"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
mode: GRACEFUL
new_leader_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:54300
I20260709 12:28:33.681465 31392 raft_consensus.cc:606] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Received request to transfer leadership to TS 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:33.684248 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:33.684882 31674 raft_consensus.cc:993] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:33.685287 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:33.685398 31531 raft_consensus.cc:993] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:33.685592 31674 raft_consensus.cc:1081] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:33.686151 31531 raft_consensus.cc:1081] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:33.686390 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:33.688314 31392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:33.689209 31392 raft_consensus.cc:493] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.691790 31390 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:33.692602 31390 raft_consensus.cc:493] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.692999 31390 raft_consensus.cc:3060] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.693857 31392 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:33.703850 31390 raft_consensus.cc:515] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.704384 31392 raft_consensus.cc:515] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.705554 31391 raft_consensus.cc:1240] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Rejecting Update request from peer 6b512fc09fe2444a95f04d456239a9dd for earlier term 2. Current term is 3. Ops: []
I20260709 12:28:33.706611 31390 leader_election.cc:290] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:33.708488 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:33.708845 31317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:33.708925 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c0212c06fb4945ad761212c1868fb1" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:33.709432 31588 raft_consensus.cc:3060] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.709547 31317 raft_consensus.cc:3055] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:33.709692 31318 raft_consensus.cc:3055] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:33.710050 31317 raft_consensus.cc:740] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:33.710230 31318 raft_consensus.cc:740] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:33.711310 31317 consensus_queue.cc:260] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.711347 31318 consensus_queue.cc:260] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.712795 31589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c8e791b40f4efd9fc22519ed30b9ed" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
W20260709 12:28:33.713414 31531 consensus_queue.cc:1175] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "451375e442ad416ebc7f5e58b8610f7f" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6b512fc09fe2444a95f04d456239a9dd for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:33.714401 31589 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:33.714885 31318 raft_consensus.cc:3060] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:33.717546 31317 raft_consensus.cc:3060] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:33.720999 31588 raft_consensus.cc:2468] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:33.713466 31392 leader_election.cc:290] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 3 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:33.722961 31358 leader_election.cc:304] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:33.724117 31650 raft_consensus.cc:2804] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:33.724689 31650 raft_consensus.cc:697] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:33.725504 31318 raft_consensus.cc:2468] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:33.725940 31650 consensus_queue.cc:237] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.726779 31317 raft_consensus.cc:2468] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 3.
I20260709 12:28:33.728008 31355 leader_election.cc:304] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:33.729080 31662 raft_consensus.cc:2804] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:33.729188 31589 raft_consensus.cc:2468] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 3.
I20260709 12:28:33.729668 31662 raft_consensus.cc:697] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:33.730772 31662 consensus_queue.cc:237] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:33.736996 31138 catalog_manager.cc:5697] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 1 to 2, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 2 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:33.755005 31138 catalog_manager.cc:5697] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 2 to 3, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 3 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:34.107800 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:34.142419 31650 raft_consensus.cc:993] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: : Instructing follower 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:34.143164 31662 raft_consensus.cc:1081] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Signalling peer 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:34.145148 31318 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dc1054676023472eab95b1e45e804e71"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:39454
I20260709 12:28:34.145780 31318 raft_consensus.cc:493] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.146297 31318 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:34.152994 31318 raft_consensus.cc:515] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.154953 31318 leader_election.cc:290] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 3 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:34.156085 31589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:34.156479 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc1054676023472eab95b1e45e804e71" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:34.156888 31589 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:34.157194 31392 raft_consensus.cc:3055] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:34.157573 31392 raft_consensus.cc:740] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:34.158455 31392 consensus_queue.cc:260] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.159627 31392 raft_consensus.cc:3060] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:34.164824 31589 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 3.
I20260709 12:28:34.166725 31284 leader_election.cc:304] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:34.167454 31318 raft_consensus.cc:1275] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:34.167606 31531 raft_consensus.cc:2804] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:34.167358 31392 raft_consensus.cc:2468] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 3.
I20260709 12:28:34.168514 31531 raft_consensus.cc:697] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 3 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:34.169457 31531 consensus_queue.cc:237] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.170428 31662 consensus_queue.cc:1048] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:34.178442 31138 catalog_manager.cc:5697] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 2 to 3, leader changed from 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 3 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.194252 31589 raft_consensus.cc:1275] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:34.195595 31318 raft_consensus.cc:1275] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:34.196444 31650 consensus_queue.cc:1048] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:34.197525 31627 consensus_queue.cc:1048] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:34.210407 31588 raft_consensus.cc:1275] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:34.212831 31662 consensus_queue.cc:1048] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:34.623252 31392 raft_consensus.cc:1275] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:34.624928 31531 consensus_queue.cc:1048] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:34.636929 31589 raft_consensus.cc:1275] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:34.640249 31687 consensus_queue.cc:1048] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:34.704950 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:34.714586 31392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
mode: GRACEFUL
new_leader_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:54300
I20260709 12:28:34.715305 31392 raft_consensus.cc:606] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Received request to transfer leadership to TS 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:34.718619 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:34.719293 31318 raft_consensus.cc:606] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:34.722853 31392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
mode: GRACEFUL
new_leader_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:54300
I20260709 12:28:34.723568 31392 raft_consensus.cc:606] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Received request to transfer leadership to TS 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:34.725039 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:34.726037 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:34.726809 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:34.843580 31662 raft_consensus.cc:993] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: : Instructing follower 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:34.844146 31662 raft_consensus.cc:1081] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Signalling peer 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:34.846058 31318 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:39454
I20260709 12:28:34.846658 31318 raft_consensus.cc:493] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.846985 31318 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:34.853752 31318 raft_consensus.cc:515] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.855610 31318 leader_election.cc:290] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 3 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:34.856829 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:34.856829 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d94e0ae944a1430a9b4def74a74cc1a3" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:34.857925 31392 raft_consensus.cc:3055] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:34.858071 31588 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:34.858258 31392 raft_consensus.cc:740] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:34.859265 31392 consensus_queue.cc:260] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.860342 31392 raft_consensus.cc:3060] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:34.865489 31588 raft_consensus.cc:2468] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 3.
I20260709 12:28:34.866570 31392 raft_consensus.cc:2468] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 3.
I20260709 12:28:34.867414 31284 leader_election.cc:304] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:34.868647 31688 raft_consensus.cc:2804] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:34.869164 31688 raft_consensus.cc:697] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 3 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:34.870172 31688 consensus_queue.cc:237] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.881170 31140 catalog_manager.cc:5697] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 2 to 3, leader changed from 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 3 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.950769 31531 raft_consensus.cc:993] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:34.951359 31688 raft_consensus.cc:1081] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:34.953187 31392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:34.953783 31392 raft_consensus.cc:493] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.954180 31392 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:34.960387 31392 raft_consensus.cc:515] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.962097 31392 leader_election.cc:290] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 5 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:34.963420 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:34.963435 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac47c2fc9c64d0e8627065d6c366bd0" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:34.964257 31588 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:34.964161 31318 raft_consensus.cc:3055] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:28:34.964893 31318 raft_consensus.cc:740] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:34.965732 31318 consensus_queue.cc:260] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.966883 31318 raft_consensus.cc:3060] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:34.971515 31588 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 5.
I20260709 12:28:34.973206 31358 leader_election.cc:304] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:34.973296 31318 raft_consensus.cc:2468] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 5.
I20260709 12:28:34.974278 31662 raft_consensus.cc:2804] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:28:34.974795 31662 raft_consensus.cc:697] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 5 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:34.975713 31662 consensus_queue.cc:237] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:34.985289 31140 catalog_manager.cc:5697] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 4 to 5, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 5 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:35.007417 31650 raft_consensus.cc:993] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: : Instructing follower 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:35.007956 31662 raft_consensus.cc:1081] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Signalling peer 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:35.009728 31318 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:39454
I20260709 12:28:35.010399 31318 raft_consensus.cc:493] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:35.010707 31318 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:35.017398 31318 raft_consensus.cc:515] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:35.019361 31318 leader_election.cc:290] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:35.020495 31588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:35.020645 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:35.021412 31588 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:35.021755 31392 raft_consensus.cc:3055] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:35.022143 31392 raft_consensus.cc:740] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:35.023146 31392 consensus_queue.cc:260] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:35.024825 31392 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:35.030042 31588 raft_consensus.cc:2468] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 2.
I20260709 12:28:35.031585 31284 leader_election.cc:304] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:35.032400 31688 raft_consensus.cc:2804] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:35.032995 31688 raft_consensus.cc:697] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:35.033507 31392 raft_consensus.cc:2468] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 2.
I20260709 12:28:35.034188 31688 consensus_queue.cc:237] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:35.043817 31140 catalog_manager.cc:5697] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 1 to 2, leader changed from 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 2 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:35.109139 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:35.357583 31318 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 5 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:28:35.359421 31662 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:28:35.371100 31589 raft_consensus.cc:1275] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 5 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:28:35.373807 31697 consensus_queue.cc:1048] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:28:35.379006 31392 raft_consensus.cc:1275] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:35.380782 31688 consensus_queue.cc:1048] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:35.395161 31589 raft_consensus.cc:1275] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:35.397891 31531 consensus_queue.cc:1048] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:35.532636 31392 raft_consensus.cc:1275] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.534420 31531 consensus_queue.cc:1048] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.546022 31589 raft_consensus.cc:1275] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.548362 31673 consensus_queue.cc:1048] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:35.734150 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:35.742353 31392 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
mode: GRACEFUL
new_leader_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:54300
I20260709 12:28:35.743049 31392 raft_consensus.cc:606] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Received request to transfer leadership to TS 6b512fc09fe2444a95f04d456239a9dd
I20260709 12:28:35.746271 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:35.746825 31318 raft_consensus.cc:606] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:35.750005 31318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
mode: GRACEFUL
new_leader_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:39484
I20260709 12:28:35.750845 31318 raft_consensus.cc:606] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Received request to transfer leadership to TS 451375e442ad416ebc7f5e58b8610f7f
I20260709 12:28:35.752298 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:35.753086 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:35.753953 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:35.953476 31688 raft_consensus.cc:993] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:35.954123 31673 raft_consensus.cc:1081] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:35.956070 31392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:35.956885 31392 raft_consensus.cc:493] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:35.957263 31392 raft_consensus.cc:3060] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:35.964084 31392 raft_consensus.cc:515] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:35.966102 31392 leader_election.cc:290] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:35.967103 31589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:35.967474 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6679bf1e8fbc4631903b685e7e5dde0a" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:35.967931 31589 raft_consensus.cc:3060] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:35.968225 31318 raft_consensus.cc:3055] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:35.968586 31318 raft_consensus.cc:740] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:35.969411 31318 consensus_queue.cc:260] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:35.970786 31318 raft_consensus.cc:3060] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:35.976917 31589 raft_consensus.cc:2468] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:35.978624 31358 leader_election.cc:304] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:35.979561 31697 raft_consensus.cc:2804] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:35.980114 31697 raft_consensus.cc:697] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:35.981297 31697 consensus_queue.cc:237] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:35.983465 31318 raft_consensus.cc:2468] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 2.
I20260709 12:28:35.990590 31140 catalog_manager.cc:5697] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 1 to 2, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 2 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:36.019238 31531 raft_consensus.cc:993] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: : Instructing follower 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:36.019874 31531 raft_consensus.cc:1081] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Signalling peer 451375e442ad416ebc7f5e58b8610f7f to start an election
I20260709 12:28:36.021766 31392 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4"
dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
 from {username='slave'} at 127.0.0.1:54278
I20260709 12:28:36.022392 31392 raft_consensus.cc:493] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:36.022711 31392 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:36.031203 31392 raft_consensus.cc:515] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:36.032992 31392 leader_election.cc:290] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 3 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733)
I20260709 12:28:36.034284 31318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
I20260709 12:28:36.034155 31589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "653fa7d1e4f549e792dc2c6e2972a3d4" candidate_uuid: "451375e442ad416ebc7f5e58b8610f7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:36.035209 31318 raft_consensus.cc:3055] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:36.035225 31589 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:36.035691 31318 raft_consensus.cc:740] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:36.036648 31318 consensus_queue.cc:260] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:36.038214 31318 raft_consensus.cc:3060] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:36.045440 31589 raft_consensus.cc:2468] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 3.
I20260709 12:28:36.047102 31358 leader_election.cc:304] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451375e442ad416ebc7f5e58b8610f7f, 67a00a279c8f4e86840326b53037f2e6; no voters: 
I20260709 12:28:36.047547 31318 raft_consensus.cc:2468] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 451375e442ad416ebc7f5e58b8610f7f in term 3.
I20260709 12:28:36.047991 31697 raft_consensus.cc:2804] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:36.048597 31697 raft_consensus.cc:697] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Becoming Leader. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:36.049585 31697 consensus_queue.cc:237] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:36.057325 31140 catalog_manager.cc:5697] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f reported cstate change: term changed from 2 to 3, leader changed from 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2) to 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3). New cstate: current_term: 3 leader_uuid: "451375e442ad416ebc7f5e58b8610f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:36.074186 31662 raft_consensus.cc:993] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: : Instructing follower 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:36.074769 31697 raft_consensus.cc:1081] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Signalling peer 6b512fc09fe2444a95f04d456239a9dd to start an election
I20260709 12:28:36.076411 31318 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb"
dest_uuid: "6b512fc09fe2444a95f04d456239a9dd"
 from {username='slave'} at 127.0.0.1:39454
I20260709 12:28:36.076946 31318 raft_consensus.cc:493] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:36.077250 31318 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:36.083235 31318 raft_consensus.cc:515] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:36.084949 31318 leader_election.cc:290] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 3 election: Requested vote from peers 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611), 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3:32901)
I20260709 12:28:36.086323 31392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "451375e442ad416ebc7f5e58b8610f7f"
I20260709 12:28:36.086323 31589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32deb0d4043d46dfae3cc0aa450a34cb" candidate_uuid: "6b512fc09fe2444a95f04d456239a9dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67a00a279c8f4e86840326b53037f2e6"
I20260709 12:28:36.087617 31392 raft_consensus.cc:3055] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:36.087617 31589 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:36.088115 31392 raft_consensus.cc:740] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 451375e442ad416ebc7f5e58b8610f7f, State: Running, Role: LEADER
I20260709 12:28:36.089031 31392 consensus_queue.cc:260] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
I20260709 12:28:36.090512 31392 raft_consensus.cc:3060] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:36.099116 31589 raft_consensus.cc:2468] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 3.
I20260709 12:28:36.100734 31392 raft_consensus.cc:2468] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b512fc09fe2444a95f04d456239a9dd in term 3.
I20260709 12:28:36.100890 31284 leader_election.cc:304] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a00a279c8f4e86840326b53037f2e6, 6b512fc09fe2444a95f04d456239a9dd; no voters: 
I20260709 12:28:36.102046 31531 raft_consensus.cc:2804] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:36.102602 31531 raft_consensus.cc:697] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 3 LEADER]: Becoming Leader. State: Replica: 6b512fc09fe2444a95f04d456239a9dd, State: Running, Role: LEADER
I20260709 12:28:36.103466 31531 consensus_queue.cc:237] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } }
W20260709 12:28:36.110306 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:36.111671 31140 catalog_manager.cc:5697] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd reported cstate change: term changed from 2 to 3, leader changed from 451375e442ad416ebc7f5e58b8610f7f (127.16.102.3) to 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2). New cstate: current_term: 3 leader_uuid: "6b512fc09fe2444a95f04d456239a9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:36.379691 31589 raft_consensus.cc:1275] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:36.381397 31697 consensus_queue.cc:1048] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:36.392057 31318 raft_consensus.cc:1275] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:36.394418 31697 consensus_queue.cc:1048] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:36.507669 31589 raft_consensus.cc:1275] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:36.509268 31697 consensus_queue.cc:1048] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:36.519057 31318 raft_consensus.cc:1275] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Refusing update from remote peer 451375e442ad416ebc7f5e58b8610f7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:36.521139 31697 consensus_queue.cc:1048] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b512fc09fe2444a95f04d456239a9dd" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:36.557345 31392 raft_consensus.cc:1275] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:36.558935 31531 consensus_queue.cc:1048] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "451375e442ad416ebc7f5e58b8610f7f" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:36.572274 31589 raft_consensus.cc:1275] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Refusing update from remote peer 6b512fc09fe2444a95f04d456239a9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:36.573920 31718 consensus_queue.cc:1048] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a00a279c8f4e86840326b53037f2e6" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:36.761219 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:36.795519 16792 tablet_server.cc:179] TabletServer@127.16.102.1:38611 shutting down...
I20260709 12:28:36.828747 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:36.829650 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.830574 16792 raft_consensus.cc:2243] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.831146 16792 raft_consensus.cc:2272] T aac47c2fc9c64d0e8627065d6c366bd0 P 67a00a279c8f4e86840326b53037f2e6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.833570 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.834312 16792 raft_consensus.cc:2243] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.834784 16792 raft_consensus.cc:2272] T 36836434931140938a0c90e537ebd1e9 P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.837349 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.838253 16792 raft_consensus.cc:2243] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.838840 16792 raft_consensus.cc:2272] T dc1054676023472eab95b1e45e804e71 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.841586 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.842262 16792 raft_consensus.cc:2243] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.842732 16792 raft_consensus.cc:2272] T d6c0212c06fb4945ad761212c1868fb1 P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.844834 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.845402 16792 raft_consensus.cc:2243] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.845923 16792 raft_consensus.cc:2272] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.848868 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.849530 16792 raft_consensus.cc:2243] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.850061 16792 raft_consensus.cc:2272] T d94e0ae944a1430a9b4def74a74cc1a3 P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.852519 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.853147 16792 raft_consensus.cc:2243] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.853622 16792 raft_consensus.cc:2272] T b8c8e791b40f4efd9fc22519ed30b9ed P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.855909 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.856444 16792 raft_consensus.cc:2243] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.856860 16792 raft_consensus.cc:2272] T 6679bf1e8fbc4631903b685e7e5dde0a P 67a00a279c8f4e86840326b53037f2e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.859622 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.860183 16792 raft_consensus.cc:2243] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.860570 16792 raft_consensus.cc:2272] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 67a00a279c8f4e86840326b53037f2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.862710 16792 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:36.863219 16792 raft_consensus.cc:2243] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.863663 16792 raft_consensus.cc:2272] T 32deb0d4043d46dfae3cc0aa450a34cb P 67a00a279c8f4e86840326b53037f2e6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.898720 16792 tablet_server.cc:196] TabletServer@127.16.102.1:38611 shutdown complete.
W20260709 12:28:36.927002 31284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260709 12:28:36.931311 31284 consensus_peers.cc:597] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:36.933370 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
I20260709 12:28:36.964129 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:36.965130 16792 tablet_replica.cc:333] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.966508 16792 raft_consensus.cc:2243] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.967233 16792 raft_consensus.cc:2272] T d6c0212c06fb4945ad761212c1868fb1 P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:36.970072 31284 consensus_peers.cc:597] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:36.971071 16792 tablet_replica.cc:333] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.971849 16792 raft_consensus.cc:2243] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:36.972846 16792 raft_consensus.cc:2272] T 32deb0d4043d46dfae3cc0aa450a34cb P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.975958 16792 tablet_replica.cc:333] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.976660 16792 raft_consensus.cc:2243] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:36.977911 16792 raft_consensus.cc:2272] T dc1054676023472eab95b1e45e804e71 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.980873 16792 tablet_replica.cc:333] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.981702 16792 raft_consensus.cc:2243] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.982366 16792 raft_consensus.cc:2272] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.986253 16792 tablet_replica.cc:333] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.987380 16792 raft_consensus.cc:2243] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:36.988515 16792 raft_consensus.cc:2272] T d94e0ae944a1430a9b4def74a74cc1a3 P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.991726 16792 tablet_replica.cc:333] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.992514 16792 raft_consensus.cc:2243] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.993108 16792 raft_consensus.cc:2272] T 6679bf1e8fbc4631903b685e7e5dde0a P 6b512fc09fe2444a95f04d456239a9dd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:36.996769 16792 tablet_replica.cc:333] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:36.997577 16792 raft_consensus.cc:2243] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:36.998238 16792 raft_consensus.cc:2272] T b8c8e791b40f4efd9fc22519ed30b9ed P 6b512fc09fe2444a95f04d456239a9dd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.000948 16792 tablet_replica.cc:333] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:37.001567 16792 raft_consensus.cc:2243] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:37.002204 16792 raft_consensus.cc:2272] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.005216 16792 tablet_replica.cc:333] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:37.005985 16792 raft_consensus.cc:2243] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:37.006520 16792 raft_consensus.cc:2272] T aac47c2fc9c64d0e8627065d6c366bd0 P 6b512fc09fe2444a95f04d456239a9dd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.009198 16792 tablet_replica.cc:333] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd: stopping tablet replica
I20260709 12:28:37.009742 16792 raft_consensus.cc:2243] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.010867 16792 raft_consensus.cc:2272] T 36836434931140938a0c90e537ebd1e9 P 6b512fc09fe2444a95f04d456239a9dd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:37.020200 31355 consensus_peers.cc:597] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f -> Peer 6b512fc09fe2444a95f04d456239a9dd (127.16.102.2:35733): Couldn't send request to peer 6b512fc09fe2444a95f04d456239a9dd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:37.037148 31358 consensus_peers.cc:597] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:37.054000 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
I20260709 12:28:37.086373 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
W20260709 12:28:37.111835 31189 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:37.119032 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:37.120061 16792 tablet_replica.cc:333] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.121315 16792 raft_consensus.cc:2243] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.122462 16792 raft_consensus.cc:2272] T d6c0212c06fb4945ad761212c1868fb1 P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.125212 16792 tablet_replica.cc:333] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.126039 16792 raft_consensus.cc:2243] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:37.126608 16792 raft_consensus.cc:2272] T 36836434931140938a0c90e537ebd1e9 P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.129242 16792 tablet_replica.cc:333] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.130383 16792 raft_consensus.cc:2243] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.131678 16792 raft_consensus.cc:2272] T 6679bf1e8fbc4631903b685e7e5dde0a P 451375e442ad416ebc7f5e58b8610f7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.135885 16792 tablet_replica.cc:333] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
W20260709 12:28:37.136005 31358 consensus_peers.cc:597] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f -> Peer 67a00a279c8f4e86840326b53037f2e6 (127.16.102.1:38611): Couldn't send request to peer 67a00a279c8f4e86840326b53037f2e6. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:37.136801 16792 raft_consensus.cc:2243] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.137986 16792 raft_consensus.cc:2272] T b8c8e791b40f4efd9fc22519ed30b9ed P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.140820 16792 tablet_replica.cc:333] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.141569 16792 raft_consensus.cc:2243] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.142591 16792 raft_consensus.cc:2272] T aac47c2fc9c64d0e8627065d6c366bd0 P 451375e442ad416ebc7f5e58b8610f7f [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.145265 16792 tablet_replica.cc:333] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.146031 16792 raft_consensus.cc:2243] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:37.146608 16792 raft_consensus.cc:2272] T 32deb0d4043d46dfae3cc0aa450a34cb P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.149664 16792 tablet_replica.cc:333] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.150440 16792 raft_consensus.cc:2243] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:37.150985 16792 raft_consensus.cc:2272] T d94e0ae944a1430a9b4def74a74cc1a3 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.154273 16792 tablet_replica.cc:333] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.155025 16792 raft_consensus.cc:2243] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.156062 16792 raft_consensus.cc:2272] T 653fa7d1e4f549e792dc2c6e2972a3d4 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.159282 16792 tablet_replica.cc:333] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.159953 16792 raft_consensus.cc:2243] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.161015 16792 raft_consensus.cc:2272] T 6c9b3981ba8b4bf6af4bcd2c9e9e5e4f P 451375e442ad416ebc7f5e58b8610f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.163887 16792 tablet_replica.cc:333] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f: stopping tablet replica
I20260709 12:28:37.164525 16792 raft_consensus.cc:2243] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:37.165047 16792 raft_consensus.cc:2272] T dc1054676023472eab95b1e45e804e71 P 451375e442ad416ebc7f5e58b8610f7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.219791 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:28:37.251577 16792 master.cc:562] Master@127.16.102.62:45863 shutting down...
I20260709 12:28:37.283953 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:37.284608 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:37.284927 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P dce1c60063764b1ca97568bbe2f03dab: stopping tablet replica
I20260709 12:28:37.309257 16792 master.cc:584] Master@127.16.102.62:45863 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13462 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:28:37.369026 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:37923
I20260709 12:28:37.370478 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:37.377039 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:37.377686 31728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:37.377924 31730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.379001 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:37.380187 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:37.380374 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:37.380491 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600117380478 us; error 0 us; skew 500 ppm
I20260709 12:28:37.381042 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:37.384246 16792 webserver.cc:533] Webserver started at http://127.16.102.62:39657/ using document root <none> and password file <none>
I20260709 12:28:37.384785 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:37.384945 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:37.385202 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:37.386569 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "f4eab84de7d54fc8a56c4e3466b72be3"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.391930 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:37.396299 31735 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.397261 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:28:37.397629 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "f4eab84de7d54fc8a56c4e3466b72be3"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.398061 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:37.413142 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:37.414927 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:37.462809 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:37923
I20260709 12:28:37.462926 31786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:37923 every 8 connection(s)
I20260709 12:28:37.467519 31787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.481890 31787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3: Bootstrap starting.
I20260709 12:28:37.488476 31787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:37.494139 31787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3: No bootstrap required, opened a new log
I20260709 12:28:37.496908 31787 raft_consensus.cc:359] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" member_type: VOTER }
I20260709 12:28:37.497430 31787 raft_consensus.cc:385] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:37.497721 31787 raft_consensus.cc:740] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eab84de7d54fc8a56c4e3466b72be3, State: Initialized, Role: FOLLOWER
I20260709 12:28:37.498451 31787 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" member_type: VOTER }
I20260709 12:28:37.499033 31787 raft_consensus.cc:399] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:37.499292 31787 raft_consensus.cc:493] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:37.499627 31787 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:37.506212 31787 raft_consensus.cc:515] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" member_type: VOTER }
I20260709 12:28:37.506944 31787 leader_election.cc:304] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4eab84de7d54fc8a56c4e3466b72be3; no voters: 
I20260709 12:28:37.508313 31787 leader_election.cc:290] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:37.508764 31790 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:37.510377 31790 raft_consensus.cc:697] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 1 LEADER]: Becoming Leader. State: Replica: f4eab84de7d54fc8a56c4e3466b72be3, State: Running, Role: LEADER
I20260709 12:28:37.511229 31790 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" member_type: VOTER }
I20260709 12:28:37.511755 31787 sys_catalog.cc:565] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:37.516947 31791 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" member_type: VOTER } }
I20260709 12:28:37.517895 31791 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:37.516289 31792 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4eab84de7d54fc8a56c4e3466b72be3. Latest consensus state: current_term: 1 leader_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4eab84de7d54fc8a56c4e3466b72be3" member_type: VOTER } }
I20260709 12:28:37.519464 31792 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:37.520157 31797 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:37.526304 31797 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:37.532200 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:37.537571 31797 catalog_manager.cc:1383] Generated new cluster ID: 11638afb538c49ee85775bcc43d28d2e
I20260709 12:28:37.538038 31797 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:37.567408 31797 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:37.569100 31797 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:37.584648 31797 catalog_manager.cc:6070] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3: Generated new TSK 0
I20260709 12:28:37.585446 31797 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:37.599761 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:37.607218 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:37.608538 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.611773 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:37.611635 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.613170 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:37.613422 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:37.613615 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600117613587 us; error 0 us; skew 500 ppm
I20260709 12:28:37.614267 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:37.616978 16792 webserver.cc:533] Webserver started at http://127.16.102.1:33007/ using document root <none> and password file <none>
I20260709 12:28:37.617475 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:37.617648 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:37.617940 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:37.619153 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "4edf51e7dd58421693aa948341d7d1a7"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.624533 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:28:37.629118 31816 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.630242 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:37.630679 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "4edf51e7dd58421693aa948341d7d1a7"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.631058 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:37.645076 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:37.646620 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:37.648319 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:37.651021 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:37.651293 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.651561 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:37.651749 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.700989 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:33063
I20260709 12:28:37.701157 31878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:33063 every 8 connection(s)
I20260709 12:28:37.707067 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:37.716770 31883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:37.718679 31884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.722323 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:37.722469 31879 heartbeater.cc:344] Connected to a master server at 127.16.102.62:37923
I20260709 12:28:37.723215 31879 heartbeater.cc:461] Registering TS with master...
W20260709 12:28:37.723549 31887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.724381 31879 heartbeater.cc:507] Master 127.16.102.62:37923 requested a full tablet report, sending...
I20260709 12:28:37.724836 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:37.725172 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:37.725386 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600117725368 us; error 0 us; skew 500 ppm
I20260709 12:28:37.726320 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:37.727473 31752 ts_manager.cc:194] Registered new tserver with Master: 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063)
I20260709 12:28:37.729760 16792 webserver.cc:533] Webserver started at http://127.16.102.2:35619/ using document root <none> and password file <none>
I20260709 12:28:37.730374 31752 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55070
I20260709 12:28:37.730496 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:37.730893 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:37.731212 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:37.732941 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "6ada7c912cd54c569166ffdb9a780ebe"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.738556 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:28:37.742779 31891 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.743755 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:28:37.744169 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "6ada7c912cd54c569166ffdb9a780ebe"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.744544 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:37.769191 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:37.770661 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:37.772418 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:37.775089 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:37.775365 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.775632 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:37.775825 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.825803 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:35519
I20260709 12:28:37.825949 31953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:35519 every 8 connection(s)
I20260709 12:28:37.831800 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:37.842644 31957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:37.842756 31958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.847517 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:37.848065 31960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:37.848407 31954 heartbeater.cc:344] Connected to a master server at 127.16.102.62:37923
I20260709 12:28:37.848969 31954 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:37.849480 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:37.849860 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:37.850102 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600117850074 us; error 0 us; skew 500 ppm
I20260709 12:28:37.850106 31954 heartbeater.cc:507] Master 127.16.102.62:37923 requested a full tablet report, sending...
I20260709 12:28:37.851094 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:37.853489 31752 ts_manager.cc:194] Registered new tserver with Master: 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519)
I20260709 12:28:37.854717 16792 webserver.cc:533] Webserver started at http://127.16.102.3:36435/ using document root <none> and password file <none>
I20260709 12:28:37.855610 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:37.855901 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:37.855901 31752 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55080
I20260709 12:28:37.856417 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:37.858327 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "a6840d903f1c4b1dbc1059dab574254c"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.864154 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:37.868321 31965 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.869283 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:28:37.869679 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "a6840d903f1c4b1dbc1059dab574254c"
format_stamp: "Formatted at 2026-07-09 12:28:37 on dist-test-slave-7xt3"
I20260709 12:28:37.870091 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:37.888463 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:37.890051 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:37.891743 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:37.894333 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:37.894590 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.894846 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:37.895030 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:37.944985 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:41513
I20260709 12:28:37.945114 32027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:41513 every 8 connection(s)
I20260709 12:28:37.961531 32028 heartbeater.cc:344] Connected to a master server at 127.16.102.62:37923
I20260709 12:28:37.962042 32028 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:37.962898 32028 heartbeater.cc:507] Master 127.16.102.62:37923 requested a full tablet report, sending...
I20260709 12:28:37.965104 31752 ts_manager.cc:194] Registered new tserver with Master: a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:37.965255 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01653423s
I20260709 12:28:37.966914 31752 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55094
I20260709 12:28:37.996016 31752 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55096:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:37.998718 31752 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:38.075985 31992 tablet_service.cc:1511] Processing CreateTablet for tablet 25f04b51011f4894bd3f33b603a9adba (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b7120157d5d74d2b95c984a360a527df]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:38.075968 31993 tablet_service.cc:1511] Processing CreateTablet for tablet 93426e95342c4fe4a107f2dde1cb8894 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b7120157d5d74d2b95c984a360a527df]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:38.077841 31992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f04b51011f4894bd3f33b603a9adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.078946 31993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93426e95342c4fe4a107f2dde1cb8894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.082351 31919 tablet_service.cc:1511] Processing CreateTablet for tablet 93426e95342c4fe4a107f2dde1cb8894 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b7120157d5d74d2b95c984a360a527df]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:38.084112 31919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93426e95342c4fe4a107f2dde1cb8894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.088472 31843 tablet_service.cc:1511] Processing CreateTablet for tablet 25f04b51011f4894bd3f33b603a9adba (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b7120157d5d74d2b95c984a360a527df]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:38.089414 31918 tablet_service.cc:1511] Processing CreateTablet for tablet 25f04b51011f4894bd3f33b603a9adba (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b7120157d5d74d2b95c984a360a527df]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:38.088466 31844 tablet_service.cc:1511] Processing CreateTablet for tablet 93426e95342c4fe4a107f2dde1cb8894 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b7120157d5d74d2b95c984a360a527df]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:38.090971 31918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f04b51011f4894bd3f33b603a9adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.091671 31843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f04b51011f4894bd3f33b603a9adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.094211 31844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93426e95342c4fe4a107f2dde1cb8894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.129631 32043 tablet_bootstrap.cc:492] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: Bootstrap starting.
I20260709 12:28:38.131553 32044 tablet_bootstrap.cc:492] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe: Bootstrap starting.
I20260709 12:28:38.141737 32043 tablet_bootstrap.cc:654] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.146139 32044 tablet_bootstrap.cc:654] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.155843 32043 tablet_bootstrap.cc:492] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: No bootstrap required, opened a new log
I20260709 12:28:38.155925 32045 tablet_bootstrap.cc:492] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7: Bootstrap starting.
I20260709 12:28:38.156509 32043 ts_tablet_manager.cc:1403] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.008s
I20260709 12:28:38.160406 32043 raft_consensus.cc:359] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.161135 32043 raft_consensus.cc:385] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.161419 32043 raft_consensus.cc:740] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.161497 32044 tablet_bootstrap.cc:492] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe: No bootstrap required, opened a new log
I20260709 12:28:38.162307 32044 ts_tablet_manager.cc:1403] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.012s
I20260709 12:28:38.162454 32043 consensus_queue.cc:260] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.165421 32045 tablet_bootstrap.cc:654] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.166121 32044 raft_consensus.cc:359] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.167213 32044 raft_consensus.cc:385] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.167553 32044 raft_consensus.cc:740] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.168520 32044 consensus_queue.cc:260] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.172303 32043 ts_tablet_manager.cc:1434] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: Time spent starting tablet: real 0.015s	user 0.016s	sys 0.000s
I20260709 12:28:38.172019 32028 heartbeater.cc:499] Master 127.16.102.62:37923 was elected leader, sending a full tablet report...
I20260709 12:28:38.173851 32043 tablet_bootstrap.cc:492] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c: Bootstrap starting.
I20260709 12:28:38.175424 32045 tablet_bootstrap.cc:492] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7: No bootstrap required, opened a new log
I20260709 12:28:38.176097 32045 ts_tablet_manager.cc:1403] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.003s
I20260709 12:28:38.180116 32045 raft_consensus.cc:359] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.181058 32045 raft_consensus.cc:385] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.181403 32045 raft_consensus.cc:740] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.182371 32043 tablet_bootstrap.cc:654] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.182664 32045 consensus_queue.cc:260] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.185891 31954 heartbeater.cc:499] Master 127.16.102.62:37923 was elected leader, sending a full tablet report...
I20260709 12:28:38.186480 32044 ts_tablet_manager.cc:1434] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe: Time spent starting tablet: real 0.024s	user 0.019s	sys 0.004s
I20260709 12:28:38.187736 32044 tablet_bootstrap.cc:492] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe: Bootstrap starting.
I20260709 12:28:38.196046 32044 tablet_bootstrap.cc:654] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.221015 31879 heartbeater.cc:499] Master 127.16.102.62:37923 was elected leader, sending a full tablet report...
I20260709 12:28:38.224319 32045 ts_tablet_manager.cc:1434] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7: Time spent starting tablet: real 0.048s	user 0.017s	sys 0.026s
I20260709 12:28:38.226241 32045 tablet_bootstrap.cc:492] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7: Bootstrap starting.
I20260709 12:28:38.226945 32043 tablet_bootstrap.cc:492] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c: No bootstrap required, opened a new log
I20260709 12:28:38.227749 32043 ts_tablet_manager.cc:1403] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c: Time spent bootstrapping tablet: real 0.054s	user 0.023s	sys 0.018s
I20260709 12:28:38.228577 32044 tablet_bootstrap.cc:492] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe: No bootstrap required, opened a new log
I20260709 12:28:38.229281 32044 ts_tablet_manager.cc:1403] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe: Time spent bootstrapping tablet: real 0.042s	user 0.017s	sys 0.020s
I20260709 12:28:38.231513 32043 raft_consensus.cc:359] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.232712 32043 raft_consensus.cc:385] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.232753 32044 raft_consensus.cc:359] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.233397 32043 raft_consensus.cc:740] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.233891 32044 raft_consensus.cc:385] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.234299 32044 raft_consensus.cc:740] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.235069 32045 tablet_bootstrap.cc:654] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.235159 32044 consensus_queue.cc:260] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.235101 32043 consensus_queue.cc:260] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.241046 32043 ts_tablet_manager.cc:1434] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.006s
I20260709 12:28:38.245002 32044 ts_tablet_manager.cc:1434] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.001s
I20260709 12:28:38.245334 32045 tablet_bootstrap.cc:492] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7: No bootstrap required, opened a new log
I20260709 12:28:38.245993 32045 ts_tablet_manager.cc:1403] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
I20260709 12:28:38.250016 32045 raft_consensus.cc:359] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.250895 32045 raft_consensus.cc:385] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.251210 32045 raft_consensus.cc:740] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.252038 32045 consensus_queue.cc:260] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.254458 32045 ts_tablet_manager.cc:1434] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:38.273169 32051 raft_consensus.cc:493] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.273845 32051 raft_consensus.cc:515] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.276360 32051 leader_election.cc:290] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:38.279312 32048 raft_consensus.cc:493] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.280148 32048 raft_consensus.cc:515] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.283566 32048 leader_election.cc:290] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519), 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063)
I20260709 12:28:38.295627 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93426e95342c4fe4a107f2dde1cb8894" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6840d903f1c4b1dbc1059dab574254c" is_pre_election: true
I20260709 12:28:38.296952 32003 raft_consensus.cc:2468] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 0.
I20260709 12:28:38.299594 31818 leader_election.cc:304] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, a6840d903f1c4b1dbc1059dab574254c; no voters: 
I20260709 12:28:38.300920 32051 raft_consensus.cc:2804] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.301374 32051 raft_consensus.cc:493] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.301743 32051 raft_consensus.cc:3060] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.302151 31854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93426e95342c4fe4a107f2dde1cb8894" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edf51e7dd58421693aa948341d7d1a7" is_pre_election: true
I20260709 12:28:38.305434 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93426e95342c4fe4a107f2dde1cb8894" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe" is_pre_election: true
I20260709 12:28:38.305316 31929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93426e95342c4fe4a107f2dde1cb8894" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe" is_pre_election: true
I20260709 12:28:38.306525 31928 raft_consensus.cc:2468] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 0.
I20260709 12:28:38.311038 32051 raft_consensus.cc:515] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.312400 31854 raft_consensus.cc:2393] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6840d903f1c4b1dbc1059dab574254c in current term 1: Already voted for candidate 4edf51e7dd58421693aa948341d7d1a7 in this term.
I20260709 12:28:38.313648 32051 leader_election.cc:290] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:38.314934 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93426e95342c4fe4a107f2dde1cb8894" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
I20260709 12:28:38.315277 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93426e95342c4fe4a107f2dde1cb8894" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
I20260709 12:28:38.315676 31928 raft_consensus.cc:3060] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.316017 32003 raft_consensus.cc:3060] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.317029 31966 leader_election.cc:304] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6840d903f1c4b1dbc1059dab574254c; no voters: 4edf51e7dd58421693aa948341d7d1a7, 6ada7c912cd54c569166ffdb9a780ebe
I20260709 12:28:38.323422 31928 raft_consensus.cc:2468] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 1.
I20260709 12:28:38.323422 32003 raft_consensus.cc:2468] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 1.
I20260709 12:28:38.324417 32048 raft_consensus.cc:2749] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:38.325317 31819 leader_election.cc:304] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, 6ada7c912cd54c569166ffdb9a780ebe; no voters: 
I20260709 12:28:38.326434 32051 raft_consensus.cc:2804] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.327824 32051 raft_consensus.cc:697] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Running, Role: LEADER
I20260709 12:28:38.328749 32051 consensus_queue.cc:237] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } }
I20260709 12:28:38.338819 31751 catalog_manager.cc:5697] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "4edf51e7dd58421693aa948341d7d1a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:38.478035 32048 raft_consensus.cc:493] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.478823 32048 raft_consensus.cc:515] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.480947 32048 leader_election.cc:290] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063), 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519)
I20260709 12:28:38.482275 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe" is_pre_election: true
I20260709 12:28:38.482084 31854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edf51e7dd58421693aa948341d7d1a7" is_pre_election: true
I20260709 12:28:38.483206 31928 raft_consensus.cc:2468] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6840d903f1c4b1dbc1059dab574254c in term 0.
I20260709 12:28:38.483291 31854 raft_consensus.cc:2468] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6840d903f1c4b1dbc1059dab574254c in term 0.
I20260709 12:28:38.484701 31968 leader_election.cc:304] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ada7c912cd54c569166ffdb9a780ebe, a6840d903f1c4b1dbc1059dab574254c; no voters: 
I20260709 12:28:38.485579 32048 raft_consensus.cc:2804] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.486001 32048 raft_consensus.cc:493] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.486290 32048 raft_consensus.cc:3060] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.493767 32048 raft_consensus.cc:515] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.495859 32048 leader_election.cc:290] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 1 election: Requested vote from peers 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063), 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519)
I20260709 12:28:38.496969 31854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edf51e7dd58421693aa948341d7d1a7"
I20260709 12:28:38.497051 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
I20260709 12:28:38.498031 31854 raft_consensus.cc:3060] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.498031 31928 raft_consensus.cc:3060] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.507341 31928 raft_consensus.cc:2468] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6840d903f1c4b1dbc1059dab574254c in term 1.
I20260709 12:28:38.507341 31854 raft_consensus.cc:2468] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6840d903f1c4b1dbc1059dab574254c in term 1.
I20260709 12:28:38.509094 31968 leader_election.cc:304] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ada7c912cd54c569166ffdb9a780ebe, a6840d903f1c4b1dbc1059dab574254c; no voters: 
I20260709 12:28:38.510042 32048 raft_consensus.cc:2804] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.511122 32048 raft_consensus.cc:697] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 LEADER]: Becoming Leader. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Running, Role: LEADER
I20260709 12:28:38.512171 32048 consensus_queue.cc:237] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.520474 31751 catalog_manager.cc:5697] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c reported cstate change: term changed from 0 to 1, leader changed from <none> to a6840d903f1c4b1dbc1059dab574254c (127.16.102.3). New cstate: current_term: 1 leader_uuid: "a6840d903f1c4b1dbc1059dab574254c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:38.528414 31800 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:38.590009 31751 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55112:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:38.592382 31751 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:38.619894 31844 tablet_service.cc:1511] Processing CreateTablet for tablet 59dd98b24d374dcc9bdd91b8e1aadcb5 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=458530dc20174160b912e8e1e0f33fa7]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:38.621479 31919 tablet_service.cc:1511] Processing CreateTablet for tablet 59dd98b24d374dcc9bdd91b8e1aadcb5 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=458530dc20174160b912e8e1e0f33fa7]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:38.621515 31844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59dd98b24d374dcc9bdd91b8e1aadcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.623199 31919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59dd98b24d374dcc9bdd91b8e1aadcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.623180 31993 tablet_service.cc:1511] Processing CreateTablet for tablet 59dd98b24d374dcc9bdd91b8e1aadcb5 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=458530dc20174160b912e8e1e0f33fa7]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:38.624694 31993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59dd98b24d374dcc9bdd91b8e1aadcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.624622 31843 tablet_service.cc:1511] Processing CreateTablet for tablet f1a17d9da995415ea5643f127f6f39cd (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=458530dc20174160b912e8e1e0f33fa7]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:38.626085 31843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a17d9da995415ea5643f127f6f39cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.638849 31918 tablet_service.cc:1511] Processing CreateTablet for tablet f1a17d9da995415ea5643f127f6f39cd (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=458530dc20174160b912e8e1e0f33fa7]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:38.640570 31918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a17d9da995415ea5643f127f6f39cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.643684 32044 tablet_bootstrap.cc:492] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: Bootstrap starting.
I20260709 12:28:38.644309 32045 tablet_bootstrap.cc:492] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7: Bootstrap starting.
I20260709 12:28:38.651317 32044 tablet_bootstrap.cc:654] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.651897 32045 tablet_bootstrap.cc:654] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.659421 32044 tablet_bootstrap.cc:492] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: No bootstrap required, opened a new log
I20260709 12:28:38.660151 32044 ts_tablet_manager.cc:1403] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.001s
I20260709 12:28:38.664121 32044 raft_consensus.cc:359] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.665122 32044 raft_consensus.cc:385] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.665326 32043 tablet_bootstrap.cc:492] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c: Bootstrap starting.
I20260709 12:28:38.665562 32044 raft_consensus.cc:740] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.665153 31992 tablet_service.cc:1511] Processing CreateTablet for tablet f1a17d9da995415ea5643f127f6f39cd (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=458530dc20174160b912e8e1e0f33fa7]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:38.667083 31992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a17d9da995415ea5643f127f6f39cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:38.666656 32044 consensus_queue.cc:260] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.681541 32045 tablet_bootstrap.cc:492] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7: No bootstrap required, opened a new log
I20260709 12:28:38.682514 32045 ts_tablet_manager.cc:1403] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7: Time spent bootstrapping tablet: real 0.038s	user 0.014s	sys 0.007s
I20260709 12:28:38.683434 32043 tablet_bootstrap.cc:654] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.686761 32045 raft_consensus.cc:359] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.687847 32045 raft_consensus.cc:385] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.688251 32045 raft_consensus.cc:740] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.689263 32045 consensus_queue.cc:260] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.690085 32044 ts_tablet_manager.cc:1434] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.002s
I20260709 12:28:38.694625 32045 ts_tablet_manager.cc:1434] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:28:38.696246 32045 tablet_bootstrap.cc:492] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: Bootstrap starting.
I20260709 12:28:38.699805 32043 tablet_bootstrap.cc:492] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c: No bootstrap required, opened a new log
I20260709 12:28:38.700865 32043 ts_tablet_manager.cc:1403] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c: Time spent bootstrapping tablet: real 0.036s	user 0.014s	sys 0.006s
I20260709 12:28:38.704867 32045 tablet_bootstrap.cc:654] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.705103 32044 tablet_bootstrap.cc:492] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe: Bootstrap starting.
I20260709 12:28:38.705355 32043 raft_consensus.cc:359] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.706427 32043 raft_consensus.cc:385] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.706827 32043 raft_consensus.cc:740] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.708017 32043 consensus_queue.cc:260] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.712374 32045 tablet_bootstrap.cc:492] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: No bootstrap required, opened a new log
I20260709 12:28:38.713291 32045 ts_tablet_manager.cc:1403] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: Time spent bootstrapping tablet: real 0.017s	user 0.016s	sys 0.001s
I20260709 12:28:38.715549 32043 ts_tablet_manager.cc:1434] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:28:38.715864 32044 tablet_bootstrap.cc:654] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.717093 32043 tablet_bootstrap.cc:492] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: Bootstrap starting.
I20260709 12:28:38.719558 32045 raft_consensus.cc:359] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.720409 32045 raft_consensus.cc:385] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.720700 32045 raft_consensus.cc:740] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.721496 32045 consensus_queue.cc:260] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.723970 32045 ts_tablet_manager.cc:1434] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:28:38.729182 32044 tablet_bootstrap.cc:492] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe: No bootstrap required, opened a new log
I20260709 12:28:38.729998 32044 ts_tablet_manager.cc:1403] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.012s
I20260709 12:28:38.730332 32043 tablet_bootstrap.cc:654] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.734251 32044 raft_consensus.cc:359] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.735399 32044 raft_consensus.cc:385] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.735729 32044 raft_consensus.cc:740] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.736652 32044 consensus_queue.cc:260] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.739533 32044 ts_tablet_manager.cc:1434] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:28:38.742879 32043 tablet_bootstrap.cc:492] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: No bootstrap required, opened a new log
I20260709 12:28:38.743695 32043 ts_tablet_manager.cc:1403] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.003s
I20260709 12:28:38.746798 32043 raft_consensus.cc:359] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.747495 32043 raft_consensus.cc:385] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.747766 32043 raft_consensus.cc:740] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.748561 32043 consensus_queue.cc:260] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.750779 32043 ts_tablet_manager.cc:1434] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:38.854245 32051 consensus_queue.cc:1048] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:38.870206 32078 consensus_queue.cc:1048] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.902886 32078 raft_consensus.cc:493] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.903527 32078 raft_consensus.cc:515] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.905699 32078 leader_election.cc:290] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:38.906462 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe" is_pre_election: true
I20260709 12:28:38.906903 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6840d903f1c4b1dbc1059dab574254c" is_pre_election: true
I20260709 12:28:38.907414 31928 raft_consensus.cc:2468] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 0.
I20260709 12:28:38.907634 32003 raft_consensus.cc:2468] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 0.
I20260709 12:28:38.908694 31819 leader_election.cc:304] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, 6ada7c912cd54c569166ffdb9a780ebe; no voters: 
I20260709 12:28:38.909570 32078 raft_consensus.cc:2804] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.909956 32078 raft_consensus.cc:493] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.910255 32078 raft_consensus.cc:3060] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.917479 32078 raft_consensus.cc:515] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.919497 32078 leader_election.cc:290] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 election: Requested vote from peers 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:38.920713 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
I20260709 12:28:38.920717 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
I20260709 12:28:38.921595 31928 raft_consensus.cc:3060] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.921595 32003 raft_consensus.cc:3060] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.930471 31928 raft_consensus.cc:2468] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 1.
I20260709 12:28:38.930501 32003 raft_consensus.cc:2468] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 1.
I20260709 12:28:38.930222 32048 consensus_queue.cc:1048] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.932066 31819 leader_election.cc:304] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, 6ada7c912cd54c569166ffdb9a780ebe; no voters: 
I20260709 12:28:38.933064 32078 raft_consensus.cc:2804] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.933653 32078 raft_consensus.cc:697] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Running, Role: LEADER
I20260709 12:28:38.934574 32078 consensus_queue.cc:237] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:38.945647 32048 consensus_queue.cc:1048] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.946550 31751 catalog_manager.cc:5697] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1). New cstate: current_term: 1 leader_uuid: "4edf51e7dd58421693aa948341d7d1a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.007977 32050 raft_consensus.cc:493] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.008783 32050 raft_consensus.cc:515] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.011407 32050 leader_election.cc:290] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:39.026503 31854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5" candidate_uuid: "6ada7c912cd54c569166ffdb9a780ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edf51e7dd58421693aa948341d7d1a7" is_pre_election: true
I20260709 12:28:39.026522 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5" candidate_uuid: "6ada7c912cd54c569166ffdb9a780ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6840d903f1c4b1dbc1059dab574254c" is_pre_election: true
I20260709 12:28:39.027421 31854 raft_consensus.cc:2468] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ada7c912cd54c569166ffdb9a780ebe in term 0.
I20260709 12:28:39.027571 32003 raft_consensus.cc:2468] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ada7c912cd54c569166ffdb9a780ebe in term 0.
I20260709 12:28:39.028735 31892 leader_election.cc:304] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, 6ada7c912cd54c569166ffdb9a780ebe; no voters: 
I20260709 12:28:39.029636 32050 raft_consensus.cc:2804] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:39.030027 32050 raft_consensus.cc:493] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:39.030345 32050 raft_consensus.cc:3060] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:39.038288 32050 raft_consensus.cc:515] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.040249 32050 leader_election.cc:290] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [CANDIDATE]: Term 1 election: Requested vote from peers 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:39.041363 31854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5" candidate_uuid: "6ada7c912cd54c569166ffdb9a780ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edf51e7dd58421693aa948341d7d1a7"
I20260709 12:28:39.042207 31854 raft_consensus.cc:3060] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:39.042610 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5" candidate_uuid: "6ada7c912cd54c569166ffdb9a780ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
I20260709 12:28:39.043395 32003 raft_consensus.cc:3060] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:39.049911 31854 raft_consensus.cc:2468] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ada7c912cd54c569166ffdb9a780ebe in term 1.
I20260709 12:28:39.050365 32003 raft_consensus.cc:2468] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ada7c912cd54c569166ffdb9a780ebe in term 1.
I20260709 12:28:39.051470 31892 leader_election.cc:304] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, 6ada7c912cd54c569166ffdb9a780ebe; no voters: 
I20260709 12:28:39.052449 32050 raft_consensus.cc:2804] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:39.053709 32050 raft_consensus.cc:697] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 LEADER]: Becoming Leader. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Running, Role: LEADER
I20260709 12:28:39.054756 32050 consensus_queue.cc:237] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.063175 31750 catalog_manager.cc:5697] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2). New cstate: current_term: 1 leader_uuid: "6ada7c912cd54c569166ffdb9a780ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.147814 32003 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba"
dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
mode: GRACEFUL
new_leader_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
 from {username='slave'} at 127.0.0.1:35810
I20260709 12:28:39.148500 32003 raft_consensus.cc:606] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 LEADER]: Received request to transfer leadership to TS 6ada7c912cd54c569166ffdb9a780ebe
I20260709 12:28:39.165905 31928 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5"
dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
mode: GRACEFUL
new_leader_uuid: "4edf51e7dd58421693aa948341d7d1a7"
 from {username='slave'} at 127.0.0.1:44958
I20260709 12:28:39.166579 31928 raft_consensus.cc:606] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 LEADER]: Received request to transfer leadership to TS 4edf51e7dd58421693aa948341d7d1a7
I20260709 12:28:39.389333 32061 raft_consensus.cc:993] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: : Instructing follower 6ada7c912cd54c569166ffdb9a780ebe to start an election
I20260709 12:28:39.390012 32084 raft_consensus.cc:1081] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 LEADER]: Signalling peer 6ada7c912cd54c569166ffdb9a780ebe to start an election
I20260709 12:28:39.391850 31928 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba"
dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
 from {username='slave'} at 127.0.0.1:44946
I20260709 12:28:39.392545 31928 raft_consensus.cc:493] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.393002 31928 raft_consensus.cc:3060] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.400883 31928 raft_consensus.cc:515] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.403697 31928 leader_election.cc:290] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [CANDIDATE]: Term 2 election: Requested vote from peers 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:39.404258 31854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba" candidate_uuid: "6ada7c912cd54c569166ffdb9a780ebe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4edf51e7dd58421693aa948341d7d1a7"
I20260709 12:28:39.404534 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f04b51011f4894bd3f33b603a9adba" candidate_uuid: "6ada7c912cd54c569166ffdb9a780ebe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
I20260709 12:28:39.405167 31854 raft_consensus.cc:3060] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.405314 32003 raft_consensus.cc:3055] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.405902 32003 raft_consensus.cc:740] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Running, Role: LEADER
I20260709 12:28:39.406883 32003 consensus_queue.cc:260] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.408382 32003 raft_consensus.cc:3060] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.418737 32003 raft_consensus.cc:2468] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ada7c912cd54c569166ffdb9a780ebe in term 2.
I20260709 12:28:39.421625 31893 leader_election.cc:304] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ada7c912cd54c569166ffdb9a780ebe, a6840d903f1c4b1dbc1059dab574254c; no voters: 
I20260709 12:28:39.422691 31854 raft_consensus.cc:2468] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ada7c912cd54c569166ffdb9a780ebe in term 2.
I20260709 12:28:39.422752 32050 raft_consensus.cc:2804] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.423700 32050 raft_consensus.cc:697] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 2 LEADER]: Becoming Leader. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Running, Role: LEADER
I20260709 12:28:39.424755 32050 consensus_queue.cc:237] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.432526 31751 catalog_manager.cc:5697] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe reported cstate change: term changed from 1 to 2, leader changed from a6840d903f1c4b1dbc1059dab574254c (127.16.102.3) to 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2). New cstate: current_term: 2 leader_uuid: "6ada7c912cd54c569166ffdb9a780ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.499637 32094 consensus_queue.cc:1048] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:39.515776 32094 consensus_queue.cc:1048] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.603281 32050 consensus_queue.cc:1048] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:39.610795 32090 raft_consensus.cc:993] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: : Instructing follower 4edf51e7dd58421693aa948341d7d1a7 to start an election
I20260709 12:28:39.611603 32050 raft_consensus.cc:1081] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 LEADER]: Signalling peer 4edf51e7dd58421693aa948341d7d1a7 to start an election
I20260709 12:28:39.614346 31853 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5"
dest_uuid: "4edf51e7dd58421693aa948341d7d1a7"
 from {username='slave'} at 127.0.0.1:43014
I20260709 12:28:39.615262 31853 raft_consensus.cc:493] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.615694 31853 raft_consensus.cc:3060] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.620018 32101 consensus_queue.cc:1048] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.626462 31853 raft_consensus.cc:515] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.631465 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
I20260709 12:28:39.632289 31928 raft_consensus.cc:3055] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.632647 32003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59dd98b24d374dcc9bdd91b8e1aadcb5" candidate_uuid: "4edf51e7dd58421693aa948341d7d1a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
I20260709 12:28:39.632834 31928 raft_consensus.cc:740] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ada7c912cd54c569166ffdb9a780ebe, State: Running, Role: LEADER
I20260709 12:28:39.632323 31853 leader_election.cc:290] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 2 election: Requested vote from peers 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519), a6840d903f1c4b1dbc1059dab574254c (127.16.102.3:41513)
I20260709 12:28:39.633551 32003 raft_consensus.cc:3060] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.634317 31928 consensus_queue.cc:260] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.635986 31928 raft_consensus.cc:3060] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.644331 32003 raft_consensus.cc:2468] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 2.
I20260709 12:28:39.645318 31928 raft_consensus.cc:2468] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edf51e7dd58421693aa948341d7d1a7 in term 2.
I20260709 12:28:39.646119 31818 leader_election.cc:304] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edf51e7dd58421693aa948341d7d1a7, a6840d903f1c4b1dbc1059dab574254c; no voters: 
I20260709 12:28:39.649533 32094 raft_consensus.cc:2804] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.650364 32094 raft_consensus.cc:697] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 2 LEADER]: Becoming Leader. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Running, Role: LEADER
I20260709 12:28:39.651361 32094 consensus_queue.cc:237] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:39.660254 31751 catalog_manager.cc:5697] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 reported cstate change: term changed from 1 to 2, leader changed from 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2) to 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1). New cstate: current_term: 2 leader_uuid: "4edf51e7dd58421693aa948341d7d1a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.815145 32003 raft_consensus.cc:1275] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Refusing update from remote peer 6ada7c912cd54c569166ffdb9a780ebe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.816799 32101 consensus_queue.cc:1048] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.826335 31853 raft_consensus.cc:1275] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Refusing update from remote peer 6ada7c912cd54c569166ffdb9a780ebe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.828743 32050 consensus_queue.cc:1048] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:39.958604 16792 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:28:39.960605 16792 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:28:39.961158 16792 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:28:39.977155 31853 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd"
dest_uuid: "4edf51e7dd58421693aa948341d7d1a7"
mode: GRACEFUL
new_leader_uuid: "a6840d903f1c4b1dbc1059dab574254c"
 from {username='slave'} at 127.0.0.1:43030
I20260709 12:28:39.977802 31853 raft_consensus.cc:606] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Received request to transfer leadership to TS a6840d903f1c4b1dbc1059dab574254c
I20260709 12:28:39.979243 16792 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:28:40.037155 31928 raft_consensus.cc:1275] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Refusing update from remote peer 4edf51e7dd58421693aa948341d7d1a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:40.038926 32094 consensus_queue.cc:1048] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:40.051688 32003 raft_consensus.cc:1275] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Refusing update from remote peer 4edf51e7dd58421693aa948341d7d1a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:40.053920 32094 consensus_queue.cc:1048] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:40.071375 32078 raft_consensus.cc:993] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: : Instructing follower a6840d903f1c4b1dbc1059dab574254c to start an election
I20260709 12:28:40.071997 32094 raft_consensus.cc:1081] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Signalling peer a6840d903f1c4b1dbc1059dab574254c to start an election
I20260709 12:28:40.073973 32003 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd"
dest_uuid: "a6840d903f1c4b1dbc1059dab574254c"
 from {username='slave'} at 127.0.0.1:35804
I20260709 12:28:40.074573 32003 raft_consensus.cc:493] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:40.075006 32003 raft_consensus.cc:3060] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:40.082048 32003 raft_consensus.cc:515] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:40.084584 32003 leader_election.cc:290] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 2 election: Requested vote from peers 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1:33063), 6ada7c912cd54c569166ffdb9a780ebe (127.16.102.2:35519)
I20260709 12:28:40.085920 31928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6ada7c912cd54c569166ffdb9a780ebe"
I20260709 12:28:40.085716 31853 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1a17d9da995415ea5643f127f6f39cd" candidate_uuid: "a6840d903f1c4b1dbc1059dab574254c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4edf51e7dd58421693aa948341d7d1a7"
I20260709 12:28:40.086839 31928 raft_consensus.cc:3060] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:40.086864 31853 raft_consensus.cc:3055] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:40.087450 31853 raft_consensus.cc:740] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4edf51e7dd58421693aa948341d7d1a7, State: Running, Role: LEADER
I20260709 12:28:40.088416 31853 consensus_queue.cc:260] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:40.089912 31853 raft_consensus.cc:3060] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:40.096590 31928 raft_consensus.cc:2468] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6840d903f1c4b1dbc1059dab574254c in term 2.
I20260709 12:28:40.098129 31968 leader_election.cc:304] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ada7c912cd54c569166ffdb9a780ebe, a6840d903f1c4b1dbc1059dab574254c; no voters: 
I20260709 12:28:40.099104 32084 raft_consensus.cc:2804] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:40.099174 31853 raft_consensus.cc:2468] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6840d903f1c4b1dbc1059dab574254c in term 2.
I20260709 12:28:40.100440 32084 raft_consensus.cc:697] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 2 LEADER]: Becoming Leader. State: Replica: a6840d903f1c4b1dbc1059dab574254c, State: Running, Role: LEADER
I20260709 12:28:40.101383 32084 consensus_queue.cc:237] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } }
I20260709 12:28:40.108593 31751 catalog_manager.cc:5697] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c reported cstate change: term changed from 1 to 2, leader changed from 4edf51e7dd58421693aa948341d7d1a7 (127.16.102.1) to a6840d903f1c4b1dbc1059dab574254c (127.16.102.3). New cstate: current_term: 2 leader_uuid: "a6840d903f1c4b1dbc1059dab574254c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edf51e7dd58421693aa948341d7d1a7" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ada7c912cd54c569166ffdb9a780ebe" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6840d903f1c4b1dbc1059dab574254c" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41513 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:40.144438 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:28:40.171037 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:40.172029 16792 tablet_replica.cc:333] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7: stopping tablet replica
I20260709 12:28:40.172780 16792 raft_consensus.cc:2243] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.173306 16792 raft_consensus.cc:2272] T f1a17d9da995415ea5643f127f6f39cd P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.176440 16792 tablet_replica.cc:333] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7: stopping tablet replica
I20260709 12:28:40.177214 16792 raft_consensus.cc:2243] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:40.178417 16792 raft_consensus.cc:2272] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.181951 16792 tablet_replica.cc:333] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7: stopping tablet replica
I20260709 12:28:40.182832 16792 raft_consensus.cc:2243] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.183372 16792 raft_consensus.cc:2272] T 25f04b51011f4894bd3f33b603a9adba P 4edf51e7dd58421693aa948341d7d1a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.186508 16792 tablet_replica.cc:333] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7: stopping tablet replica
I20260709 12:28:40.187101 16792 raft_consensus.cc:2243] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:40.188032 16792 raft_consensus.cc:2272] T 93426e95342c4fe4a107f2dde1cb8894 P 4edf51e7dd58421693aa948341d7d1a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.214982 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
I20260709 12:28:40.232283 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
I20260709 12:28:40.255187 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:40.255888 16792 tablet_replica.cc:333] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe: stopping tablet replica
I20260709 12:28:40.256498 16792 raft_consensus.cc:2243] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.256994 16792 raft_consensus.cc:2272] T f1a17d9da995415ea5643f127f6f39cd P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.260453 16792 tablet_replica.cc:333] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe: stopping tablet replica
I20260709 12:28:40.261139 16792 raft_consensus.cc:2243] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.261662 16792 raft_consensus.cc:2272] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.264655 16792 tablet_replica.cc:333] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe: stopping tablet replica
I20260709 12:28:40.265287 16792 raft_consensus.cc:2243] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:40.266428 16792 raft_consensus.cc:2272] T 25f04b51011f4894bd3f33b603a9adba P 6ada7c912cd54c569166ffdb9a780ebe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.269347 16792 tablet_replica.cc:333] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe: stopping tablet replica
I20260709 12:28:40.270022 16792 raft_consensus.cc:2243] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.270516 16792 raft_consensus.cc:2272] T 93426e95342c4fe4a107f2dde1cb8894 P 6ada7c912cd54c569166ffdb9a780ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.287966 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
I20260709 12:28:40.305073 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
I20260709 12:28:40.329336 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:40.330168 16792 tablet_replica.cc:333] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: stopping tablet replica
I20260709 12:28:40.330869 16792 raft_consensus.cc:2243] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:40.331629 16792 pending_rounds.cc:70] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: Trying to abort 1 pending ops.
I20260709 12:28:40.331813 16792 pending_rounds.cc:77] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305626091944988672 op_type: NO_OP noop_request { }
I20260709 12:28:40.332098 16792 raft_consensus.cc:2889] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:40.332361 16792 raft_consensus.cc:2272] T f1a17d9da995415ea5643f127f6f39cd P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.335148 16792 tablet_replica.cc:333] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c: stopping tablet replica
I20260709 12:28:40.335760 16792 raft_consensus.cc:2243] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.336217 16792 raft_consensus.cc:2272] T 59dd98b24d374dcc9bdd91b8e1aadcb5 P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.339498 16792 tablet_replica.cc:333] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c: stopping tablet replica
I20260709 12:28:40.340073 16792 raft_consensus.cc:2243] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.340519 16792 raft_consensus.cc:2272] T 93426e95342c4fe4a107f2dde1cb8894 P a6840d903f1c4b1dbc1059dab574254c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.342833 16792 tablet_replica.cc:333] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c: stopping tablet replica
I20260709 12:28:40.343343 16792 raft_consensus.cc:2243] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:40.343777 16792 raft_consensus.cc:2272] T 25f04b51011f4894bd3f33b603a9adba P a6840d903f1c4b1dbc1059dab574254c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.358865 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:28:40.375226 16792 master.cc:562] Master@127.16.102.62:37923 shutting down...
I20260709 12:28:40.396003 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:40.396800 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:40.397197 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P f4eab84de7d54fc8a56c4e3466b72be3: stopping tablet replica
I20260709 12:28:40.418226 16792 master.cc:584] Master@127.16.102.62:37923 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (3085 ms)
[----------] 6 tests from LeaderRebalancerTest (541109 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:28:40.454707 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:36697
I20260709 12:28:40.456152 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:40.462365 32107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:40.462788 32108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:40.464219 32110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.465571 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:40.466718 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:40.466958 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:40.467130 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600120467112 us; error 0 us; skew 500 ppm
I20260709 12:28:40.467772 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:40.470584 16792 webserver.cc:533] Webserver started at http://127.16.102.62:44495/ using document root <none> and password file <none>
I20260709 12:28:40.471163 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:40.471393 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:40.471695 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:40.472986 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "7e5ec5953a9c4fe5ae361d351b70e001"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.478446 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:28:40.482460 32115 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.483355 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:28:40.483711 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "7e5ec5953a9c4fe5ae361d351b70e001"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.484064 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:40.502405 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:40.503715 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:40.546613 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:36697
I20260709 12:28:40.546741 32166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:36697 every 8 connection(s)
I20260709 12:28:40.551293 32167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:40.566356 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001: Bootstrap starting.
I20260709 12:28:40.571830 32167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:40.577117 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001: No bootstrap required, opened a new log
I20260709 12:28:40.579828 32167 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" member_type: VOTER }
I20260709 12:28:40.580366 32167 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:40.580641 32167 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5ec5953a9c4fe5ae361d351b70e001, State: Initialized, Role: FOLLOWER
I20260709 12:28:40.581394 32167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" member_type: VOTER }
I20260709 12:28:40.582000 32167 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:40.582221 32167 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:40.582540 32167 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:40.589394 32167 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" member_type: VOTER }
I20260709 12:28:40.590157 32167 leader_election.cc:304] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e5ec5953a9c4fe5ae361d351b70e001; no voters: 
I20260709 12:28:40.591536 32167 leader_election.cc:290] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:40.591972 32170 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:40.593640 32170 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 1 LEADER]: Becoming Leader. State: Replica: 7e5ec5953a9c4fe5ae361d351b70e001, State: Running, Role: LEADER
I20260709 12:28:40.594489 32170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" member_type: VOTER }
I20260709 12:28:40.595099 32167 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:40.597868 32172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e5ec5953a9c4fe5ae361d351b70e001. Latest consensus state: current_term: 1 leader_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" member_type: VOTER } }
I20260709 12:28:40.597868 32171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5ec5953a9c4fe5ae361d351b70e001" member_type: VOTER } }
I20260709 12:28:40.598847 32172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:40.598954 32171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:40.604486 32175 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:40.613181 32175 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:40.614496 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:40.624543 32175 catalog_manager.cc:1383] Generated new cluster ID: 6c750da23dcf458eabd5a78132eb6618
I20260709 12:28:40.624945 32175 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:40.647119 32175 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:40.649049 32175 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:40.665499 32175 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001: Generated new TSK 0
I20260709 12:28:40.666390 32175 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:40.681984 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:40.689317 32188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:40.690601 32189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.694243 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:40.694151 32191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.700210 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:40.700481 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:40.700623 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600120700610 us; error 0 us; skew 500 ppm
I20260709 12:28:40.701180 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:40.704077 16792 webserver.cc:533] Webserver started at http://127.16.102.1:46419/ using document root <none> and password file <none>
I20260709 12:28:40.704615 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:40.704787 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:40.705037 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:40.706405 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "922eda0357a64c13b1fed5dfbeba3cde"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.711752 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:40.716053 32196 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.717072 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:28:40.717536 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "922eda0357a64c13b1fed5dfbeba3cde"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.717923 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:40.733289 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:40.734850 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:40.736663 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:40.739534 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:40.739811 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.740125 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:40.740383 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.788805 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:35775
I20260709 12:28:40.788944 32258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:35775 every 8 connection(s)
I20260709 12:28:40.795176 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:40.804790 32263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:40.806411 32265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.848371 16792 server_base.cc:1061] running on GCE node
I20260709 12:28:40.849509 32259 heartbeater.cc:344] Connected to a master server at 127.16.102.62:36697
W20260709 12:28:40.850360 32268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.850564 32259 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:40.852368 32259 heartbeater.cc:507] Master 127.16.102.62:36697 requested a full tablet report, sending...
I20260709 12:28:40.852825 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:40.853204 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:40.853457 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600120853436 us; error 0 us; skew 500 ppm
I20260709 12:28:40.854584 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:40.856223 32132 ts_manager.cc:194] Registered new tserver with Master: 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775)
I20260709 12:28:40.859084 16792 webserver.cc:533] Webserver started at http://127.16.102.2:34791/ using document root <none> and password file <none>
I20260709 12:28:40.859973 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:40.859861 32132 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46778
I20260709 12:28:40.860466 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:40.860955 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:40.863310 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "342106197b2d444cb5be73592926bf91"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.869457 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20260709 12:28:40.874442 32272 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.875455 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:28:40.875851 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "342106197b2d444cb5be73592926bf91"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.876264 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:40.889292 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:40.891021 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:40.893088 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:40.896512 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:40.896814 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.897261 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:40.897428 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.953014 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:38883
I20260709 12:28:40.953244 32334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:38883 every 8 connection(s)
I20260709 12:28:40.960758 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:40.971912 32338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:40.973328 32339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.975944 16792 server_base.cc:1061] running on GCE node
W20260709 12:28:40.976999 32341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:40.978477 32335 heartbeater.cc:344] Connected to a master server at 127.16.102.62:36697
I20260709 12:28:40.978559 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:40.979156 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:40.979223 32335 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:40.979374 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600120979355 us; error 0 us; skew 500 ppm
I20260709 12:28:40.980225 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:40.980435 32335 heartbeater.cc:507] Master 127.16.102.62:36697 requested a full tablet report, sending...
I20260709 12:28:40.983321 32132 ts_manager.cc:194] Registered new tserver with Master: 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:40.984457 16792 webserver.cc:533] Webserver started at http://127.16.102.3:45797/ using document root <none> and password file <none>
I20260709 12:28:40.985137 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:40.985376 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:40.985266 32132 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46788
I20260709 12:28:40.986049 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:40.987888 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "1daede7f0f194ea7847349205a419c57"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:40.993630 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20260709 12:28:40.998337 32346 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:40.999311 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:28:40.999679 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "1daede7f0f194ea7847349205a419c57"
format_stamp: "Formatted at 2026-07-09 12:28:40 on dist-test-slave-7xt3"
I20260709 12:28:41.000046 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:41.025954 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:41.027524 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:41.029452 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:41.032541 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:41.032850 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:41.033129 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:41.033327 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:41.088325 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:46317
I20260709 12:28:41.088445 32408 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:46317 every 8 connection(s)
I20260709 12:28:41.106899 32409 heartbeater.cc:344] Connected to a master server at 127.16.102.62:36697
I20260709 12:28:41.107620 32409 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:41.108693 32409 heartbeater.cc:507] Master 127.16.102.62:36697 requested a full tablet report, sending...
I20260709 12:28:41.111325 32132 ts_manager.cc:194] Registered new tserver with Master: 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:41.111598 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019438639s
I20260709 12:28:41.113137 32132 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46794
I20260709 12:28:41.146343 32132 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46810:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:41.150075 32132 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:41.246289 32374 tablet_service.cc:1511] Processing CreateTablet for tablet 8d9107ba28254d7f9dba13f18e4baae7 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:41.249513 32373 tablet_service.cc:1511] Processing CreateTablet for tablet ce2d4f84a8154744ab96875f21abb7e5 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:41.249513 32372 tablet_service.cc:1511] Processing CreateTablet for tablet 4d378c3205784a9d82c5d29c71db02c3 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:41.252593 32374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9107ba28254d7f9dba13f18e4baae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.253916 32371 tablet_service.cc:1511] Processing CreateTablet for tablet 5317da388b6d4e5398f870ae9c5ca79c (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:41.257251 32370 tablet_service.cc:1511] Processing CreateTablet for tablet c863ac57cd4547758e355ec99b4033c1 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:41.258963 32373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2d4f84a8154744ab96875f21abb7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.261386 32371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5317da388b6d4e5398f870ae9c5ca79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.264456 32372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d378c3205784a9d82c5d29c71db02c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.267706 32370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c863ac57cd4547758e355ec99b4033c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.276914 32369 tablet_service.cc:1511] Processing CreateTablet for tablet b4a724d0b531408a91fb85a91d558129 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:41.279388 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a724d0b531408a91fb85a91d558129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.287174 32300 tablet_service.cc:1511] Processing CreateTablet for tablet 8d9107ba28254d7f9dba13f18e4baae7 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:41.289155 32300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9107ba28254d7f9dba13f18e4baae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.291322 32299 tablet_service.cc:1511] Processing CreateTablet for tablet ce2d4f84a8154744ab96875f21abb7e5 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:41.293126 32299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2d4f84a8154744ab96875f21abb7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.295261 32298 tablet_service.cc:1511] Processing CreateTablet for tablet 4d378c3205784a9d82c5d29c71db02c3 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:41.297055 32298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d378c3205784a9d82c5d29c71db02c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.296789 32368 tablet_service.cc:1511] Processing CreateTablet for tablet 2b1a719b6a2648bb81fa22310ace5a34 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:41.298800 32368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b1a719b6a2648bb81fa22310ace5a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.299271 32297 tablet_service.cc:1511] Processing CreateTablet for tablet 5317da388b6d4e5398f870ae9c5ca79c (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:41.300983 32297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5317da388b6d4e5398f870ae9c5ca79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.303428 32296 tablet_service.cc:1511] Processing CreateTablet for tablet c863ac57cd4547758e355ec99b4033c1 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:41.305269 32296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c863ac57cd4547758e355ec99b4033c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.308589 32367 tablet_service.cc:1511] Processing CreateTablet for tablet 47cff3af1b654c3faaa22d43155cab22 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:41.308789 32295 tablet_service.cc:1511] Processing CreateTablet for tablet b4a724d0b531408a91fb85a91d558129 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:41.311655 32295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a724d0b531408a91fb85a91d558129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.313428 32294 tablet_service.cc:1511] Processing CreateTablet for tablet 2b1a719b6a2648bb81fa22310ace5a34 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:41.314580 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cff3af1b654c3faaa22d43155cab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.315155 32294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b1a719b6a2648bb81fa22310ace5a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.314821 32293 tablet_service.cc:1511] Processing CreateTablet for tablet 47cff3af1b654c3faaa22d43155cab22 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:41.316340 32293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cff3af1b654c3faaa22d43155cab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.321945 32292 tablet_service.cc:1511] Processing CreateTablet for tablet 6dcb57f0ba7f4f378d3a400a57706cf0 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:41.323848 32292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dcb57f0ba7f4f378d3a400a57706cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.357431 32224 tablet_service.cc:1511] Processing CreateTablet for tablet 8d9107ba28254d7f9dba13f18e4baae7 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:41.359392 32224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9107ba28254d7f9dba13f18e4baae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.363535 32223 tablet_service.cc:1511] Processing CreateTablet for tablet ce2d4f84a8154744ab96875f21abb7e5 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:41.365478 32223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2d4f84a8154744ab96875f21abb7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.372176 32425 tablet_bootstrap.cc:492] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:41.376214 32220 tablet_service.cc:1511] Processing CreateTablet for tablet c863ac57cd4547758e355ec99b4033c1 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:41.378193 32220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c863ac57cd4547758e355ec99b4033c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.379644 32221 tablet_service.cc:1511] Processing CreateTablet for tablet 5317da388b6d4e5398f870ae9c5ca79c (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:41.383900 32222 tablet_service.cc:1511] Processing CreateTablet for tablet 4d378c3205784a9d82c5d29c71db02c3 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:41.389622 32219 tablet_service.cc:1511] Processing CreateTablet for tablet b4a724d0b531408a91fb85a91d558129 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:41.392882 32218 tablet_service.cc:1511] Processing CreateTablet for tablet 2b1a719b6a2648bb81fa22310ace5a34 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:41.395500 32217 tablet_service.cc:1511] Processing CreateTablet for tablet 47cff3af1b654c3faaa22d43155cab22 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:41.400130 32216 tablet_service.cc:1511] Processing CreateTablet for tablet 6dcb57f0ba7f4f378d3a400a57706cf0 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:41.400568 32374 tablet_service.cc:1511] Processing CreateTablet for tablet 6dcb57f0ba7f4f378d3a400a57706cf0 (DEFAULT_TABLE table=test-workload [id=fcb1b6f82f15445da6083054f155c2ec]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:41.406169 32374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dcb57f0ba7f4f378d3a400a57706cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.425335 32221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5317da388b6d4e5398f870ae9c5ca79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.426460 32217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cff3af1b654c3faaa22d43155cab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.427217 32218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b1a719b6a2648bb81fa22310ace5a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.431263 32424 tablet_bootstrap.cc:492] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.434170 32216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dcb57f0ba7f4f378d3a400a57706cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.435650 32219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a724d0b531408a91fb85a91d558129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.442397 32222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d378c3205784a9d82c5d29c71db02c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:41.454319 32425 tablet_bootstrap.cc:654] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.502866 32426 tablet_bootstrap.cc:492] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:41.501950 32424 tablet_bootstrap.cc:654] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.519851 32426 tablet_bootstrap.cc:654] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.528244 32425 tablet_bootstrap.cc:492] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:41.528926 32425 ts_tablet_manager.cc:1403] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.157s	user 0.019s	sys 0.008s
I20260709 12:28:41.542625 32425 raft_consensus.cc:359] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.545979 32425 raft_consensus.cc:385] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.546418 32425 raft_consensus.cc:740] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.547873 32424 tablet_bootstrap.cc:492] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.548636 32424 ts_tablet_manager.cc:1403] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.118s	user 0.015s	sys 0.013s
I20260709 12:28:41.547408 32425 consensus_queue.cc:260] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.554621 32335 heartbeater.cc:499] Master 127.16.102.62:36697 was elected leader, sending a full tablet report...
I20260709 12:28:41.556205 32424 raft_consensus.cc:359] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.559196 32424 raft_consensus.cc:385] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.559605 32424 raft_consensus.cc:740] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.560545 32424 consensus_queue.cc:260] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.558646 32425 ts_tablet_manager.cc:1434] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.002s
I20260709 12:28:41.563673 32425 tablet_bootstrap.cc:492] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:41.571300 32424 ts_tablet_manager.cc:1434] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.006s
I20260709 12:28:41.572415 32409 heartbeater.cc:499] Master 127.16.102.62:36697 was elected leader, sending a full tablet report...
I20260709 12:28:41.572816 32426 tablet_bootstrap.cc:492] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:41.573414 32425 tablet_bootstrap.cc:654] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.573555 32426 ts_tablet_manager.cc:1403] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.071s	user 0.013s	sys 0.010s
I20260709 12:28:41.578119 32426 raft_consensus.cc:359] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.579269 32426 raft_consensus.cc:385] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.579622 32426 raft_consensus.cc:740] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.580643 32426 consensus_queue.cc:260] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.585083 32424 tablet_bootstrap.cc:492] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.587757 32425 tablet_bootstrap.cc:492] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:41.588690 32425 ts_tablet_manager.cc:1403] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.008s
I20260709 12:28:41.592715 32425 raft_consensus.cc:359] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.594775 32424 tablet_bootstrap.cc:654] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.595041 32425 raft_consensus.cc:385] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.595647 32425 raft_consensus.cc:740] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.596948 32425 consensus_queue.cc:260] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.603216 32425 ts_tablet_manager.cc:1434] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260709 12:28:41.605122 32425 tablet_bootstrap.cc:492] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
W20260709 12:28:41.607430 32180 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:41.616453 32430 raft_consensus.cc:493] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.623510 32430 raft_consensus.cc:515] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.631235 32259 heartbeater.cc:499] Master 127.16.102.62:36697 was elected leader, sending a full tablet report...
I20260709 12:28:41.631985 32426 ts_tablet_manager.cc:1434] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.058s	user 0.019s	sys 0.034s
I20260709 12:28:41.633421 32426 tablet_bootstrap.cc:492] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:41.636950 32424 tablet_bootstrap.cc:492] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.637969 32424 ts_tablet_manager.cc:1403] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.053s	user 0.025s	sys 0.022s
I20260709 12:28:41.641465 32426 tablet_bootstrap.cc:654] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.641935 32425 tablet_bootstrap.cc:654] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.644949 32424 raft_consensus.cc:359] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.646071 32424 raft_consensus.cc:385] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.646386 32424 raft_consensus.cc:740] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.647408 32424 consensus_queue.cc:260] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.655059 32424 ts_tablet_manager.cc:1434] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.001s
I20260709 12:28:41.656845 32424 tablet_bootstrap.cc:492] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.659897 32430 leader_election.cc:290] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:41.666253 32424 tablet_bootstrap.cc:654] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.671222 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dcb57f0ba7f4f378d3a400a57706cf0" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
W20260709 12:28:41.674947 32275 leader_election.cc:343] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.676259 32430 raft_consensus.cc:493] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.677086 32430 raft_consensus.cc:515] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.680226 32424 tablet_bootstrap.cc:492] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.681098 32424 ts_tablet_manager.cc:1403] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.004s
I20260709 12:28:41.685412 32424 raft_consensus.cc:359] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.686399 32424 raft_consensus.cc:385] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.686744 32424 raft_consensus.cc:740] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.687891 32424 consensus_queue.cc:260] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.694576 32424 ts_tablet_manager.cc:1434] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.001s
I20260709 12:28:41.694929 32430 leader_election.cc:290] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:41.696074 32424 tablet_bootstrap.cc:492] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.698617 32425 tablet_bootstrap.cc:492] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:41.699474 32425 ts_tablet_manager.cc:1403] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.095s	user 0.017s	sys 0.000s
I20260709 12:28:41.705555 32425 raft_consensus.cc:359] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.706606 32425 raft_consensus.cc:385] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.706951 32425 raft_consensus.cc:740] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.707613 32424 tablet_bootstrap.cc:654] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.710083 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dcb57f0ba7f4f378d3a400a57706cf0" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:41.710395 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:41.711293 32383 raft_consensus.cc:2468] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
W20260709 12:28:41.712138 32274 leader_election.cc:343] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.710192 32426 tablet_bootstrap.cc:492] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:41.712747 32274 leader_election.cc:304] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 342106197b2d444cb5be73592926bf91; no voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:41.713577 32426 ts_tablet_manager.cc:1403] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.080s	user 0.043s	sys 0.028s
I20260709 12:28:41.714908 32274 leader_election.cc:304] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:41.717603 32426 raft_consensus.cc:359] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.720247 32426 raft_consensus.cc:385] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.720636 32424 tablet_bootstrap.cc:492] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.718148 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:41.719646 32430 raft_consensus.cc:2749] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:41.721441 32424 ts_tablet_manager.cc:1403] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.010s
I20260709 12:28:41.720628 32426 raft_consensus.cc:740] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.721941 32430 raft_consensus.cc:2804] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:41.722534 32430 raft_consensus.cc:493] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:41.723027 32430 raft_consensus.cc:3060] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:41.722754 32426 consensus_queue.cc:260] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.724102 32234 raft_consensus.cc:2468] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:41.718789 32425 consensus_queue.cc:260] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.725476 32424 raft_consensus.cc:359] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.726466 32424 raft_consensus.cc:385] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.726806 32424 raft_consensus.cc:740] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.727756 32424 consensus_queue.cc:260] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.729125 32425 ts_tablet_manager.cc:1434] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.029s	user 0.011s	sys 0.000s
I20260709 12:28:41.730583 32424 ts_tablet_manager.cc:1434] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.003s
I20260709 12:28:41.733193 32424 tablet_bootstrap.cc:492] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.731034 32426 ts_tablet_manager.cc:1434] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:28:41.738415 32426 tablet_bootstrap.cc:492] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:41.741463 32424 tablet_bootstrap.cc:654] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.742130 32425 tablet_bootstrap.cc:492] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:41.748605 32426 tablet_bootstrap.cc:654] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.766326 32430 raft_consensus.cc:515] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.769356 32425 tablet_bootstrap.cc:654] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.769454 32424 tablet_bootstrap.cc:492] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.770673 32424 ts_tablet_manager.cc:1403] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.038s	user 0.018s	sys 0.019s
I20260709 12:28:41.770642 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:41.771713 32234 raft_consensus.cc:3060] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:41.772805 32426 tablet_bootstrap.cc:492] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:41.773458 32426 ts_tablet_manager.cc:1403] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.035s	user 0.012s	sys 0.009s
I20260709 12:28:41.775300 32424 raft_consensus.cc:359] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.776328 32424 raft_consensus.cc:385] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.776701 32424 raft_consensus.cc:740] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.777179 32426 raft_consensus.cc:359] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.778117 32426 raft_consensus.cc:385] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.778537 32426 raft_consensus.cc:740] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.777856 32424 consensus_queue.cc:260] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.779520 32426 consensus_queue.cc:260] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.782123 32426 ts_tablet_manager.cc:1434] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.008s	user 0.011s	sys 0.000s
I20260709 12:28:41.783972 32426 tablet_bootstrap.cc:492] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:41.789644 32424 ts_tablet_manager.cc:1434] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:28:41.791399 32424 tablet_bootstrap.cc:492] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.799463 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:41.800549 32383 raft_consensus.cc:3060] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:41.802942 32426 tablet_bootstrap.cc:654] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.805358 32424 tablet_bootstrap.cc:654] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.812924 32430 leader_election.cc:290] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:41.816907 32426 tablet_bootstrap.cc:492] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:41.817773 32426 ts_tablet_manager.cc:1403] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.034s	user 0.009s	sys 0.012s
I20260709 12:28:41.819905 32424 tablet_bootstrap.cc:492] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.820689 32424 ts_tablet_manager.cc:1403] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.030s	user 0.013s	sys 0.009s
I20260709 12:28:41.822017 32383 raft_consensus.cc:2468] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:41.822131 32426 raft_consensus.cc:359] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.823313 32426 raft_consensus.cc:385] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.823662 32426 raft_consensus.cc:740] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.824424 32274 leader_election.cc:304] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:41.824818 32426 consensus_queue.cc:260] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.825081 32424 raft_consensus.cc:359] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.825975 32430 raft_consensus.cc:2804] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:41.828703 32424 raft_consensus.cc:385] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.828971 32430 raft_consensus.cc:697] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:41.829161 32424 raft_consensus.cc:740] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.830273 32430 consensus_queue.cc:237] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.830466 32424 consensus_queue.cc:260] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.831328 32425 tablet_bootstrap.cc:492] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:41.832049 32425 ts_tablet_manager.cc:1403] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.095s	user 0.026s	sys 0.042s
I20260709 12:28:41.827564 32426 ts_tablet_manager.cc:1434] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:28:41.817747 32234 raft_consensus.cc:2468] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:41.828102 32433 raft_consensus.cc:493] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.835296 32426 tablet_bootstrap.cc:492] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:41.834928 32433 raft_consensus.cc:515] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.835971 32425 raft_consensus.cc:359] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.836889 32425 raft_consensus.cc:385] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.837229 32425 raft_consensus.cc:740] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.838361 32425 consensus_queue.cc:260] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.842065 32425 ts_tablet_manager.cc:1434] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20260709 12:28:41.843708 32425 tablet_bootstrap.cc:492] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:41.845358 32426 tablet_bootstrap.cc:654] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.852262 32433 leader_election.cc:290] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:41.854322 32424 ts_tablet_manager.cc:1434] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.002s
I20260709 12:28:41.855705 32425 tablet_bootstrap.cc:654] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.857758 32424 tablet_bootstrap.cc:492] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:41.861682 32431 raft_consensus.cc:493] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.862468 32431 raft_consensus.cc:515] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.869469 32431 leader_election.cc:290] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775)
I20260709 12:28:41.882246 32433 raft_consensus.cc:493] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.883956 32433 raft_consensus.cc:515] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.886483 32433 leader_election.cc:290] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:41.891743 32424 tablet_bootstrap.cc:654] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.903769 32426 tablet_bootstrap.cc:492] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:41.904462 32426 ts_tablet_manager.cc:1403] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.069s	user 0.018s	sys 0.028s
I20260709 12:28:41.908116 32426 raft_consensus.cc:359] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.909005 32426 raft_consensus.cc:385] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.909332 32426 raft_consensus.cc:740] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.907845 32130 catalog_manager.cc:5697] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:41.910382 32426 consensus_queue.cc:260] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.919811 32425 tablet_bootstrap.cc:492] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:41.920375 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:41.920758 32425 ts_tablet_manager.cc:1403] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.077s	user 0.008s	sys 0.011s
I20260709 12:28:41.921330 32310 raft_consensus.cc:2468] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:41.925407 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:41.929327 32200 leader_election.cc:304] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:41.930338 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:41.931722 32433 raft_consensus.cc:2804] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:41.916491 32426 ts_tablet_manager.cc:1434] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:28:41.932288 32433 raft_consensus.cc:493] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:41.934024 32433 raft_consensus.cc:3060] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:41.934508 32198 leader_election.cc:343] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.937937 32424 tablet_bootstrap.cc:492] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:41.938701 32424 ts_tablet_manager.cc:1403] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.081s	user 0.015s	sys 0.016s
W20260709 12:28:41.938941 32200 leader_election.cc:343] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.933109 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:41.940938 32383 raft_consensus.cc:2468] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:41.942425 32431 raft_consensus.cc:493] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.943138 32431 raft_consensus.cc:515] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.938606 32426 tablet_bootstrap.cc:492] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:41.946650 32198 leader_election.cc:304] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 342106197b2d444cb5be73592926bf91
I20260709 12:28:41.945593 32431 leader_election.cc:290] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:41.949306 32425 raft_consensus.cc:359] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.956182 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:41.959474 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:41.962752 32433 raft_consensus.cc:515] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
W20260709 12:28:41.963346 32350 leader_election.cc:343] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.965128 32425 raft_consensus.cc:385] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.967706 32425 raft_consensus.cc:740] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:41.970569 32458 raft_consensus.cc:2804] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:41.972821 32458 raft_consensus.cc:493] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:41.973239 32458 raft_consensus.cc:3060] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:41.973551 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:41.977183 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:41.978431 32424 raft_consensus.cc:359] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:41.979516 32424 raft_consensus.cc:385] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:41.979902 32424 raft_consensus.cc:740] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
W20260709 12:28:41.980239 32349 leader_election.cc:343] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.981127 32349 leader_election.cc:304] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1daede7f0f194ea7847349205a419c57; no voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:41.971029 32426 tablet_bootstrap.cc:654] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:41.972131 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:41.984393 32431 raft_consensus.cc:2749] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:41.971374 32433 leader_election.cc:290] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:41.971740 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:41.986900 32310 raft_consensus.cc:3060] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:41.972170 32425 consensus_queue.cc:260] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:41.995308 32309 raft_consensus.cc:2468] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:41.996755 32426 tablet_bootstrap.cc:492] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:41.997480 32426 ts_tablet_manager.cc:1403] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.063s	user 0.015s	sys 0.001s
I20260709 12:28:41.997659 32350 leader_election.cc:304] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:41.998735 32431 raft_consensus.cc:2804] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:41.999164 32431 raft_consensus.cc:493] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:41.999526 32431 raft_consensus.cc:3060] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.000598 32425 ts_tablet_manager.cc:1434] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.079s	user 0.012s	sys 0.000s
I20260709 12:28:42.001081 32426 raft_consensus.cc:359] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.002008 32426 raft_consensus.cc:385] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.002351 32426 raft_consensus.cc:740] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
W20260709 12:28:42.002554 32198 leader_election.cc:343] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:41.980953 32424 consensus_queue.cc:260] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.004839 32425 tablet_bootstrap.cc:492] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:42.011806 32431 raft_consensus.cc:515] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:42.013724 32458 raft_consensus.cc:515] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:42.014130 32426 consensus_queue.cc:260] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.015322 32234 raft_consensus.cc:2393] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1daede7f0f194ea7847349205a419c57 in current term 1: Already voted for candidate 922eda0357a64c13b1fed5dfbeba3cde in this term.
I20260709 12:28:42.015758 32424 ts_tablet_manager.cc:1434] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.077s	user 0.005s	sys 0.005s
I20260709 12:28:42.017424 32424 tablet_bootstrap.cc:492] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:42.013619 32310 raft_consensus.cc:2468] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:42.022274 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:42.023305 32383 raft_consensus.cc:2393] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 922eda0357a64c13b1fed5dfbeba3cde in current term 1: Already voted for candidate 1daede7f0f194ea7847349205a419c57 in this term.
I20260709 12:28:42.024425 32430 raft_consensus.cc:493] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.025002 32200 leader_election.cc:304] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 1daede7f0f194ea7847349205a419c57
I20260709 12:28:42.026583 32433 raft_consensus.cc:2804] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.026832 32458 leader_election.cc:290] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Requested vote from peers 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.029963 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:42.030884 32310 raft_consensus.cc:3060] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.032298 32425 tablet_bootstrap.cc:654] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.025089 32430 raft_consensus.cc:515] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.036412 32430 leader_election.cc:290] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.036484 32424 tablet_bootstrap.cc:654] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.038682 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cff3af1b654c3faaa22d43155cab22" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:42.039589 32234 raft_consensus.cc:2468] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:42.040905 32310 raft_consensus.cc:2468] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:42.041951 32425 tablet_bootstrap.cc:492] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:42.042593 32425 ts_tablet_manager.cc:1403] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.041s	user 0.015s	sys 0.000s
I20260709 12:28:42.042254 32350 leader_election.cc:304] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:42.042146 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cff3af1b654c3faaa22d43155cab22" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:42.043610 32462 raft_consensus.cc:2804] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.043735 32383 raft_consensus.cc:2468] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:42.044874 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:42.045646 32275 leader_election.cc:304] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.046357 32309 raft_consensus.cc:2393] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 922eda0357a64c13b1fed5dfbeba3cde in current term 1: Already voted for candidate 1daede7f0f194ea7847349205a419c57 in this term.
I20260709 12:28:42.047555 32462 raft_consensus.cc:697] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:42.048789 32462 consensus_queue.cc:237] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:42.049167 32425 raft_consensus.cc:359] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.050091 32425 raft_consensus.cc:385] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.050577 32431 leader_election.cc:290] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775)
I20260709 12:28:42.050459 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:42.051482 32200 leader_election.cc:304] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 922eda0357a64c13b1fed5dfbeba3cde; no voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91
I20260709 12:28:42.050459 32425 raft_consensus.cc:740] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.053578 32425 consensus_queue.cc:260] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.051532 32234 raft_consensus.cc:2393] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1daede7f0f194ea7847349205a419c57 in current term 1: Already voted for candidate 922eda0357a64c13b1fed5dfbeba3cde in this term.
I20260709 12:28:42.054181 32424 tablet_bootstrap.cc:492] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:42.054584 32430 raft_consensus.cc:2804] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:42.054872 32424 ts_tablet_manager.cc:1403] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.038s	user 0.008s	sys 0.006s
I20260709 12:28:42.055258 32433 raft_consensus.cc:697] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:42.058979 32463 raft_consensus.cc:2749] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:42.064824 32433 consensus_queue.cc:237] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.066926 32425 ts_tablet_manager.cc:1434] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:28:42.067847 32441 raft_consensus.cc:493] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.067559 32424 raft_consensus.cc:359] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.068475 32424 raft_consensus.cc:385] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.068452 32441 raft_consensus.cc:515] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.068933 32424 raft_consensus.cc:740] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.055081 32430 raft_consensus.cc:493] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.069991 32430 raft_consensus.cc:3060] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.069872 32424 consensus_queue.cc:260] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.072477 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:42.073098 32424 ts_tablet_manager.cc:1434] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:28:42.073413 32234 raft_consensus.cc:2468] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:42.075853 32431 raft_consensus.cc:493] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.080741 32431 raft_consensus.cc:515] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.076553 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:42.083278 32383 raft_consensus.cc:2468] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:42.081452 32275 leader_election.cc:304] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.084225 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:42.079200 32466 raft_consensus.cc:493] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.085029 32233 raft_consensus.cc:2468] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:42.085163 32466 raft_consensus.cc:515] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.079422 32441 leader_election.cc:290] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.091044 32425 tablet_bootstrap.cc:492] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:42.092624 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:42.093710 32310 raft_consensus.cc:2393] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1daede7f0f194ea7847349205a419c57 in current term 1: Already voted for candidate 922eda0357a64c13b1fed5dfbeba3cde in this term.
I20260709 12:28:42.095498 32466 leader_election.cc:290] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.085175 32426 ts_tablet_manager.cc:1434] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.087s	user 0.006s	sys 0.002s
I20260709 12:28:42.086876 32468 raft_consensus.cc:2804] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:42.087189 32349 leader_election.cc:304] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.097039 32466 raft_consensus.cc:2804] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:42.097461 32466 raft_consensus.cc:493] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.097522 32426 tablet_bootstrap.cc:492] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:42.097920 32466 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.098280 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:42.100481 32349 leader_election.cc:304] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1daede7f0f194ea7847349205a419c57; no voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:42.107695 32468 raft_consensus.cc:493] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.108433 32468 raft_consensus.cc:3060] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.108815 32431 leader_election.cc:290] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.074652 32424 tablet_bootstrap.cc:492] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:42.109596 32431 raft_consensus.cc:3060] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.110256 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
W20260709 12:28:42.112475 32350 leader_election.cc:343] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:42.112058 32131 catalog_manager.cc:5697] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 1 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:42.116329 32466 raft_consensus.cc:515] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.117240 32430 raft_consensus.cc:515] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.121392 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
W20260709 12:28:42.123531 32350 leader_election.cc:343] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:42.124289 32466 leader_election.cc:290] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.126045 32426 tablet_bootstrap.cc:654] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.127345 32424 tablet_bootstrap.cc:654] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.134614 32430 leader_election.cc:290] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.135835 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:42.135938 32234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cff3af1b654c3faaa22d43155cab22" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:42.136744 32233 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.137137 32234 raft_consensus.cc:3060] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.140908 32431 raft_consensus.cc:2749] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:42.129252 32130 catalog_manager.cc:5697] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:42.143292 32468 raft_consensus.cc:515] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.146868 32431 raft_consensus.cc:493] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.147743 32431 raft_consensus.cc:515] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.154145 32468 leader_election.cc:290] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.158196 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:42.159195 32232 raft_consensus.cc:3060] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.160269 32233 raft_consensus.cc:2468] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:42.162328 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:42.163421 32231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
W20260709 12:28:42.164564 32350 leader_election.cc:343] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:42.165446 32349 leader_election.cc:304] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 342106197b2d444cb5be73592926bf91
I20260709 12:28:42.166631 32462 raft_consensus.cc:2804] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:28:42.167078 32349 leader_election.cc:343] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:42.167388 32462 raft_consensus.cc:697] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:42.167680 32349 leader_election.cc:304] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1daede7f0f194ea7847349205a419c57; no voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:42.168900 32462 consensus_queue.cc:237] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.170878 32425 tablet_bootstrap.cc:654] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.172433 32470 raft_consensus.cc:2749] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:42.175065 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47cff3af1b654c3faaa22d43155cab22" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:42.175894 32383 raft_consensus.cc:3060] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.179575 32234 raft_consensus.cc:2468] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:42.181303 32275 leader_election.cc:304] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.182381 32468 raft_consensus.cc:2804] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.183053 32468 raft_consensus.cc:697] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:42.183665 32232 raft_consensus.cc:2468] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:42.184288 32468 consensus_queue.cc:237] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.185339 32275 leader_election.cc:304] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.186663 32441 raft_consensus.cc:2804] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.186756 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:42.187460 32441 raft_consensus.cc:697] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:42.189122 32384 raft_consensus.cc:3060] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.190721 32383 raft_consensus.cc:2468] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:42.191313 32441 consensus_queue.cc:237] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.194206 32424 tablet_bootstrap.cc:492] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:42.158325 32431 leader_election.cc:290] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.197683 32424 ts_tablet_manager.cc:1403] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.123s	user 0.024s	sys 0.027s
I20260709 12:28:42.216355 32384 raft_consensus.cc:2468] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:42.221972 32426 tablet_bootstrap.cc:492] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:42.221725 32424 raft_consensus.cc:359] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.222724 32426 ts_tablet_manager.cc:1403] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.126s	user 0.019s	sys 0.040s
I20260709 12:28:42.222960 32424 raft_consensus.cc:385] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.223443 32424 raft_consensus.cc:740] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.224594 32424 consensus_queue.cc:260] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.228240 32424 ts_tablet_manager.cc:1434] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.003s
I20260709 12:28:42.227835 32426 raft_consensus.cc:359] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.228821 32426 raft_consensus.cc:385] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.229172 32426 raft_consensus.cc:740] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.230041 32426 consensus_queue.cc:260] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.229791 32132 catalog_manager.cc:5697] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:42.234391 32426 ts_tablet_manager.cc:1434] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:28:42.235491 32426 tablet_bootstrap.cc:492] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:42.234457 32132 catalog_manager.cc:5697] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:42.243139 32425 tablet_bootstrap.cc:492] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:42.243813 32425 ts_tablet_manager.cc:1403] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.153s	user 0.026s	sys 0.012s
I20260709 12:28:42.247466 32425 raft_consensus.cc:359] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.248412 32425 raft_consensus.cc:385] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.248760 32425 raft_consensus.cc:740] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.249718 32431 raft_consensus.cc:493] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.250049 32425 consensus_queue.cc:260] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.250491 32431 raft_consensus.cc:515] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.253537 32425 ts_tablet_manager.cc:1434] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:28:42.253681 32426 tablet_bootstrap.cc:654] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.254546 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dcb57f0ba7f4f378d3a400a57706cf0" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:42.255111 32425 tablet_bootstrap.cc:492] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:42.255342 32310 raft_consensus.cc:2468] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:42.256181 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dcb57f0ba7f4f378d3a400a57706cf0" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:42.256991 32233 raft_consensus.cc:2468] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:42.257191 32350 leader_election.cc:304] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:42.258258 32431 leader_election.cc:290] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.258567 32462 raft_consensus.cc:2804] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:42.259032 32462 raft_consensus.cc:493] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.259393 32462 raft_consensus.cc:3060] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.266148 32425 tablet_bootstrap.cc:654] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.265834 32130 catalog_manager.cc:5697] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:42.277732 32426 tablet_bootstrap.cc:492] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:42.278409 32426 ts_tablet_manager.cc:1403] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.043s	user 0.008s	sys 0.023s
I20260709 12:28:42.279675 32425 tablet_bootstrap.cc:492] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:42.279256 32462 raft_consensus.cc:515] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.280339 32425 ts_tablet_manager.cc:1403] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.003s
I20260709 12:28:42.281766 32426 raft_consensus.cc:359] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.282760 32426 raft_consensus.cc:385] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.283062 32426 raft_consensus.cc:740] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.282955 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dcb57f0ba7f4f378d3a400a57706cf0" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:42.283892 32233 raft_consensus.cc:3060] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.284107 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dcb57f0ba7f4f378d3a400a57706cf0" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:42.284816 32310 raft_consensus.cc:3060] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.291637 32462 leader_election.cc:290] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:42.284116 32426 consensus_queue.cc:260] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.292744 32468 consensus_queue.cc:1048] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:42.294886 32310 raft_consensus.cc:2468] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:42.295126 32426 ts_tablet_manager.cc:1434] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 12:28:42.296594 32426 tablet_bootstrap.cc:492] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:42.296533 32350 leader_election.cc:304] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:42.298103 32431 raft_consensus.cc:2804] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.298645 32431 raft_consensus.cc:697] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:42.299575 32431 consensus_queue.cc:237] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.307770 32426 tablet_bootstrap.cc:654] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.319949 32426 tablet_bootstrap.cc:492] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:42.322086 32426 ts_tablet_manager.cc:1403] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.008s
I20260709 12:28:42.319846 32233 raft_consensus.cc:2468] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:42.320888 32425 raft_consensus.cc:359] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.323700 32425 raft_consensus.cc:385] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.324146 32425 raft_consensus.cc:740] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.325389 32425 consensus_queue.cc:260] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.325888 32426 raft_consensus.cc:359] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.327128 32426 raft_consensus.cc:385] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.327613 32426 raft_consensus.cc:740] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.328586 32426 consensus_queue.cc:260] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.328959 32425 ts_tablet_manager.cc:1434] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.048s	user 0.009s	sys 0.003s
I20260709 12:28:42.331269 32426 ts_tablet_manager.cc:1434] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:28:42.331635 32425 tablet_bootstrap.cc:492] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:42.339083 32430 raft_consensus.cc:493] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.339977 32430 raft_consensus.cc:515] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.342685 32425 tablet_bootstrap.cc:654] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:42.341846 32129 catalog_manager.cc:5697] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:42.343669 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:42.343935 32430 leader_election.cc:290] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.344568 32233 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:42.346251 32275 leader_election.cc:304] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.347281 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:42.347503 32430 raft_consensus.cc:2804] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:42.348132 32430 raft_consensus.cc:493] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.348291 32384 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:42.348567 32430 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.360246 32430 raft_consensus.cc:515] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.361564 32468 consensus_queue.cc:1048] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.364557 32430 leader_election.cc:290] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.368407 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:42.369249 32384 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.374348 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:42.375187 32233 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.376065 32425 tablet_bootstrap.cc:492] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:42.376739 32425 ts_tablet_manager.cc:1403] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.045s	user 0.020s	sys 0.008s
I20260709 12:28:42.378506 32384 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:42.380230 32274 leader_election.cc:304] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:42.381400 32430 raft_consensus.cc:2804] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.382263 32430 raft_consensus.cc:697] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:42.383275 32430 consensus_queue.cc:237] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.386368 32233 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:42.380223 32425 raft_consensus.cc:359] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.388665 32425 raft_consensus.cc:385] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:42.388984 32425 raft_consensus.cc:740] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:42.390043 32425 consensus_queue.cc:260] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.392588 32425 ts_tablet_manager.cc:1434] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:28:42.404021 32131 catalog_manager.cc:5697] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:42.500635 32431 consensus_queue.cc:1048] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:42.514128 32466 consensus_queue.cc:1048] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.538686 32463 consensus_queue.cc:1048] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:42.542513 32463 consensus_queue.cc:1048] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.596052 32310 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.604830 32466 consensus_queue.cc:1048] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:42.615909 32180 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:42.619503 32466 consensus_queue.cc:1048] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.652034 32430 consensus_queue.cc:1048] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:42.666181 32468 consensus_queue.cc:1048] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.683759 32463 raft_consensus.cc:493] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.684333 32463 raft_consensus.cc:515] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.686681 32463 leader_election.cc:290] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.687379 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:42.687882 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:42.688318 32310 raft_consensus.cc:2468] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:42.688658 32383 raft_consensus.cc:2468] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:42.689600 32200 leader_election.cc:304] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.690603 32463 raft_consensus.cc:2804] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:42.690912 32463 raft_consensus.cc:493] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.691205 32463 raft_consensus.cc:3060] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.698392 32463 raft_consensus.cc:515] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.700273 32463 leader_election.cc:290] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:42.701457 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:42.701457 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:42.702380 32383 raft_consensus.cc:3060] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.702486 32310 raft_consensus.cc:3060] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:42.709537 32310 raft_consensus.cc:2468] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:42.709537 32383 raft_consensus.cc:2468] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:42.711226 32200 leader_election.cc:304] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:42.712116 32463 raft_consensus.cc:2804] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:42.712689 32463 raft_consensus.cc:697] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:42.713526 32463 consensus_queue.cc:237] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:42.723928 32131 catalog_manager.cc:5697] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 1 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:42.750409 32468 consensus_queue.cc:1048] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:42.763482 32441 consensus_queue.cc:1048] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.820901 32441 consensus_queue.cc:1048] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:42.833591 32441 consensus_queue.cc:1048] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.854238 32466 consensus_queue.cc:1048] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:42.869608 32462 consensus_queue.cc:1048] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:43.076956 32383 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:49058
I20260709 12:28:43.077714 32383 raft_consensus.cc:606] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:43.085280 32462 raft_consensus.cc:993] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.086081 32470 raft_consensus.cc:1081] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.088186 32233 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39056
I20260709 12:28:43.088932 32233 raft_consensus.cc:493] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.089234 32233 raft_consensus.cc:3060] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.096160 32310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:48882
I20260709 12:28:43.097057 32310 raft_consensus.cc:606] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:43.098503 32233 raft_consensus.cc:515] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:43.100929 32233 leader_election.cc:290] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:43.101783 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.102576 32383 raft_consensus.cc:3055] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.102924 32383 raft_consensus.cc:740] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:43.103300 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d9107ba28254d7f9dba13f18e4baae7" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:43.103971 32310 raft_consensus.cc:3060] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.103798 32383 consensus_queue.cc:260] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:43.104626 32309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:48882
I20260709 12:28:43.105209 32383 raft_consensus.cc:3060] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.105293 32309 raft_consensus.cc:606] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:43.112372 32309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:48882
I20260709 12:28:43.113140 32309 raft_consensus.cc:606] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:43.113443 32310 raft_consensus.cc:2468] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.113664 32383 raft_consensus.cc:2468] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.115160 32200 leader_election.cc:304] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:43.116186 32463 raft_consensus.cc:2804] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.116814 32463 raft_consensus.cc:697] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:43.117807 32463 consensus_queue.cc:237] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } }
I20260709 12:28:43.124064 32383 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
mode: GRACEFUL
new_leader_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:49058
I20260709 12:28:43.124881 32383 raft_consensus.cc:606] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Received request to transfer leadership to TS 342106197b2d444cb5be73592926bf91
I20260709 12:28:43.130291 32132 catalog_manager.cc:5697] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 1daede7f0f194ea7847349205a419c57 (127.16.102.3) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:43.142990 32233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
mode: GRACEFUL
new_leader_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:39058
I20260709 12:28:43.143672 32233 raft_consensus.cc:606] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Received request to transfer leadership to TS 342106197b2d444cb5be73592926bf91
I20260709 12:28:43.150177 32233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
mode: GRACEFUL
new_leader_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:39058
I20260709 12:28:43.150844 32233 raft_consensus.cc:606] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Received request to transfer leadership to TS 342106197b2d444cb5be73592926bf91
I20260709 12:28:43.159466 32493 raft_consensus.cc:993] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: : Instructing follower 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:43.160116 32463 raft_consensus.cc:1081] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Signalling peer 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:43.162154 32309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34"
dest_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:48858
I20260709 12:28:43.162825 32309 raft_consensus.cc:493] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.163185 32309 raft_consensus.cc:3060] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.171030 32309 raft_consensus.cc:515] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.173287 32309 leader_election.cc:290] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:43.174327 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:43.174655 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b1a719b6a2648bb81fa22310ace5a34" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.175108 32233 raft_consensus.cc:3055] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.175359 32383 raft_consensus.cc:3060] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.175495 32233 raft_consensus.cc:740] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:43.176416 32233 consensus_queue.cc:260] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.177639 32233 raft_consensus.cc:3060] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.183300 32383 raft_consensus.cc:2468] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:43.184937 32274 leader_election.cc:304] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:43.184937 32233 raft_consensus.cc:2468] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:43.186324 32468 raft_consensus.cc:2804] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.186889 32468 raft_consensus.cc:697] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:43.187858 32468 consensus_queue.cc:237] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.198104 32131 catalog_manager.cc:5697] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 1 to 2, leader changed from 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1) to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:43.256246 32463 consensus_queue.cc:1048] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:43.270393 32463 consensus_queue.cc:1048] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:43.281190 32463 raft_consensus.cc:993] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: : Instructing follower 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:43.281735 32468 raft_consensus.cc:993] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.281759 32463 raft_consensus.cc:1081] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Signalling peer 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:43.282512 32430 raft_consensus.cc:1081] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.284874 32233 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39048
I20260709 12:28:43.284662 32309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129"
dest_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:48858
I20260709 12:28:43.285612 32233 raft_consensus.cc:493] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.286290 32233 raft_consensus.cc:3060] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.286473 32309 raft_consensus.cc:493] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.287041 32309 raft_consensus.cc:3060] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.294109 32233 raft_consensus.cc:515] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.294463 32309 raft_consensus.cc:515] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.297612 32309 leader_election.cc:290] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:43.297984 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:43.298491 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d378c3205784a9d82c5d29c71db02c3" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.299045 32310 raft_consensus.cc:3055] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.299501 32383 raft_consensus.cc:3060] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.299091 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:43.298207 32233 leader_election.cc:290] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:43.300745 32232 raft_consensus.cc:3055] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.301151 32232 raft_consensus.cc:740] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:43.301309 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4a724d0b531408a91fb85a91d558129" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.302065 32384 raft_consensus.cc:3060] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.302173 32232 consensus_queue.cc:260] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.303925 32232 raft_consensus.cc:3060] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.299501 32310 raft_consensus.cc:740] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:43.305214 32310 consensus_queue.cc:260] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.306793 32310 raft_consensus.cc:3060] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.311156 32383 raft_consensus.cc:2468] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.312947 32198 leader_election.cc:304] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:43.314162 32463 raft_consensus.cc:2804] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.314186 32232 raft_consensus.cc:2468] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:43.314764 32463 raft_consensus.cc:697] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:43.315554 32384 raft_consensus.cc:2468] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:43.316067 32275 leader_election.cc:304] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:43.315886 32463 consensus_queue.cc:237] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.317533 32430 raft_consensus.cc:2804] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.317853 32310 raft_consensus.cc:2468] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.318177 32430 raft_consensus.cc:697] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:43.319324 32430 consensus_queue.cc:237] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.326341 32468 raft_consensus.cc:993] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.326941 32430 raft_consensus.cc:1081] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.326982 32132 catalog_manager.cc:5697] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:43.328863 32232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39048
I20260709 12:28:43.329485 32232 raft_consensus.cc:493] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.329931 32232 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.329005 32131 catalog_manager.cc:5697] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 1 to 2, leader changed from 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1) to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:43.339927 32232 raft_consensus.cc:515] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.344174 32232 leader_election.cc:290] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:43.345430 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:43.345944 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.346256 32310 raft_consensus.cc:3055] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.346702 32384 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.346706 32310 raft_consensus.cc:740] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:43.347945 32310 consensus_queue.cc:260] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.349172 32310 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.354200 32384 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.355736 32198 leader_election.cc:304] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:43.356629 32463 raft_consensus.cc:2804] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.357332 32463 raft_consensus.cc:697] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:43.358589 32310 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.358428 32463 consensus_queue.cc:237] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.366807 32131 catalog_manager.cc:5697] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:43.494038 32430 raft_consensus.cc:993] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.494617 32509 raft_consensus.cc:1081] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:43.496495 32232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39048
I20260709 12:28:43.497090 32232 raft_consensus.cc:493] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.497367 32232 raft_consensus.cc:3060] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.504146 32232 raft_consensus.cc:515] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.506484 32232 leader_election.cc:290] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:43.507704 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.507704 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2d4f84a8154744ab96875f21abb7e5" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:43.508771 32384 raft_consensus.cc:3060] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.508850 32310 raft_consensus.cc:3055] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.509379 32310 raft_consensus.cc:740] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:43.510257 32310 consensus_queue.cc:260] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.511782 32310 raft_consensus.cc:3060] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.516062 32384 raft_consensus.cc:2468] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.517907 32198 leader_election.cc:304] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:43.518870 32463 raft_consensus.cc:2804] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.519172 32310 raft_consensus.cc:2468] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:43.519380 32463 raft_consensus.cc:697] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:43.520589 32463 consensus_queue.cc:237] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.529646 32131 catalog_manager.cc:5697] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:43.581132 32462 raft_consensus.cc:993] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: : Instructing follower 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:43.581712 32462 raft_consensus.cc:1081] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Signalling peer 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:43.583565 32310 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1"
dest_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:48866
I20260709 12:28:43.584209 32310 raft_consensus.cc:493] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:43.584563 32310 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.591377 32310 raft_consensus.cc:515] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.593393 32310 leader_election.cc:290] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:43.594596 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:43.594669 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:43.595715 32232 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.595921 32384 raft_consensus.cc:3055] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:43.596418 32384 raft_consensus.cc:740] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:43.597446 32384 consensus_queue.cc:260] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.599020 32384 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:43.605544 32232 raft_consensus.cc:2468] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:43.607200 32275 leader_election.cc:304] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:43.608183 32509 raft_consensus.cc:2804] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:43.608227 32384 raft_consensus.cc:2468] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:43.608790 32509 raft_consensus.cc:697] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:43.610023 32509 consensus_queue.cc:237] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:43.618409 32131 catalog_manager.cc:5697] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 1 to 2, leader changed from 1daede7f0f194ea7847349205a419c57 (127.16.102.3) to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:43.634270 32384 raft_consensus.cc:1275] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.635730 32463 consensus_queue.cc:1048] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:43.646994 32310 raft_consensus.cc:1275] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.648909 32493 consensus_queue.cc:1048] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.702325 32310 raft_consensus.cc:1275] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.703837 32493 consensus_queue.cc:1048] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:43.714716 32384 raft_consensus.cc:1275] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.716661 32463 consensus_queue.cc:1048] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.721410 32383 raft_consensus.cc:1275] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.724071 32509 consensus_queue.cc:1048] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.735669 32232 raft_consensus.cc:1275] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.737671 32430 consensus_queue.cc:1048] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.799786 32310 raft_consensus.cc:1275] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.801405 32463 consensus_queue.cc:1048] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:43.810675 32384 raft_consensus.cc:1275] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.813741 32493 consensus_queue.cc:1048] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.853982 32384 raft_consensus.cc:1275] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.855716 32430 consensus_queue.cc:1048] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:43.865870 32232 raft_consensus.cc:1275] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.869045 32509 consensus_queue.cc:1048] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.901608 32310 raft_consensus.cc:1275] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.903403 32493 consensus_queue.cc:1048] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:43.914306 32384 raft_consensus.cc:1275] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.916635 32493 consensus_queue.cc:1048] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:44.063747 32232 raft_consensus.cc:1275] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:44.065356 32509 consensus_queue.cc:1048] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:44.077450 32384 raft_consensus.cc:1275] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:44.087453 32468 consensus_queue.cc:1048] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:46.193207 32131 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59906:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:46.195873 32131 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:46.255208 32219 tablet_service.cc:1511] Processing CreateTablet for tablet 09073e2cfcc344fc87cd7b4e8dc3d4e4 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:46.256645 32219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09073e2cfcc344fc87cd7b4e8dc3d4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.258363 32295 tablet_service.cc:1511] Processing CreateTablet for tablet 09073e2cfcc344fc87cd7b4e8dc3d4e4 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:46.259382 32374 tablet_service.cc:1511] Processing CreateTablet for tablet 09073e2cfcc344fc87cd7b4e8dc3d4e4 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:46.259941 32295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09073e2cfcc344fc87cd7b4e8dc3d4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.260816 32374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09073e2cfcc344fc87cd7b4e8dc3d4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.262559 32216 tablet_service.cc:1511] Processing CreateTablet for tablet ad5339dac1fd4281a629838a43425f82 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:46.263625 32297 tablet_service.cc:1511] Processing CreateTablet for tablet ad5339dac1fd4281a629838a43425f82 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:46.264137 32216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad5339dac1fd4281a629838a43425f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.265102 32297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad5339dac1fd4281a629838a43425f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.267035 32221 tablet_service.cc:1511] Processing CreateTablet for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:46.268613 32221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.269299 32300 tablet_service.cc:1511] Processing CreateTablet for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:46.270900 32300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.272482 32368 tablet_service.cc:1511] Processing CreateTablet for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:46.272482 32372 tablet_service.cc:1511] Processing CreateTablet for tablet ad5339dac1fd4281a629838a43425f82 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:46.275658 32222 tablet_service.cc:1511] Processing CreateTablet for tablet 009f6f95dd3940908cc89c65101c2404 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:46.277266 32222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009f6f95dd3940908cc89c65101c2404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.274175 32368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.281241 32372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad5339dac1fd4281a629838a43425f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.301589 32367 tablet_service.cc:1511] Processing CreateTablet for tablet 009f6f95dd3940908cc89c65101c2404 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:46.303563 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009f6f95dd3940908cc89c65101c2404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.307418 32296 tablet_service.cc:1511] Processing CreateTablet for tablet 009f6f95dd3940908cc89c65101c2404 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:46.309293 32296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009f6f95dd3940908cc89c65101c2404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.312598 32220 tablet_service.cc:1511] Processing CreateTablet for tablet 1963e8657255457683fb8f3884b836e5 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:46.314415 32220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1963e8657255457683fb8f3884b836e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.331212 32298 tablet_service.cc:1511] Processing CreateTablet for tablet 1963e8657255457683fb8f3884b836e5 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:46.332875 32298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1963e8657255457683fb8f3884b836e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.337577 32369 tablet_service.cc:1511] Processing CreateTablet for tablet 1963e8657255457683fb8f3884b836e5 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:46.339675 32369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1963e8657255457683fb8f3884b836e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.339939 32217 tablet_service.cc:1511] Processing CreateTablet for tablet 1912f4ba565e408dae59d2f999bca9b3 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:46.341586 32217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1912f4ba565e408dae59d2f999bca9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.342612 32294 tablet_service.cc:1511] Processing CreateTablet for tablet 1912f4ba565e408dae59d2f999bca9b3 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:46.353150 32294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1912f4ba565e408dae59d2f999bca9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.362550 32370 tablet_service.cc:1511] Processing CreateTablet for tablet 1912f4ba565e408dae59d2f999bca9b3 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:46.364449 32370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1912f4ba565e408dae59d2f999bca9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.364584 32535 tablet_bootstrap.cc:492] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.378600 32220 tablet_service.cc:1511] Processing CreateTablet for tablet 205414dd32024892b3e7a8785fc044f7 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:46.380292 32220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205414dd32024892b3e7a8785fc044f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.397713 32367 tablet_service.cc:1511] Processing CreateTablet for tablet 205414dd32024892b3e7a8785fc044f7 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:46.399618 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205414dd32024892b3e7a8785fc044f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.400390 32300 tablet_service.cc:1511] Processing CreateTablet for tablet 205414dd32024892b3e7a8785fc044f7 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:46.402773 32535 tablet_bootstrap.cc:654] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.419020 32293 tablet_service.cc:1511] Processing CreateTablet for tablet 03c67f09414e4f6cac46bf806026ba01 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:46.419410 32222 tablet_service.cc:1511] Processing CreateTablet for tablet 03c67f09414e4f6cac46bf806026ba01 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:46.426872 32222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c67f09414e4f6cac46bf806026ba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.429731 32293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c67f09414e4f6cac46bf806026ba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.437719 32300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205414dd32024892b3e7a8785fc044f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.446533 32367 tablet_service.cc:1511] Processing CreateTablet for tablet 03c67f09414e4f6cac46bf806026ba01 (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:46.448287 32367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03c67f09414e4f6cac46bf806026ba01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.454870 32299 tablet_service.cc:1511] Processing CreateTablet for tablet 69c479a9a227487bba2ae9f419198b1a (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:46.456758 32299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c479a9a227487bba2ae9f419198b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.456799 32538 tablet_bootstrap.cc:492] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.461983 32216 tablet_service.cc:1511] Processing CreateTablet for tablet 69c479a9a227487bba2ae9f419198b1a (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:46.464129 32216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c479a9a227487bba2ae9f419198b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.466804 32537 tablet_bootstrap.cc:492] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.472841 32372 tablet_service.cc:1511] Processing CreateTablet for tablet 69c479a9a227487bba2ae9f419198b1a (DEFAULT_TABLE table=soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:46.474502 32372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c479a9a227487bba2ae9f419198b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:46.504817 32538 tablet_bootstrap.cc:654] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.518443 32537 tablet_bootstrap.cc:654] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.520869 32535 tablet_bootstrap.cc:492] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.528923 32535 ts_tablet_manager.cc:1403] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.165s	user 0.011s	sys 0.011s
I20260709 12:28:46.529639 32538 tablet_bootstrap.cc:492] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:46.530498 32538 ts_tablet_manager.cc:1403] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.074s	user 0.003s	sys 0.019s
I20260709 12:28:46.534340 32538 raft_consensus.cc:359] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.535423 32538 raft_consensus.cc:385] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.535815 32538 raft_consensus.cc:740] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.536921 32538 consensus_queue.cc:260] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.532764 32535 raft_consensus.cc:359] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.538539 32535 raft_consensus.cc:385] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.538859 32535 raft_consensus.cc:740] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.539710 32535 consensus_queue.cc:260] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.541678 32535 ts_tablet_manager.cc:1434] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:28:46.543023 32535 tablet_bootstrap.cc:492] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.545295 32537 tablet_bootstrap.cc:492] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:46.546029 32537 ts_tablet_manager.cc:1403] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.080s	user 0.011s	sys 0.006s
I20260709 12:28:46.550061 32537 raft_consensus.cc:359] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.550921 32537 raft_consensus.cc:385] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.551261 32537 raft_consensus.cc:740] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.551350 32535 tablet_bootstrap.cc:654] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.552479 32537 consensus_queue.cc:260] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.554910 32538 ts_tablet_manager.cc:1434] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:28:46.556404 32538 tablet_bootstrap.cc:492] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.566793 32537 ts_tablet_manager.cc:1434] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.001s
I20260709 12:28:46.568732 32537 tablet_bootstrap.cc:492] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.570286 32538 tablet_bootstrap.cc:654] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.580966 32537 tablet_bootstrap.cc:654] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.585254 32535 tablet_bootstrap.cc:492] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.586081 32535 ts_tablet_manager.cc:1403] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.043s	user 0.021s	sys 0.017s
I20260709 12:28:46.589124 32538 tablet_bootstrap.cc:492] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:46.589915 32538 ts_tablet_manager.cc:1403] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.012s
I20260709 12:28:46.589984 32535 raft_consensus.cc:359] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.590931 32535 raft_consensus.cc:385] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.591270 32535 raft_consensus.cc:740] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.592304 32535 consensus_queue.cc:260] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.593837 32538 raft_consensus.cc:359] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.594727 32538 raft_consensus.cc:385] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.595058 32538 raft_consensus.cc:740] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.595431 32535 ts_tablet_manager.cc:1434] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.005s
I20260709 12:28:46.595948 32538 consensus_queue.cc:260] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.596760 32535 tablet_bootstrap.cc:492] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.598951 32538 ts_tablet_manager.cc:1434] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.005s
I20260709 12:28:46.601240 32538 tablet_bootstrap.cc:492] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.605742 32535 tablet_bootstrap.cc:654] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.609490 32538 tablet_bootstrap.cc:654] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.612725 32537 tablet_bootstrap.cc:492] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:46.613582 32537 ts_tablet_manager.cc:1403] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.013s
I20260709 12:28:46.618116 32537 raft_consensus.cc:359] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.618968 32537 raft_consensus.cc:385] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.619290 32537 raft_consensus.cc:740] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.619799 32535 tablet_bootstrap.cc:492] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.620393 32538 tablet_bootstrap.cc:492] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:46.620507 32535 ts_tablet_manager.cc:1403] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.024s	user 0.019s	sys 0.001s
I20260709 12:28:46.620137 32537 consensus_queue.cc:260] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.620998 32538 ts_tablet_manager.cc:1403] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.010s
I20260709 12:28:46.622927 32537 ts_tablet_manager.cc:1434] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:28:46.624095 32537 tablet_bootstrap.cc:492] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.624464 32535 raft_consensus.cc:359] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.624790 32538 raft_consensus.cc:359] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.625344 32535 raft_consensus.cc:385] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.625617 32538 raft_consensus.cc:385] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.625731 32535 raft_consensus.cc:740] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.626106 32538 raft_consensus.cc:740] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.626950 32535 consensus_queue.cc:260] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.629293 32538 consensus_queue.cc:260] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.632735 32537 tablet_bootstrap.cc:654] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.633615 32535 ts_tablet_manager.cc:1434] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.005s
I20260709 12:28:46.634867 32535 tablet_bootstrap.cc:492] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.635937 32538 ts_tablet_manager.cc:1434] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.001s
I20260709 12:28:46.637200 32538 tablet_bootstrap.cc:492] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.643224 32535 tablet_bootstrap.cc:654] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.650067 32538 tablet_bootstrap.cc:654] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.656505 32537 tablet_bootstrap.cc:492] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:46.656512 32535 tablet_bootstrap.cc:492] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.657276 32535 ts_tablet_manager.cc:1403] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.009s
I20260709 12:28:46.657276 32537 ts_tablet_manager.cc:1403] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.033s	user 0.016s	sys 0.004s
I20260709 12:28:46.661130 32535 raft_consensus.cc:359] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.661988 32535 raft_consensus.cc:385] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.662322 32535 raft_consensus.cc:740] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.663170 32536 raft_consensus.cc:493] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.663228 32535 consensus_queue.cc:260] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.663861 32536 raft_consensus.cc:515] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
W20260709 12:28:46.664186 32180 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:46.666218 32535 ts_tablet_manager.cc:1434] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:28:46.666448 32538 tablet_bootstrap.cc:492] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:46.667212 32538 ts_tablet_manager.cc:1403] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.003s
I20260709 12:28:46.667541 32535 tablet_bootstrap.cc:492] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.668226 32537 raft_consensus.cc:359] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.668809 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:46.669183 32537 raft_consensus.cc:385] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.669549 32537 raft_consensus.cc:740] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.670537 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:46.676231 32535 tablet_bootstrap.cc:654] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.670827 32537 consensus_queue.cc:260] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.677366 32232 raft_consensus.cc:2468] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:46.671109 32538 raft_consensus.cc:359] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.669587 32384 raft_consensus.cc:2468] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:46.680763 32538 raft_consensus.cc:385] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.681145 32538 raft_consensus.cc:740] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.673961 32536 leader_election.cc:290] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.683940 32537 ts_tablet_manager.cc:1434] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:28:46.686960 32275 leader_election.cc:304] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:46.687776 32535 tablet_bootstrap.cc:492] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.688429 32535 ts_tablet_manager.cc:1403] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.000s
I20260709 12:28:46.688118 32538 consensus_queue.cc:260] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.688732 32468 raft_consensus.cc:2804] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:46.689199 32468 raft_consensus.cc:493] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.689543 32468 raft_consensus.cc:3060] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.690968 32538 ts_tablet_manager.cc:1434] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:28:46.685601 32537 tablet_bootstrap.cc:492] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.692126 32535 raft_consensus.cc:359] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.692988 32535 raft_consensus.cc:385] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.693298 32535 raft_consensus.cc:740] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.694346 32535 consensus_queue.cc:260] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.697180 32535 ts_tablet_manager.cc:1434] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.008s	user 0.011s	sys 0.000s
I20260709 12:28:46.697613 32538 tablet_bootstrap.cc:492] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.717382 32535 tablet_bootstrap.cc:492] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.719939 32537 tablet_bootstrap.cc:654] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.731324 32538 tablet_bootstrap.cc:654] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.735818 32535 tablet_bootstrap.cc:654] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.749478 32468 raft_consensus.cc:515] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.754585 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:46.755331 32232 raft_consensus.cc:3060] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.759243 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:46.760032 32384 raft_consensus.cc:3060] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.760600 32538 tablet_bootstrap.cc:492] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:46.761446 32538 ts_tablet_manager.cc:1403] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.064s	user 0.022s	sys 0.023s
I20260709 12:28:46.757583 32468 leader_election.cc:290] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.765390 32232 raft_consensus.cc:2468] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:46.766328 32537 tablet_bootstrap.cc:492] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:46.767007 32537 ts_tablet_manager.cc:1403] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.082s	user 0.019s	sys 0.043s
I20260709 12:28:46.767364 32275 leader_election.cc:304] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:46.767016 32538 raft_consensus.cc:359] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.768369 32468 raft_consensus.cc:2804] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:46.768999 32468 raft_consensus.cc:697] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:46.769570 32384 raft_consensus.cc:2468] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:46.770021 32468 consensus_queue.cc:237] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.770884 32537 raft_consensus.cc:359] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.771715 32537 raft_consensus.cc:385] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.772039 32537 raft_consensus.cc:740] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.772696 32535 tablet_bootstrap.cc:492] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.773331 32535 ts_tablet_manager.cc:1403] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.075s	user 0.024s	sys 0.000s
I20260709 12:28:46.772884 32537 consensus_queue.cc:260] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.768136 32538 raft_consensus.cc:385] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.774324 32538 raft_consensus.cc:740] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.775738 32538 consensus_queue.cc:260] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.777576 32535 raft_consensus.cc:359] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.778649 32535 raft_consensus.cc:385] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.778932 32535 raft_consensus.cc:740] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.779775 32535 consensus_queue.cc:260] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.783556 32535 ts_tablet_manager.cc:1434] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.003s
I20260709 12:28:46.784691 32535 tablet_bootstrap.cc:492] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.786427 32537 ts_tablet_manager.cc:1434] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:28:46.788702 32538 ts_tablet_manager.cc:1434] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.001s
I20260709 12:28:46.792519 32535 tablet_bootstrap.cc:654] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.791791 32132 catalog_manager.cc:5697] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:46.795307 32538 tablet_bootstrap.cc:492] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.805917 32537 tablet_bootstrap.cc:492] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.810508 32462 raft_consensus.cc:493] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.811280 32462 raft_consensus.cc:515] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.816061 32535 tablet_bootstrap.cc:492] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.816815 32535 ts_tablet_manager.cc:1403] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.032s	user 0.019s	sys 0.008s
I20260709 12:28:46.816769 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:46.817746 32310 raft_consensus.cc:2468] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:46.818125 32537 tablet_bootstrap.cc:654] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.820842 32535 raft_consensus.cc:359] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.821700 32535 raft_consensus.cc:385] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.822039 32535 raft_consensus.cc:740] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.823201 32462 leader_election.cc:290] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:46.822868 32535 consensus_queue.cc:260] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.826639 32535 ts_tablet_manager.cc:1434] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:28:46.826915 32350 leader_election.cc:304] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:46.827852 32535 tablet_bootstrap.cc:492] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.828303 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
W20260709 12:28:46.830282 32349 leader_election.cc:343] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:46.831532 32462 raft_consensus.cc:2804] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:46.833987 32462 raft_consensus.cc:493] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.834563 32462 raft_consensus.cc:3060] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.836406 32535 tablet_bootstrap.cc:654] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.840472 32538 tablet_bootstrap.cc:654] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.833362 32537 tablet_bootstrap.cc:492] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:46.845722 32537 ts_tablet_manager.cc:1403] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.009s
I20260709 12:28:46.848009 32462 raft_consensus.cc:515] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.849648 32537 raft_consensus.cc:359] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.850642 32537 raft_consensus.cc:385] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.851049 32537 raft_consensus.cc:740] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.852047 32537 consensus_queue.cc:260] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.853152 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
W20260709 12:28:46.854970 32349 leader_election.cc:343] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:46.856345 32537 ts_tablet_manager.cc:1434] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:28:46.857616 32537 tablet_bootstrap.cc:492] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.857487 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:46.858287 32309 raft_consensus.cc:3060] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.864784 32538 tablet_bootstrap.cc:492] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:46.865438 32524 raft_consensus.cc:493] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.870863 32524 raft_consensus.cc:515] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.871646 32551 raft_consensus.cc:493] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.872272 32551 raft_consensus.cc:515] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.873541 32524 leader_election.cc:290] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.877315 32468 raft_consensus.cc:493] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.878386 32468 raft_consensus.cc:515] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.869012 32462 leader_election.cc:290] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:46.880410 32537 tablet_bootstrap.cc:654] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.881901 32538 ts_tablet_manager.cc:1403] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.087s	user 0.016s	sys 0.003s
I20260709 12:28:46.883530 32468 leader_election.cc:290] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.885771 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:46.886955 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:46.890436 32310 raft_consensus.cc:2468] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:46.888422 32551 leader_election.cc:290] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.892740 32383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1963e8657255457683fb8f3884b836e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:46.893584 32383 raft_consensus.cc:2468] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:46.893620 32382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:46.894699 32382 raft_consensus.cc:2468] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:46.896339 32274 leader_election.cc:304] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:46.896258 32198 leader_election.cc:304] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:46.897791 32564 raft_consensus.cc:2804] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:46.898468 32309 raft_consensus.cc:2468] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:46.898329 32564 raft_consensus.cc:493] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.885879 32538 raft_consensus.cc:359] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.899391 32564 raft_consensus.cc:3060] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.890512 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1963e8657255457683fb8f3884b836e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:46.906607 32563 raft_consensus.cc:493] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.900098 32350 leader_election.cc:304] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:46.898255 32468 raft_consensus.cc:2804] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:46.907392 32233 raft_consensus.cc:2468] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:46.907862 32462 raft_consensus.cc:2804] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:46.907385 32563 raft_consensus.cc:515] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.887600 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:46.909844 32462 raft_consensus.cc:697] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:46.910290 32232 raft_consensus.cc:2468] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:46.899559 32538 raft_consensus.cc:385] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.911015 32462 consensus_queue.cc:237] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.911674 32538 raft_consensus.cc:740] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.911801 32275 leader_election.cc:304] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:46.912915 32551 raft_consensus.cc:2804] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:46.907927 32468 raft_consensus.cc:493] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.913548 32551 raft_consensus.cc:493] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.913985 32468 raft_consensus.cc:3060] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.915400 32537 tablet_bootstrap.cc:492] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:46.912879 32563 leader_election.cc:290] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:46.914080 32551 raft_consensus.cc:3060] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.922655 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:46.924173 32232 raft_consensus.cc:2468] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:46.927305 32537 ts_tablet_manager.cc:1403] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.070s	user 0.016s	sys 0.024s
I20260709 12:28:46.888970 32535 tablet_bootstrap.cc:492] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:46.929122 32349 leader_election.cc:304] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:46.930315 32563 raft_consensus.cc:2804] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:46.912837 32538 consensus_queue.cc:260] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.930871 32563 raft_consensus.cc:493] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.931448 32563 raft_consensus.cc:3060] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.926916 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:46.931730 32537 raft_consensus.cc:359] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.932834 32537 raft_consensus.cc:385] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.932953 32309 raft_consensus.cc:2468] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 0.
I20260709 12:28:46.933547 32537 raft_consensus.cc:740] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.926182 32551 raft_consensus.cc:515] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.932010 32384 raft_consensus.cc:2468] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:46.935568 32535 ts_tablet_manager.cc:1403] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.108s	user 0.009s	sys 0.013s
I20260709 12:28:46.937165 32538 ts_tablet_manager.cc:1434] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.055s	user 0.007s	sys 0.002s
I20260709 12:28:46.939160 32538 tablet_bootstrap.cc:492] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:46.947494 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:46.947759 32551 leader_election.cc:290] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.948295 32232 raft_consensus.cc:3060] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.948872 32537 consensus_queue.cc:260] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.951839 32537 ts_tablet_manager.cc:1434] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:28:46.952144 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:46.952916 32384 raft_consensus.cc:3060] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.953301 32537 tablet_bootstrap.cc:492] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:46.952253 32535 raft_consensus.cc:359] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.959766 32468 raft_consensus.cc:515] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.960263 32535 raft_consensus.cc:385] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:46.960704 32535 raft_consensus.cc:740] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:46.961753 32535 consensus_queue.cc:260] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.957423 32129 catalog_manager.cc:5697] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:46.963814 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1963e8657255457683fb8f3884b836e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:46.964587 32233 raft_consensus.cc:3060] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.964942 32535 ts_tablet_manager.cc:1434] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 12:28:46.966295 32535 tablet_bootstrap.cc:492] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: Bootstrap starting.
I20260709 12:28:46.967545 32384 raft_consensus.cc:2468] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:46.969079 32274 leader_election.cc:304] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:46.970134 32551 raft_consensus.cc:2804] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:46.970678 32551 raft_consensus.cc:697] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:46.971657 32551 consensus_queue.cc:237] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.975275 32233 raft_consensus.cc:2468] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:46.985203 32275 leader_election.cc:304] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:46.959203 32564 raft_consensus.cc:515] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:46.977480 32537 tablet_bootstrap.cc:654] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.979343 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1963e8657255457683fb8f3884b836e5" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:46.986284 32536 raft_consensus.cc:2804] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:46.986858 32384 raft_consensus.cc:3060] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.998274 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:46.999184 32309 raft_consensus.cc:3060] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:46.999794 32538 tablet_bootstrap.cc:654] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:47.002682 32128 catalog_manager.cc:5697] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.005290 32536 raft_consensus.cc:697] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:47.006564 32536 consensus_queue.cc:237] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.007120 32232 raft_consensus.cc:2468] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:46.980021 32468 leader_election.cc:290] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:46.983047 32535 tablet_bootstrap.cc:654] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:46.983189 32563 raft_consensus.cc:515] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.012542 32382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.013571 32382 raft_consensus.cc:3060] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.015201 32233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:47.020623 32233 raft_consensus.cc:3060] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.018392 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.022287 32310 raft_consensus.cc:3060] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.015898 32564 leader_election.cc:290] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.028081 32382 raft_consensus.cc:2468] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:47.021194 32563 leader_election.cc:290] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:47.009559 32309 raft_consensus.cc:2468] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:47.039816 32535 tablet_bootstrap.cc:492] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: No bootstrap required, opened a new log
I20260709 12:28:47.040738 32198 leader_election.cc:304] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.035585 32468 raft_consensus.cc:493] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.041424 32535 ts_tablet_manager.cc:1403] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: Time spent bootstrapping tablet: real 0.075s	user 0.013s	sys 0.007s
I20260709 12:28:47.042574 32564 raft_consensus.cc:2804] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:47.043632 32310 raft_consensus.cc:2468] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:47.043969 32564 raft_consensus.cc:697] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.028555 32537 tablet_bootstrap.cc:492] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:47.045099 32350 leader_election.cc:304] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:47.045574 32537 ts_tablet_manager.cc:1403] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.093s	user 0.006s	sys 0.017s
I20260709 12:28:47.045289 32564 consensus_queue.cc:237] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.041873 32468 raft_consensus.cc:515] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.048136 32563 raft_consensus.cc:2804] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:47.049578 32563 raft_consensus.cc:697] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:47.052668 32563 consensus_queue.cc:237] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.044551 32128 catalog_manager.cc:5697] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.071385 32382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1912f4ba565e408dae59d2f999bca9b3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
W20260709 12:28:47.073433 32274 leader_election.cc:343] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:47.061578 32537 raft_consensus.cc:359] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.073628 32131 catalog_manager.cc:5697] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 1 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.071450 32233 raft_consensus.cc:2468] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 1.
I20260709 12:28:47.074314 32537 raft_consensus.cc:385] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.076391 32537 raft_consensus.cc:740] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.063915 32535 raft_consensus.cc:359] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.064726 32538 tablet_bootstrap.cc:492] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:47.077718 32537 consensus_queue.cc:260] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.079315 32538 ts_tablet_manager.cc:1403] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.140s	user 0.020s	sys 0.024s
I20260709 12:28:47.081789 32537 ts_tablet_manager.cc:1434] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.008s
I20260709 12:28:47.083671 32538 raft_consensus.cc:359] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.084540 32537 tablet_bootstrap.cc:492] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:47.085352 32535 raft_consensus.cc:385] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.085790 32535 raft_consensus.cc:740] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.086884 32535 consensus_queue.cc:260] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.068048 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1912f4ba565e408dae59d2f999bca9b3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde" is_pre_election: true
I20260709 12:28:47.068361 32468 leader_election.cc:290] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.067092 32384 raft_consensus.cc:2468] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:47.088378 32232 raft_consensus.cc:2468] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 0.
I20260709 12:28:47.084728 32538 raft_consensus.cc:385] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.089530 32538 raft_consensus.cc:740] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.090051 32275 leader_election.cc:304] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 1daede7f0f194ea7847349205a419c57
I20260709 12:28:47.091658 32468 raft_consensus.cc:2804] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:47.091361 32538 consensus_queue.cc:260] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.092108 32468 raft_consensus.cc:493] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.092550 32468 raft_consensus.cc:3060] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.094424 32538 ts_tablet_manager.cc:1434] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.006s
I20260709 12:28:47.095902 32538 tablet_bootstrap.cc:492] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:47.098094 32535 ts_tablet_manager.cc:1434] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.005s
I20260709 12:28:47.103609 32537 tablet_bootstrap.cc:654] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:47.104821 32564 raft_consensus.cc:493] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.105662 32564 raft_consensus.cc:515] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.108834 32564 leader_election.cc:290] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.110957 32538 tablet_bootstrap.cc:654] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:47.115803 32129 catalog_manager.cc:5697] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:47.117161 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:47.118841 32310 raft_consensus.cc:2393] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 922eda0357a64c13b1fed5dfbeba3cde in current term 1: Already voted for candidate 1daede7f0f194ea7847349205a419c57 in this term.
I20260709 12:28:47.124032 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:47.128470 32198 leader_election.cc:304] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 922eda0357a64c13b1fed5dfbeba3cde; no voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91
I20260709 12:28:47.130144 32564 raft_consensus.cc:3060] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.131254 32538 tablet_bootstrap.cc:492] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:47.131989 32538 ts_tablet_manager.cc:1403] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.009s
I20260709 12:28:47.135730 32538 raft_consensus.cc:359] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.136538 32538 raft_consensus.cc:385] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.136821 32538 raft_consensus.cc:740] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.137733 32538 consensus_queue.cc:260] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.140223 32538 ts_tablet_manager.cc:1434] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260709 12:28:47.141296 32538 tablet_bootstrap.cc:492] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: Bootstrap starting.
I20260709 12:28:47.105297 32468 raft_consensus.cc:515] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.148612 32538 tablet_bootstrap.cc:654] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:47.151391 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1912f4ba565e408dae59d2f999bca9b3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:47.152506 32232 raft_consensus.cc:3060] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.156386 32382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1912f4ba565e408dae59d2f999bca9b3" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.157639 32382 raft_consensus.cc:3060] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.159349 32468 leader_election.cc:290] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.160893 32538 tablet_bootstrap.cc:492] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: No bootstrap required, opened a new log
I20260709 12:28:47.162040 32538 ts_tablet_manager.cc:1403] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.007s
I20260709 12:28:47.168203 32538 raft_consensus.cc:359] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.169231 32538 raft_consensus.cc:385] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.169543 32538 raft_consensus.cc:740] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.170377 32538 consensus_queue.cc:260] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.172801 32538 ts_tablet_manager.cc:1434] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:28:47.173369 32564 raft_consensus.cc:2749] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:47.175261 32232 raft_consensus.cc:2468] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:47.176447 32382 raft_consensus.cc:2468] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 1.
I20260709 12:28:47.176806 32275 leader_election.cc:304] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.177289 32537 tablet_bootstrap.cc:492] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:47.177989 32537 ts_tablet_manager.cc:1403] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.094s	user 0.034s	sys 0.039s
I20260709 12:28:47.178041 32468 raft_consensus.cc:2804] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:47.179563 32468 raft_consensus.cc:697] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:47.180536 32468 consensus_queue.cc:237] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.181483 32537 raft_consensus.cc:359] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.183069 32537 raft_consensus.cc:385] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.183382 32537 raft_consensus.cc:740] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.184293 32537 consensus_queue.cc:260] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.186839 32537 ts_tablet_manager.cc:1434] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 12:28:47.188450 32537 tablet_bootstrap.cc:492] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: Bootstrap starting.
I20260709 12:28:47.195883 32537 tablet_bootstrap.cc:654] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:47.202252 32128 catalog_manager.cc:5697] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.212329 32537 tablet_bootstrap.cc:492] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: No bootstrap required, opened a new log
I20260709 12:28:47.213163 32537 ts_tablet_manager.cc:1403] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.009s
I20260709 12:28:47.213214 32468 consensus_queue.cc:1048] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.216675 32537 raft_consensus.cc:359] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.217494 32537 raft_consensus.cc:385] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:47.217760 32537 raft_consensus.cc:740] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Initialized, Role: FOLLOWER
I20260709 12:28:47.218313 32468 consensus_queue.cc:1048] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.218492 32537 consensus_queue.cc:260] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.221277 32537 ts_tablet_manager.cc:1434] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:28:47.228757 32564 raft_consensus.cc:493] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.229453 32564 raft_consensus.cc:515] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.233475 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:47.234452 32310 raft_consensus.cc:2468] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:47.236130 32200 leader_election.cc:304] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.237486 32524 raft_consensus.cc:2804] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:47.238070 32524 raft_consensus.cc:493] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.238587 32524 raft_consensus.cc:3060] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.242550 32564 leader_election.cc:290] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.257408 32382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:47.258769 32382 raft_consensus.cc:2468] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:47.266662 32524 raft_consensus.cc:515] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.275425 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.276197 32310 raft_consensus.cc:3060] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.279119 32524 leader_election.cc:290] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.281049 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.281757 32384 raft_consensus.cc:3060] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.284466 32310 raft_consensus.cc:2468] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:47.285920 32200 leader_election.cc:304] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.286789 32524 raft_consensus.cc:2804] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:47.287376 32524 raft_consensus.cc:697] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.288293 32524 consensus_queue.cc:237] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.289541 32384 raft_consensus.cc:2468] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:47.298810 32129 catalog_manager.cc:5697] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 1 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.397243 32563 consensus_queue.cc:1048] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:47.404001 32563 consensus_queue.cc:1048] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.410656 32569 consensus_queue.cc:1048] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.428426 32536 consensus_queue.cc:1048] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.440062 32564 raft_consensus.cc:493] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.440730 32564 raft_consensus.cc:515] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.443264 32564 leader_election.cc:290] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.444185 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91" is_pre_election: true
I20260709 12:28:47.445043 32310 raft_consensus.cc:2468] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:47.446065 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57" is_pre_election: true
I20260709 12:28:47.446491 32200 leader_election.cc:304] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.446930 32384 raft_consensus.cc:2468] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 0.
I20260709 12:28:47.448055 32564 raft_consensus.cc:2804] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:47.448480 32564 raft_consensus.cc:493] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.448808 32564 raft_consensus.cc:3060] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.450390 32566 consensus_queue.cc:1048] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.458398 32536 consensus_queue.cc:1048] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.458043 32564 raft_consensus.cc:515] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.461256 32564 leader_election.cc:290] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.462636 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.463310 32310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.463521 32384 raft_consensus.cc:3060] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.464010 32310 raft_consensus.cc:3060] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:47.469847 32564 consensus_queue.cc:1048] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.473261 32384 raft_consensus.cc:2468] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:47.477074 32198 leader_election.cc:304] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.478021 32524 raft_consensus.cc:2804] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:47.484043 32524 raft_consensus.cc:697] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.493180 32551 consensus_queue.cc:1048] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.494597 32524 consensus_queue.cc:237] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.498572 32310 raft_consensus.cc:2468] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 1.
I20260709 12:28:47.519886 32462 consensus_queue.cc:1048] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.560588 32598 consensus_queue.cc:1048] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.563519 32129 catalog_manager.cc:5697] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 0 to 1, leader changed from <none> to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 1 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.631536 32551 consensus_queue.cc:1048] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:47.643776 32536 consensus_queue.cc:1048] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.769047 32309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:48882
I20260709 12:28:47.769680 32309 raft_consensus.cc:606] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:47.779310 32382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:49058
I20260709 12:28:47.779948 32382 raft_consensus.cc:606] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:47.786958 32309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:48882
I20260709 12:28:47.787776 32309 raft_consensus.cc:606] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Received request to transfer leadership to TS 922eda0357a64c13b1fed5dfbeba3cde
I20260709 12:28:47.790036 32569 raft_consensus.cc:993] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:47.790537 32536 raft_consensus.cc:1081] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:47.792198 32232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39048
I20260709 12:28:47.792820 32232 raft_consensus.cc:493] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:47.793252 32232 raft_consensus.cc:3060] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.802893 32232 raft_consensus.cc:515] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.803335 32384 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
mode: GRACEFUL
new_leader_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:49058
I20260709 12:28:47.804082 32384 raft_consensus.cc:606] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Received request to transfer leadership to TS 342106197b2d444cb5be73592926bf91
I20260709 12:28:47.806028 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.806524 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009f6f95dd3940908cc89c65101c2404" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.806866 32309 raft_consensus.cc:3055] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:47.807173 32384 raft_consensus.cc:3060] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.807286 32309 raft_consensus.cc:740] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:47.808312 32309 consensus_queue.cc:260] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.809372 32309 raft_consensus.cc:3060] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.812506 32232 leader_election.cc:290] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.814293 32384 raft_consensus.cc:2468] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:47.815860 32198 leader_election.cc:304] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.816829 32598 raft_consensus.cc:2804] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:47.817373 32598 raft_consensus.cc:697] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.818318 32598 consensus_queue.cc:237] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.820374 32564 consensus_queue.cc:1048] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.821892 32232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
mode: GRACEFUL
new_leader_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:39058
I20260709 12:28:47.822576 32232 raft_consensus.cc:606] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Received request to transfer leadership to TS 342106197b2d444cb5be73592926bf91
I20260709 12:28:47.818864 32309 raft_consensus.cc:2468] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:47.830191 32232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
mode: GRACEFUL
new_leader_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:39058
I20260709 12:28:47.830885 32232 raft_consensus.cc:606] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Received request to transfer leadership to TS 1daede7f0f194ea7847349205a419c57
I20260709 12:28:47.832878 32131 catalog_manager.cc:5697] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.840270 32564 consensus_queue.cc:1048] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:47.849804 32564 raft_consensus.cc:993] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: : Instructing follower 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:47.850432 32564 raft_consensus.cc:1081] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Signalling peer 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:47.852824 32309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01"
dest_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:48858
I20260709 12:28:47.853562 32309 raft_consensus.cc:493] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:47.853958 32309 raft_consensus.cc:3060] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.861235 32309 raft_consensus.cc:515] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.863637 32309 leader_election.cc:290] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.864493 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:47.865298 32232 raft_consensus.cc:3055] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:47.865631 32232 raft_consensus.cc:740] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.865520 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03c67f09414e4f6cac46bf806026ba01" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.866436 32384 raft_consensus.cc:3060] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.866773 32232 consensus_queue.cc:260] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.867950 32232 raft_consensus.cc:3060] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.873148 32384 raft_consensus.cc:2468] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:47.874632 32274 leader_election.cc:304] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:47.875547 32468 raft_consensus.cc:2804] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:47.876065 32468 raft_consensus.cc:697] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:47.876924 32468 consensus_queue.cc:237] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.877720 32232 raft_consensus.cc:2468] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:47.886646 32128 catalog_manager.cc:5697] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 1 to 2, leader changed from 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1) to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.887509 32536 raft_consensus.cc:993] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:47.888195 32468 raft_consensus.cc:1081] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:47.889856 32232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39048
I20260709 12:28:47.890369 32232 raft_consensus.cc:493] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:47.890672 32232 raft_consensus.cc:3060] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.897574 32232 raft_consensus.cc:515] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.899396 32232 leader_election.cc:290] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.900487 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.900352 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09073e2cfcc344fc87cd7b4e8dc3d4e4" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.901253 32384 raft_consensus.cc:3060] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.901335 32309 raft_consensus.cc:3055] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:47.901775 32309 raft_consensus.cc:740] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:47.902675 32309 consensus_queue.cc:260] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.904088 32309 raft_consensus.cc:3060] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.909754 32384 raft_consensus.cc:2468] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:47.911295 32198 leader_election.cc:304] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.912127 32564 raft_consensus.cc:2804] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:47.912122 32309 raft_consensus.cc:2468] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:47.912971 32564 raft_consensus.cc:697] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.914022 32564 consensus_queue.cc:237] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.922482 32128 catalog_manager.cc:5697] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.927347 32563 raft_consensus.cc:993] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: : Instructing follower 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:47.928086 32462 raft_consensus.cc:1081] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Signalling peer 922eda0357a64c13b1fed5dfbeba3cde to start an election
I20260709 12:28:47.930157 32232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
 from {username='slave'} at 127.0.0.1:39056
I20260709 12:28:47.930862 32232 raft_consensus.cc:493] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:47.931268 32232 raft_consensus.cc:3060] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.938016 32232 raft_consensus.cc:515] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.939765 32232 leader_election.cc:290] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Requested vote from peers 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:47.939859 32564 consensus_queue.cc:1048] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:47.941037 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.941035 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e00cc7bf6ea54b1d9dd3722d7351c3c9" candidate_uuid: "922eda0357a64c13b1fed5dfbeba3cde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:47.942034 32309 raft_consensus.cc:3060] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.942121 32384 raft_consensus.cc:3055] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:47.942596 32384 raft_consensus.cc:740] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:47.943756 32384 consensus_queue.cc:260] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.945266 32384 raft_consensus.cc:3060] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.952471 32309 raft_consensus.cc:2468] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:47.954222 32200 leader_election.cc:304] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:47.955034 32384 raft_consensus.cc:2468] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922eda0357a64c13b1fed5dfbeba3cde in term 2.
I20260709 12:28:47.955323 32598 raft_consensus.cc:2804] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:47.956046 32598 raft_consensus.cc:697] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Leader. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.956796 32599 consensus_queue.cc:1048] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:47.957072 32598 consensus_queue.cc:237] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.972380 32598 raft_consensus.cc:993] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: : Instructing follower 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:47.972997 32595 raft_consensus.cc:1081] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Signalling peer 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:47.973353 32128 catalog_manager.cc:5697] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde reported cstate change: term changed from 1 to 2, leader changed from 1daede7f0f194ea7847349205a419c57 (127.16.102.3) to 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1). New cstate: current_term: 2 leader_uuid: "922eda0357a64c13b1fed5dfbeba3cde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:47.975363 32384 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:28:47.976012 32384 raft_consensus.cc:493] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:47.976347 32384 raft_consensus.cc:3060] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.986191 32384 raft_consensus.cc:515] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.988921 32384 leader_election.cc:290] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:47.989297 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:47.989748 32309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69c479a9a227487bba2ae9f419198b1a" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:47.990612 32309 raft_consensus.cc:3060] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.990842 32232 raft_consensus.cc:3055] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:47.991329 32232 raft_consensus.cc:740] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:47.992220 32232 consensus_queue.cc:260] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:47.993408 32232 raft_consensus.cc:3060] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:47.998703 32309 raft_consensus.cc:2468] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 2.
I20260709 12:28:48.000133 32350 leader_election.cc:304] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 342106197b2d444cb5be73592926bf91; no voters: 
I20260709 12:28:48.001026 32462 raft_consensus.cc:2804] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:48.001502 32462 raft_consensus.cc:697] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 2 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:48.002411 32462 consensus_queue.cc:237] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:48.006022 32232 raft_consensus.cc:2468] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 2.
I20260709 12:28:48.012545 32128 catalog_manager.cc:5697] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 1 to 2, leader changed from 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1) to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:48.139956 32563 raft_consensus.cc:993] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: : Instructing follower 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:48.140569 32563 raft_consensus.cc:1081] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Signalling peer 342106197b2d444cb5be73592926bf91 to start an election
I20260709 12:28:48.142634 32309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7"
dest_uuid: "342106197b2d444cb5be73592926bf91"
 from {username='slave'} at 127.0.0.1:48866
I20260709 12:28:48.143249 32309 raft_consensus.cc:493] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:48.143628 32309 raft_consensus.cc:3060] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:48.150908 32309 raft_consensus.cc:515] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:48.153045 32309 leader_election.cc:290] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317)
I20260709 12:28:48.154201 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:48.154377 32384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "205414dd32024892b3e7a8785fc044f7" candidate_uuid: "342106197b2d444cb5be73592926bf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daede7f0f194ea7847349205a419c57"
I20260709 12:28:48.155051 32232 raft_consensus.cc:3060] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:48.155138 32384 raft_consensus.cc:3055] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:48.155596 32384 raft_consensus.cc:740] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:48.156384 32384 consensus_queue.cc:260] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:48.157477 32384 raft_consensus.cc:3060] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:48.161677 32232 raft_consensus.cc:2468] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:48.163244 32275 leader_election.cc:304] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342106197b2d444cb5be73592926bf91, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:48.163635 32384 raft_consensus.cc:2468] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342106197b2d444cb5be73592926bf91 in term 2.
I20260709 12:28:48.164289 32468 raft_consensus.cc:2804] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:48.164780 32468 raft_consensus.cc:697] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Becoming Leader. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:48.165591 32468 consensus_queue.cc:237] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:48.173589 32128 catalog_manager.cc:5697] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 reported cstate change: term changed from 1 to 2, leader changed from 1daede7f0f194ea7847349205a419c57 (127.16.102.3) to 342106197b2d444cb5be73592926bf91 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "342106197b2d444cb5be73592926bf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:48.283593 32309 raft_consensus.cc:1275] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.285323 32599 consensus_queue.cc:1048] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.296132 32384 raft_consensus.cc:1275] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.298043 32599 consensus_queue.cc:1048] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:48.342675 32382 raft_consensus.cc:1275] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.344192 32599 consensus_queue.cc:1048] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.352285 32232 raft_consensus.cc:1275] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.354269 32309 raft_consensus.cc:1275] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.356477 32598 consensus_queue.cc:1048] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.357667 32468 consensus_queue.cc:1048] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:48.372565 32382 raft_consensus.cc:1275] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.374387 32536 consensus_queue.cc:1048] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.459753 32309 raft_consensus.cc:1275] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.461352 32598 consensus_queue.cc:1048] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.474028 32382 raft_consensus.cc:1275] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 922eda0357a64c13b1fed5dfbeba3cde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.475845 32599 consensus_queue.cc:1048] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:48.546448 32309 raft_consensus.cc:1275] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.547938 32563 consensus_queue.cc:1048] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.556680 32232 raft_consensus.cc:1275] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.558513 32609 consensus_queue.cc:1048] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.570081 32232 raft_consensus.cc:1275] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.571813 32468 consensus_queue.cc:1048] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:48.581928 32382 raft_consensus.cc:1275] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Refusing update from remote peer 342106197b2d444cb5be73592926bf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:48.583839 32612 consensus_queue.cc:1048] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:50.835353 32131 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59906:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:28:50.860982 32131 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59906:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:28:50.959789 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:28:51.363734 32659 log.cc:927] Time spent T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:28:51.366919 32669 log.cc:927] Time spent T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: Append to log took a long time: real 0.053s	user 0.002s	sys 0.000s
I20260709 12:28:51.366556 32306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1963e8657255457683fb8f3884b836e5"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:44224
I20260709 12:28:51.368196 32306 raft_consensus.cc:606] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Received request to transfer leadership to TS 1daede7f0f194ea7847349205a419c57
W20260709 12:28:51.366732 32658 log.cc:927] Time spent T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:28:51.409578 32671 tablet.cc:1724] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.442864 32676 tablet.cc:1724] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.462409 32670 tablet.cc:1724] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.465224 32680 tablet.cc:1724] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.466871 32654 raft_consensus.cc:993] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: : Instructing follower 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.467875 32612 raft_consensus.cc:1081] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Signalling peer 1daede7f0f194ea7847349205a419c57 to start an election
W20260709 12:28:51.474948 32672 log.cc:927] Time spent T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:28:51.507113 32384 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1963e8657255457683fb8f3884b836e5"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:49032
I20260709 12:28:51.509749 32384 raft_consensus.cc:493] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:51.510401 32384 raft_consensus.cc:3060] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:51.523811 32336 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:1963e8657255457683fb8f3884b836e5)
I20260709 12:28:51.480747 32675 tablet.cc:1724] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.534072 32688 tablet.cc:1724] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.527699 32682 tablet.cc:1724] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.572149 32675 tablet.cc:1724] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.542244 32685 tablet.cc:1724] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.577101 32234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
mode: GRACEFUL
new_leader_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:48300
I20260709 12:28:51.578145 32234 raft_consensus.cc:606] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Received request to transfer leadership to TS 1daede7f0f194ea7847349205a419c57
I20260709 12:28:51.597214 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
W20260709 12:28:51.537395 32198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:46317, user_credentials={real_user=slave}} blocked reactor thread for 103716us
I20260709 12:28:51.599895 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:51.608105 32688 tablet.cc:1724] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.617367 32599 raft_consensus.cc:993] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: : Instructing follower 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.618113 32599 raft_consensus.cc:1081] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Signalling peer 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.622365 32689 tablet.cc:1724] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.631978 32380 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:28:51.632906 32380 raft_consensus.cc:493] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 12:28:51.602180 32336 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:009f6f95dd3940908cc89c65101c2404)
W20260709 12:28:51.636695 32410 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:1963e8657255457683fb8f3884b836e5)
I20260709 12:28:51.633349 32380 raft_consensus.cc:3060] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:51.652458 32682 tablet.cc:1724] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.662556 32691 tablet.cc:1724] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.666309 32692 tablet.cc:1724] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.637634 32384 raft_consensus.cc:515] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.677635 32131 catalog_manager.cc:5789] TS 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883) does not have the latest schema for tablet 205414dd32024892b3e7a8785fc044f7 (table soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]). Expected version 1 got 0
I20260709 12:28:51.677623 32693 tablet.cc:1724] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.679731 32131 catalog_manager.cc:5789] TS 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883) does not have the latest schema for tablet 03c67f09414e4f6cac46bf806026ba01 (table soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]). Expected version 1 got 0
I20260709 12:28:51.686841 32676 tablet.cc:1724] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.700531 32675 tablet.cc:1724] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.704085 32384 leader_election.cc:290] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:51.705473 32306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1963e8657255457683fb8f3884b836e5" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:51.706583 32306 raft_consensus.cc:3055] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:51.706974 32306 raft_consensus.cc:740] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:51.708114 32306 consensus_queue.cc:260] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.705603 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1963e8657255457683fb8f3884b836e5" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:51.692727 32671 tablet.cc:1724] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.713073 32232 raft_consensus.cc:3060] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:51.723526 32306 raft_consensus.cc:3060] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:51.731216 32380 raft_consensus.cc:515] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.732954 32234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c"
dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
mode: GRACEFUL
new_leader_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:48300
I20260709 12:28:51.733930 32234 raft_consensus.cc:606] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Received request to transfer leadership to TS 1daede7f0f194ea7847349205a419c57
I20260709 12:28:51.739662 32379 raft_consensus.cc:1240] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Rejecting Update request from peer 922eda0357a64c13b1fed5dfbeba3cde for earlier term 1. Current term is 2. Ops: []
I20260709 12:28:51.741779 32230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:51.742784 32230 raft_consensus.cc:3055] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:51.743155 32230 raft_consensus.cc:740] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:51.744252 32230 consensus_queue.cc:260] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.746829 32680 tablet.cc:1724] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.745697 32230 raft_consensus.cc:3060] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:51.751752 32232 raft_consensus.cc:2468] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 2.
I20260709 12:28:51.754015 32349 leader_election.cc:304] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:51.755311 32661 raft_consensus.cc:2804] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:51.755867 32661 raft_consensus.cc:697] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 2 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:51.756916 32661 consensus_queue.cc:237] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.767460 32305 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1"
dest_uuid: "342106197b2d444cb5be73592926bf91"
mode: GRACEFUL
new_leader_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:44224
I20260709 12:28:51.768487 32305 raft_consensus.cc:606] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Received request to transfer leadership to TS 1daede7f0f194ea7847349205a419c57
I20260709 12:28:51.770517 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:28:51.771487 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:51.773936 32306 raft_consensus.cc:2468] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 2.
I20260709 12:28:51.773273 32670 tablet.cc:1724] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.774739 32230 raft_consensus.cc:2468] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 2.
I20260709 12:28:51.776484 32349 leader_election.cc:304] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:51.777648 32661 raft_consensus.cc:2804] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:51.778245 32661 raft_consensus.cc:697] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 2 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:51.779374 32661 consensus_queue.cc:237] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.782058 32380 leader_election.cc:290] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 2 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:51.783751 32305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ad5339dac1fd4281a629838a43425f82" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:51.784655 32305 raft_consensus.cc:3060] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:51.786445 32689 tablet.cc:1724] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.788689 32688 tablet.cc:1724] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.780789 32128 catalog_manager.cc:5697] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 1 to 2, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:51.788494 32685 tablet.cc:1724] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.810016 32664 raft_consensus.cc:993] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: : Instructing follower 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.810812 32664 raft_consensus.cc:1081] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Signalling peer 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.802232 32670 tablet.cc:1724] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.793301 32129 catalog_manager.cc:5789] TS 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775) does not have the latest schema for tablet e00cc7bf6ea54b1d9dd3722d7351c3c9 (table soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]). Expected version 1 got 0
I20260709 12:28:51.818233 32380 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:28:51.818977 32380 raft_consensus.cc:493] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:51.819427 32380 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:51.831574 32683 raft_consensus.cc:993] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: : Instructing follower 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.832310 32683 raft_consensus.cc:1081] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Signalling peer 1daede7f0f194ea7847349205a419c57 to start an election
I20260709 12:28:51.834815 32376 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1"
dest_uuid: "1daede7f0f194ea7847349205a419c57"
 from {username='slave'} at 127.0.0.1:49032
I20260709 12:28:51.835655 32376 raft_consensus.cc:493] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:51.833062 32693 tablet.cc:1724] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.836170 32376 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:51.865934 32305 raft_consensus.cc:2468] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 2.
I20260709 12:28:51.864969 32691 tablet.cc:1724] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:51.892262 32376 raft_consensus.cc:515] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.894949 32376 leader_election.cc:290] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 3 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:51.895957 32230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:51.896725 32230 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:51.897197 32380 raft_consensus.cc:515] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.899976 32380 leader_election.cc:290] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 3 election: Requested vote from peers 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775), 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883)
I20260709 12:28:51.900456 32232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "922eda0357a64c13b1fed5dfbeba3cde"
I20260709 12:28:51.901211 32232 raft_consensus.cc:3055] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:51.901526 32232 raft_consensus.cc:740] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 922eda0357a64c13b1fed5dfbeba3cde, State: Running, Role: LEADER
I20260709 12:28:51.901599 32306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5317da388b6d4e5398f870ae9c5ca79c" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:51.902405 32306 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:51.902570 32232 consensus_queue.cc:260] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.904240 32232 raft_consensus.cc:3060] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:51.910281 32305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c863ac57cd4547758e355ec99b4033c1" candidate_uuid: "1daede7f0f194ea7847349205a419c57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "342106197b2d444cb5be73592926bf91"
I20260709 12:28:51.911111 32305 raft_consensus.cc:3055] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:51.911430 32305 raft_consensus.cc:740] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 342106197b2d444cb5be73592926bf91, State: Running, Role: LEADER
I20260709 12:28:51.912356 32305 consensus_queue.cc:260] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.913017 32232 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 3.
I20260709 12:28:51.914565 32349 leader_election.cc:304] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:51.915632 32661 raft_consensus.cc:2804] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:51.916154 32661 raft_consensus.cc:697] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 3 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:51.916390 32230 raft_consensus.cc:2468] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 3.
I20260709 12:28:51.918054 32305 raft_consensus.cc:3060] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:51.917263 32661 consensus_queue.cc:237] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.922648 32349 leader_election.cc:304] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daede7f0f194ea7847349205a419c57, 922eda0357a64c13b1fed5dfbeba3cde; no voters: 
I20260709 12:28:51.923847 32665 raft_consensus.cc:2804] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:51.927443 32665 raft_consensus.cc:697] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 3 LEADER]: Becoming Leader. State: Replica: 1daede7f0f194ea7847349205a419c57, State: Running, Role: LEADER
I20260709 12:28:51.928581 32665 consensus_queue.cc:237] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } }
I20260709 12:28:51.937860 32306 raft_consensus.cc:2468] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 3.
I20260709 12:28:51.947947 32305 raft_consensus.cc:2468] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daede7f0f194ea7847349205a419c57 in term 3.
I20260709 12:28:51.954756 32128 catalog_manager.cc:5697] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 1 to 2, leader changed from 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1) to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 2 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:51.956781 32128 catalog_manager.cc:5789] TS 1daede7f0f194ea7847349205a419c57 (127.16.102.3:46317) does not have the latest schema for tablet ad5339dac1fd4281a629838a43425f82 (table soft_deleted_table [id=be4c3a62106549b09baebc2737fee606]). Expected version 1 got 0
I20260709 12:28:51.971307 32131 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001: soft_deleted_table [id=be4c3a62106549b09baebc2737fee606] alter complete (version 1)
I20260709 12:28:51.986747 32128 catalog_manager.cc:5697] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 2 to 3, leader changed from 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1) to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 3 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:51.989099 32128 catalog_manager.cc:5697] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 reported cstate change: term changed from 2 to 3, leader changed from 342106197b2d444cb5be73592926bf91 (127.16.102.2) to 1daede7f0f194ea7847349205a419c57 (127.16.102.3). New cstate: current_term: 3 leader_uuid: "1daede7f0f194ea7847349205a419c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daede7f0f194ea7847349205a419c57" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:52.313484 32305 raft_consensus.cc:1275] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:52.315232 32661 consensus_queue.cc:1048] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:52.332144 32306 raft_consensus.cc:1275] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:52.335706 32661 consensus_queue.cc:1048] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:52.338178 32232 raft_consensus.cc:1275] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:52.340600 32698 consensus_queue.cc:1048] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:52.360683 32691 tablet.cc:1724] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:52.366827 32230 raft_consensus.cc:1275] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:52.369027 32661 consensus_queue.cc:1048] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:52.401491 32691 tablet.cc:1719] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: Skipping requested alter to schema version 1, tablet already version 1
I20260709 12:28:52.409646 32706 tablet.cc:1719] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: Skipping requested alter to schema version 1, tablet already version 1
I20260709 12:28:52.410491 32705 tablet.cc:1719] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: Skipping requested alter to schema version 1, tablet already version 1
I20260709 12:28:52.463346 32305 raft_consensus.cc:1275] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:52.465217 32703 consensus_queue.cc:1048] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:52.476706 32232 raft_consensus.cc:1275] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:52.483637 32661 consensus_queue.cc:1048] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:52.485407 32305 raft_consensus.cc:1275] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:52.486025 32232 raft_consensus.cc:1275] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Refusing update from remote peer 1daede7f0f194ea7847349205a419c57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:52.487602 32661 consensus_queue.cc:1048] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342106197b2d444cb5be73592926bf91" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:52.488456 32665 consensus_queue.cc:1048] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922eda0357a64c13b1fed5dfbeba3cde" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:52.772950 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:52.779932 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:52.780689 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:52.786633 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:52.787387 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:53.788915 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:53.794968 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:53.795562 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:53.800593 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:53.801081 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:54.802544 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:54.809674 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:54.810321 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:54.817562 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:54.818212 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:55.819445 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:55.825700 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:55.826388 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:55.832082 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:55.832736 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:56.834168 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:56.840112 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:56.840718 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:56.845980 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:56.846701 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:57.847923 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:57.854190 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:57.854851 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:57.859946 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:57.860730 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:58.862246 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:58.870055 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:58.870688 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:58.876034 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:58.876626 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:59.878015 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:59.884536 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:59.885247 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:59.892190 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:59.892753 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:00.893971 16792 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:29:00.899991 16792 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:29:00.900580 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:00.906213 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:00.906788 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:01.983426 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:29:02.036051 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:02.037091 16792 tablet_replica.cc:333] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.038550 16792 raft_consensus.cc:2243] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.039546 16792 raft_consensus.cc:2272] T 69c479a9a227487bba2ae9f419198b1a P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.043375 16792 tablet_replica.cc:333] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.044310 16792 raft_consensus.cc:2243] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.045320 16792 raft_consensus.cc:2272] T 1912f4ba565e408dae59d2f999bca9b3 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.050879 16792 tablet_replica.cc:333] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.051872 16792 raft_consensus.cc:2243] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.053020 16792 raft_consensus.cc:2272] T 205414dd32024892b3e7a8785fc044f7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.057199 16792 tablet_replica.cc:333] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.058338 16792 raft_consensus.cc:2243] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.058964 16792 raft_consensus.cc:2272] T 1963e8657255457683fb8f3884b836e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.062323 16792 tablet_replica.cc:333] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.063134 16792 raft_consensus.cc:2243] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.063992 16792 raft_consensus.cc:2272] T ad5339dac1fd4281a629838a43425f82 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.067415 16792 tablet_replica.cc:333] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.068125 16792 raft_consensus.cc:2243] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.069365 16792 raft_consensus.cc:2272] T 8d9107ba28254d7f9dba13f18e4baae7 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.072492 16792 tablet_replica.cc:333] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.073457 16792 raft_consensus.cc:2243] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.074177 16792 raft_consensus.cc:2272] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.077979 16792 tablet_replica.cc:333] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.078866 16792 raft_consensus.cc:2243] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.079857 16792 raft_consensus.cc:2272] T b4a724d0b531408a91fb85a91d558129 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.083689 16792 tablet_replica.cc:333] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.084725 16792 raft_consensus.cc:2243] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.085495 16792 raft_consensus.cc:2272] T 5317da388b6d4e5398f870ae9c5ca79c P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.089323 16792 tablet_replica.cc:333] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.090283 16792 raft_consensus.cc:2243] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.091755 16792 raft_consensus.cc:2272] T 4d378c3205784a9d82c5d29c71db02c3 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.095412 16792 tablet_replica.cc:333] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.096159 16792 raft_consensus.cc:2243] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.096892 16792 raft_consensus.cc:2272] T c863ac57cd4547758e355ec99b4033c1 P 922eda0357a64c13b1fed5dfbeba3cde [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.101017 16792 tablet_replica.cc:333] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.102036 16792 raft_consensus.cc:2243] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.102897 16792 raft_consensus.cc:2272] T 2b1a719b6a2648bb81fa22310ace5a34 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.107200 16792 tablet_replica.cc:333] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.107985 16792 raft_consensus.cc:2243] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.109228 16792 raft_consensus.cc:2272] T ce2d4f84a8154744ab96875f21abb7e5 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.112555 16792 tablet_replica.cc:333] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.113444 16792 raft_consensus.cc:2243] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.114771 16792 raft_consensus.cc:2272] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.118026 16792 tablet_replica.cc:333] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.118685 16792 raft_consensus.cc:2243] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.119876 16792 raft_consensus.cc:2272] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.123381 16792 tablet_replica.cc:333] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.124063 16792 raft_consensus.cc:2243] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.124784 16792 raft_consensus.cc:2272] T 03c67f09414e4f6cac46bf806026ba01 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.127991 16792 tablet_replica.cc:333] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.128650 16792 raft_consensus.cc:2243] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.129792 16792 raft_consensus.cc:2272] T 009f6f95dd3940908cc89c65101c2404 P 922eda0357a64c13b1fed5dfbeba3cde [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.133119 16792 tablet_replica.cc:333] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde: stopping tablet replica
I20260709 12:29:02.133764 16792 raft_consensus.cc:2243] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.134476 16792 raft_consensus.cc:2272] T 47cff3af1b654c3faaa22d43155cab22 P 922eda0357a64c13b1fed5dfbeba3cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.199262 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
W20260709 12:29:02.217914 32349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260709 12:29:02.226657 32349 consensus_peers.cc:597] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.232043 32275 consensus_peers.cc:597] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.277401 32349 consensus_peers.cc:597] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.278654 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
W20260709 12:29:02.286384 32350 consensus_peers.cc:597] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 -> Peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Couldn't send request to peer 342106197b2d444cb5be73592926bf91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.287832 32350 consensus_peers.cc:597] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 -> Peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Couldn't send request to peer 342106197b2d444cb5be73592926bf91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.290736 32275 consensus_peers.cc:597] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.296347 32350 consensus_peers.cc:597] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 -> Peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Couldn't send request to peer 342106197b2d444cb5be73592926bf91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.323959 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:02.325050 16792 tablet_replica.cc:333] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.326049 16792 raft_consensus.cc:2243] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.326787 16792 raft_consensus.cc:2272] T 69c479a9a227487bba2ae9f419198b1a P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.329882 16792 tablet_replica.cc:333] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.330710 16792 raft_consensus.cc:2243] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.331485 16792 raft_consensus.cc:2272] T 1963e8657255457683fb8f3884b836e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.334525 16792 tablet_replica.cc:333] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.335314 16792 raft_consensus.cc:2243] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.335954 16792 raft_consensus.cc:2272] T ad5339dac1fd4281a629838a43425f82 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.338991 16792 tablet_replica.cc:333] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.339706 16792 raft_consensus.cc:2243] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.340796 16792 raft_consensus.cc:2272] T 205414dd32024892b3e7a8785fc044f7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.343755 16792 tablet_replica.cc:333] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.344472 16792 raft_consensus.cc:2243] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.345804 16792 raft_consensus.cc:2272] T 1912f4ba565e408dae59d2f999bca9b3 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.350243 16792 tablet_replica.cc:333] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.351261 16792 raft_consensus.cc:2243] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.351907 16792 raft_consensus.cc:2272] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.355383 16792 tablet_replica.cc:333] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.356388 16792 raft_consensus.cc:2243] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.357299 16792 raft_consensus.cc:2272] T 8d9107ba28254d7f9dba13f18e4baae7 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.360666 16792 tablet_replica.cc:333] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.361397 16792 raft_consensus.cc:2243] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.362552 16792 raft_consensus.cc:2272] T b4a724d0b531408a91fb85a91d558129 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.365545 16792 tablet_replica.cc:333] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.366299 16792 raft_consensus.cc:2243] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.366940 16792 raft_consensus.cc:2272] T 5317da388b6d4e5398f870ae9c5ca79c P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.370085 16792 tablet_replica.cc:333] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.370743 16792 raft_consensus.cc:2243] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.371294 16792 raft_consensus.cc:2272] T 4d378c3205784a9d82c5d29c71db02c3 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.374073 16792 tablet_replica.cc:333] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.374673 16792 raft_consensus.cc:2243] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.375180 16792 raft_consensus.cc:2272] T c863ac57cd4547758e355ec99b4033c1 P 342106197b2d444cb5be73592926bf91 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.377907 16792 tablet_replica.cc:333] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.378468 16792 raft_consensus.cc:2243] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.379674 16792 raft_consensus.cc:2272] T 2b1a719b6a2648bb81fa22310ace5a34 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.383708 16792 tablet_replica.cc:333] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.384929 16792 raft_consensus.cc:2243] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.386030 16792 raft_consensus.cc:2272] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:02.386550 32349 consensus_peers.cc:597] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.387259 32349 consensus_peers.cc:597] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.390170 16792 tablet_replica.cc:333] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
W20260709 12:29:02.390447 32349 consensus_peers.cc:597] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.391315 16792 raft_consensus.cc:2243] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.392118 16792 raft_consensus.cc:2272] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.396241 16792 tablet_replica.cc:333] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
W20260709 12:29:02.396751 32275 consensus_peers.cc:597] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.397028 16792 raft_consensus.cc:2243] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.397780 16792 raft_consensus.cc:2272] T ce2d4f84a8154744ab96875f21abb7e5 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.400768 16792 tablet_replica.cc:333] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.401448 16792 raft_consensus.cc:2243] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.402709 16792 raft_consensus.cc:2272] T 03c67f09414e4f6cac46bf806026ba01 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.405545 16792 tablet_replica.cc:333] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.406214 16792 raft_consensus.cc:2243] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.406960 16792 raft_consensus.cc:2272] T 009f6f95dd3940908cc89c65101c2404 P 342106197b2d444cb5be73592926bf91 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.409742 16792 tablet_replica.cc:333] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91: stopping tablet replica
I20260709 12:29:02.410476 16792 raft_consensus.cc:2243] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.411549 16792 raft_consensus.cc:2272] T 47cff3af1b654c3faaa22d43155cab22 P 342106197b2d444cb5be73592926bf91 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:02.424158 32350 consensus_peers.cc:597] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 -> Peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Couldn't send request to peer 342106197b2d444cb5be73592926bf91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:02.443704 32350 consensus_peers.cc:597] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 -> Peer 342106197b2d444cb5be73592926bf91 (127.16.102.2:38883): Couldn't send request to peer 342106197b2d444cb5be73592926bf91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.470429 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
W20260709 12:29:02.473196 32349 consensus_peers.cc:597] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 -> Peer 922eda0357a64c13b1fed5dfbeba3cde (127.16.102.1:35775): Couldn't send request to peer 922eda0357a64c13b1fed5dfbeba3cde. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:02.526360 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
I20260709 12:29:02.571655 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:02.572528 16792 tablet_replica.cc:333] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.573433 16792 raft_consensus.cc:2243] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.574671 16792 raft_consensus.cc:2272] T 69c479a9a227487bba2ae9f419198b1a P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.577625 16792 tablet_replica.cc:333] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.578373 16792 raft_consensus.cc:2243] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.579505 16792 raft_consensus.cc:2272] T ad5339dac1fd4281a629838a43425f82 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.582495 16792 tablet_replica.cc:333] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.583334 16792 raft_consensus.cc:2243] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.584036 16792 raft_consensus.cc:2272] T 1912f4ba565e408dae59d2f999bca9b3 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.587008 16792 tablet_replica.cc:333] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.587702 16792 raft_consensus.cc:2243] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.588343 16792 raft_consensus.cc:2272] T 205414dd32024892b3e7a8785fc044f7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.591250 16792 tablet_replica.cc:333] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.591974 16792 raft_consensus.cc:2243] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.592973 16792 raft_consensus.cc:2272] T 1963e8657255457683fb8f3884b836e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.595829 16792 tablet_replica.cc:333] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.596536 16792 raft_consensus.cc:2243] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.597167 16792 raft_consensus.cc:2272] T 03c67f09414e4f6cac46bf806026ba01 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.599906 16792 tablet_replica.cc:333] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.600602 16792 raft_consensus.cc:2243] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.601191 16792 raft_consensus.cc:2272] T 47cff3af1b654c3faaa22d43155cab22 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.604087 16792 tablet_replica.cc:333] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.604821 16792 raft_consensus.cc:2243] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.605496 16792 raft_consensus.cc:2272] T 009f6f95dd3940908cc89c65101c2404 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.608269 16792 tablet_replica.cc:333] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.608891 16792 raft_consensus.cc:2243] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.609427 16792 raft_consensus.cc:2272] T 8d9107ba28254d7f9dba13f18e4baae7 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.612104 16792 tablet_replica.cc:333] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.612697 16792 raft_consensus.cc:2243] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.613281 16792 raft_consensus.cc:2272] T b4a724d0b531408a91fb85a91d558129 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.616113 16792 tablet_replica.cc:333] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.616727 16792 raft_consensus.cc:2243] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.617791 16792 raft_consensus.cc:2272] T 6dcb57f0ba7f4f378d3a400a57706cf0 P 1daede7f0f194ea7847349205a419c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.620916 16792 tablet_replica.cc:333] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.621680 16792 raft_consensus.cc:2243] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.622478 16792 raft_consensus.cc:2272] T 2b1a719b6a2648bb81fa22310ace5a34 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.625857 16792 tablet_replica.cc:333] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.626554 16792 raft_consensus.cc:2243] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.627063 16792 raft_consensus.cc:2272] T 4d378c3205784a9d82c5d29c71db02c3 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.629616 16792 tablet_replica.cc:333] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.630280 16792 raft_consensus.cc:2243] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.631258 16792 raft_consensus.cc:2272] T c863ac57cd4547758e355ec99b4033c1 P 1daede7f0f194ea7847349205a419c57 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.633769 16792 tablet_replica.cc:333] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.634392 16792 raft_consensus.cc:2243] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.634887 16792 raft_consensus.cc:2272] T 09073e2cfcc344fc87cd7b4e8dc3d4e4 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.637355 16792 tablet_replica.cc:333] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.638002 16792 raft_consensus.cc:2243] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.638509 16792 raft_consensus.cc:2272] T e00cc7bf6ea54b1d9dd3722d7351c3c9 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.640954 16792 tablet_replica.cc:333] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.641490 16792 raft_consensus.cc:2243] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:02.642014 16792 raft_consensus.cc:2272] T ce2d4f84a8154744ab96875f21abb7e5 P 1daede7f0f194ea7847349205a419c57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.644536 16792 tablet_replica.cc:333] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57: stopping tablet replica
I20260709 12:29:02.645040 16792 raft_consensus.cc:2243] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.646068 16792 raft_consensus.cc:2272] T 5317da388b6d4e5398f870ae9c5ca79c P 1daede7f0f194ea7847349205a419c57 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.696440 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:29:02.747944 16792 master.cc:562] Master@127.16.102.62:36697 shutting down...
I20260709 12:29:02.782274 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:02.783170 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:02.783623 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7e5ec5953a9c4fe5ae361d351b70e001: stopping tablet replica
I20260709 12:29:02.808293 16792 master.cc:584] Master@127.16.102.62:36697 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (22423 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:29:02.878662 16792 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.102.62:35193
I20260709 12:29:02.880120 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:29:02.886798 32740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:02.886785 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:02.889614 32742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:29:02.890578 16792 server_base.cc:1061] running on GCE node
I20260709 12:29:02.891793 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:29:02.891993 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:29:02.892113 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600142892101 us; error 0 us; skew 500 ppm
I20260709 12:29:02.892629 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:29:02.895435 16792 webserver.cc:533] Webserver started at http://127.16.102.62:38099/ using document root <none> and password file <none>
I20260709 12:29:02.895953 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:29:02.896119 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:29:02.896351 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:29:02.897670 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/master-0-root/instance:
uuid: "32730eefe9ff4ef1ab309329feffbe4b"
format_stamp: "Formatted at 2026-07-09 12:29:02 on dist-test-slave-7xt3"
I20260709 12:29:02.904017 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:29:02.908452 32747 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:02.909422 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:29:02.909735 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/master-0-root
uuid: "32730eefe9ff4ef1ab309329feffbe4b"
format_stamp: "Formatted at 2026-07-09 12:29:02 on dist-test-slave-7xt3"
I20260709 12:29:02.910144 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:29:02.937393 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:29:02.939023 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:29:02.987325 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.62:35193
I20260709 12:29:02.987463   330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.62:35193 every 8 connection(s)
I20260709 12:29:02.992259   331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.008315   331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b: Bootstrap starting.
I20260709 12:29:03.014639   331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:03.020174   331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b: No bootstrap required, opened a new log
I20260709 12:29:03.022991   331 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32730eefe9ff4ef1ab309329feffbe4b" member_type: VOTER }
I20260709 12:29:03.023557   331 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:03.023852   331 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32730eefe9ff4ef1ab309329feffbe4b, State: Initialized, Role: FOLLOWER
I20260709 12:29:03.024641   331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32730eefe9ff4ef1ab309329feffbe4b" member_type: VOTER }
I20260709 12:29:03.025161   331 raft_consensus.cc:399] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:29:03.025377   331 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:29:03.025606   331 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:03.032249   331 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32730eefe9ff4ef1ab309329feffbe4b" member_type: VOTER }
I20260709 12:29:03.032881   331 leader_election.cc:304] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32730eefe9ff4ef1ab309329feffbe4b; no voters: 
I20260709 12:29:03.034430   331 leader_election.cc:290] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:29:03.035106   334 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:03.036741   334 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 1 LEADER]: Becoming Leader. State: Replica: 32730eefe9ff4ef1ab309329feffbe4b, State: Running, Role: LEADER
I20260709 12:29:03.037590   334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32730eefe9ff4ef1ab309329feffbe4b" member_type: VOTER }
I20260709 12:29:03.038277   331 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:29:03.044003   336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32730eefe9ff4ef1ab309329feffbe4b. Latest consensus state: current_term: 1 leader_uuid: "32730eefe9ff4ef1ab309329feffbe4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32730eefe9ff4ef1ab309329feffbe4b" member_type: VOTER } }
I20260709 12:29:03.045512   336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [sys.catalog]: This master's current role is: LEADER
I20260709 12:29:03.044121   335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32730eefe9ff4ef1ab309329feffbe4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32730eefe9ff4ef1ab309329feffbe4b" member_type: VOTER } }
I20260709 12:29:03.046239   335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [sys.catalog]: This master's current role is: LEADER
I20260709 12:29:03.048856   342 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:29:03.054703   342 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:29:03.058220 16792 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:29:03.066382   342 catalog_manager.cc:1383] Generated new cluster ID: 9f88e08ee9de4868ae5e86c1b2a6c75a
I20260709 12:29:03.066797   342 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:29:03.094908   342 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:29:03.096647   342 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:29:03.110370   342 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b: Generated new TSK 0
I20260709 12:29:03.111227   342 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:29:03.125914 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:29:03.133565   352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:03.134964   353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:29:03.137842 16792 server_base.cc:1061] running on GCE node
W20260709 12:29:03.137873   355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:29:03.139469 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:29:03.139694 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:29:03.139830 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600143139818 us; error 0 us; skew 500 ppm
I20260709 12:29:03.140503 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:29:03.143318 16792 webserver.cc:533] Webserver started at http://127.16.102.1:38289/ using document root <none> and password file <none>
I20260709 12:29:03.143811 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:29:03.143972 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:29:03.144194 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:29:03.145468 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-0-root/instance:
uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
format_stamp: "Formatted at 2026-07-09 12:29:03 on dist-test-slave-7xt3"
I20260709 12:29:03.150816 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:29:03.155066   360 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.155982 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:29:03.156333 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-0-root
uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
format_stamp: "Formatted at 2026-07-09 12:29:03 on dist-test-slave-7xt3"
I20260709 12:29:03.156659 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:29:03.182742 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:29:03.184286 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:29:03.186075 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:29:03.188863 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:29:03.189137 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.189428 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:29:03.189636 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.244946 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.1:36833
I20260709 12:29:03.245075   422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.1:36833 every 8 connection(s)
I20260709 12:29:03.251288 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:29:03.260535   427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:03.262239   428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:03.266228   430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:29:03.267526 16792 server_base.cc:1061] running on GCE node
I20260709 12:29:03.268839 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:29:03.269124 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:29:03.269130   423 heartbeater.cc:344] Connected to a master server at 127.16.102.62:35193
I20260709 12:29:03.269389 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600143269368 us; error 0 us; skew 500 ppm
I20260709 12:29:03.269919   423 heartbeater.cc:461] Registering TS with master...
I20260709 12:29:03.270368 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:29:03.270972   423 heartbeater.cc:507] Master 127.16.102.62:35193 requested a full tablet report, sending...
I20260709 12:29:03.273577 16792 webserver.cc:533] Webserver started at http://127.16.102.2:43163/ using document root <none> and password file <none>
I20260709 12:29:03.274343 32764 ts_manager.cc:194] Registered new tserver with Master: 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833)
I20260709 12:29:03.274564 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:29:03.275020 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:29:03.275429 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:29:03.277335 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-1-root/instance:
uuid: "4c73adb282914cd0a9ab735eda3a2178"
format_stamp: "Formatted at 2026-07-09 12:29:03 on dist-test-slave-7xt3"
I20260709 12:29:03.277534 32764 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53442
I20260709 12:29:03.283474 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:29:03.288195   435 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.289230 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:29:03.289625 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-1-root
uuid: "4c73adb282914cd0a9ab735eda3a2178"
format_stamp: "Formatted at 2026-07-09 12:29:03 on dist-test-slave-7xt3"
I20260709 12:29:03.290041 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:29:03.301155 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:29:03.302767 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:29:03.304543 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:29:03.307219 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:29:03.307485 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.307754 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:29:03.307947 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.363262 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.2:44003
I20260709 12:29:03.363413   497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.2:44003 every 8 connection(s)
I20260709 12:29:03.369313 16792 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:29:03.378902   501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:03.381225   502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:29:03.383361   504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:29:03.384516 16792 server_base.cc:1061] running on GCE node
I20260709 12:29:03.385308   498 heartbeater.cc:344] Connected to a master server at 127.16.102.62:35193
I20260709 12:29:03.385807   498 heartbeater.cc:461] Registering TS with master...
I20260709 12:29:03.386119 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:29:03.386476 16792 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:29:03.386696 16792 hybrid_clock.cc:648] HybridClock initialized: now 1783600143386677 us; error 0 us; skew 500 ppm
I20260709 12:29:03.386905   498 heartbeater.cc:507] Master 127.16.102.62:35193 requested a full tablet report, sending...
I20260709 12:29:03.387495 16792 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:29:03.389726 32764 ts_manager.cc:194] Registered new tserver with Master: 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:03.391008 16792 webserver.cc:533] Webserver started at http://127.16.102.3:43263/ using document root <none> and password file <none>
I20260709 12:29:03.391815 32764 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53444
I20260709 12:29:03.391870 16792 fs_manager.cc:362] Metadata directory not provided
I20260709 12:29:03.392441 16792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:29:03.392807 16792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:29:03.394567 16792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-2-root/instance:
uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
format_stamp: "Formatted at 2026-07-09 12:29:03 on dist-test-slave-7xt3"
I20260709 12:29:03.400681 16792 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20260709 12:29:03.405113   509 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.406152 16792 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:29:03.406554 16792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-2-root
uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
format_stamp: "Formatted at 2026-07-09 12:29:03 on dist-test-slave-7xt3"
I20260709 12:29:03.406916 16792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskDkZd3Q/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599579151043-16792-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:29:03.419509 16792 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:29:03.421195 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:29:03.423309 16792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:29:03.426374 16792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:29:03.426733 16792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.426968 16792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:29:03.427124 16792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:29:03.479125 16792 rpc_server.cc:307] RPC server started. Bound to: 127.16.102.3:41701
I20260709 12:29:03.479262   571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.102.3:41701 every 8 connection(s)
I20260709 12:29:03.496677   572 heartbeater.cc:344] Connected to a master server at 127.16.102.62:35193
I20260709 12:29:03.497280   572 heartbeater.cc:461] Registering TS with master...
I20260709 12:29:03.498272   572 heartbeater.cc:507] Master 127.16.102.62:35193 requested a full tablet report, sending...
I20260709 12:29:03.500844 32764 ts_manager.cc:194] Registered new tserver with Master: 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:03.501947 16792 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018952394s
I20260709 12:29:03.502696 32764 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53450
I20260709 12:29:03.535477 32764 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55896:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:29:03.538897 32764 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:29:03.624091   537 tablet_service.cc:1511] Processing CreateTablet for tablet 6f531bc8ca7f4f2bac07a02d7d562d47 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:03.625988   537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f531bc8ca7f4f2bac07a02d7d562d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.626976   387 tablet_service.cc:1511] Processing CreateTablet for tablet c985c2255f224d56813909db7551f037 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:03.628918   387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c985c2255f224d56813909db7551f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.631249   388 tablet_service.cc:1511] Processing CreateTablet for tablet 6f531bc8ca7f4f2bac07a02d7d562d47 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:03.633203   388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f531bc8ca7f4f2bac07a02d7d562d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.635229   536 tablet_service.cc:1511] Processing CreateTablet for tablet c985c2255f224d56813909db7551f037 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:03.639308   536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c985c2255f224d56813909db7551f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.639415   385 tablet_service.cc:1511] Processing CreateTablet for tablet 69700dd13eb644fc87075764655a7c8d (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:03.641327   385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69700dd13eb644fc87075764655a7c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.643298   386 tablet_service.cc:1511] Processing CreateTablet for tablet 40ce481100e74572b992deb470778bb1 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:03.643585   384 tablet_service.cc:1511] Processing CreateTablet for tablet 6a32c85c69854a579b8c8c9eb2f490bc (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:03.645200   386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40ce481100e74572b992deb470778bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.646072   384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a32c85c69854a579b8c8c9eb2f490bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.647776   535 tablet_service.cc:1511] Processing CreateTablet for tablet 40ce481100e74572b992deb470778bb1 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:03.649531   535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40ce481100e74572b992deb470778bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.651829   534 tablet_service.cc:1511] Processing CreateTablet for tablet 69700dd13eb644fc87075764655a7c8d (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:03.655452   533 tablet_service.cc:1511] Processing CreateTablet for tablet 6a32c85c69854a579b8c8c9eb2f490bc (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:03.658898   534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69700dd13eb644fc87075764655a7c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.663069   533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a32c85c69854a579b8c8c9eb2f490bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.666208   587 tablet_bootstrap.cc:492] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:03.683409   463 tablet_service.cc:1511] Processing CreateTablet for tablet 6f531bc8ca7f4f2bac07a02d7d562d47 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:03.685329   460 tablet_service.cc:1511] Processing CreateTablet for tablet 69700dd13eb644fc87075764655a7c8d (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:03.687296   461 tablet_service.cc:1511] Processing CreateTablet for tablet 40ce481100e74572b992deb470778bb1 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:03.688406   459 tablet_service.cc:1511] Processing CreateTablet for tablet 6a32c85c69854a579b8c8c9eb2f490bc (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:03.690448   383 tablet_service.cc:1511] Processing CreateTablet for tablet 3fd387cc70584d6aa36ef41af2f18cfb (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:03.692497   383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd387cc70584d6aa36ef41af2f18cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.695724   458 tablet_service.cc:1511] Processing CreateTablet for tablet 3fd387cc70584d6aa36ef41af2f18cfb (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:03.693938   462 tablet_service.cc:1511] Processing CreateTablet for tablet c985c2255f224d56813909db7551f037 (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:03.685477   463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f531bc8ca7f4f2bac07a02d7d562d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.698194   462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c985c2255f224d56813909db7551f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.700335   460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69700dd13eb644fc87075764655a7c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.702303   461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40ce481100e74572b992deb470778bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.707193   459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a32c85c69854a579b8c8c9eb2f490bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.710947   537 tablet_service.cc:1511] Processing CreateTablet for tablet 3fd387cc70584d6aa36ef41af2f18cfb (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:03.713037   537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd387cc70584d6aa36ef41af2f18cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.709076   458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd387cc70584d6aa36ef41af2f18cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.726239   587 tablet_bootstrap.cc:654] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:03.735424   382 tablet_service.cc:1511] Processing CreateTablet for tablet 4b229e0d49c44c37adb4c95d73bdd6fe (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:03.737336   382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b229e0d49c44c37adb4c95d73bdd6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.748207   457 tablet_service.cc:1511] Processing CreateTablet for tablet 4b229e0d49c44c37adb4c95d73bdd6fe (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:03.750309   457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b229e0d49c44c37adb4c95d73bdd6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.756845   532 tablet_service.cc:1511] Processing CreateTablet for tablet 4b229e0d49c44c37adb4c95d73bdd6fe (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:03.762800   532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b229e0d49c44c37adb4c95d73bdd6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.779785   587 tablet_bootstrap.cc:492] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:03.780647   587 ts_tablet_manager.cc:1403] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.115s	user 0.011s	sys 0.005s
I20260709 12:29:03.780381   456 tablet_service.cc:1511] Processing CreateTablet for tablet 644ee91daa20415db01046db3df21c5f (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:03.782126   456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644ee91daa20415db01046db3df21c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.786867   386 tablet_service.cc:1511] Processing CreateTablet for tablet 5d27dc670d7a42f39da938e5e12dd82d (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:03.790437   536 tablet_service.cc:1511] Processing CreateTablet for tablet 644ee91daa20415db01046db3df21c5f (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:03.791945   386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d27dc670d7a42f39da938e5e12dd82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.792164   536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644ee91daa20415db01046db3df21c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.800477   455 tablet_service.cc:1511] Processing CreateTablet for tablet 5d27dc670d7a42f39da938e5e12dd82d (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:03.802953   455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d27dc670d7a42f39da938e5e12dd82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.806802   385 tablet_service.cc:1511] Processing CreateTablet for tablet 644ee91daa20415db01046db3df21c5f (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:03.814148   385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644ee91daa20415db01046db3df21c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.839651   589 tablet_bootstrap.cc:492] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:03.852087   587 raft_consensus.cc:359] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:03.853231   587 raft_consensus.cc:385] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:03.853642   587 raft_consensus.cc:740] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
W20260709 12:29:03.856793 32751 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.102.1:36833, user_credentials={real_user=slave}} blocked reactor thread for 43924.1us
I20260709 12:29:03.870476   587 consensus_queue.cc:260] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:03.863231   534 tablet_service.cc:1511] Processing CreateTablet for tablet 5d27dc670d7a42f39da938e5e12dd82d (DEFAULT_TABLE table=test-workload [id=adfdb0df25a748ce862dfaf4828df68f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:03.880780   534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d27dc670d7a42f39da938e5e12dd82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:03.949277   591 tablet_bootstrap.cc:492] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:03.954139   589 tablet_bootstrap.cc:654] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:03.961052   572 heartbeater.cc:499] Master 127.16.102.62:35193 was elected leader, sending a full tablet report...
I20260709 12:29:03.990352   591 tablet_bootstrap.cc:654] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:03.985792   587 ts_tablet_manager.cc:1434] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.205s	user 0.020s	sys 0.033s
I20260709 12:29:03.994585   590 raft_consensus.cc:493] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.995172   590 raft_consensus.cc:515] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:03.997469   587 tablet_bootstrap.cc:492] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.000753   590 leader_election.cc:290] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.008981   587 tablet_bootstrap.cc:654] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.016317   589 tablet_bootstrap.cc:492] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.023892   589 ts_tablet_manager.cc:1403] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.185s	user 0.026s	sys 0.019s
I20260709 12:29:04.029309   587 tablet_bootstrap.cc:492] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.030162   587 ts_tablet_manager.cc:1403] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.007s
I20260709 12:29:04.033620   587 raft_consensus.cc:359] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.034785   587 raft_consensus.cc:385] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.035401   587 raft_consensus.cc:740] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.036284   587 consensus_queue.cc:260] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.040562   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.041092   587 ts_tablet_manager.cc:1434] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 12:29:04.042433   587 tablet_bootstrap.cc:492] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.033609   589 raft_consensus.cc:359] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:04.044317   511 leader_election.cc:343] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.044487   589 raft_consensus.cc:385] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.044996   589 raft_consensus.cc:740] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.046133   589 consensus_queue.cc:260] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.047001   591 tablet_bootstrap.cc:492] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.047902   591 ts_tablet_manager.cc:1403] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.099s	user 0.018s	sys 0.030s
I20260709 12:29:04.050568   587 tablet_bootstrap.cc:654] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
W20260709 12:29:04.052114   344 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:29:04.054646   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
W20260709 12:29:04.057395   513 leader_election.cc:343] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.058228   513 leader_election.cc:304] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:04.059316   590 raft_consensus.cc:2749] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:04.053707   591 raft_consensus.cc:359] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.060779   591 raft_consensus.cc:385] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.061232   591 raft_consensus.cc:740] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.062641   591 consensus_queue.cc:260] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.063170   423 heartbeater.cc:499] Master 127.16.102.62:35193 was elected leader, sending a full tablet report...
I20260709 12:29:04.063807   589 ts_tablet_manager.cc:1434] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.039s	user 0.011s	sys 0.008s
I20260709 12:29:04.064407   587 tablet_bootstrap.cc:492] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.065083   587 ts_tablet_manager.cc:1403] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.009s
I20260709 12:29:04.065444   589 tablet_bootstrap.cc:492] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.069070   587 raft_consensus.cc:359] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.070086   587 raft_consensus.cc:385] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.070415   587 raft_consensus.cc:740] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.071456   587 consensus_queue.cc:260] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.076478   589 tablet_bootstrap.cc:654] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.077562   587 ts_tablet_manager.cc:1434] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:29:04.079125   587 tablet_bootstrap.cc:492] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.087255   498 heartbeater.cc:499] Master 127.16.102.62:35193 was elected leader, sending a full tablet report...
I20260709 12:29:04.089002   591 ts_tablet_manager.cc:1434] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.041s	user 0.025s	sys 0.008s
I20260709 12:29:04.091456   591 tablet_bootstrap.cc:492] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.101671   597 raft_consensus.cc:493] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.102286   591 tablet_bootstrap.cc:654] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.102608   597 raft_consensus.cc:515] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.115126   587 tablet_bootstrap.cc:654] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.119418   591 tablet_bootstrap.cc:492] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
W20260709 12:29:04.119846   499 tablet.cc:2406] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:29:04.120579   591 ts_tablet_manager.cc:1403] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.009s
I20260709 12:29:04.124127   597 leader_election.cc:290] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.126297   591 raft_consensus.cc:359] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.127933   591 raft_consensus.cc:385] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.128448   591 raft_consensus.cc:740] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.130024   591 consensus_queue.cc:260] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.132643   590 raft_consensus.cc:493] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.132752   591 ts_tablet_manager.cc:1434] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20260709 12:29:04.133523   590 raft_consensus.cc:515] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.135011   591 tablet_bootstrap.cc:492] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.137392   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
W20260709 12:29:04.140025   513 leader_election.cc:343] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.141275   590 leader_election.cc:290] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.142756   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
W20260709 12:29:04.145241   511 leader_election.cc:343] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.145872   511 leader_election.cc:304] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:04.148036   590 raft_consensus.cc:2749] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:04.149508   591 tablet_bootstrap.cc:654] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.149390   599 raft_consensus.cc:493] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.149960   589 tablet_bootstrap.cc:492] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.150660   589 ts_tablet_manager.cc:1403] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.086s	user 0.046s	sys 0.024s
I20260709 12:29:04.150293   599 raft_consensus.cc:515] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.154628   589 raft_consensus.cc:359] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.154827   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.155586   589 raft_consensus.cc:385] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.156013   589 raft_consensus.cc:740] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
W20260709 12:29:04.170331   363 leader_election.cc:343] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.158221   599 leader_election.cc:290] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.178892   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.157667   589 consensus_queue.cc:260] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:04.181315   362 leader_election.cc:343] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.181924   362 leader_election.cc:304] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c; no voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:04.182080   589 ts_tablet_manager.cc:1434] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.031s	user 0.011s	sys 0.000s
I20260709 12:29:04.183545   589 tablet_bootstrap.cc:492] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.183576   597 raft_consensus.cc:2749] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:04.191926   589 tablet_bootstrap.cc:654] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.193595   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
W20260709 12:29:04.195389   439 leader_election.cc:343] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.196564   587 tablet_bootstrap.cc:492] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.197266   587 ts_tablet_manager.cc:1403] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.118s	user 0.020s	sys 0.015s
I20260709 12:29:04.197412   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.198369   547 raft_consensus.cc:2468] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.200067   438 leader_election.cc:304] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:04.201306   599 raft_consensus.cc:2804] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.201607   591 tablet_bootstrap.cc:492] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.201205   587 raft_consensus.cc:359] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.201993   599 raft_consensus.cc:493] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.202363   591 ts_tablet_manager.cc:1403] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.068s	user 0.025s	sys 0.011s
I20260709 12:29:04.202394   587 raft_consensus.cc:385] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.202576   599 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.203182   587 raft_consensus.cc:740] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.204205   587 consensus_queue.cc:260] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.205543   591 raft_consensus.cc:359] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.206454   591 raft_consensus.cc:385] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.206817   591 raft_consensus.cc:740] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.206966   587 ts_tablet_manager.cc:1434] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:29:04.207726   591 consensus_queue.cc:260] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.208309   587 tablet_bootstrap.cc:492] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.218487   591 ts_tablet_manager.cc:1434] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.008s
I20260709 12:29:04.220956   608 raft_consensus.cc:493] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.221439   591 tablet_bootstrap.cc:492] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.222942   608 raft_consensus.cc:515] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.224449   599 raft_consensus.cc:515] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.233153   608 leader_election.cc:290] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.235150   591 tablet_bootstrap.cc:654] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.235615   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40ce481100e74572b992deb470778bb1" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.236498   547 raft_consensus.cc:2468] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.236690   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:04.237557   546 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.240999   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.237709   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40ce481100e74572b992deb470778bb1" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.242830   398 raft_consensus.cc:2468] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.243598   589 tablet_bootstrap.cc:492] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.244388   589 ts_tablet_manager.cc:1403] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.061s	user 0.011s	sys 0.012s
W20260709 12:29:04.244915   439 leader_election.cc:343] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.245955   439 leader_election.cc:304] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:04.247287   610 raft_consensus.cc:2804] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.247751   610 raft_consensus.cc:493] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.248091   610 raft_consensus.cc:3060] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.248564   589 raft_consensus.cc:359] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.249282   589 raft_consensus.cc:385] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.249544   589 raft_consensus.cc:740] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.254642   546 raft_consensus.cc:2468] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.255520   589 consensus_queue.cc:260] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.256579   438 leader_election.cc:304] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:04.257635   608 raft_consensus.cc:2804] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.239174   599 leader_election.cc:290] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.261937   589 ts_tablet_manager.cc:1434] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.008s
I20260709 12:29:04.265295   589 tablet_bootstrap.cc:492] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.264700   610 raft_consensus.cc:515] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.268230   610 leader_election.cc:290] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.269567   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40ce481100e74572b992deb470778bb1" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.270720   398 raft_consensus.cc:3060] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.270741   587 tablet_bootstrap.cc:654] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.278455   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40ce481100e74572b992deb470778bb1" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:04.279464   547 raft_consensus.cc:3060] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.280817   589 tablet_bootstrap.cc:654] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.282217   608 raft_consensus.cc:697] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:04.283921   608 consensus_queue.cc:237] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.292784   587 tablet_bootstrap.cc:492] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.293853   587 ts_tablet_manager.cc:1403] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.086s	user 0.020s	sys 0.002s
I20260709 12:29:04.297284   547 raft_consensus.cc:2468] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.298069   587 raft_consensus.cc:359] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.300325   587 raft_consensus.cc:385] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.300678   587 raft_consensus.cc:740] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.301082   398 raft_consensus.cc:2468] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.303186   591 tablet_bootstrap.cc:492] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.301692   587 consensus_queue.cc:260] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.304150   591 ts_tablet_manager.cc:1403] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.084s	user 0.030s	sys 0.023s
I20260709 12:29:04.305140   589 tablet_bootstrap.cc:492] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.306104   589 ts_tablet_manager.cc:1403] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.041s	user 0.007s	sys 0.013s
I20260709 12:29:04.306927   587 ts_tablet_manager.cc:1434] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:29:04.303098   439 leader_election.cc:304] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:04.307989   610 raft_consensus.cc:2804] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.308461   587 tablet_bootstrap.cc:492] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.308580   610 raft_consensus.cc:697] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:04.310040   610 consensus_queue.cc:237] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.311657   589 raft_consensus.cc:359] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.312819   589 raft_consensus.cc:385] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.313148   589 raft_consensus.cc:740] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.314304   589 consensus_queue.cc:260] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.318853   587 tablet_bootstrap.cc:654] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.323746   591 raft_consensus.cc:359] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.325055   591 raft_consensus.cc:385] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.325489   591 raft_consensus.cc:740] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.326542   591 consensus_queue.cc:260] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.329020   589 ts_tablet_manager.cc:1434] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.005s
I20260709 12:29:04.329371   591 ts_tablet_manager.cc:1434] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:29:04.332051   589 tablet_bootstrap.cc:492] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.334520   591 tablet_bootstrap.cc:492] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.337633 32762 catalog_manager.cc:5697] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:04.342348   610 raft_consensus.cc:493] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.343039   589 tablet_bootstrap.cc:654] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.343134   610 raft_consensus.cc:515] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.347620   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
W20260709 12:29:04.350358   439 leader_election.cc:343] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.353148   610 leader_election.cc:290] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.359570   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
W20260709 12:29:04.361560   438 leader_election.cc:343] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.362218   438 leader_election.cc:304] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:04.364034   610 raft_consensus.cc:2749] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:04.364383   591 tablet_bootstrap.cc:654] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.366806   587 tablet_bootstrap.cc:492] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.366974   589 tablet_bootstrap.cc:492] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.367791   587 ts_tablet_manager.cc:1403] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.060s	user 0.024s	sys 0.006s
I20260709 12:29:04.368014   589 ts_tablet_manager.cc:1403] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.015s
I20260709 12:29:04.373021   589 raft_consensus.cc:359] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.375130   587 raft_consensus.cc:359] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.376035   589 raft_consensus.cc:385] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.376065   587 raft_consensus.cc:385] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.376672   589 raft_consensus.cc:740] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.376708   587 raft_consensus.cc:740] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.378036   591 tablet_bootstrap.cc:492] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.377956   587 consensus_queue.cc:260] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.378051   589 consensus_queue.cc:260] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.379154   591 ts_tablet_manager.cc:1403] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.010s
I20260709 12:29:04.380827   587 ts_tablet_manager.cc:1434] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:29:04.384613   591 raft_consensus.cc:359] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.385617   590 raft_consensus.cc:493] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.386060   591 raft_consensus.cc:385] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.386332   590 raft_consensus.cc:515] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.386623   591 raft_consensus.cc:740] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.387336   587 tablet_bootstrap.cc:492] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.388296   591 consensus_queue.cc:260] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.390303   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.391268   398 raft_consensus.cc:2468] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.392441   591 ts_tablet_manager.cc:1434] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.013s	user 0.014s	sys 0.000s
I20260709 12:29:04.393524   513 leader_election.cc:304] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.394642   589 ts_tablet_manager.cc:1434] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 12:29:04.396009   589 tablet_bootstrap.cc:492] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.393868   591 tablet_bootstrap.cc:492] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.404405   589 tablet_bootstrap.cc:654] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.407616   623 raft_consensus.cc:2804] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.408300   623 raft_consensus.cc:493] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.409639   591 tablet_bootstrap.cc:654] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.409965   587 tablet_bootstrap.cc:654] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.408661   623 raft_consensus.cc:3060] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.424754   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.424866   591 tablet_bootstrap.cc:492] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.426496   591 ts_tablet_manager.cc:1403] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.033s	user 0.020s	sys 0.003s
W20260709 12:29:04.427527   511 leader_election.cc:343] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.428582 32762 catalog_manager.cc:5697] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:04.424870   587 tablet_bootstrap.cc:492] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.430562   587 ts_tablet_manager.cc:1403] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.001s
I20260709 12:29:04.430327   591 raft_consensus.cc:359] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.431260   591 raft_consensus.cc:385] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.431596   591 raft_consensus.cc:740] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.432482   591 consensus_queue.cc:260] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.432833   610 raft_consensus.cc:493] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.433648   590 leader_election.cc:290] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.434149   610 raft_consensus.cc:515] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.434311   587 raft_consensus.cc:359] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.435154   587 raft_consensus.cc:385] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.435485   587 raft_consensus.cc:740] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.436751   591 ts_tablet_manager.cc:1434] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 12:29:04.438213   591 tablet_bootstrap.cc:492] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.438853   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d27dc670d7a42f39da938e5e12dd82d" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.439214   589 tablet_bootstrap.cc:492] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.439884   589 ts_tablet_manager.cc:1403] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.044s	user 0.027s	sys 0.004s
W20260709 12:29:04.441097   439 leader_election.cc:343] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.443194   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d27dc670d7a42f39da938e5e12dd82d" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.443979   589 raft_consensus.cc:359] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.444945   589 raft_consensus.cc:385] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.445276   589 raft_consensus.cc:740] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.446384   591 tablet_bootstrap.cc:654] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.446522   589 consensus_queue.cc:260] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.448607   610 leader_election.cc:290] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.449230   589 ts_tablet_manager.cc:1434] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
W20260709 12:29:04.446265   438 leader_election.cc:343] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.449806   438 leader_election.cc:304] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:04.450606   589 tablet_bootstrap.cc:492] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.451303   610 raft_consensus.cc:2749] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:04.454663   591 tablet_bootstrap.cc:492] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.455457   591 ts_tablet_manager.cc:1403] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20260709 12:29:04.455216   587 consensus_queue.cc:260] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.458057   587 ts_tablet_manager.cc:1434] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.005s
I20260709 12:29:04.454509   623 raft_consensus.cc:515] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.459594   591 raft_consensus.cc:359] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.460522   591 raft_consensus.cc:385] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.460901   591 raft_consensus.cc:740] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.461872   591 consensus_queue.cc:260] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.462297   623 leader_election.cc:290] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.463423   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.464241   398 raft_consensus.cc:3060] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.464898   591 ts_tablet_manager.cc:1434] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:29:04.466354   591 tablet_bootstrap.cc:492] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.466264   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
W20260709 12:29:04.468139   511 leader_election.cc:343] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:04.468694   589 tablet_bootstrap.cc:654] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.476257   591 tablet_bootstrap.cc:654] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.479974   587 tablet_bootstrap.cc:492] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.485757   398 raft_consensus.cc:2468] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.495163   513 leader_election.cc:304] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:04.496320   623 raft_consensus.cc:2804] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.497155   623 raft_consensus.cc:697] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:04.498375   623 consensus_queue.cc:237] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.499437   591 tablet_bootstrap.cc:492] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.500201   591 ts_tablet_manager.cc:1403] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.015s
I20260709 12:29:04.496246   587 tablet_bootstrap.cc:654] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.502990   589 tablet_bootstrap.cc:492] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.503868   589 ts_tablet_manager.cc:1403] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.054s	user 0.022s	sys 0.018s
I20260709 12:29:04.504160   591 raft_consensus.cc:359] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.505326   591 raft_consensus.cc:385] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.505698   591 raft_consensus.cc:740] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.507103   591 consensus_queue.cc:260] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.512916   589 raft_consensus.cc:359] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.514446   589 raft_consensus.cc:385] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.514923   589 raft_consensus.cc:740] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.516211   589 consensus_queue.cc:260] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.519279   610 raft_consensus.cc:493] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.520287   610 raft_consensus.cc:515] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.524823   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.525319   610 leader_election.cc:290] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.525620   398 raft_consensus.cc:2468] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.525990   591 ts_tablet_manager.cc:1434] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.025s	user 0.012s	sys 0.000s
I20260709 12:29:04.527283   591 tablet_bootstrap.cc:492] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:04.527696   439 leader_election.cc:304] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:04.526794   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.529299   547 raft_consensus.cc:2468] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.529309   610 raft_consensus.cc:2804] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.530107   610 raft_consensus.cc:493] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.530476   610 raft_consensus.cc:3060] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.529505 32761 catalog_manager.cc:5697] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:04.534493   589 ts_tablet_manager.cc:1434] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.001s
I20260709 12:29:04.535969   589 tablet_bootstrap.cc:492] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.536868   591 tablet_bootstrap.cc:654] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.544569   589 tablet_bootstrap.cc:654] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.550076   587 tablet_bootstrap.cc:492] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.550943   587 ts_tablet_manager.cc:1403] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.071s	user 0.015s	sys 0.019s
I20260709 12:29:04.560518   623 raft_consensus.cc:493] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.556391   610 raft_consensus.cc:515] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.561210   623 raft_consensus.cc:515] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.563901   623 leader_election.cc:290] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.564865   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.565739   398 raft_consensus.cc:3060] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.566905   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644ee91daa20415db01046db3df21c5f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.568042   473 raft_consensus.cc:2468] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.571118   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644ee91daa20415db01046db3df21c5f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.572234   397 raft_consensus.cc:2468] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.573779   511 leader_election.cc:304] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.559875   587 raft_consensus.cc:359] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.576228   587 raft_consensus.cc:385] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.576548   587 raft_consensus.cc:740] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.577440   587 consensus_queue.cc:260] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.578696   590 raft_consensus.cc:2804] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.578923   398 raft_consensus.cc:2468] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.579164   590 raft_consensus.cc:493] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.579691   590 raft_consensus.cc:3060] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.580034   587 ts_tablet_manager.cc:1434] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 12:29:04.580628   439 leader_election.cc:304] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:04.581243   587 tablet_bootstrap.cc:492] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:04.581703   608 raft_consensus.cc:2804] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.582293   608 raft_consensus.cc:697] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:04.583268   608 consensus_queue.cc:237] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.587684   623 raft_consensus.cc:493] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.588734   587 tablet_bootstrap.cc:654] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.589974   591 tablet_bootstrap.cc:492] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:04.590626   597 raft_consensus.cc:493] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.591503   597 raft_consensus.cc:515] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:04.590947   590 raft_consensus.cc:515] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.588430   623 raft_consensus.cc:515] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.594647   597 leader_election.cc:290] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.594796   589 tablet_bootstrap.cc:492] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.595584   589 ts_tablet_manager.cc:1403] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.060s	user 0.012s	sys 0.026s
I20260709 12:29:04.596537   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.597704   547 raft_consensus.cc:2393] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46fd946b83f34e42b80480fe7a6f7f0c in current term 1: Already voted for candidate 4c73adb282914cd0a9ab735eda3a2178 in this term.
I20260709 12:29:04.598778   590 leader_election.cc:290] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.596382   623 leader_election.cc:290] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.601058   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644ee91daa20415db01046db3df21c5f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.602025   396 raft_consensus.cc:3060] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.602989   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.604089   472 raft_consensus.cc:2468] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.604007 32762 catalog_manager.cc:5697] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:04.590622   591 ts_tablet_manager.cc:1403] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.064s	user 0.026s	sys 0.015s
I20260709 12:29:04.596760   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.608954   362 leader_election.cc:304] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c; no voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:04.610002   597 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.620441   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.629062   597 raft_consensus.cc:2749] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:04.629526   397 raft_consensus.cc:2468] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.622216   589 raft_consensus.cc:359] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.623696   511 leader_election.cc:304] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.623728   591 raft_consensus.cc:359] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.631012   591 raft_consensus.cc:385] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.631349   591 raft_consensus.cc:740] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.631593   396 raft_consensus.cc:2468] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.632174   591 consensus_queue.cc:260] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.632980   513 leader_election.cc:304] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.624927   610 leader_election.cc:290] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.623534   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644ee91daa20415db01046db3df21c5f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:04.634299   473 raft_consensus.cc:3060] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.634512   634 raft_consensus.cc:2804] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.635000   634 raft_consensus.cc:493] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.635362   634 raft_consensus.cc:3060] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.637564   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:04.638258   547 raft_consensus.cc:3060] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.644069   634 raft_consensus.cc:515] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.645231   591 ts_tablet_manager.cc:1434] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.000s
I20260709 12:29:04.646361   590 raft_consensus.cc:2804] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.646857   547 raft_consensus.cc:2468] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.647025   590 raft_consensus.cc:697] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:04.648195   590 consensus_queue.cc:237] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.628794   587 tablet_bootstrap.cc:492] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:04.630600   589 raft_consensus.cc:385] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.649714   634 leader_election.cc:290] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.650048   589 raft_consensus.cc:740] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.651275   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:04.651995   472 raft_consensus.cc:3060] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.652594   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.653406   396 raft_consensus.cc:3060] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.650883   589 consensus_queue.cc:260] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.656122   589 ts_tablet_manager.cc:1434] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.000s
I20260709 12:29:04.650868   587 ts_tablet_manager.cc:1403] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.070s	user 0.015s	sys 0.003s
I20260709 12:29:04.657735   589 tablet_bootstrap.cc:492] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:04.662222   473 raft_consensus.cc:2468] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.673871   472 raft_consensus.cc:2468] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.671594   587 raft_consensus.cc:359] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.675102   587 raft_consensus.cc:385] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.675545   587 raft_consensus.cc:740] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.675921   511 leader_election.cc:304] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.677296   634 raft_consensus.cc:2804] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.678042   634 raft_consensus.cc:697] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:04.676781   587 consensus_queue.cc:260] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.679083   634 consensus_queue.cc:237] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.681391   587 ts_tablet_manager.cc:1434] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.023s	user 0.011s	sys 0.000s
I20260709 12:29:04.683367   589 tablet_bootstrap.cc:654] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:04.685725   396 raft_consensus.cc:2468] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.686343   623 raft_consensus.cc:493] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.687158   623 raft_consensus.cc:515] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.692250   623 leader_election.cc:290] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.692708   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.693257   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.693728   396 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.694126   472 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.695552   511 leader_election.cc:304] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.697598   634 raft_consensus.cc:2804] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.698500   634 raft_consensus.cc:493] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.699077   634 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.699716 32761 catalog_manager.cc:5697] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:04.704809   589 tablet_bootstrap.cc:492] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:04.705560   589 ts_tablet_manager.cc:1403] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.048s	user 0.017s	sys 0.007s
I20260709 12:29:04.708010   634 raft_consensus.cc:515] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.709889   589 raft_consensus.cc:359] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.710562   634 leader_election.cc:290] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.711036   589 raft_consensus.cc:385] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:04.711403   589 raft_consensus.cc:740] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:04.711647   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.712389   396 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.712512   589 consensus_queue.cc:260] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.713943   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:04.714668   472 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.715180   589 ts_tablet_manager.cc:1434] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260709 12:29:04.723191 32762 catalog_manager.cc:5697] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:04.725600   634 raft_consensus.cc:493] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.725383   472 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.726397   634 raft_consensus.cc:515] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.727427   511 leader_election.cc:304] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.728520   623 raft_consensus.cc:2804] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.725383   396 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.729103   623 raft_consensus.cc:697] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:04.730330   623 consensus_queue.cc:237] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.730579   634 leader_election.cc:290] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.730189   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d27dc670d7a42f39da938e5e12dd82d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.737521   397 raft_consensus.cc:2468] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.737677   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d27dc670d7a42f39da938e5e12dd82d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:04.739019   513 leader_election.cc:304] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.739126   472 raft_consensus.cc:2468] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:04.740022   590 raft_consensus.cc:2804] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.740450   590 raft_consensus.cc:493] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.740775   590 raft_consensus.cc:3060] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.749245 32762 catalog_manager.cc:5697] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:04.750615   590 raft_consensus.cc:515] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.753571   590 leader_election.cc:290] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:04.753980   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d27dc670d7a42f39da938e5e12dd82d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.754769   397 raft_consensus.cc:3060] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.756752   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d27dc670d7a42f39da938e5e12dd82d" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:04.757541   472 raft_consensus.cc:3060] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.766806   397 raft_consensus.cc:2468] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.767144   472 raft_consensus.cc:2468] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:04.768394   513 leader_election.cc:304] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:04.769315   590 raft_consensus.cc:2804] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.769940   590 raft_consensus.cc:697] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:04.770782   590 consensus_queue.cc:237] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.780256 32762 catalog_manager.cc:5697] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:04.789978   610 raft_consensus.cc:493] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.790838   610 raft_consensus.cc:515] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.793325   610 leader_election.cc:290] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.794210   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fd387cc70584d6aa36ef41af2f18cfb" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.794546   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fd387cc70584d6aa36ef41af2f18cfb" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.795024   397 raft_consensus.cc:2468] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.795280   547 raft_consensus.cc:2468] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:04.796373   439 leader_election.cc:304] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:04.797297   610 raft_consensus.cc:2804] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:04.797712   610 raft_consensus.cc:493] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.798065   610 raft_consensus.cc:3060] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.805325   610 raft_consensus.cc:515] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.807772   610 leader_election.cc:290] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.808734   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fd387cc70584d6aa36ef41af2f18cfb" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:04.809062   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fd387cc70584d6aa36ef41af2f18cfb" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:04.809512   397 raft_consensus.cc:3060] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.809805   547 raft_consensus.cc:3060] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.816901   397 raft_consensus.cc:2468] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.816901   547 raft_consensus.cc:2468] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:04.818444   439 leader_election.cc:304] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:04.819309   610 raft_consensus.cc:2804] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:04.819878   610 raft_consensus.cc:697] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:04.820765   610 consensus_queue.cc:237] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.829375 32762 catalog_manager.cc:5697] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:04.836169   610 consensus_queue.cc:1048] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:04.850481   612 consensus_queue.cc:1048] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:04.892916   608 consensus_queue.cc:1048] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:04.903817   472 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:41666
I20260709 12:29:04.904743   472 raft_consensus.cc:606] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:04.908798   608 consensus_queue.cc:1048] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:04.914343   472 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c985c2255f224d56813909db7551f037"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:41666
I20260709 12:29:04.915309   472 raft_consensus.cc:606] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:04.924922   472 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "40ce481100e74572b992deb470778bb1"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:41666
I20260709 12:29:04.925757   472 raft_consensus.cc:606] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:04.942986   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:04.943786   547 raft_consensus.cc:606] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:04.950735   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:04.951345   547 raft_consensus.cc:606] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:04.961141   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:04.961731   547 raft_consensus.cc:606] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:04.967782   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "644ee91daa20415db01046db3df21c5f"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:04.968621   547 raft_consensus.cc:606] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:04.981969   608 raft_consensus.cc:493] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.982682   608 raft_consensus.cc:515] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:04.984956   608 leader_election.cc:290] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:04.986115   397 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:04.986115   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:04.987383   397 raft_consensus.cc:2393] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c73adb282914cd0a9ab735eda3a2178 in current term 1: Already voted for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in this term.
I20260709 12:29:04.988502   439 leader_election.cc:304] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:04.989292   608 raft_consensus.cc:3060] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:04.994560   590 consensus_queue.cc:1048] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:04.999138   608 raft_consensus.cc:2749] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:05.003423   623 raft_consensus.cc:993] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.004119   634 raft_consensus.cc:1081] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.006156   396 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36718
I20260709 12:29:05.007189   396 raft_consensus.cc:493] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.007663   396 raft_consensus.cc:3060] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.012413   634 consensus_queue.cc:1048] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.020115   396 raft_consensus.cc:515] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.024289   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:05.025164   472 raft_consensus.cc:3060] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.026332   396 leader_election.cc:290] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:05.029263   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69700dd13eb644fc87075764655a7c8d" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.030104   547 raft_consensus.cc:3055] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.030507   547 raft_consensus.cc:740] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:05.031554   547 consensus_queue.cc:260] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.032953   547 raft_consensus.cc:3060] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.036876   472 raft_consensus.cc:2468] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.038618   362 leader_election.cc:304] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:05.039702   597 raft_consensus.cc:2804] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.041167   597 raft_consensus.cc:697] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:05.041998   547 raft_consensus.cc:2468] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.042337   597 consensus_queue.cc:237] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.051136 32762 catalog_manager.cc:5697] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.100030   608 consensus_queue.cc:1048] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:05.114733   608 consensus_queue.cc:1048] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:05.126379   612 raft_consensus.cc:993] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.127079   608 raft_consensus.cc:1081] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.129191   396 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c985c2255f224d56813909db7551f037"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36734
I20260709 12:29:05.129928   396 raft_consensus.cc:493] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.130313   396 raft_consensus.cc:3060] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.136649   634 consensus_queue.cc:1048] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.139842   396 raft_consensus.cc:515] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.143633   396 leader_election.cc:290] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:05.144088   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:05.144917   472 raft_consensus.cc:3055] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.145301   472 raft_consensus.cc:740] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:05.145390   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c985c2255f224d56813909db7551f037" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.146301   472 consensus_queue.cc:260] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.147056   547 raft_consensus.cc:3060] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.147972   472 raft_consensus.cc:3060] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.157258   547 raft_consensus.cc:2468] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.159356   623 consensus_queue.cc:1048] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:05.159845   363 leader_election.cc:304] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:05.161104   597 raft_consensus.cc:2804] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.161650   597 raft_consensus.cc:697] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:05.162817   597 consensus_queue.cc:237] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.164705   623 consensus_queue.cc:1048] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.165910   472 raft_consensus.cc:2468] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.177259   590 raft_consensus.cc:993] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:05.176293 32760 catalog_manager.cc:5697] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.177918   590 raft_consensus.cc:1081] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:05.180353   473 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:41648
I20260709 12:29:05.181595   473 raft_consensus.cc:493] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.182094   473 raft_consensus.cc:3060] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.187709   590 consensus_queue.cc:1048] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.194018   623 raft_consensus.cc:993] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:05.194730   590 raft_consensus.cc:1081] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:05.196071   473 raft_consensus.cc:515] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.197273   472 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "644ee91daa20415db01046db3df21c5f"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:41648
I20260709 12:29:05.198057   472 raft_consensus.cc:493] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.198488   472 raft_consensus.cc:3060] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.209542   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:05.209755   473 leader_election.cc:290] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:05.211150   472 raft_consensus.cc:515] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.211993   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a32c85c69854a579b8c8c9eb2f490bc" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.212735   547 raft_consensus.cc:3055] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.213088   547 raft_consensus.cc:740] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:05.214003   472 leader_election.cc:290] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:05.214972   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644ee91daa20415db01046db3df21c5f" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:05.214076   547 consensus_queue.cc:260] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.215353   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644ee91daa20415db01046db3df21c5f" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.215710   396 raft_consensus.cc:3060] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.216138   546 raft_consensus.cc:3055] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.216531   546 raft_consensus.cc:740] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:05.217010   547 raft_consensus.cc:3060] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.217554   546 consensus_queue.cc:260] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.219036   546 raft_consensus.cc:3060] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.227970   547 raft_consensus.cc:2468] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:05.229734   438 leader_election.cc:304] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:05.230229   396 raft_consensus.cc:2468] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:05.230293   546 raft_consensus.cc:2468] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:05.230572   623 consensus_queue.cc:1048] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.231257   612 raft_consensus.cc:2804] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.231849   439 leader_election.cc:304] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:05.231860   612 raft_consensus.cc:697] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:05.232919   608 raft_consensus.cc:2804] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.233760   608 raft_consensus.cc:697] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:05.234061   612 consensus_queue.cc:237] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.234735   608 consensus_queue.cc:237] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.244974 32762 catalog_manager.cc:5697] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.248682   634 consensus_queue.cc:1048] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.264233 32762 catalog_manager.cc:5697] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.286877   634 consensus_queue.cc:1048] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:05.297183   610 raft_consensus.cc:993] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.297853   610 raft_consensus.cc:1081] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.297911   623 raft_consensus.cc:993] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:05.298491   623 raft_consensus.cc:1081] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:05.299515   634 consensus_queue.cc:1048] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.299937   396 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36734
I20260709 12:29:05.296444   612 consensus_queue.cc:1048] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:29:05.300696   396 raft_consensus.cc:493] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.301148   396 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.302567   473 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:41648
I20260709 12:29:05.305891   473 raft_consensus.cc:493] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.306454   473 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.314477   396 raft_consensus.cc:515] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:05.321660   663 consensus_queue.cc:1048] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:05.325058   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:05.326015   472 raft_consensus.cc:3055] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.326408   472 raft_consensus.cc:740] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:05.326380   473 raft_consensus.cc:515] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.327633   472 consensus_queue.cc:260] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:05.329032   472 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.325206   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.330993   546 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.332777   396 leader_election.cc:290] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:05.342832   473 leader_election.cc:290] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:05.345893   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:05.346777   398 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.344249   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.348589   545 raft_consensus.cc:3055] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.348977   545 raft_consensus.cc:740] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:05.349984   545 consensus_queue.cc:260] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.351821   545 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.359114   398 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:05.361057   439 leader_election.cc:304] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:05.363354   663 raft_consensus.cc:2804] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.363976   663 raft_consensus.cc:697] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:05.364931   663 consensus_queue.cc:237] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.366847   546 raft_consensus.cc:2468] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.367350   545 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:05.368790   363 leader_election.cc:304] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:05.364954   472 raft_consensus.cc:2468] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.370955   597 raft_consensus.cc:2804] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.371577   597 raft_consensus.cc:697] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:05.372579   597 consensus_queue.cc:237] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:05.378876 32762 catalog_manager.cc:5697] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.384895 32761 catalog_manager.cc:5697] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.456923   545 raft_consensus.cc:1275] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.458690   597 consensus_queue.cc:1048] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.468508   472 raft_consensus.cc:1275] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.471316   655 consensus_queue.cc:1048] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:05.526888   608 raft_consensus.cc:993] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.527518   663 raft_consensus.cc:1081] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:05.529577   398 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "40ce481100e74572b992deb470778bb1"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36734
I20260709 12:29:05.530230   398 raft_consensus.cc:493] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.530588   398 raft_consensus.cc:3060] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.537822   398 raft_consensus.cc:515] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.540292   398 leader_election.cc:290] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:05.541527   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40ce481100e74572b992deb470778bb1" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:05.541409   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40ce481100e74572b992deb470778bb1" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:05.542528   545 raft_consensus.cc:3060] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.542572   472 raft_consensus.cc:3055] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:05.543087   472 raft_consensus.cc:740] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:05.544123   472 consensus_queue.cc:260] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.545604   472 raft_consensus.cc:3060] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.552402   545 raft_consensus.cc:2468] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.554169   363 leader_election.cc:304] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:05.554714   472 raft_consensus.cc:2468] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:05.555055   655 raft_consensus.cc:2804] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:05.555640   655 raft_consensus.cc:697] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:05.556624   655 consensus_queue.cc:237] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:05.564323 32761 catalog_manager.cc:5697] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:05.623980   545 raft_consensus.cc:1275] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.625571   663 consensus_queue.cc:1048] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.635485   398 raft_consensus.cc:1275] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.637717   608 consensus_queue.cc:1048] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:05.642804   396 raft_consensus.cc:3060] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:05.652896   396 raft_consensus.cc:1275] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.654496   608 consensus_queue.cc:1048] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.663976   545 raft_consensus.cc:1275] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.667387   608 consensus_queue.cc:1048] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:05.749311   396 raft_consensus.cc:1275] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.751035   608 consensus_queue.cc:1048] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.752384   545 raft_consensus.cc:1275] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.754004   655 consensus_queue.cc:1048] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.760613   547 raft_consensus.cc:1275] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.763775   610 consensus_queue.cc:1048] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:05.767006   472 raft_consensus.cc:1275] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.768885   655 consensus_queue.cc:1048] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:05.847191   472 raft_consensus.cc:1275] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.849203   655 consensus_queue.cc:1048] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.863912   545 raft_consensus.cc:1275] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.866055   597 consensus_queue.cc:1048] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:05.941154   545 raft_consensus.cc:1275] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.942723   597 consensus_queue.cc:1048] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:05.953539   472 raft_consensus.cc:1275] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:05.955731   597 consensus_queue.cc:1048] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:08.017155 32763 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55902:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:29:08.020401 32763 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:29:08.081538   386 tablet_service.cc:1511] Processing CreateTablet for tablet 4e772bdd32994e9d9db75376cbbdc5cd (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:08.083158   386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e772bdd32994e9d9db75376cbbdc5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.082736   460 tablet_service.cc:1511] Processing CreateTablet for tablet 4e772bdd32994e9d9db75376cbbdc5cd (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:08.084518   460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e772bdd32994e9d9db75376cbbdc5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.084609   534 tablet_service.cc:1511] Processing CreateTablet for tablet 4e772bdd32994e9d9db75376cbbdc5cd (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:08.086251   534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e772bdd32994e9d9db75376cbbdc5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.089751   462 tablet_service.cc:1511] Processing CreateTablet for tablet 529ee0b0b3b243b8b9f5f8d500eefbc4 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:08.091566   462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529ee0b0b3b243b8b9f5f8d500eefbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.092170   458 tablet_service.cc:1511] Processing CreateTablet for tablet d83869da929a44a6be7379e8b01bbf4c (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:08.093681   458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83869da929a44a6be7379e8b01bbf4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:29:08.101722   344 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:29:08.104646   532 tablet_service.cc:1511] Processing CreateTablet for tablet 529ee0b0b3b243b8b9f5f8d500eefbc4 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:08.105084   461 tablet_service.cc:1511] Processing CreateTablet for tablet 422991c944394d5e9024d49c2cdb1222 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:08.107062   537 tablet_service.cc:1511] Processing CreateTablet for tablet d83869da929a44a6be7379e8b01bbf4c (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:08.108278   461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422991c944394d5e9024d49c2cdb1222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.112751   382 tablet_service.cc:1511] Processing CreateTablet for tablet 529ee0b0b3b243b8b9f5f8d500eefbc4 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:08.114691   382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529ee0b0b3b243b8b9f5f8d500eefbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.115048   383 tablet_service.cc:1511] Processing CreateTablet for tablet 422991c944394d5e9024d49c2cdb1222 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:08.116113   533 tablet_service.cc:1511] Processing CreateTablet for tablet de48d2cd9e654eda8d32522d64717448 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:08.112906   459 tablet_service.cc:1511] Processing CreateTablet for tablet de48d2cd9e654eda8d32522d64717448 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:08.112988   385 tablet_service.cc:1511] Processing CreateTablet for tablet d83869da929a44a6be7379e8b01bbf4c (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:08.116113   536 tablet_service.cc:1511] Processing CreateTablet for tablet 422991c944394d5e9024d49c2cdb1222 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:08.116596   383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422991c944394d5e9024d49c2cdb1222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.118937   459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de48d2cd9e654eda8d32522d64717448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.124189   384 tablet_service.cc:1511] Processing CreateTablet for tablet de48d2cd9e654eda8d32522d64717448 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:08.124506   455 tablet_service.cc:1511] Processing CreateTablet for tablet 7512ad59cefc40bbbc57250610c02648 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:08.126097   455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7512ad59cefc40bbbc57250610c02648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.127131   387 tablet_service.cc:1511] Processing CreateTablet for tablet 7512ad59cefc40bbbc57250610c02648 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:08.130729   463 tablet_service.cc:1511] Processing CreateTablet for tablet 13f8cc8ae4cf46bd9aebc62a521857f9 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:08.132429   463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f8cc8ae4cf46bd9aebc62a521857f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.137485   384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de48d2cd9e654eda8d32522d64717448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.137662   535 tablet_service.cc:1511] Processing CreateTablet for tablet 7512ad59cefc40bbbc57250610c02648 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:08.137946   531 tablet_service.cc:1511] Processing CreateTablet for tablet 13f8cc8ae4cf46bd9aebc62a521857f9 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:08.138324   388 tablet_service.cc:1511] Processing CreateTablet for tablet 13f8cc8ae4cf46bd9aebc62a521857f9 (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:08.139298   385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83869da929a44a6be7379e8b01bbf4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.140697   388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f8cc8ae4cf46bd9aebc62a521857f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.141567   381 tablet_service.cc:1511] Processing CreateTablet for tablet 89ea3aeecf65438b8000e3b8c10d9a7b (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:08.143204   387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7512ad59cefc40bbbc57250610c02648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.143970   381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ea3aeecf65438b8000e3b8c10d9a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.117498   532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529ee0b0b3b243b8b9f5f8d500eefbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.146068   533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de48d2cd9e654eda8d32522d64717448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.147212   531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f8cc8ae4cf46bd9aebc62a521857f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.150347   536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422991c944394d5e9024d49c2cdb1222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.154361   535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7512ad59cefc40bbbc57250610c02648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.158079   537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83869da929a44a6be7379e8b01bbf4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.181990   456 tablet_service.cc:1511] Processing CreateTablet for tablet 89ea3aeecf65438b8000e3b8c10d9a7b (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:08.183773   456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ea3aeecf65438b8000e3b8c10d9a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.187114   530 tablet_service.cc:1511] Processing CreateTablet for tablet 89ea3aeecf65438b8000e3b8c10d9a7b (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:08.189059   530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89ea3aeecf65438b8000e3b8c10d9a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.202648   380 tablet_service.cc:1511] Processing CreateTablet for tablet 157d51caa8704cda9be3b34a5291145f (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:08.204741   380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157d51caa8704cda9be3b34a5291145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.211308   457 tablet_service.cc:1511] Processing CreateTablet for tablet 157d51caa8704cda9be3b34a5291145f (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:08.213232   457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157d51caa8704cda9be3b34a5291145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:08.260106   529 tablet_service.cc:1511] Processing CreateTablet for tablet 157d51caa8704cda9be3b34a5291145f (DEFAULT_TABLE table=soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:08.262120   529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157d51caa8704cda9be3b34a5291145f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:29:08.243494   362 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:44003, user_credentials={real_user=slave}} blocked reactor thread for 111873us
I20260709 12:29:08.308583   691 tablet_bootstrap.cc:492] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.318924   687 tablet_bootstrap.cc:492] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.338757   687 tablet_bootstrap.cc:654] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.368126   691 tablet_bootstrap.cc:654] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
W20260709 12:29:08.377461   438 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:41701, user_credentials={real_user=slave}} blocked reactor thread for 46806.6us
I20260709 12:29:08.434406   693 tablet_bootstrap.cc:492] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.436662   687 tablet_bootstrap.cc:492] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.437639   687 ts_tablet_manager.cc:1403] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.119s	user 0.017s	sys 0.004s
I20260709 12:29:08.443367   693 tablet_bootstrap.cc:654] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.443311   687 raft_consensus.cc:359] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.444643   687 raft_consensus.cc:385] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.445111   687 raft_consensus.cc:740] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.446219   687 consensus_queue.cc:260] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.447280   691 tablet_bootstrap.cc:492] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.447976   691 ts_tablet_manager.cc:1403] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.140s	user 0.014s	sys 0.006s
I20260709 12:29:08.449425   687 ts_tablet_manager.cc:1434] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:29:08.450979   687 tablet_bootstrap.cc:492] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.452493   691 raft_consensus.cc:359] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.453452   691 raft_consensus.cc:385] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.453789   691 raft_consensus.cc:740] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.454870   691 consensus_queue.cc:260] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.457921   691 ts_tablet_manager.cc:1434] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.004s
I20260709 12:29:08.459501   691 tablet_bootstrap.cc:492] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.462505   687 tablet_bootstrap.cc:654] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.465957   692 raft_consensus.cc:493] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.466691   692 raft_consensus.cc:515] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.470659   692 leader_election.cc:290] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.471953   691 tablet_bootstrap.cc:654] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.472148   693 tablet_bootstrap.cc:492] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.471983   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:08.472882   693 ts_tablet_manager.cc:1403] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.012s
W20260709 12:29:08.474514   438 leader_election.cc:343] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.477900   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.478894   396 raft_consensus.cc:2468] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:08.481045   693 raft_consensus.cc:359] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.482296   693 raft_consensus.cc:385] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.482700   693 raft_consensus.cc:740] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.483333   439 leader_election.cc:304] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:08.483904   693 consensus_queue.cc:260] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.486685   663 raft_consensus.cc:2804] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.487300   663 raft_consensus.cc:493] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.487751   663 raft_consensus.cc:3060] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.488142   693 ts_tablet_manager.cc:1434] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.004s
I20260709 12:29:08.489476   693 tablet_bootstrap.cc:492] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.496194   687 tablet_bootstrap.cc:492] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.497014   687 ts_tablet_manager.cc:1403] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.046s	user 0.022s	sys 0.007s
I20260709 12:29:08.500967   687 raft_consensus.cc:359] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.501895   687 raft_consensus.cc:385] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.502223   687 raft_consensus.cc:740] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.503782   687 consensus_queue.cc:260] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.506603   687 ts_tablet_manager.cc:1434] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.004s
I20260709 12:29:08.508358   687 tablet_bootstrap.cc:492] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.510671   691 tablet_bootstrap.cc:492] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.510845   693 tablet_bootstrap.cc:654] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.512022   691 ts_tablet_manager.cc:1403] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.053s	user 0.021s	sys 0.018s
I20260709 12:29:08.518405   687 tablet_bootstrap.cc:654] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.521944   663 raft_consensus.cc:515] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.521996   691 raft_consensus.cc:359] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.523321   691 raft_consensus.cc:385] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.523741   691 raft_consensus.cc:740] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.524629   691 consensus_queue.cc:260] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.527384   691 ts_tablet_manager.cc:1434] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260709 12:29:08.532368   691 tablet_bootstrap.cc:492] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.536408   693 tablet_bootstrap.cc:492] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.536393   687 tablet_bootstrap.cc:492] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.537423   693 ts_tablet_manager.cc:1403] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.048s	user 0.015s	sys 0.004s
I20260709 12:29:08.537498   687 ts_tablet_manager.cc:1403] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.009s
I20260709 12:29:08.539439   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:08.540256   396 raft_consensus.cc:3060] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.541458   687 raft_consensus.cc:359] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.541874   693 raft_consensus.cc:359] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.542400   687 raft_consensus.cc:385] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.542969   687 raft_consensus.cc:740] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.543422   693 raft_consensus.cc:385] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.543862   693 raft_consensus.cc:740] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.545189   691 tablet_bootstrap.cc:654] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.547075   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:08.547880   547 raft_consensus.cc:3060] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.548897   687 consensus_queue.cc:260] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.556888   687 ts_tablet_manager.cc:1434] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.019s	user 0.010s	sys 0.000s
I20260709 12:29:08.557953   663 leader_election.cc:290] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.544741   693 consensus_queue.cc:260] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.559028   687 tablet_bootstrap.cc:492] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.569940   693 ts_tablet_manager.cc:1434] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.001s
I20260709 12:29:08.557538   547 raft_consensus.cc:2468] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:08.571986   438 leader_election.cc:304] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:08.572626   687 tablet_bootstrap.cc:654] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.576653   396 raft_consensus.cc:2468] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:08.578799   663 raft_consensus.cc:2804] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:08.579510   663 raft_consensus.cc:697] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:08.580562   663 consensus_queue.cc:237] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.582859   693 tablet_bootstrap.cc:492] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.583415   691 tablet_bootstrap.cc:492] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.584020   691 ts_tablet_manager.cc:1403] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.052s	user 0.020s	sys 0.014s
I20260709 12:29:08.588112   691 raft_consensus.cc:359] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.589094   691 raft_consensus.cc:385] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.589447   691 raft_consensus.cc:740] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.590440   691 consensus_queue.cc:260] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.594762   687 tablet_bootstrap.cc:492] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.594851   691 ts_tablet_manager.cc:1434] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.003s
I20260709 12:29:08.597873   691 tablet_bootstrap.cc:492] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.598271   693 tablet_bootstrap.cc:654] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.603796   687 ts_tablet_manager.cc:1403] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.045s	user 0.020s	sys 0.003s
I20260709 12:29:08.612625   692 raft_consensus.cc:493] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.613233   687 raft_consensus.cc:359] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.614257   687 raft_consensus.cc:385] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.613294   692 raft_consensus.cc:515] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.614727   687 raft_consensus.cc:740] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.613929   693 tablet_bootstrap.cc:492] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.616736   693 ts_tablet_manager.cc:1403] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.034s	user 0.011s	sys 0.013s
I20260709 12:29:08.611341   691 tablet_bootstrap.cc:654] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.619664   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7512ad59cefc40bbbc57250610c02648" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.620736   693 raft_consensus.cc:359] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.621639   693 raft_consensus.cc:385] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.615828   687 consensus_queue.cc:260] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:08.623624   439 leader_election.cc:343] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.623770   663 raft_consensus.cc:493] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.624461   663 raft_consensus.cc:515] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.625058   687 ts_tablet_manager.cc:1434] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:29:08.625239 32759 catalog_manager.cc:5697] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:08.626412   687 tablet_bootstrap.cc:492] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.627013   663 leader_election.cc:290] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.622102   693 raft_consensus.cc:740] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.628509   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:08.628718   693 consensus_queue.cc:260] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.630264   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7512ad59cefc40bbbc57250610c02648" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:08.631316   547 raft_consensus.cc:2468] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:08.631544   693 ts_tablet_manager.cc:1434] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:29:08.633031   693 tablet_bootstrap.cc:492] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.633996   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.634833   396 raft_consensus.cc:2468] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
W20260709 12:29:08.636778   438 leader_election.cc:343] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.637807   438 leader_election.cc:304] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:08.639093   687 tablet_bootstrap.cc:654] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.639492   439 leader_election.cc:304] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:08.640615   663 raft_consensus.cc:2804] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.641332   663 raft_consensus.cc:493] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.641700   663 raft_consensus.cc:3060] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.644732   688 raft_consensus.cc:2804] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.645323   691 tablet_bootstrap.cc:492] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.645536   688 raft_consensus.cc:493] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.646183   691 ts_tablet_manager.cc:1403] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.049s	user 0.017s	sys 0.000s
I20260709 12:29:08.646261   688 raft_consensus.cc:3060] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.652669   692 leader_election.cc:290] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.650306   690 raft_consensus.cc:493] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.656958   691 raft_consensus.cc:359] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.658506   691 raft_consensus.cc:385] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.658964   691 raft_consensus.cc:740] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.659157   690 raft_consensus.cc:515] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.660068   691 consensus_queue.cc:260] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.662295   690 leader_election.cc:290] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.662503   687 tablet_bootstrap.cc:492] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.663110   691 ts_tablet_manager.cc:1434] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.000s
I20260709 12:29:08.663273   687 ts_tablet_manager.cc:1403] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.037s	user 0.016s	sys 0.015s
I20260709 12:29:08.641621   693 tablet_bootstrap.cc:654] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.664924   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
W20260709 12:29:08.666906   363 leader_election.cc:343] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.668148   687 raft_consensus.cc:359] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.669112   687 raft_consensus.cc:385] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.669474   687 raft_consensus.cc:740] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.669730   691 tablet_bootstrap.cc:492] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.670571   687 consensus_queue.cc:260] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.673221   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:08.676059   687 ts_tablet_manager.cc:1434] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:29:08.677009   688 raft_consensus.cc:515] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:08.677980   362 leader_election.cc:343] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.678859   362 leader_election.cc:304] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c; no voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:08.679345   663 raft_consensus.cc:515] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.677979   687 tablet_bootstrap.cc:492] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.680002   690 raft_consensus.cc:2749] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:08.683509   691 tablet_bootstrap.cc:654] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.684664   693 tablet_bootstrap.cc:492] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.685384   693 ts_tablet_manager.cc:1403] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.053s	user 0.006s	sys 0.007s
I20260709 12:29:08.687743   688 leader_election.cc:290] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.687484   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:08.688998   396 raft_consensus.cc:3060] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.689525   663 leader_election.cc:290] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.689709   693 raft_consensus.cc:359] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.690779   693 raft_consensus.cc:385] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.691205   693 raft_consensus.cc:740] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.692302   693 consensus_queue.cc:260] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.695111   693 ts_tablet_manager.cc:1434] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.009s	user 0.011s	sys 0.000s
I20260709 12:29:08.696100   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7512ad59cefc40bbbc57250610c02648" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:08.696955   545 raft_consensus.cc:3060] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.696331   693 tablet_bootstrap.cc:492] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.702725   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7512ad59cefc40bbbc57250610c02648" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:08.705152   691 tablet_bootstrap.cc:492] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.706064   691 ts_tablet_manager.cc:1403] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.005s
I20260709 12:29:08.706190   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:08.710778   691 raft_consensus.cc:359] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:08.711828   438 leader_election.cc:343] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:29:08.711835   439 leader_election.cc:343] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.712014   691 raft_consensus.cc:385] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.713872   691 raft_consensus.cc:740] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.715047   691 consensus_queue.cc:260] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.717875   693 tablet_bootstrap.cc:654] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.718674   691 ts_tablet_manager.cc:1434] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:29:08.720165   687 tablet_bootstrap.cc:654] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.720175   691 tablet_bootstrap.cc:492] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.729655   545 raft_consensus.cc:2468] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:08.732242   438 leader_election.cc:304] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:08.734030   663 raft_consensus.cc:2804] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:08.734572   663 raft_consensus.cc:697] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:08.735487   663 consensus_queue.cc:237] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.736689   396 raft_consensus.cc:2468] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:08.741382   693 tablet_bootstrap.cc:492] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.741183   439 leader_election.cc:304] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:08.742100   693 ts_tablet_manager.cc:1403] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.046s	user 0.011s	sys 0.012s
I20260709 12:29:08.742837   691 tablet_bootstrap.cc:654] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.743232   688 raft_consensus.cc:2804] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:08.744040   688 raft_consensus.cc:697] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:08.746160   693 raft_consensus.cc:359] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.747088   693 raft_consensus.cc:385] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.747449   693 raft_consensus.cc:740] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.748744   687 tablet_bootstrap.cc:492] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.749387   687 ts_tablet_manager.cc:1403] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.072s	user 0.013s	sys 0.005s
I20260709 12:29:08.745296   688 consensus_queue.cc:237] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.753145   687 raft_consensus.cc:359] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.758450   687 raft_consensus.cc:385] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.758950   687 raft_consensus.cc:740] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.760542   693 consensus_queue.cc:260] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.765163   693 ts_tablet_manager.cc:1434] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:29:08.762153   687 consensus_queue.cc:260] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.768630   634 raft_consensus.cc:493] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.769517   634 raft_consensus.cc:515] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.774228   693 tablet_bootstrap.cc:492] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.774281   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.775561   396 raft_consensus.cc:2468] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:08.777319   513 leader_election.cc:304] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:08.787561   634 leader_election.cc:290] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:08.795828   687 ts_tablet_manager.cc:1434] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.046s	user 0.010s	sys 0.001s
I20260709 12:29:08.797510   687 tablet_bootstrap.cc:492] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.800802   693 tablet_bootstrap.cc:654] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.801105   634 raft_consensus.cc:2804] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.806253   688 raft_consensus.cc:493] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.807175   634 raft_consensus.cc:493] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.807677   634 raft_consensus.cc:3060] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.807447   688 raft_consensus.cc:515] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.811412   688 leader_election.cc:290] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.812175   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.816139   396 raft_consensus.cc:2468] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 0.
I20260709 12:29:08.815250   687 tablet_bootstrap.cc:654] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.818578   439 leader_election.cc:304] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:08.819765   688 raft_consensus.cc:2804] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.820348   688 raft_consensus.cc:493] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.820837   688 raft_consensus.cc:3060] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.815241 32763 catalog_manager.cc:5697] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:08.830785 32763 catalog_manager.cc:5697] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:08.839334   693 tablet_bootstrap.cc:492] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.840929   693 ts_tablet_manager.cc:1403] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.073s	user 0.009s	sys 0.014s
I20260709 12:29:08.846383   717 raft_consensus.cc:493] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.846561   691 tablet_bootstrap.cc:492] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.847492   691 ts_tablet_manager.cc:1403] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.128s	user 0.044s	sys 0.022s
I20260709 12:29:08.849258   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:08.849516   693 raft_consensus.cc:359] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.850560   693 raft_consensus.cc:385] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.850762   473 raft_consensus.cc:2468] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:08.854216   717 raft_consensus.cc:515] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.854826   693 raft_consensus.cc:740] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.855832   693 consensus_queue.cc:260] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.856755   687 tablet_bootstrap.cc:492] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:08.857932   687 ts_tablet_manager.cc:1403] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.061s	user 0.019s	sys 0.020s
I20260709 12:29:08.858424   717 leader_election.cc:290] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:08.858722   693 ts_tablet_manager.cc:1434] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:29:08.852178   688 raft_consensus.cc:515] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.860958   688 leader_election.cc:290] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:08.853196   691 raft_consensus.cc:359] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.864256   693 tablet_bootstrap.cc:492] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.861622   687 raft_consensus.cc:359] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.862228   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:08.863919   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.882543   398 raft_consensus.cc:3060] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.882843   396 raft_consensus.cc:2468] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:08.884414   513 leader_election.cc:304] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:08.865849   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:08.877756   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:08.888108   546 raft_consensus.cc:3060] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.828276   634 raft_consensus.cc:515] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.888988   693 tablet_bootstrap.cc:654] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.861629   691 raft_consensus.cc:385] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.890010   691 raft_consensus.cc:740] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.891409   691 consensus_queue.cc:260] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.893101   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:08.893970   395 raft_consensus.cc:3060] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.894603   691 ts_tablet_manager.cc:1434] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.012s
I20260709 12:29:08.895370   634 leader_election.cc:290] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:08.896175   691 tablet_bootstrap.cc:492] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.898427   687 raft_consensus.cc:385] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.903581   687 raft_consensus.cc:740] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.904698   691 tablet_bootstrap.cc:654] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.863474   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:08.904863   687 consensus_queue.cc:260] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.907955   687 ts_tablet_manager.cc:1434] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 12:29:08.893915   717 raft_consensus.cc:2804] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.912279   717 raft_consensus.cc:493] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.912763   717 raft_consensus.cc:3060] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.919785   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:08.920727   473 raft_consensus.cc:3060] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.929319   395 raft_consensus.cc:2468] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:08.931188   513 leader_election.cc:304] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:08.932914   634 raft_consensus.cc:2804] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:29:08.933615   511 leader_election.cc:343] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.934578   634 raft_consensus.cc:697] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:08.936152   691 tablet_bootstrap.cc:492] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:08.936295   718 raft_consensus.cc:493] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.936980   691 ts_tablet_manager.cc:1403] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.041s	user 0.021s	sys 0.002s
I20260709 12:29:08.935607   634 consensus_queue.cc:237] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.936995   718 raft_consensus.cc:515] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.939801   718 leader_election.cc:290] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:08.940894   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:08.924309   546 raft_consensus.cc:2468] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:08.941645   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:08.943313   693 tablet_bootstrap.cc:492] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:08.942953   438 leader_election.cc:304] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:08.944713   693 ts_tablet_manager.cc:1403] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.081s	user 0.015s	sys 0.019s
I20260709 12:29:08.912767   687 tablet_bootstrap.cc:492] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:08.941929   396 raft_consensus.cc:2468] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:08.945254   688 raft_consensus.cc:2804] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:08.946823   688 raft_consensus.cc:697] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:08.947929   688 consensus_queue.cc:237] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:08.949163   511 leader_election.cc:343] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:08.941161   691 raft_consensus.cc:359] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.951517   691 raft_consensus.cc:385] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.949283   693 raft_consensus.cc:359] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.952471   693 raft_consensus.cc:385] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:08.951215   513 leader_election.cc:304] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:08.952859   693 raft_consensus.cc:740] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.953589   718 raft_consensus.cc:2804] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:08.954082   718 raft_consensus.cc:493] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.954581   718 raft_consensus.cc:3060] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.953910   693 consensus_queue.cc:260] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.951974   691 raft_consensus.cc:740] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:08.956247   691 consensus_queue.cc:260] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.959514   691 ts_tablet_manager.cc:1434] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.004s
I20260709 12:29:08.960963   691 tablet_bootstrap.cc:492] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:08.963541   473 raft_consensus.cc:2468] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:08.946825   717 raft_consensus.cc:515] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:08.971359   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:08.971414   688 consensus_queue.cc:1048] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:08.972055   396 raft_consensus.cc:3060] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:08.973352 32762 catalog_manager.cc:5697] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 1 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:08.982877   691 tablet_bootstrap.cc:654] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.988770   693 ts_tablet_manager.cc:1434] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.007s
I20260709 12:29:08.990319   693 tablet_bootstrap.cc:492] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
I20260709 12:29:08.994035   687 tablet_bootstrap.cc:654] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.967054   398 raft_consensus.cc:2468] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 1.
I20260709 12:29:08.997884   717 leader_election.cc:290] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:08.999209   693 tablet_bootstrap.cc:654] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:08.996560 32762 catalog_manager.cc:5697] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:08.988862   396 raft_consensus.cc:2468] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:09.002424   718 raft_consensus.cc:515] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.007455   688 raft_consensus.cc:493] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.007381   513 leader_election.cc:304] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.008208   688 raft_consensus.cc:515] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.009138   717 raft_consensus.cc:2804] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:09.010078   717 raft_consensus.cc:697] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:09.010403   718 leader_election.cc:290] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:09.011173   688 leader_election.cc:290] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:09.012558   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:09.011070   717 consensus_queue.cc:237] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.014129   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:09.016813   396 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:09.015695   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:09.015712   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
W20260709 12:29:09.020550   511 leader_election.cc:343] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:09.022060   691 tablet_bootstrap.cc:492] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:09.022784   691 ts_tablet_manager.cc:1403] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.062s	user 0.016s	sys 0.003s
W20260709 12:29:09.022266   511 leader_election.cc:343] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:09.020643   395 raft_consensus.cc:3060] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:29:09.028425   438 leader_election.cc:343] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:09.031781   691 raft_consensus.cc:359] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.032716   691 raft_consensus.cc:385] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:09.032985   691 raft_consensus.cc:740] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:09.033772   691 consensus_queue.cc:260] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:09.035588   439 leader_election.cc:343] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:09.036434   691 ts_tablet_manager.cc:1434] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:29:09.037642   691 tablet_bootstrap.cc:492] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: Bootstrap starting.
I20260709 12:29:09.036463   439 leader_election.cc:304] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:09.043632   688 raft_consensus.cc:2749] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:09.044533   688 consensus_queue.cc:1048] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:09.045571   691 tablet_bootstrap.cc:654] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:09.053531   687 tablet_bootstrap.cc:492] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:09.054360   687 ts_tablet_manager.cc:1403] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.142s	user 0.016s	sys 0.007s
I20260709 12:29:09.070675   687 raft_consensus.cc:359] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.071539   687 raft_consensus.cc:385] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:09.071849   687 raft_consensus.cc:740] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:09.072544 32761 catalog_manager.cc:5697] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:09.075309   693 tablet_bootstrap.cc:492] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:09.076087   693 ts_tablet_manager.cc:1403] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.086s	user 0.009s	sys 0.023s
I20260709 12:29:09.080802   693 raft_consensus.cc:359] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.084769   693 raft_consensus.cc:385] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:09.072660   687 consensus_queue.cc:260] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.089092   395 raft_consensus.cc:2468] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:09.085289   693 raft_consensus.cc:740] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:09.093453   687 ts_tablet_manager.cc:1434] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 12:29:09.094319   693 consensus_queue.cc:260] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.094928   687 tablet_bootstrap.cc:492] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: Bootstrap starting.
I20260709 12:29:09.095625   513 leader_election.cc:304] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:09.096395   693 ts_tablet_manager.cc:1434] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.000s
I20260709 12:29:09.097518   717 raft_consensus.cc:2804] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:09.098269   717 raft_consensus.cc:697] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:09.099597   717 consensus_queue.cc:237] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.097604   693 tablet_bootstrap.cc:492] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: Bootstrap starting.
W20260709 12:29:09.113739   344 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:29:09.136456   687 tablet_bootstrap.cc:654] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:09.138747   691 tablet_bootstrap.cc:492] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: No bootstrap required, opened a new log
I20260709 12:29:09.138849   693 tablet_bootstrap.cc:654] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:09.139727   691 ts_tablet_manager.cc:1403] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent bootstrapping tablet: real 0.102s	user 0.020s	sys 0.028s
I20260709 12:29:09.143828   691 raft_consensus.cc:359] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.144639   691 raft_consensus.cc:385] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:09.144774   687 tablet_bootstrap.cc:492] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: No bootstrap required, opened a new log
I20260709 12:29:09.144968   691 raft_consensus.cc:740] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Initialized, Role: FOLLOWER
I20260709 12:29:09.145426   687 ts_tablet_manager.cc:1403] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: Time spent bootstrapping tablet: real 0.051s	user 0.008s	sys 0.005s
I20260709 12:29:09.145898   691 consensus_queue.cc:260] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.148452   691 ts_tablet_manager.cc:1434] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:29:09.148975   687 raft_consensus.cc:359] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.149866   687 raft_consensus.cc:385] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:09.150171   687 raft_consensus.cc:740] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Initialized, Role: FOLLOWER
I20260709 12:29:09.151171   687 consensus_queue.cc:260] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.152552 32762 catalog_manager.cc:5697] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:09.155851   693 tablet_bootstrap.cc:492] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: No bootstrap required, opened a new log
I20260709 12:29:09.156586   693 ts_tablet_manager.cc:1403] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent bootstrapping tablet: real 0.059s	user 0.020s	sys 0.005s
I20260709 12:29:09.160710   687 ts_tablet_manager.cc:1434] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:29:09.160859   693 raft_consensus.cc:359] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.162074   693 raft_consensus.cc:385] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:09.162441   693 raft_consensus.cc:740] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Initialized, Role: FOLLOWER
I20260709 12:29:09.163321   693 consensus_queue.cc:260] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.165990   693 ts_tablet_manager.cc:1434] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:29:09.204772   546 raft_consensus.cc:3060] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.207682   688 consensus_queue.cc:1048] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.215837   688 consensus_queue.cc:1048] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:09.231549   671 raft_consensus.cc:493] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.232170   671 raft_consensus.cc:515] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.239244   671 leader_election.cc:290] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:09.242240   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7512ad59cefc40bbbc57250610c02648" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:09.243201   545 raft_consensus.cc:2393] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46fd946b83f34e42b80480fe7a6f7f0c in current term 1: Already voted for candidate 4c73adb282914cd0a9ab735eda3a2178 in this term.
I20260709 12:29:09.243736   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7512ad59cefc40bbbc57250610c02648" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:09.245450   362 leader_election.cc:304] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c; no voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:09.246367   671 raft_consensus.cc:3060] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.254519   671 raft_consensus.cc:2749] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:09.271514   717 raft_consensus.cc:493] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.272281   717 raft_consensus.cc:515] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.274612   717 leader_election.cc:290] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:09.275974   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "157d51caa8704cda9be3b34a5291145f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:09.276314   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "157d51caa8704cda9be3b34a5291145f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:09.277135   472 raft_consensus.cc:2468] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:09.277118   395 raft_consensus.cc:2468] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:09.279404   513 leader_election.cc:304] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.280438   717 raft_consensus.cc:2804] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:09.281023   717 raft_consensus.cc:493] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:09.281424   717 raft_consensus.cc:3060] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.282469   663 consensus_queue.cc:1048] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.284088   688 consensus_queue.cc:1048] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:09.303758   717 raft_consensus.cc:515] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.311851   717 leader_election.cc:290] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:09.314613   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "157d51caa8704cda9be3b34a5291145f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:09.315526   395 raft_consensus.cc:3060] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.315985   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "157d51caa8704cda9be3b34a5291145f" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:09.316830   472 raft_consensus.cc:3060] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.326531   395 raft_consensus.cc:2468] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:09.328457   513 leader_election.cc:304] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.329604   717 raft_consensus.cc:2804] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:09.330478   717 raft_consensus.cc:697] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:09.330987   718 consensus_queue.cc:1048] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.331583   717 consensus_queue.cc:237] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.332283   472 raft_consensus.cc:2468] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:09.349783   718 consensus_queue.cc:1048] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.354471 32761 catalog_manager.cc:5697] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:09.416672   717 raft_consensus.cc:493] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.417341   717 raft_consensus.cc:515] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.420037   717 leader_election.cc:290] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:09.420794   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:09.421176   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178" is_pre_election: true
I20260709 12:29:09.421592   395 raft_consensus.cc:2468] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:09.422040   472 raft_consensus.cc:2468] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 0.
I20260709 12:29:09.422817   513 leader_election.cc:304] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.423707   717 raft_consensus.cc:2804] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:09.424095   717 raft_consensus.cc:493] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:09.424400   717 raft_consensus.cc:3060] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.431051   634 consensus_queue.cc:1048] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.433610   717 raft_consensus.cc:515] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.437495   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:09.437783   717 leader_election.cc:290] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:09.438428   398 raft_consensus.cc:3060] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.439147   688 consensus_queue.cc:1048] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.440676   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:09.441526   472 raft_consensus.cc:3060] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.449949   398 raft_consensus.cc:2468] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:09.452119   513 leader_election.cc:304] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.453411   723 raft_consensus.cc:2804] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:09.454114   723 raft_consensus.cc:697] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:09.455309   723 consensus_queue.cc:237] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.457101   472 raft_consensus.cc:2468] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 1.
I20260709 12:29:09.458344   473 raft_consensus.cc:3060] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.467934   663 consensus_queue.cc:1048] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.471998   718 consensus_queue.cc:1048] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.474356   688 raft_consensus.cc:493] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.475085   688 raft_consensus.cc:515] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.480573   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" is_pre_election: true
I20260709 12:29:09.483831   688 leader_election.cc:290] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:09.490374   394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" is_pre_election: true
I20260709 12:29:09.491882   394 raft_consensus.cc:2393] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c73adb282914cd0a9ab735eda3a2178 in current term 1: Already voted for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in this term.
I20260709 12:29:09.494479   439 leader_election.cc:304] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178; no voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:09.495954   688 raft_consensus.cc:3060] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:09.508100   688 raft_consensus.cc:2749] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:09.514689   634 consensus_queue.cc:1048] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.529645 32761 catalog_manager.cc:5697] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 1 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:09.533418   634 consensus_queue.cc:1048] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.729192   634 consensus_queue.cc:1048] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:09.742740   634 consensus_queue.cc:1048] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:09.788646   472 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:41666
I20260709 12:29:09.789330   472 raft_consensus.cc:606] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:09.795607   472 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:41666
I20260709 12:29:09.796229   472 raft_consensus.cc:606] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:09.801608   545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:09.802284   545 raft_consensus.cc:606] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:09.808393   472 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:41666
I20260709 12:29:09.809078   472 raft_consensus.cc:606] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Received request to transfer leadership to TS 46fd946b83f34e42b80480fe7a6f7f0c
I20260709 12:29:09.815062   545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:09.815768   545 raft_consensus.cc:606] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:09.825322   545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:09.825995   545 raft_consensus.cc:606] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:09.831555   545 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
mode: GRACEFUL
new_leader_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:47538
I20260709 12:29:09.832342   545 raft_consensus.cc:606] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Received request to transfer leadership to TS 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:09.863557   718 raft_consensus.cc:993] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:09.864225   634 raft_consensus.cc:1081] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:09.865921   472 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:41648
I20260709 12:29:09.866452   472 raft_consensus.cc:493] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:09.866717   472 raft_consensus.cc:3060] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.872777   472 raft_consensus.cc:515] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.874446   472 leader_election.cc:290] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:09.875751   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:09.876061   394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13f8cc8ae4cf46bd9aebc62a521857f9" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:09.876645   545 raft_consensus.cc:3055] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:09.877107   545 raft_consensus.cc:740] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:09.877146   394 raft_consensus.cc:3060] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.878312   545 consensus_queue.cc:260] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.879921   545 raft_consensus.cc:3060] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.887676   749 raft_consensus.cc:993] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:09.888439   749 raft_consensus.cc:1081] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:09.888615   394 raft_consensus.cc:2468] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:09.890395   439 leader_election.cc:304] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:09.890787   398 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36734
I20260709 12:29:09.891546   749 raft_consensus.cc:2804] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:09.891626   398 raft_consensus.cc:493] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:09.892164   749 raft_consensus.cc:697] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:09.892254   398 raft_consensus.cc:3060] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.892570   545 raft_consensus.cc:2468] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:09.893211   749 consensus_queue.cc:237] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.901081   398 raft_consensus.cc:515] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.904374   398 leader_election.cc:290] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:09.904524   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:09.904006 32759 catalog_manager.cc:5697] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:09.905223   472 raft_consensus.cc:3055] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:09.905933   472 raft_consensus.cc:740] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:09.905223   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422991c944394d5e9024d49c2cdb1222" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:09.907171   545 raft_consensus.cc:3060] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.906855   472 consensus_queue.cc:260] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.909368   472 raft_consensus.cc:3060] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.916718   545 raft_consensus.cc:2468] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:09.916906   472 raft_consensus.cc:2468] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:09.918092   363 leader_election.cc:304] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.918915   690 raft_consensus.cc:2804] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:09.919457   690 raft_consensus.cc:697] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:09.920467   690 consensus_queue.cc:237] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.928390 32759 catalog_manager.cc:5697] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:09.941372   688 raft_consensus.cc:993] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:09.941888   749 raft_consensus.cc:1081] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:09.943643   398 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36734
I20260709 12:29:09.944263   398 raft_consensus.cc:493] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:09.944620   398 raft_consensus.cc:3060] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.953269   398 raft_consensus.cc:515] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.955420   398 leader_election.cc:290] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:09.956378   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:09.956595   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e772bdd32994e9d9db75376cbbdc5cd" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:09.957240   472 raft_consensus.cc:3055] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:09.957362   545 raft_consensus.cc:3060] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.957587   472 raft_consensus.cc:740] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:09.958563   472 consensus_queue.cc:260] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.959753   472 raft_consensus.cc:3060] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:09.965101   545 raft_consensus.cc:2468] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:09.965895   472 raft_consensus.cc:2468] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:09.966701   363 leader_election.cc:304] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:09.967552   690 raft_consensus.cc:2804] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:09.968062   690 raft_consensus.cc:697] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:09.968945   690 consensus_queue.cc:237] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:09.977783 32759 catalog_manager.cc:5697] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:10.002167   634 consensus_queue.cc:1048] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:10.002810   717 raft_consensus.cc:993] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:10.003489   717 raft_consensus.cc:1081] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:10.005326   472 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:41648
I20260709 12:29:10.005903   472 raft_consensus.cc:493] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:10.006575   472 raft_consensus.cc:3060] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.015146   472 raft_consensus.cc:515] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.017261   472 leader_election.cc:290] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:10.018512   717 raft_consensus.cc:993] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:10.018239   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:10.019084   717 raft_consensus.cc:1081] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:10.019249   398 raft_consensus.cc:3060] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.021040   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89ea3aeecf65438b8000e3b8c10d9a7b" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:10.021811   545 raft_consensus.cc:3055] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:10.022163   545 raft_consensus.cc:740] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:10.023001   545 consensus_queue.cc:260] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.024467   545 raft_consensus.cc:3060] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.027125   394 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36718
I20260709 12:29:10.027899   394 raft_consensus.cc:493] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:10.028367   394 raft_consensus.cc:3060] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.036474   545 raft_consensus.cc:2468] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:10.039410   717 consensus_queue.cc:1048] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:10.039669   438 leader_election.cc:304] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:10.039307   394 raft_consensus.cc:515] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.039342   398 raft_consensus.cc:2468] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:10.041185   749 raft_consensus.cc:2804] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:10.041754   749 raft_consensus.cc:697] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:10.042825   394 leader_election.cc:290] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:10.042760   749 consensus_queue.cc:237] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.044942   473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:10.048782   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d83869da929a44a6be7379e8b01bbf4c" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:10.049571   545 raft_consensus.cc:3055] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:10.049926   545 raft_consensus.cc:740] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:10.051016   545 consensus_queue.cc:260] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
W20260709 12:29:10.051755   717 consensus_queue.cc:1175] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4c73adb282914cd0a9ab735eda3a2178" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:29:10.052807   545 raft_consensus.cc:3060] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.054415 32761 catalog_manager.cc:5697] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:10.061014   545 raft_consensus.cc:2468] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:10.062448   363 leader_election.cc:304] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 4c73adb282914cd0a9ab735eda3a2178
I20260709 12:29:10.063438   690 raft_consensus.cc:2804] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:10.063969   690 raft_consensus.cc:697] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:10.064908   690 consensus_queue.cc:237] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.072827 32759 catalog_manager.cc:5697] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:10.096154   723 raft_consensus.cc:993] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: : Instructing follower 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:10.096685   717 raft_consensus.cc:1081] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Signalling peer 4c73adb282914cd0a9ab735eda3a2178 to start an election
I20260709 12:29:10.098664   472 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
 from {username='slave'} at 127.0.0.1:41648
I20260709 12:29:10.099237   472 raft_consensus.cc:493] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:10.099573   472 raft_consensus.cc:3060] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.106225   472 raft_consensus.cc:515] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.108604   472 leader_election.cc:290] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:10.109326   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:10.109148   398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de48d2cd9e654eda8d32522d64717448" candidate_uuid: "4c73adb282914cd0a9ab735eda3a2178" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:10.110190   398 raft_consensus.cc:3060] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.110724   749 raft_consensus.cc:993] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:10.110821   545 raft_consensus.cc:3055] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:10.111490   545 raft_consensus.cc:740] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:10.111519   688 raft_consensus.cc:1081] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Signalling peer 46fd946b83f34e42b80480fe7a6f7f0c to start an election
I20260709 12:29:10.112488   545 consensus_queue.cc:260] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.114091   545 raft_consensus.cc:3060] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.114974   394 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
 from {username='slave'} at 127.0.0.1:36734
I20260709 12:29:10.115679   394 raft_consensus.cc:493] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:10.116132   394 raft_consensus.cc:3060] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.121521   545 raft_consensus.cc:2468] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:10.123216   438 leader_election.cc:304] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:10.124313   688 raft_consensus.cc:2804] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:10.125145   688 raft_consensus.cc:697] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Leader. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:10.126127   688 consensus_queue.cc:237] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.126902   394 raft_consensus.cc:515] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.127830   398 raft_consensus.cc:2468] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c73adb282914cd0a9ab735eda3a2178 in term 2.
I20260709 12:29:10.129683   394 leader_election.cc:290] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Requested vote from peers 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003), 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3:41701)
I20260709 12:29:10.130993   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:10.131927   472 raft_consensus.cc:3055] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:10.132300   472 raft_consensus.cc:740] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:10.130987   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "529ee0b0b3b243b8b9f5f8d500eefbc4" candidate_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
I20260709 12:29:10.133131   545 raft_consensus.cc:3060] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.133643   472 consensus_queue.cc:260] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.135286   472 raft_consensus.cc:3060] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.138042 32759 catalog_manager.cc:5697] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 reported cstate change: term changed from 1 to 2, leader changed from 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3) to 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2). New cstate: current_term: 2 leader_uuid: "4c73adb282914cd0a9ab735eda3a2178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:10.146944   472 raft_consensus.cc:2468] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:10.148797   362 leader_election.cc:304] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 4c73adb282914cd0a9ab735eda3a2178; no voters: 
I20260709 12:29:10.149906   690 raft_consensus.cc:2804] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:10.150441   690 raft_consensus.cc:697] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Leader. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:10.151412   690 consensus_queue.cc:237] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:10.154095   545 raft_consensus.cc:2468] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fd946b83f34e42b80480fe7a6f7f0c in term 2.
I20260709 12:29:10.162304 32759 catalog_manager.cc:5697] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c reported cstate change: term changed from 1 to 2, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1). New cstate: current_term: 2 leader_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:10.286994   546 raft_consensus.cc:1275] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.288625   688 consensus_queue.cc:1048] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:10.298297   394 raft_consensus.cc:1275] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.302109   692 consensus_queue.cc:1048] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.426659   472 raft_consensus.cc:1275] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.428758   690 consensus_queue.cc:1048] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:10.439335   546 raft_consensus.cc:1275] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.442996   671 consensus_queue.cc:1048] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.460598   394 raft_consensus.cc:1275] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.462464   692 consensus_queue.cc:1048] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.467531   546 raft_consensus.cc:1275] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.469336   671 consensus_queue.cc:1048] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:10.475754   545 raft_consensus.cc:1275] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.478432   472 raft_consensus.cc:3060] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:10.484773   692 consensus_queue.cc:1048] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.491880   472 raft_consensus.cc:1275] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.493798   690 consensus_queue.cc:1048] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.494611   473 raft_consensus.cc:1275] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.497448   758 consensus_queue.cc:1048] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.507316   546 raft_consensus.cc:1275] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.511528   671 consensus_queue.cc:1048] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.537803   473 raft_consensus.cc:1275] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.539490   762 consensus_queue.cc:1048] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:10.552350   546 raft_consensus.cc:1275] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 46fd946b83f34e42b80480fe7a6f7f0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.554419   671 consensus_queue.cc:1048] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:10.609047   394 raft_consensus.cc:1275] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.610555   692 consensus_queue.cc:1048] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:10.621481   545 raft_consensus.cc:1275] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Refusing update from remote peer 4c73adb282914cd0a9ab735eda3a2178: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:10.623651   688 consensus_queue.cc:1048] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:12.840341 32761 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55902:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:29:12.869365 32761 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55902:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:29:12.964787 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:29:13.008003   785 log.cc:927] Time spent T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: Append to log took a long time: real 0.087s	user 0.003s	sys 0.004s
W20260709 12:29:13.163880   796 log.cc:927] Time spent T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
W20260709 12:29:13.172905   798 log.cc:927] Time spent T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: Append to log took a long time: real 0.098s	user 0.002s	sys 0.000s
W20260709 12:29:13.212203   801 log.cc:927] Time spent T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
W20260709 12:29:13.251628   363 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.3:41701, user_credentials={real_user=slave}} blocked reactor thread for 151555us
W20260709 12:29:13.274698   362 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.102.2:44003, user_credentials={real_user=slave}} blocked reactor thread for 151102us
I20260709 12:29:13.327637   825 tablet.cc:1724] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.379045   828 tablet.cc:1724] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:13.485309   830 log.cc:927] Time spent T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:29:13.485868   468 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe"
dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
mode: GRACEFUL
new_leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
 from {username='slave'} at 127.0.0.1:41668
I20260709 12:29:13.487020   468 raft_consensus.cc:606] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Received request to transfer leadership to TS 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:13.484354   839 tablet.cc:1724] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.383741   826 tablet.cc:1724] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.474658   834 tablet.cc:1724] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.520452   841 tablet.cc:1724] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.511579   842 tablet.cc:1724] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:13.560288   835 log.cc:927] Time spent T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: Append to log took a long time: real 0.052s	user 0.002s	sys 0.000s
I20260709 12:29:13.591509   837 tablet.cc:1724] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:13.618847   804 log.cc:927] Time spent T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: Append to log took a long time: real 0.486s	user 0.003s	sys 0.008s
I20260709 12:29:13.598333   843 tablet.cc:1724] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.636873   692 raft_consensus.cc:993] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: : Instructing follower 5e64426dfb04470687fbf9ebfc7eeee4 to start an election
I20260709 12:29:13.649439   828 tablet.cc:1724] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.650847   692 raft_consensus.cc:1081] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Signalling peer 5e64426dfb04470687fbf9ebfc7eeee4 to start an election
I20260709 12:29:13.655746   545 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
 from {username='slave'} at 127.0.0.1:47524
I20260709 12:29:13.656800   545 raft_consensus.cc:493] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:13.514827   846 tablet.cc:1724] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.662020   847 tablet.cc:1724] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.657511   545 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:13.681962   841 tablet.cc:1724] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.683736   848 tablet.cc:1724] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.637418   839 tablet.cc:1724] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.706857   825 tablet.cc:1724] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.708609   828 tablet.cc:1724] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.681414   826 tablet.cc:1724] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.713719   837 tablet.cc:1724] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.726840   828 tablet.cc:1724] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.728125   842 tablet.cc:1724] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.728813 32759 catalog_manager.cc:5789] TS 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833) does not have the latest schema for tablet 4e772bdd32994e9d9db75376cbbdc5cd (table soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]). Expected version 1 got 0
I20260709 12:29:13.743471 32761 catalog_manager.cc:5789] TS 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003) does not have the latest schema for tablet 7512ad59cefc40bbbc57250610c02648 (table soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]). Expected version 1 got 0
I20260709 12:29:13.745371   828 tablet.cc:1724] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.746384   846 tablet.cc:1724] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.750512 32761 catalog_manager.cc:5789] TS 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003) does not have the latest schema for tablet 13f8cc8ae4cf46bd9aebc62a521857f9 (table soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]). Expected version 1 got 0
I20260709 12:29:13.751516 32761 catalog_manager.cc:5789] TS 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003) does not have the latest schema for tablet 89ea3aeecf65438b8000e3b8c10d9a7b (table soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09]). Expected version 1 got 0
I20260709 12:29:13.757959   834 tablet.cc:1724] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.760442   395 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47"
dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
mode: GRACEFUL
new_leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
 from {username='slave'} at 127.0.0.1:53306
I20260709 12:29:13.761333   395 raft_consensus.cc:606] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Received request to transfer leadership to TS 5e64426dfb04470687fbf9ebfc7eeee4
I20260709 12:29:13.763597 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:13.764456 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:13.771274   545 raft_consensus.cc:515] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:13.775488   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:13.776525   395 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:13.798422   834 tablet.cc:1724] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.801677   545 leader_election.cc:290] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 3 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:13.803267   471 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b229e0d49c44c37adb4c95d73bdd6fe" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:13.806807   471 raft_consensus.cc:3055] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:29:13.807217   471 raft_consensus.cc:740] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c73adb282914cd0a9ab735eda3a2178, State: Running, Role: LEADER
I20260709 12:29:13.808622   471 consensus_queue.cc:260] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:13.810442   471 raft_consensus.cc:3060] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:29:13.814438   439 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.816253   395 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 3.
I20260709 12:29:13.818452   513 leader_election.cc:304] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fd946b83f34e42b80480fe7a6f7f0c, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:13.819882   823 raft_consensus.cc:2804] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:29:13.820418   823 raft_consensus.cc:697] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:13.821499   823 consensus_queue.cc:237] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } }
I20260709 12:29:13.804531   847 tablet.cc:1724] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.846766   826 tablet.cc:1724] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:13.864087   471 raft_consensus.cc:2468] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 3.
I20260709 12:29:13.872404 32763 catalog_manager.cc:5697] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 2 to 3, leader changed from 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2) to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 3 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:13.874490   803 raft_consensus.cc:993] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: : Instructing follower 5e64426dfb04470687fbf9ebfc7eeee4 to start an election
I20260709 12:29:13.880795   803 raft_consensus.cc:1081] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Signalling peer 5e64426dfb04470687fbf9ebfc7eeee4 to start an election
I20260709 12:29:13.884555   546 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47"
dest_uuid: "5e64426dfb04470687fbf9ebfc7eeee4"
 from {username='slave'} at 127.0.0.1:47512
I20260709 12:29:13.885454   546 raft_consensus.cc:493] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:13.885939   546 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:13.900468   546 raft_consensus.cc:515] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:13.901458 32760 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b: soft_deleted_table [id=a2b0e2477a9145d9a4de5134849b5b09] alter complete (version 1)
I20260709 12:29:13.904657   546 leader_election.cc:290] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 3 election: Requested vote from peers 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833), 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003)
I20260709 12:29:13.904836   395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "46fd946b83f34e42b80480fe7a6f7f0c"
I20260709 12:29:13.905719   395 raft_consensus.cc:3055] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:29:13.906670   395 raft_consensus.cc:740] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46fd946b83f34e42b80480fe7a6f7f0c, State: Running, Role: LEADER
I20260709 12:29:13.907671   395 consensus_queue.cc:260] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:13.909056   472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f531bc8ca7f4f2bac07a02d7d562d47" candidate_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c73adb282914cd0a9ab735eda3a2178"
I20260709 12:29:13.909614   395 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:13.909953   472 raft_consensus.cc:3060] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:13.922286   472 raft_consensus.cc:2468] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 3.
I20260709 12:29:13.923321   395 raft_consensus.cc:2468] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e64426dfb04470687fbf9ebfc7eeee4 in term 3.
I20260709 12:29:13.924373   511 leader_election.cc:304] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c73adb282914cd0a9ab735eda3a2178, 5e64426dfb04470687fbf9ebfc7eeee4; no voters: 
I20260709 12:29:13.925621   823 raft_consensus.cc:2804] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:29:13.926235   823 raft_consensus.cc:697] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 LEADER]: Becoming Leader. State: Replica: 5e64426dfb04470687fbf9ebfc7eeee4, State: Running, Role: LEADER
I20260709 12:29:13.927337   823 consensus_queue.cc:237] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } }
I20260709 12:29:13.937389 32763 catalog_manager.cc:5697] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 reported cstate change: term changed from 2 to 3, leader changed from 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1) to 5e64426dfb04470687fbf9ebfc7eeee4 (127.16.102.3). New cstate: current_term: 3 leader_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e64426dfb04470687fbf9ebfc7eeee4" member_type: VOTER last_known_addr { host: "127.16.102.3" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:14.236137   472 raft_consensus.cc:1275] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Refusing update from remote peer 5e64426dfb04470687fbf9ebfc7eeee4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:29:14.237516   823 consensus_queue.cc:1048] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:29:14.249238   395 raft_consensus.cc:1275] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Refusing update from remote peer 5e64426dfb04470687fbf9ebfc7eeee4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:29:14.252964   823 consensus_queue.cc:1048] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:29:14.377030   472 raft_consensus.cc:1275] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Refusing update from remote peer 5e64426dfb04470687fbf9ebfc7eeee4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:29:14.379201   823 consensus_queue.cc:1048] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c73adb282914cd0a9ab735eda3a2178" member_type: VOTER last_known_addr { host: "127.16.102.2" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:29:14.395144   395 raft_consensus.cc:1275] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Refusing update from remote peer 5e64426dfb04470687fbf9ebfc7eeee4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:29:14.397024   865 consensus_queue.cc:1048] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46fd946b83f34e42b80480fe7a6f7f0c" member_type: VOTER last_known_addr { host: "127.16.102.1" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:29:14.765646 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:14.771816 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:14.772461 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:15.773751 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:15.779950 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:15.780576 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:16.781999 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:16.789574 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:16.790307 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:17.791811 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:17.799041 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:17.799925 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:18.801368 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:18.808212 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:18.808981 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:20.398329 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:20.404974 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:20.405503 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:29:21.097756   568 debug-util.cc:398] Leaking SignalData structure 0x7b08001f1e60 after lost signal to thread 16795
W20260709 12:29:21.103899   568 debug-util.cc:398] Leaking SignalData structure 0x7b0800196fe0 after lost signal to thread 330
W20260709 12:29:21.117764   568 debug-util.cc:398] Leaking SignalData structure 0x7b0800208a20 after lost signal to thread 422
W20260709 12:29:21.131636   568 debug-util.cc:398] Leaking SignalData structure 0x7b080027a460 after lost signal to thread 497
W20260709 12:29:21.137274   568 debug-util.cc:398] Leaking SignalData structure 0x7b08002166a0 after lost signal to thread 571
I20260709 12:29:21.407282 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:21.419993 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:21.420658 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:22.422327 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:22.428246 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:22.428851 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:23.430297 16792 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:23.437263 16792 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:23.437844 16792 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:24.522763 16792 tablet_server.cc:179] TabletServer@127.16.102.1:0 shutting down...
I20260709 12:29:24.591661 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:24.593021 16792 tablet_replica.cc:333] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.594671 16792 raft_consensus.cc:2243] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.596627 16792 raft_consensus.cc:2272] T d83869da929a44a6be7379e8b01bbf4c P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.601002 16792 tablet_replica.cc:333] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.602294 16792 raft_consensus.cc:2243] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.603645 16792 raft_consensus.cc:2272] T 7512ad59cefc40bbbc57250610c02648 P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.608873 16792 tablet_replica.cc:333] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.609968 16792 raft_consensus.cc:2243] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.610915 16792 raft_consensus.cc:2272] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.614804 16792 tablet_replica.cc:333] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.615777 16792 raft_consensus.cc:2243] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.616518 16792 raft_consensus.cc:2272] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.620013 16792 tablet_replica.cc:333] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.621176 16792 raft_consensus.cc:2243] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.622081 16792 raft_consensus.cc:2272] T 5d27dc670d7a42f39da938e5e12dd82d P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.625967 16792 tablet_replica.cc:333] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.626758 16792 raft_consensus.cc:2243] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.628094 16792 raft_consensus.cc:2272] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.632153 16792 tablet_replica.cc:333] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.632972 16792 raft_consensus.cc:2243] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.633807 16792 raft_consensus.cc:2272] T de48d2cd9e654eda8d32522d64717448 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.637454 16792 tablet_replica.cc:333] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.638463 16792 raft_consensus.cc:2243] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.639256 16792 raft_consensus.cc:2272] T 644ee91daa20415db01046db3df21c5f P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.643536 16792 tablet_replica.cc:333] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.644687 16792 raft_consensus.cc:2243] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.646860 16792 raft_consensus.cc:2272] T 422991c944394d5e9024d49c2cdb1222 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.651247 16792 tablet_replica.cc:333] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.652184 16792 raft_consensus.cc:2243] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.652845 16792 raft_consensus.cc:2272] T 3fd387cc70584d6aa36ef41af2f18cfb P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.656404 16792 tablet_replica.cc:333] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.657284 16792 raft_consensus.cc:2243] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.658216 16792 raft_consensus.cc:2272] T 4b229e0d49c44c37adb4c95d73bdd6fe P 46fd946b83f34e42b80480fe7a6f7f0c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.661911 16792 tablet_replica.cc:333] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.662721 16792 raft_consensus.cc:2243] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.664316 16792 raft_consensus.cc:2272] T c985c2255f224d56813909db7551f037 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:24.667891   513 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 13 similar messages]
I20260709 12:29:24.669176 16792 tablet_replica.cc:333] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.670337 16792 raft_consensus.cc:2243] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.671279 16792 raft_consensus.cc:2272] T 157d51caa8704cda9be3b34a5291145f P 46fd946b83f34e42b80480fe7a6f7f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.676443 16792 tablet_replica.cc:333] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
W20260709 12:29:24.676478   513 consensus_peers.cc:597] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:24.678660 16792 raft_consensus.cc:2243] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.680675 16792 raft_consensus.cc:2272] T 4e772bdd32994e9d9db75376cbbdc5cd P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:24.685124   439 consensus_peers.cc:597] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:24.685523 16792 tablet_replica.cc:333] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.686697 16792 raft_consensus.cc:2243] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.687772 16792 raft_consensus.cc:2272] T 89ea3aeecf65438b8000e3b8c10d9a7b P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.691797 16792 tablet_replica.cc:333] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.692555 16792 raft_consensus.cc:2243] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.694039 16792 raft_consensus.cc:2272] T 40ce481100e74572b992deb470778bb1 P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.697706 16792 tablet_replica.cc:333] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.698541 16792 raft_consensus.cc:2243] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.700125 16792 raft_consensus.cc:2272] T 69700dd13eb644fc87075764655a7c8d P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.703991 16792 tablet_replica.cc:333] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c: stopping tablet replica
I20260709 12:29:24.704763 16792 raft_consensus.cc:2243] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.705574 16792 raft_consensus.cc:2272] T 6a32c85c69854a579b8c8c9eb2f490bc P 46fd946b83f34e42b80480fe7a6f7f0c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:24.731842   513 consensus_peers.cc:597] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:24.769631 16792 tablet_server.cc:196] TabletServer@127.16.102.1:0 shutdown complete.
W20260709 12:29:24.811020   513 consensus_peers.cc:597] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:36833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:24.835887   439 consensus_peers.cc:597] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:36833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:24.836819   439 consensus_peers.cc:597] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:36833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:24.853035 16792 tablet_server.cc:179] TabletServer@127.16.102.2:0 shutting down...
W20260709 12:29:24.881508   511 consensus_peers.cc:597] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Couldn't send request to peer 4c73adb282914cd0a9ab735eda3a2178. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:24.889644   511 consensus_peers.cc:597] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Couldn't send request to peer 4c73adb282914cd0a9ab735eda3a2178. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:24.903594   513 consensus_peers.cc:597] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 46fd946b83f34e42b80480fe7a6f7f0c (127.16.102.1:36833): Couldn't send request to peer 46fd946b83f34e42b80480fe7a6f7f0c. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.1:36833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:24.911520 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:24.912554 16792 tablet_replica.cc:333] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.913806 16792 raft_consensus.cc:2243] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:29:24.914448   438 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:24.915083 16792 raft_consensus.cc:2272] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.918391 16792 tablet_replica.cc:333] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.919158 16792 raft_consensus.cc:2243] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.919832 16792 raft_consensus.cc:2272] T d83869da929a44a6be7379e8b01bbf4c P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.922758 16792 tablet_replica.cc:333] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.923482 16792 raft_consensus.cc:2243] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.924080 16792 raft_consensus.cc:2272] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.928650 16792 tablet_replica.cc:333] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.929589 16792 raft_consensus.cc:2243] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.930814 16792 raft_consensus.cc:2272] T 7512ad59cefc40bbbc57250610c02648 P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.933897 16792 tablet_replica.cc:333] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.934652 16792 raft_consensus.cc:2243] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.935230 16792 raft_consensus.cc:2272] T 69700dd13eb644fc87075764655a7c8d P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.938200 16792 tablet_replica.cc:333] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.938958 16792 raft_consensus.cc:2243] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.939544 16792 raft_consensus.cc:2272] T c985c2255f224d56813909db7551f037 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.942799 16792 tablet_replica.cc:333] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.943742 16792 raft_consensus.cc:2243] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.945031 16792 raft_consensus.cc:2272] T 6a32c85c69854a579b8c8c9eb2f490bc P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.948351 16792 tablet_replica.cc:333] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.949131 16792 raft_consensus.cc:2243] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.949750 16792 raft_consensus.cc:2272] T 422991c944394d5e9024d49c2cdb1222 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.952500 16792 tablet_replica.cc:333] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.953223 16792 raft_consensus.cc:2243] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.954319 16792 raft_consensus.cc:2272] T 3fd387cc70584d6aa36ef41af2f18cfb P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.957056 16792 tablet_replica.cc:333] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.957710 16792 raft_consensus.cc:2243] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.958846 16792 raft_consensus.cc:2272] T de48d2cd9e654eda8d32522d64717448 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.961515 16792 tablet_replica.cc:333] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.962239 16792 raft_consensus.cc:2243] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.963323 16792 raft_consensus.cc:2272] T 644ee91daa20415db01046db3df21c5f P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.966254 16792 tablet_replica.cc:333] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.967054 16792 raft_consensus.cc:2243] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.967577 16792 raft_consensus.cc:2272] T 4b229e0d49c44c37adb4c95d73bdd6fe P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.970314 16792 tablet_replica.cc:333] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.970932 16792 raft_consensus.cc:2243] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.971561 16792 raft_consensus.cc:2272] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 4c73adb282914cd0a9ab735eda3a2178 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.974332 16792 tablet_replica.cc:333] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.975020 16792 raft_consensus.cc:2243] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.975591 16792 raft_consensus.cc:2272] T 5d27dc670d7a42f39da938e5e12dd82d P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.978696 16792 tablet_replica.cc:333] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.979439 16792 raft_consensus.cc:2243] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.980202 16792 raft_consensus.cc:2272] T 157d51caa8704cda9be3b34a5291145f P 4c73adb282914cd0a9ab735eda3a2178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.983139 16792 tablet_replica.cc:333] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.983920 16792 raft_consensus.cc:2243] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.984587 16792 raft_consensus.cc:2272] T 4e772bdd32994e9d9db75376cbbdc5cd P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.987735 16792 tablet_replica.cc:333] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.988406 16792 raft_consensus.cc:2243] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:24.989702 16792 raft_consensus.cc:2272] T 89ea3aeecf65438b8000e3b8c10d9a7b P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:24.992843 16792 tablet_replica.cc:333] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178: stopping tablet replica
I20260709 12:29:24.993496 16792 raft_consensus.cc:2243] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:24.994171 16792 raft_consensus.cc:2272] T 40ce481100e74572b992deb470778bb1 P 4c73adb282914cd0a9ab735eda3a2178 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.048766 16792 tablet_server.cc:196] TabletServer@127.16.102.2:0 shutdown complete.
I20260709 12:29:25.114379 16792 tablet_server.cc:179] TabletServer@127.16.102.3:0 shutting down...
I20260709 12:29:25.156709 16792 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:25.157735 16792 tablet_replica.cc:333] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.159096 16792 raft_consensus.cc:2243] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.159966 16792 raft_consensus.cc:2272] T d83869da929a44a6be7379e8b01bbf4c P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:25.163466   511 consensus_peers.cc:597] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 -> Peer 4c73adb282914cd0a9ab735eda3a2178 (127.16.102.2:44003): Couldn't send request to peer 4c73adb282914cd0a9ab735eda3a2178. Status: Network error: Client connection negotiation failed: client connection to 127.16.102.2:44003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:25.164148 16792 tablet_replica.cc:333] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.165046 16792 raft_consensus.cc:2243] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.165623 16792 raft_consensus.cc:2272] T 529ee0b0b3b243b8b9f5f8d500eefbc4 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.168865 16792 tablet_replica.cc:333] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.169545 16792 raft_consensus.cc:2243] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.170238 16792 raft_consensus.cc:2272] T 7512ad59cefc40bbbc57250610c02648 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.173121 16792 tablet_replica.cc:333] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.173877 16792 raft_consensus.cc:2243] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.174517 16792 raft_consensus.cc:2272] T 13f8cc8ae4cf46bd9aebc62a521857f9 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.177563 16792 tablet_replica.cc:333] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.178684 16792 raft_consensus.cc:2243] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:29:25.179901 16792 raft_consensus.cc:2272] T 6f531bc8ca7f4f2bac07a02d7d562d47 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.182978 16792 tablet_replica.cc:333] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.183751 16792 raft_consensus.cc:2243] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:25.184916 16792 raft_consensus.cc:2272] T 5d27dc670d7a42f39da938e5e12dd82d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.188028 16792 tablet_replica.cc:333] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.188862 16792 raft_consensus.cc:2243] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.189546 16792 raft_consensus.cc:2272] T 422991c944394d5e9024d49c2cdb1222 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.192415 16792 tablet_replica.cc:333] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.193183 16792 raft_consensus.cc:2243] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.193804 16792 raft_consensus.cc:2272] T 3fd387cc70584d6aa36ef41af2f18cfb P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.196699 16792 tablet_replica.cc:333] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.197392 16792 raft_consensus.cc:2243] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.198072 16792 raft_consensus.cc:2272] T de48d2cd9e654eda8d32522d64717448 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.200747 16792 tablet_replica.cc:333] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.201431 16792 raft_consensus.cc:2243] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.202111 16792 raft_consensus.cc:2272] T 644ee91daa20415db01046db3df21c5f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.205025 16792 tablet_replica.cc:333] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.205729 16792 raft_consensus.cc:2243] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.206364 16792 raft_consensus.cc:2272] T 6a32c85c69854a579b8c8c9eb2f490bc P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.209111 16792 tablet_replica.cc:333] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.209789 16792 raft_consensus.cc:2243] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.210443 16792 raft_consensus.cc:2272] T 69700dd13eb644fc87075764655a7c8d P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.213161 16792 tablet_replica.cc:333] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.213892 16792 raft_consensus.cc:2243] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:25.214970 16792 raft_consensus.cc:2272] T 157d51caa8704cda9be3b34a5291145f P 5e64426dfb04470687fbf9ebfc7eeee4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.217721 16792 tablet_replica.cc:333] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.218490 16792 raft_consensus.cc:2243] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.219091 16792 raft_consensus.cc:2272] T 4e772bdd32994e9d9db75376cbbdc5cd P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.221757 16792 tablet_replica.cc:333] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.222465 16792 raft_consensus.cc:2243] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.223119 16792 raft_consensus.cc:2272] T 89ea3aeecf65438b8000e3b8c10d9a7b P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.225972 16792 tablet_replica.cc:333] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.226583 16792 raft_consensus.cc:2243] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.227172 16792 raft_consensus.cc:2272] T 40ce481100e74572b992deb470778bb1 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.229902 16792 tablet_replica.cc:333] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.230516 16792 raft_consensus.cc:2243] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:25.231084 16792 raft_consensus.cc:2272] T c985c2255f224d56813909db7551f037 P 5e64426dfb04470687fbf9ebfc7eeee4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.233897 16792 tablet_replica.cc:333] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4: stopping tablet replica
I20260709 12:29:25.234517 16792 raft_consensus.cc:2243] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:29:25.235636 16792 raft_consensus.cc:2272] T 4b229e0d49c44c37adb4c95d73bdd6fe P 5e64426dfb04470687fbf9ebfc7eeee4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.306340 16792 tablet_server.cc:196] TabletServer@127.16.102.3:0 shutdown complete.
I20260709 12:29:25.359140 16792 master.cc:562] Master@127.16.102.62:35193 shutting down...
I20260709 12:29:25.391607 16792 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:25.392354 16792 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:25.392665 16792 tablet_replica.cc:333] T 00000000000000000000000000000000 P 32730eefe9ff4ef1ab309329feffbe4b: stopping tablet replica
I20260709 12:29:25.418113 16792 master.cc:584] Master@127.16.102.62:35193 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (22604 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (45028 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (586138 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 12:29:25.485623 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2243 messages since previous log ~33 seconds ago
I20260709 12:29:25.485961 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2243 messages since previous log ~33 seconds ago
I20260709 12:29:25.486192 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2235 messages since previous log ~33 seconds ago
I20260709 12:29:25.486424 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~54 seconds ago
I20260709 12:29:25.486641 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~54 seconds ago
I20260709 12:29:25.487006 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~0 seconds ago
I20260709 12:29:25.487365 16792 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 53 messages since previous log ~21 seconds ago
